2. Variants of estimation algorithms

Let us compare the efficiency of moving object area identification using the sets ˜ ° of parameters hx; hy and <sup>ð</sup>ρ; <sup>φ</sup> <sup>Þ</sup>. The following method for estimating the shift vectors' field H is used. At the first stage, stochastic gradient descent algorithm sequentially processes all nodes of the reference image row by row with the set of ˜ ° parameters hx; hy . It processes each row i bidirectionally [14], first, from the left to the right, getting, according to Eq. (1), the estimates

$$\hat{h}^{l}\_{(i,j+1)\mathbf{x}} = \hat{h}^{l}\_{(ij)\mathbf{x}} - \lambda\_{h} \text{sign} \left(\beta\_{\mathbf{k}\mathbf{x}} \left(\mathbf{z}^{\mathbf{d}}\_{i,j+1}, \hat{\boldsymbol{h}}\_{i,j}\right)\right), \hat{h}^{l}\_{(i,1)\mathbf{x}} = \hat{h}^{l}\_{(i-1,1)\mathbf{x}}.$$

$$\hat{h}^{l}\_{(i,j+1)\mathbf{y}} = \hat{h}^{l}\_{(ij)\mathbf{y}} - \lambda\_{h} \text{sign} \left(\beta\_{\mathbf{y}} \left(\mathbf{z}^{\mathbf{d}}\_{i,j+1}, \hat{\boldsymbol{h}}\_{i,j}\right)\right), \hat{h}^{l}\_{(i,1)\mathbf{y}} = \hat{h}^{l}\_{(i-1,1)\mathbf{y}}.$$

and then from the right to the left getting the estimates

$$\begin{split} \hat{h}^{r}\_{\{i,N\_{\mathcal{V}}-j\}\mathbf{x}} &= \hat{h}^{r}\_{\{i,N\_{\mathcal{V}}-j+1\}\mathbf{x}} - \lambda\_{h} \operatorname{sign} \left( \beta\_{\text{lux}} \Big( z^{\mathbf{d}}\_{i,N\_{\mathcal{V}}-j}, \hat{\boldsymbol{\tilde{h}}}\_{i,N\_{\mathcal{V}}-j+1} \Big) \right), \hat{h}^{r}\_{\{i,N\_{\mathcal{V}}\}\mathbf{x}} = \hat{h}^{r}\_{\{i-1,N\_{\mathcal{V}}\}\mathbf{x}}, \\ \hat{h}^{r}\_{\{i,N\_{\mathcal{V}}-j\}\mathbf{y}} &= \hat{h}^{r}\_{\{i,N\_{\mathcal{V}}-j+1\}\mathbf{y}} - \lambda\_{h} \operatorname{sign} \left( \beta\_{\text{hy}} \Big( z^{\mathbf{d}}\_{i,N\_{\mathcal{V}}-j}, \hat{\boldsymbol{\tilde{h}}}\_{i,N\_{\mathcal{V}}-j+1} \Big) \right), \hat{h}^{r}\_{\{i,N\_{\mathcal{V}}\}\mathbf{y}} = \hat{h}^{r}\_{\{i-1,N\_{\mathcal{V}}\}\mathbf{y}}. \end{split}$$

where parameter λ<sup>h</sup> is determined by the maximum speed of moving objects.

When using the set of parameters (ρ, <sup>φ</sup>), considering that <sup>∂</sup>x=∂<sup>ρ</sup> <sup>¼</sup> cos <sup>φ</sup>, <sup>∂</sup>x=∂<sup>φ</sup> ¼ �<sup>ρ</sup> sin <sup>φ</sup>, <sup>∂</sup>y=∂<sup>ρ</sup> <sup>¼</sup> sin <sup>φ</sup>, and <sup>∂</sup>y=∂<sup>φ</sup> <sup>¼</sup> <sup>ρ</sup> cos <sup>φ</sup>, for gradient estimation ˜ °<sup>T</sup> <sup>β</sup> <sup>¼</sup> βρ; βφ , we get [11]

� � � � � � � � ~ ~ ~ ˛ ˝ ˛ ˝ <sup>s</sup> <sup>s</sup> <sup>s</sup> <sup>s</sup> <sup>d</sup> <sup>β</sup> z zz <sup>2</sup><sup>z</sup> <sup>ρ</sup> <sup>¼</sup> <sup>x</sup>þΔx, <sup>y</sup> <sup>x</sup>�Δx, <sup>y</sup> <sup>~</sup>zxþΔx, <sup>y</sup> <sup>þ</sup> <sup>x</sup>�Δx, <sup>y</sup> i,j cos <sup>φ</sup> ~ ~ ~ ˛ ˝ ˛ ˝ <sup>s</sup> <sup>s</sup> <sup>s</sup> <sup>s</sup> <sup>d</sup> <sup>þ</sup> z z<sup>~</sup> <sup>þ</sup> <sup>z</sup> <sup>2</sup><sup>z</sup> sin <sup>φ</sup>, <sup>Δ</sup><sup>y</sup> zx, <sup>y</sup>þΔ<sup>y</sup> x, <sup>y</sup>� x, <sup>y</sup>þΔ<sup>y</sup> x, <sup>y</sup>�Δ<sup>y</sup> i,j ~ ~ ~ ˛ ˝ ˛ ˝ <sup>s</sup> <sup>s</sup> <sup>s</sup> <sup>s</sup> <sup>d</sup> βφ ¼ � z z <sup>x</sup>þΔx, <sup>y</sup> <sup>x</sup>�Δx, <sup>y</sup> <sup>~</sup>zxþΔx, <sup>y</sup> <sup>þ</sup> zx�Δx, <sup>y</sup> <sup>2</sup>zi,j <sup>ρ</sup> sin <sup>φ</sup> ~ ~ ~ ˛ ˝ ˛ ˝ <sup>s</sup> <sup>s</sup> <sup>s</sup> <sup>s</sup> <sup>d</sup> z zz x, <sup>y</sup>� <sup>þ</sup> <sup>~</sup> <sup>þ</sup> <sup>2</sup><sup>z</sup> <sup>ρ</sup> cos <sup>φ</sup>: <sup>Δ</sup><sup>y</sup> zx, <sup>y</sup>þΔ<sup>y</sup> x, <sup>y</sup>þΔ<sup>y</sup> x, <sup>y</sup>�Δ<sup>y</sup> i,j

Then according to Eq. (1), we get the estimates

$$\hat{\rho}\_{i,j+1}^{l} = \hat{\rho}\_{i,j}^{l} - \lambda\_{\rho} \operatorname{sign} \left( \beta\_{\rho} \left( \mathbf{z}\_{i,j+1}^{\mathbf{d}}, \hat{\rho}\_{i,j}, \hat{\rho}\_{i,j} \right) \right),$$

$$\hat{\rho}\_{i,j+1}^{l} = \hat{\rho}\_{i,j}^{l} - \lambda\_{\rho} \operatorname{sign} \left( \beta\_{\rho} \left( \mathbf{z}\_{i,j+1}^{\mathbf{d}}, \hat{\rho}\_{i,j}, \hat{\rho}\_{i,j} \right) \right),$$

$$\hat{\rho}\_{i,N\_{\mathcal{V}} - j}^{r} = \hat{\rho}\_{i,N\_{\mathcal{V}} - j + 1}^{r} - \lambda\_{\rho} \operatorname{sign} \left( \beta\_{\rho} \left( \mathbf{z}\_{i,N\_{\mathcal{V}} - j}^{\mathbf{d}}, \hat{\rho}\_{i,N\_{\mathcal{V}} - j + 1}, \hat{\rho}\_{i,N\_{\mathcal{V}} - j + 1} \right) \right),$$

$$\hat{\rho}\_{i,N\_{\mathcal{V}} - j}^{r} = \hat{\rho}\_{i,N\_{\mathcal{V}} - j + 1}^{r} - \lambda\_{\rho} \operatorname{sign} \left( \beta\_{\rho} \left( \mathbf{z}\_{i,N\_{\mathcal{V}} - j}^{\mathbf{d}}, \hat{\rho}\_{i,N\_{\mathcal{V}} - j + 1}, \hat{\rho}\_{i,N\_{\mathcal{V}} - j + 1} \right) \right),$$

where the parameters λρ and λφ are also determined by the maximum speed of moving objects.

h<sup>l</sup> h ^ ^<sup>r</sup> ^ At the second stage, for each node ð Þ i; j , the estimates and are ð Þ <sup>i</sup>;<sup>j</sup> <sup>x</sup> ð Þ <sup>i</sup>;<sup>j</sup> <sup>x</sup> processed jointly [15]. The optimal value of h <sup>i</sup>;<sup>j</sup> ð Þ<sup>x</sup> is found between the estimates with some step Δh. Step Δ<sup>h</sup> is determined by the required accuracy. If the absolute <sup>h</sup><sup>l</sup> difference between the estimates ð Þ <sup>i</sup>;<sup>j</sup> <sup>x</sup> ^ <sup>h</sup><sup>r</sup> and is less than <sup>Δ</sup>h, then the estimate ð Þ <sup>i</sup>;<sup>j</sup> <sup>x</sup> ^ hð Þ <sup>i</sup>;<sup>j</sup> <sup>x</sup> of of poss ^ the defor ible values mation parameter is assumed to stimate <sup>h</sup><sup>l</sup> be equal . Otherwise, the set ð Þ <sup>i</sup>;<sup>j</sup> <sup>x</sup> is given by ^ of the defor mation p arameter e

$$
\hat{h}^{m}\_{(i,j)\mathbf{x}} = \hat{h}^{l}\_{(i,j)\mathbf{x}} + m\Delta\_{h}, \\
m = \overline{\mathbf{0}, k+1}, \\
k = \left| \hat{h}^{r}\_{(i,j)\mathbf{x}} - \hat{h}^{l}\_{(i,j)\mathbf{x}} \right| / \Delta\_{h}.
$$

The ^ optimal value of <sup>h</sup> <sup>i</sup>;<sup>j</sup> from the resulting set is found using some criterion. ð Þ<sup>x</sup>

#### 2.1 Criteria for the formation of the resulting estimation

Two criteria for the formation of the resulting estimation are studied [16]:

• Gradient estimation minimum

$$\min\_{\lambda\_m = \overline{0\_0}, k+1} \beta\_h \quad \hat{h}^{l}\_{(ij)\chi} + m\Delta\_h \Big),\tag{2}$$

• Correlation coefficient maximum (CC)

$$\max\_{m=\overline{0,1}} \text{CC} \left\{ \bar{z}\_{\mathbf{x}(m)+p,\mathbf{y}(m)+\varsigma}^{\mathbf{s}}, z\_{i+p,j+\varsigma}^{\mathbf{d}} \right\}, \mathbf{p} = \overline{-a,a}, \mathbf{s} = \overline{-b,b}, \tag{3}$$

^ ^ where (x mð Þ, y mð Þ) are the coordinates of the point ð Þ <sup>i</sup>; <sup>j</sup> of the image <sup>Z</sup><sup>s</sup> in the image Z<sup>d</sup> for the estimate h<sup>m</sup> i,j and ð2a þ 1Þ x ð2b þ 1Þ is the window size for calculation of CC. The optimal value h <sup>i</sup>;<sup>j</sup> is determined similarly. ð Þ<sup>y</sup>

Joint processing of estimates allows compensating the inertia of the recursive estimation. The results given below were obtained with a ¼ b ¼ 1.

The joint processing of estimates ρ^<sup>l</sup> i,j , ρ^<sup>i</sup> r ,j and φ^<sup>i</sup> l ,j , and φ^<sup>i</sup> r ,j as well as finding the optimal values of ρ^i,j and φ^i,j are performed as described above using the same criteria of gradient estimation minimum and CC maximum.

Thus for joint processing of the estimates, depending on the criterion, we obtain two versions of the estimation algorithm:

Algorithm A: reverse processing using the criterion Eq. (2); Algorithm B: reverse processing using the criterion Eq. (3).

#### 2.2 Comparison of the estimation algorithm efficiency

Let us first consider the efficiency of the criterion Eq. (2). Figure 2 shows an example of images used for the study. These are two adjacent frames of a video sequence where the vehicle located in the center is moving and the vehicle on the right is motionless. The parameters of inter-frame spatial shift of the moving vehicle are hx <sup>¼</sup> 3, hy <sup>¼</sup> <sup>2</sup>, <sup>95</sup> or <sup>ρ</sup> <sup>¼</sup> <sup>4</sup>, 2, and <sup>φ</sup> <sup>¼</sup> <sup>45</sup><sup>∘</sup> .

Formation of Inter-Frame Deformation Field of Images Using Reverse Stochastic Gradient… DOI: http://dx.doi.org/10.5772/intechopen.83489

Figure 2.

An example of adjacent frames of a video sequence with a moving object.

For example, Figure 3 shows typical results of shift magnitude estimation for a single row of the reference image (Figure 2) while using Algorithm A. For a correct comparison of the estimates ρ^i,j , φ^i,j and h ^ <sup>i</sup>;<sup>j</sup> , and h ^ <sup>i</sup>;<sup>j</sup> , the latter is recalculated to polar parameters: ð Þx ð Þy

$$
\rho(h) = \sqrt{\left(\hat{h}\_{(ij)\mathbf{x}}\right)^2 + \left(\hat{h}\_{(ij)\mathbf{y}}\right)^2}, \\
\rho(h) = \text{arctg}\left(\hat{h}\_{(ij)\mathbf{x}}/\hat{h}\_{(ij)\mathbf{y}}\right).
$$

� � Figure 3a shows dependencies ρ^<sup>i</sup> l ,j and ρ^<sup>i</sup> r ,j on i point in a row, Figure 3b shows the result of their joint processing, Figure 3c shows dependencies <sup>ρ</sup> hl and <sup>ρ</sup> <sup>h</sup><sup>r</sup> ð Þ on i, and Figure 3d shows the result of their joint processing. The solid gray line represents the true value of the deformation parameter. The results for parameters ðρ; φÞ are visually better. Estimates given in Table 1 also confirm that. Table 1 shows mean value <sup>m</sup> and variance <sup>σ</sup>^<sup>2</sup> of estimation error for the area with and <sup>h</sup> <sup>h</sup> without motion for results shown in Figure 3b and d.

� � Note that with the use of parameters ðρ; φÞ in the motion area, the mean value of the estimation error is lower, and the variance is more than two times less than with the use of the set of parameters hx; hy . In the area without motion, the use of parameters ðρ; φÞ is also preferable since the bias of the estimates is less despite the

Figure 3. An example of estimates of shift magnitude for a row using criterion Eq. (2).


Table 1.

Estimation error of shift vectors of deformation field.

slightly larger variance. For comparison, Table 1 also shows the results for the mean value and estimation of the variance <sup>σ</sup>^<sup>2</sup> of estimation errors averaged over the <sup>h</sup> entire image.

Figure 4 shows the results of the joint processing of the sets of estimates ˜ ° ˜ ° <sup>h</sup> ^ð Þ <sup>i</sup>;j x; <sup>h</sup> ^ <sup>i</sup>;j y ρ^<sup>l</sup> i,j ; <sup>ρ</sup>^<sup>r</sup> ð Þ and i,j using the criterion Eq. (3) of the formation of the resulting estimate (Algorithm B). The results are for the same row as the results in ˛ ˝ Figure 3. Figure 4a corresponds to the set of parameters hx; hy , and Figure 4b corresponds to ðρ; φ Þ. Table 1 summarizes the numerical data of the estimation error for the row and averaging over the entire image.

The results show that the use of CC maximum as the criterion can significantly improve the results of the joint processing of the estimates. For both sets of parameters, the mean value of the error for the motion area does not exceed 0.2. The ˛ ˝ variance of the error is approximately halved for the set of parameters hx; hy and 15 times less for parameters ðρ; φÞ. For the area without motion, the mean error is less by about 2.6 and 3.3 times. The variance of the estimates also decreases. However, when using CC, the computational complexity of the estimation procedure increases significantly.

Figure 4. The resulting estimates of shift magnitude for a row using criterion Eq. (3). Formation of Inter-Frame Deformation Field of Images Using Reverse Stochastic Gradient… DOI: http://dx.doi.org/10.5772/intechopen.83489

Figure 5 shows the visualization of estimates of the shift vectors' field H for the four estimation algorithms described above. It shows the magnitudes of the estimated vectors as a function of coordinates of nodes of the reference image. Figure 5a and b shows the results for criteria Eqs. (2) and (3) for the set of ˜ ° parameters hx; hy and Figure 5c and d for the same criteria when parameters ðρ; φÞ are used. The figures illustrate and confirm the conclusions made above.

#### 2.3 Use of inter-row correlation of adjacent rows of the image

The approach to estimate the field H considered above processes images row by row as one-dimensional signals. Let us consider the possibility to increase processing efficiency by taking into account correlation of the adjacent rows of image [13, 17]. For each node ð Þ i; j of the reference image, we will also form two i,j of the parameters of the vector hi,j. To form the estimate <sup>1</sup> i,j and ^ ^<sup>2</sup> ^<sup>1</sup> estimates <sup>h</sup> <sup>h</sup> hi,j , the stochastic gradient procedure processes rows one after the other with the change in direction after each row, for example, the odd rows from the left to the i,j , and the even rows, from the right to the left, forming ^1<sup>l</sup> right, forming estimates <sup>h</sup> ^1<sup>r</sup> ^<sup>1</sup> <sup>h</sup> i,j , obtained from the previous row. If the estimate estimates h <sup>1</sup>,j . The current shift estimate <sup>1</sup>,j in a node is compared with the i,j is considered better iþ iþ ^<sup>1</sup> ^<sup>1</sup> estimate <sup>h</sup> <sup>h</sup> using some criterion, then it is taken as the current estimate.

As the criterion we use CC maximum Eq. (3) with the difference that here the i,j . The estimate i,j is ^<sup>1</sup> ^<sup>1</sup> ^<sup>2</sup> set of possible values is limited by the estimates h h h <sup>1</sup>,j <sup>i</sup><sup>þ</sup> and also formed in a similar way with the difference that, when processing, the odd rows are processed from the right to the left and the even rows from the left to the right.

i,j is carried out as described above. <sup>1</sup> i,j and ^ ^<sup>2</sup> The joint processing of estimates <sup>h</sup> <sup>h</sup> On the one hand, when taking into account inter-row correlation, the formation of i,j requires <sup>a</sup> greater amount of computation. On the other hand, the set of <sup>1</sup> i,j and ^ <sup>2</sup> <sup>h</sup> ^ h

Figure 5. Visualization of estimating shift vectors' field by algorithms A and B with different criteria.

possible estimates for their joint processing turns out to be significantly smaller, which reduces the computational complexity.

Thus we have two more algorithms to estimate the shift vectors' field:


#### 2.4 Comparison of algorithms C and D

Figure 6 shows the typical results of joint processing of the sets of estimates ^ ^ ð Þ <sup>h</sup> <sup>i</sup>;<sup>j</sup> <sup>ρ</sup>i,j <sup>h</sup> <sup>i</sup>;j x, ð Þ<sup>y</sup> (Figure 6a) and ^ , and <sup>φ</sup>^i,j (Figure 6b) using Algorithm C. Figure 6c and d shows the results for the same sets and Algorithm D. It can be seen from the figures that the use of inter-row correlation can significantly improve the estima- ˜ ° tion results. For the parameters hx; hy and Algorithm C, mean value of the error for motion area decreases by 1.2 times, error variance by 1.1 times, and for Algorithm D by 3 times and 29 times, respectively. For the set of parameters ðρ; φÞ and Algorithm C, the mean value of the error for motion area decreases by 5 times, variance by 2.1 times, and for Algorithm D by 10 times and 2.5 times, respectively. The numerical values are listed in Table 1.

Figure 7 shows visualization of the estimates of shift vectors' field H for the estimation algorithms. Figure 7a and b corresponds to the algorithms C and D and ˜ ° set of parameters hx; hy , and Figure 7c and d corresponds to the same algorithms and the set of parameters ðρ; φÞ. It can be seen that criterion Eq. (3) provides a greater accuracy of estimation (but also a larger computational costs). When maximum spatial accuracy is required, it is advisable to use one of algorithms B and D. The error for the Algorithm D is smaller, but the algorithm has a greater computational complexity.

Figure 8 shows moving object area and its contour obtained from the results of algorithms C (Figure 8a) and D (Figure 8b) by thresholding with the threshold equal to 0.1 of the maximum value of shift magnitude. Note that there are practically no errors of the second kind for Algorithm D.

#### Figure 6. Example of estimates of shift magnitude for a row using inter-row correlation.

Formation of Inter-Frame Deformation Field of Images Using Reverse Stochastic Gradient… DOI: http://dx.doi.org/10.5772/intechopen.83489

Figure 7. Visualization of shift vectors' field while using inter-row correlation.

Figure 8. Results of moving object area detection.

#### 3. Efficiency of the algorithms for complex motion

Let us consider the possibilities of the developed algorithms when estimating shift vectors' field for more complex types of motion, in particular, for the case when inter-frame geometric deformations of images of a moving object can be ˜ ° described by <sup>a</sup> similarity model with <sup>a</sup> vector of parameters <sup>α</sup> <sup>¼</sup> <sup>h</sup>; <sup>φ</sup>; <sup>κ</sup> . To compare the results of estimation of the shift vectors' field with the results obtained for the images in Figure 2, the right image was modeled with the corresponding <sup>T</sup> parameters of inter-frame deformations: <sup>h</sup> ¼ ð2; <sup>3</sup><sup>Þ</sup> , <sup>φ</sup> <sup>¼</sup> <sup>4</sup><sup>∘</sup> , and κ ¼ 1.

The results presented below are for algorithms B and D which were shown to have the highest efficiency with the use of Eq. (3) as a criterion. Figure 9 shows typical results for a single row of the image; here graphs (a) and (b) correspond to ˜ ° the Algorithm <sup>B</sup> and the set of parameters hx; hy and <sup>ð</sup>ρ; <sup>φ</sup>Þ, respectively, and graphs (c) and (d) correspond to Algorithm D. In this case the estimates for the set of parameters ðρ; φÞ are closer to the true values (gray line). Figure 10 shows visualization of estimates of the shift vectors' field for the entire image. Figure 10a and b corresponds to Algorithm B, and Figure 10c and d corresponds to Algorithm ˜ ° <sup>D</sup> with the set of parameters hx; hy and <sup>ð</sup>ρ; <sup>φ</sup>Þ, respectively.

Figure 9. Example of estimates of shift magnitude for a row in case of complex motion of an object.

Figure 10. Visualization of shift vectors' field in case of complex motion of an object.

In the presented figures, in order to remove outliers significantly exceeding the true value of the shift, the estimation module was limited to ρmax ¼ 10. When using Algorithm B, we note a significant number of false estimates ρ≥ ρmax both for and ˜ ° outside the area of motion. In particular, when using the parameters hx; hy , the estimates for a significant number of bottom rows of the image exceed the threshold. The use of the inter-row correlation significantly reduces the number of outliers for ˜ ° the set of parameters hx; hy and virtually eliminates them for the set <sup>ð</sup>ρ; <sup>φ</sup>Þ.
