**Motion Tracking and Potentially Dangerous Situations Recognition in Complex Environment**

Houssam Salmane, Yassine Ruichek and Louahdi Khoudour

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/68141

#### Abstract

[116] Campos T, Liu Q, Barnard M. S3A Speaker Tracking with Kinect2. 2017. Available from:

[117] Bernardin K, Stiefelhagen R. Evaluating multiple object tracking performance: the CLEAR MOT metrics. EURASIP Journal on Image and Video Processing. 2008;2008(1):1–10 [118] Li Y, Huang C, Nevatia R. Learning to associate: Hybridboosted multi-target tracker for crowded scene. In: CVPR 2009. IEEE Conference on Computer Vision and Pattern

[119] Wu B, Nevatia R. Tracking of multiple, partially occluded humans based on static body part detection. In: 2006 IEEE Computer Society Conference on Computer Vision and

[120] Ristic B, et al. A metric for performance evaluation of multi-target tracking algorithms.

[121] Schuhmacher D, Vo B-T, Vo B-N. A consistent metric for performance evaluation of multi-object filters. IEEE Transactions on Signal Processing. 2008;56(8):3447–3457 [122] Barnard M, et al. Robust multi-speaker tracking via dictionary learning and identity

[123] Pham NT, Huang W, Ong SH. Tracking multiple objects using probability hypothesis density filter and color measurements. In: IEEE International Conference on Multimedia

IEEE Transactions on Signal Processing. 2011;59(7):3452–3457

modeling. IEEE Transactions on Multimedia. 2014;16(3):864–880

http://epubs.surrey.ac.uk/807708/

Pattern Recognition (CVPR'06). IEEE; 2006

Recognition. IEEE; 2009

74 Motion Tracking and Gesture Recognition

and Expo. IEEE; 2007

In recent years, video surveillance systems have been playing a significantly important role in the human safety and security field by monitoring public or private areas. In this chapter, we have discussed the development of an intelligent surveillance system to detect, track and identify potentially hazardous events that may occur at level crossings (LC). This system starts by detecting and tracking objects on the level crossing. Then, a danger evaluation method is built using hidden Markov model in order to predict trajectories of the detected objects. The trajectories are analyzed with a credibility model to evaluate dangerous situations at level crossings. Synthetics and real data are used to test the effectiveness and the robustness of the proposed algorithms and the whole approach by considering various scenarios within several situations.

Keywords: video surveillance system, tracking and recognition, level crossing, hidden Markov model (HMM)

#### 1. Introduction

Improving safety at level crossing (LC) became an important academic research topic in the transportation field and took increasingly railway undertaking concerns. European countries and European projects try to upgarde level crossing safety which is quite weak today. These projects, like SELCAT "Safer European Level Crossing Appraisal and Technology" [1], has set up some databases of accidents at European level. United States presents very well equipped level crossing with advanced means for sensing and telecommunication [2]. Selectra Vision Company in Italy [3] has developed a surveillance system for detecting obstacles in the monitored area of a level crossing using a 3D laser scanner. Nevertheless, developing a new LC safety

© 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

system which allows quantifying the risk within the LC environment and transmitting it to road users, rail managers and even train drivers still is the main focus for technical solutions.

One of the objectives of the proposed work is to perform a video analysis-based system in order to evaluate the degree of danger of each detected and tracked moving object at level crossing.

The first step of our proposed video surveillance system starts by robustly detecting and separating moving objects crossing the LC. Many approaches are used in the literature to detect objects in real time; examples are Independent Components Analysis [4], Histogram of Oriented Gradients [5], Wavelet [6], Eigen backgrounds [7], kernel and contour tracking [8, 9] or Kalman and particle filters [10, 11]. However, these techniques require further development to distinguish between detected objects.

That is why our approach consists of detecting all moving pixels based on a background subtraction approach. To obtain separated objects, we propose a model based on clustering the detected pixels, affected by motion, by comparing a specific energy vector associated to each target. Finally, the tracking of each pixel detected within a moving object is achieved by using a Harris cornersbased optical flow propagation technique, followed by a Kalman filtering-based rectification.

The second step is focused on predicting trajectories of the detected moving objects such as to avoid potentially dangerous level crossing accident scenarios (vehicle stopped at LC for example). Gaussian mixture model (GMM) [12], hidden Markov model (HMM) [13], Hierarchical and Couple Hidden Markov Model [14, 15] are usually used for representing and recognizing objects' trajectories. However, these methods need a high number of statistical measures to be accurate. Using a real-time hidden Markov model, the degree of dangerousness related to each object is instantly estimated by analyzing each object's trajectory considering different sources of danger (position, velocity, acceleration…). All the information obtained from the sources of danger is fused using Dempster-Shafer technique [16]. Figure 1 illustrates the synopsis of the proposed video surveillance security system.

Figure 1. Synoptic of the video surveillance security system installed at Level crossing.

The remaining of the chapter is organized as follows. Section2 is dealing with object detection and separation. In Section3, the tracking process is developed. Section4 describes the proposed method for evaluating and recognizing dangerous situations in a level crossing environment. In Section5, some evaluation results are provided based on typical accident scenarios played in a real level crossing environment. Finally, in Section6, some conclusions and short-term perspectives are provided.

#### 2. Object detection and separation

system which allows quantifying the risk within the LC environment and transmitting it to road

One of the objectives of the proposed work is to perform a video analysis-based system in order to evaluate the degree of danger of each detected and tracked moving object at level crossing.

The first step of our proposed video surveillance system starts by robustly detecting and separating moving objects crossing the LC. Many approaches are used in the literature to detect objects in real time; examples are Independent Components Analysis [4], Histogram of Oriented Gradients [5], Wavelet [6], Eigen backgrounds [7], kernel and contour tracking [8, 9] or Kalman and particle filters [10, 11]. However, these techniques require further development

That is why our approach consists of detecting all moving pixels based on a background subtraction approach. To obtain separated objects, we propose a model based on clustering the detected pixels, affected by motion, by comparing a specific energy vector associated to each target. Finally, the tracking of each pixel detected within a moving object is achieved by using a Harris cornersbased optical flow propagation technique, followed by a Kalman filtering-based rectification.

The second step is focused on predicting trajectories of the detected moving objects such as to avoid potentially dangerous level crossing accident scenarios (vehicle stopped at LC for example). Gaussian mixture model (GMM) [12], hidden Markov model (HMM) [13], Hierarchical and Couple Hidden Markov Model [14, 15] are usually used for representing and recognizing objects' trajectories. However, these methods need a high number of statistical measures to be accurate. Using a real-time hidden Markov model, the degree of dangerousness related to each object is instantly estimated by analyzing each object's trajectory considering different sources of danger (position, velocity, acceleration…). All the information obtained from the sources of danger is fused using Dempster-Shafer technique [16]. Figure 1 illustrates the synopsis of the

users, rail managers and even train drivers still is the main focus for technical solutions.

to distinguish between detected objects.

76 Motion Tracking and Gesture Recognition

proposed video surveillance security system.

Figure 1. Synoptic of the video surveillance security system installed at Level crossing.

In this section, a method has been developed to detect and separate moving objects in the level crossing surveillance zone. This method starts by detecting pixels affected by motion, by using background subtraction technique as a preprocessing phase; each image processed at each step, a subtraction from the background image is carried out. The main aim of this procedure is to extract the moving pixels in the current image (Figure 2b). Furthermore, a subtraction from the previous image is also carried out in order to obtain the moving pixels situated on the edges of the object. (Figure 2c). The procedure continues by determining in the current image the required targets. A target in the image is defined by a set of connected pixels affected by motion. A bounding box is then associated for each group of connected pixels (Figure 3).

Each created bounding box may belong to an existing target extracted from the previous frame, or representing a new target extracted from the current frame (Figure 3). The intersection between all created bounding boxes in the current frame and those representing the targets extracted from the previous frame is analyzed in order to determine the number and the shape of all moving objects (targets) in the current frame; a bounding box created from the current frame is considered as a new target if and only if it does not intersect any existing bounding box representing a target extracted from the previous frame. On the other case, if a bounding box created from the current frame intersects existing targets, an iterative separation method is applied. During each iteration, a pixel in the current bounding box should be assigned to one of the existing targets.

The pixels clustering process starts by defined two energy vectors. The first energy vector Ei target, initialized to zero, is concerned with each existing target number i. This energy is then updated iteratively. The second energy vector E<sup>i</sup> pixel is defined for each pixel located at the position ðx, yÞ with respect to the target number i. This energy is expressed as follows:

Figure 2. Detection of moving pixels. (a) Current image. (b) Detected moving pixels. (c) Moving pixels situated in the contour of the objects.

$$E\_{\rm pixel}^i = \begin{bmatrix} E\_{\rm D'}^i \ E\_{F'}^i \ E\_{I'}^i \ E\_G^i \end{bmatrix}^T \tag{1}$$

where E<sup>i</sup> <sup>G</sup>, <sup>E</sup><sup>i</sup> <sup>I</sup>, Ei <sup>F</sup>, <sup>E</sup><sup>i</sup> <sup>D</sup> are, respectively, the distance, optical flow, intensity and gradient energies.

For each iteration, the pixel ðx, yÞ is assigned to the target that provides the maximum number of closest components between the energy vectors Ei pixel and <sup>E</sup><sup>i</sup> target, if the pixel ðx, yÞ is assigned to the target number p. The energy vector E<sup>p</sup> target is then updated as follows:

$$E\_{\text{target}}^p = \left(\frac{\text{N} \ast E\_{\text{target}}^p + E\_{\text{pixel}}^i}{N + 1}\right) \tag{2}$$

where N is the number of pixels in the target number p, before adding the pixel ðx, yÞ. Figure 4 shows the results of the multiobjects separation method.

Figure 3. Bounding boxes extraction.

Figure 4. Multiobjects separation result. (a) Original frame with five moving objects. (b) Objects separation result.

### 3. Event tracking

Ei

Ep

of closest components between the energy vectors Ei

shows the results of the multiobjects separation method.

to the target number p. The energy vector E<sup>p</sup>

where E<sup>i</sup>

gies.

<sup>G</sup>, <sup>E</sup><sup>i</sup> <sup>I</sup>, Ei <sup>F</sup>, <sup>E</sup><sup>i</sup>

78 Motion Tracking and Gesture Recognition

Figure 3. Bounding boxes extraction.

pixel <sup>¼</sup> <sup>E</sup><sup>i</sup>

target <sup>¼</sup> <sup>N</sup>�Ep

<sup>D</sup>, <sup>E</sup><sup>i</sup> <sup>F</sup>, <sup>E</sup><sup>i</sup> <sup>I</sup>, <sup>E</sup><sup>i</sup> G

For each iteration, the pixel ðx, yÞ is assigned to the target that provides the maximum number

where N is the number of pixels in the target number p, before adding the pixel ðx, yÞ. Figure 4

Figure 4. Multiobjects separation result. (a) Original frame with five moving objects. (b) Objects separation result.

<sup>D</sup> are, respectively, the distance, optical flow, intensity and gradient ener-

pixel and <sup>E</sup><sup>i</sup>

pixel

target <sup>þ</sup> <sup>E</sup><sup>i</sup>

N þ 1 !

target is then updated as follows:

� �<sup>T</sup> <sup>ð</sup>1<sup>Þ</sup>

target, if the pixel ðx, yÞ is assigned

ð2Þ

#### 3.1. Objects tracking

Once the targets are extracted from the current frame, the objective is to develop a dense optical flow computation algorithm to track them. We firstly estimate the optical flow of Harris points using an iterative Lucas-Kanade algorithm [17]. We consider that these particular points have a stable optical flow. The optical flow for every pixel of the detected objects is then estimated by propagating the optical flow of Harris points using a Gaussian distribution. The mean and standard deviation of the distribution are taken as the mean and standard deviation of the Harris points' optical flow [19]. The results of the optical flow propagation process are then processed by Kalman filtering to correct the optical flow of all the pixels of the detected objects [18, 19].

The tracking process is tested and evaluated in [18, 19]. Figure 5 shows an example of multiobjects tracking by combining the objects detection and separation method, and the tracking process.

Figure 5. Tracking process: from right to left.

#### 3.2. Optical flow–based object segmentation

Given a target, the objective is to partition it into multiple rectangular boxes representing different regions based on optical flow of its pixels. To achieve that, we use a recursive algorithm, which compares neighboring pixels to extract regions in which the pixels have a homogeneous optical flow. Only regions with a significant size are conserved (determined experimentally in dependence of the camera's view around the level crossing area and the resolution of the camera). Figure 6 presents optical flow–based segmentation results for a moving object tracked in an image sequence. In order to predict the normal (supposed) trajectories, the extracted regions are represented by the gravity centers of the boxes surrounding them. Then, each specific trajectory should be linked to the gravity center of its extracted region.

#### 3.3. Ideal trajectory prediction

Let us consider, thanks to optical flow, an extracted region. When we consider the center of the region, two trajectories could be defined: current ideal trajectory and predicted ideal trajectory. The current ideal trajectory corresponds to the trajectory that the center of the region should follow to avoid potential dangerous situations (Figure 9). The predicted ideal trajectory corresponds to the trajectory that should be followed to come back toward the current ideal trajectory (Figure 7). A statistical approach based on a hidden Markov model (HMM) is

Figure 6. Segmentation of an object using optical flow procedure.

proposed to predict the new ideal trajectory: the predicted ideal trajectory of the considered region center at time instant t (state qt : initial state of the HMM) is constructed from the states <sup>ð</sup>qtþ<sup>1</sup>, …, qtþtf Þ generated by the HMM using Forward-Backward, Viterbi and Baum-Welch algorithms [20, 21]. We also associate to the considered region center the four following parameters: velocity ðVtþ<sup>1</sup>, …, VtþtfÞ, acceleration ðatþ<sup>1</sup>, …, atþtfÞ, orientation ðotþ<sup>1</sup>, …, otþtfÞ, position <sup>ð</sup>ptþ<sup>1</sup>, …, ptþtf Þ and the distance ðDtþ<sup>1</sup>, …, DtþtfÞ from the region center to the current ideal trajectory.

Figure 8 shows the general architecture of the proposed HMM and how the predicted ideal trajectory is performed from the considered region center.

As shown in Figure 8a, the random hidden state variable qt corresponds to the position of the considered region center at time t. The random observation variable ut represents simultaneously the acceleration, orientation, velocity and position of the considered region center at time t. at <sup>t</sup>þ<sup>1</sup> represents the transition probability from state qt to state qtþ<sup>1</sup>, and bt represents the distribution of the observation at time t.

As illustrated in Figure 8b, given the velocity vector V<sup>t</sup> ! at time t, calculated from optical flow, the state qtþ<sup>1</sup> in the HMM is reached from the state qt with a probability of 1. Given the acceleration, orientation and velocity at time t, the velocity V<sup>t</sup>þ<sup>1</sup> ! at time t þ 1 is then predicted. Motion Tracking and Potentially Dangerous Situations Recognition in Complex Environment http://dx.doi.org/10.5772/68141 81

Figure 7. Ideal trajectory prediction by using HMM.

proposed to predict the new ideal trajectory: the predicted ideal trajectory of the considered

algorithms [20, 21]. We also associate to the considered region center the four following parameters: velocity ðVtþ<sup>1</sup>, …, VtþtfÞ, acceleration ðatþ<sup>1</sup>, …, atþtfÞ, orientation ðotþ<sup>1</sup>, …, otþtfÞ,

Figure 8 shows the general architecture of the proposed HMM and how the predicted ideal

As shown in Figure 8a, the random hidden state variable qt corresponds to the position of the considered region center at time t. The random observation variable ut represents simultaneously the acceleration, orientation, velocity and position of the considered region center at

the state qtþ<sup>1</sup> in the HMM is reached from the state qt with a probability of 1. Given the

<sup>t</sup>þ<sup>1</sup> represents the transition probability from state qt to state qtþ<sup>1</sup>, and bt represents the

!

!

Þ generated by the HMM using Forward-Backward, Viterbi and Baum-Welch

Þ and the distance ðDtþ<sup>1</sup>, …, DtþtfÞ from the region center to the current

: initial state of the HMM) is constructed from the states

at time t, calculated from optical flow,

at time t þ 1 is then predicted.

region center at time instant t (state qt

80 Motion Tracking and Gesture Recognition

Figure 6. Segmentation of an object using optical flow procedure.

distribution of the observation at time t.

trajectory is performed from the considered region center.

As illustrated in Figure 8b, given the velocity vector V<sup>t</sup>

acceleration, orientation and velocity at time t, the velocity V<sup>t</sup>þ<sup>1</sup>

<sup>ð</sup>qtþ<sup>1</sup>, …, qtþtf

ideal trajectory.

time t. at

position <sup>ð</sup>ptþ<sup>1</sup>, …, ptþtf

Figure 8. Schematic representation of the HMM for ideal trajectory prediction. (a) HMM model. (b) Trajectory prediction model.

As illustrated in Figure 8b, we next define a circle C. The center of the circle C is the end point of the velocity vector V<sup>t</sup>þ<sup>1</sup> �! and the radius of this circle is the maximum absolute acceleration.

#### 4. Evaluation of the level of dangerousness (recognition of dangerous situations)

On the basis of the previous steps, we present in this section a method to evaluate and recognize potential dangerous situations when a moving object is detected within the monitored area of a level crossing.

To analyze the predicted ideal trajectory, various sources of dangerousness are considered based on Dempster-Shafer theory [16]. This theory combines the dangers produced by the different sources in order to obtain a measure of the degree of danger.

For each region center, five sources of danger are considered: acceleration, orientation, velocity, position and distance between the predicted and the current ideal trajectories. A mass assignment is then defined for each source of danger. Let m<sup>i</sup> be the belief mass related to the danger source number i. The belief masses are defined as follows:

The mass assignment m<sup>1</sup> (position) is computed from the distance between the predicted position ptþtf at time instant t þ tf and the barrier of the level crossing:

$$m^1 = \frac{|P\_d - 0.5|}{0.5} \tag{3}$$

$$P\_d = \int\_{-\infty}^{d\_c^f} G\_{d\_N, o\_d}(\mathbf{x}) d\mathbf{x} \,\,\sigma\_d = \sqrt{D\_{\text{max}}} \,\,\, d\_N = 0 \tag{4}$$

where GdN,<sup>σ</sup><sup>d</sup> ðxÞ is a Gaussian distribution of the variable x. dN ¼ 0 and σ<sup>d</sup> are, respectively, its mean and standard deviation. Dmax is the maximum distance that an object can traverse in the image. d tf <sup>c</sup> is a function given as follows:

$$d\_c^{t\prime} = \begin{cases} W & \text{if } \overrightarrow{D}\_p \cdot \overrightarrow{V}\_c^{t\prime} \le 0 \\\ W + \overrightarrow{D}\_p^\* \cdot \overrightarrow{V}\_c^{t\prime} & \text{if } \overrightarrow{D}\_p \cdot \overrightarrow{V}\_c^{t\prime} > 0 \end{cases} \tag{5}$$

$$\mathcal{W} = \text{coef} \ast \text{sqrt}(D\_s) \; D\_s = \frac{D\_{\text{max}}}{5} \tag{6}$$

$$\text{coef} = \frac{D\_s}{D\_p + D\_i} \ D\_i = \begin{cases} 0 & \text{if inside probability zone} \\ D\_{\text{max}} & \text{if outside probability zone} \end{cases} \tag{7}$$

$$
\overrightarrow{D\_p^\*} = \begin{bmatrix} \frac{1}{D\_{p\_x}} & \frac{1}{D\_{p\_y}} \end{bmatrix}^T \tag{8}
$$

where Dp ! ¼ ½ Dpx Dpy � <sup>T</sup> is the distance from the region center to the barrier of the level crossing. Vtf c ! is the velocity of the considered region center at time tf . The parameter W depends on the position of the region center in the level crossing zone (inside or outside a prohibited LC zone). More the value of d tf <sup>c</sup> is greater than zero more the belief mass m<sup>1</sup> will increase (degree of dangerousness increments).

As illustrated in Figure 8b, we next define a circle C. The center of the circle C is the end point

On the basis of the previous steps, we present in this section a method to evaluate and recognize potential dangerous situations when a moving object is detected within the moni-

To analyze the predicted ideal trajectory, various sources of dangerousness are considered based on Dempster-Shafer theory [16]. This theory combines the dangers produced by the

For each region center, five sources of danger are considered: acceleration, orientation, velocity, position and distance between the predicted and the current ideal trajectories. A mass assignment is then defined for each source of danger. Let m<sup>i</sup> be the belief mass related to the danger

The mass assignment m<sup>1</sup> (position) is computed from the distance between the predicted

<sup>m</sup><sup>1</sup> <sup>¼</sup> <sup>j</sup>Pd � <sup>0</sup>:5<sup>j</sup>

GdN,<sup>σ</sup><sup>d</sup> <sup>ð</sup>xÞdx <sup>σ</sup><sup>d</sup> <sup>¼</sup> ffiffiffiffiffiffiffiffiffiffi

where GdN,<sup>σ</sup><sup>d</sup> ðxÞ is a Gaussian distribution of the variable x. dN ¼ 0 and σ<sup>d</sup> are, respectively, its mean and standard deviation. Dmax is the maximum distance that an object can traverse in the

<sup>W</sup> <sup>¼</sup> coef �sqrtðDs<sup>Þ</sup> Ds <sup>¼</sup> Dmax

<sup>¼</sup> <sup>1</sup> Dpx

<sup>c</sup> <sup>¼</sup> <sup>W</sup> if Dp

�

D� p �!

Wþ D� p ! : Vtf c !

Dmax

! : Vtf c !

if Dp ! : Vtf c !

Di <sup>¼</sup> <sup>0</sup> if inside prohibited zone Dmax if outside prohibited zone

> 1 Dpy h i<sup>T</sup>

≤ 0

> 0

<sup>0</sup>:<sup>5</sup> <sup>ð</sup>3<sup>Þ</sup>

<sup>p</sup> dN <sup>¼</sup> <sup>0</sup> <sup>ð</sup>4<sup>Þ</sup>

<sup>5</sup> <sup>ð</sup>6<sup>Þ</sup>

ð5Þ

ð7Þ

ð8Þ

at time instant t þ tf and the barrier of the level crossing:

different sources in order to obtain a measure of the degree of danger.

source number i. The belief masses are defined as follows:

Pd ¼ ð d t f c

<sup>c</sup> is a function given as follows:

d tf

coef <sup>¼</sup> Ds

Dp þ Di

�∞

8 ><

>:

4. Evaluation of the level of dangerousness (recognition of dangerous

and the radius of this circle is the maximum absolute acceleration.

of the velocity vector V<sup>t</sup>þ<sup>1</sup>

82 Motion Tracking and Gesture Recognition

tored area of a level crossing.

situations)

position ptþtf

image. d tf �!

The mass assignment m<sup>2</sup> (velocity) is computed from the difference between the predicted velocity Vtf <sup>c</sup> at time instant tf and a prefixed nominal velocity VN.

$$m^2 = \begin{cases} 0.01 & \text{if } (V\_c^{t\_f} - V\_N) \le 0\\ \frac{P\_v - 0.5}{0.5} & \text{if } (V\_c^{t\_f} - V\_N) > 0 \end{cases} \tag{9}$$

$$P\_v = \int\_{-\infty}^{V\_\varepsilon^\circ} G\_{V\_\mathbf{n}, \sigma\_v}(\mathbf{x}) d\mathbf{x} \,\sigma\_v = \frac{V\_N}{4} \tag{10}$$

where Vtf <sup>c</sup> is the velocity of the considered region center at time tf . VN represents the maximal velocity that a target can reach in the image. GVn,<sup>σ</sup><sup>v</sup> ðxÞ is a Gaussian distribution, with a mean equal to VN and a standard deviation equal to σv.

The mass assignment m<sup>3</sup> (orientation) is computed by comparing the angle of the predicted velocity Vt at time instant t and the angle of the current ideal trajectory.

$$m^3 = \frac{|P\_o - 0.5|}{0.5} \tag{11}$$

$$P\_o = \int\_{-\infty}^{\frac{\delta^\circ}{\sigma^\circ\_\varepsilon}} G\_{o\_\mathcal{N}, \sigma\_o}(\mathbf{x}) \, d\mathbf{x} \, \sigma\_o = \frac{2 \ast \pi}{7} \tag{12}$$

where o tf <sup>c</sup> is the velocity orientation of the considered region center at time tf . ON is the orientation of the current ideal trajectory. GoN,<sup>σ</sup><sup>o</sup> ðxÞ is a Gaussian distribution, with a mean equal to ON and a standard deviation equal to σo.

The mass assignment m<sup>4</sup> (acceleration) is computed from the difference between the predicted accelerations at and atþtf at time instants t and t þ tf respectively.

$$m^4 = \begin{cases} 0.01 & \text{if } (a\_c^{t\_f} - a\_N) \le 0\\ \frac{P\_d - 0.5}{0.5} & \text{if } (a\_c^{t\_f} - a\_N) > 0 \end{cases} \tag{13}$$

$$P\_a = \int\_{-\infty}^{\frac{\omega}{d\_c}} G\_{a\_N, \sigma\_a}(\mathbf{x}) \, d\mathbf{x} \, \sigma\_a = \frac{a\_N}{4} \tag{14}$$

$$a\_c^{t\_f} = \frac{V\_c^{t\_f} - V\_c^{t\_i}}{n \ast T\_{misjour}} \tag{15}$$

where a tf <sup>c</sup> is the acceleration of the considered region center at time tf . aN represents the maximal acceleration that a target can reach in the image. GaN,<sup>σ</sup><sup>a</sup> ðxÞ is a Gaussian distribution, with a mean equal to aN and a standard deviation equal to σa.

Finally, the mass assignment m<sup>5</sup> (distance) is computed from the distance between the predicted position ptf at time instant tf and the current ideal trajectory:

$$m^5 = \frac{|P\_D - 0.5|}{0.5} \tag{16}$$

$$P\_D = \int\_{-\infty}^{D^\dagger} G\_{D\nu, \sigma\_D}(\mathbf{x}) \, d\mathbf{x} \quad \sigma\_D = 2 \ast V\_n \quad D\_N = 0 \tag{17}$$

where <sup>D</sup>tf is the the distance between the predicted position ptf of the considered region center at time tf and the current ideal trajectory. GDN,<sup>σ</sup><sup>D</sup> ðxÞ is a Gaussian distribution, with a mean equal to DN ¼ 0 and a standard deviation equal to σD.

Once the degrees of dangerousness are computed for the five sources, Dempster-Shafer [16] combination is used to determine the degree of danger related to the considered region center:

$$\text{Danger} = \text{Demoster} - \text{Shafter}(m^1, m^2, m^3, m^4, m^5) \tag{18}$$

To determine the degree of danger of the target, we take simply the maximum value among the degrees of danger of all regions composing the target.

#### 5. Video surveillance experimental results

To validate our work, we apply the proposed dangerous situation method on four typical accidental scenarios. These scenarios, registered at a level crossing in the north of France (Mouzon), correspond to real situations occurred in LC accidents (dangerous situations: vehicle zigzagging between the closed half barriers, presence of obstacle in the level crossing and pedestrian crossing level crossing area). Each analyzed scenario includes a sequence with more than 500 frames. Table 1 presents the datasets and materials used in the analysis of our method.

#### 5.1. Experimental methodology

In the framework of this chapter, we determine a pure quantitative degree of dangerousness from different scenarios identified at level crossing (see Eq. (18)). This system is able to detect potentially dangerous situations occurring at the LC both in the two cases (states of the barriers): barriers opened or barriers closed.


Table 1. Dataset and materials.

a tf <sup>c</sup> <sup>¼</sup> <sup>V</sup>tf

with a mean equal to aN and a standard deviation equal to σa.

PD ¼

equal to DN ¼ 0 and a standard deviation equal to σD.

the degrees of danger of all regions composing the target.

5. Video surveillance experimental results

5.1. Experimental methodology

barriers): barriers opened or barriers closed.

D ð t f

�∞

where a tf

predicted position ptf

84 Motion Tracking and Gesture Recognition

<sup>c</sup> � <sup>V</sup>ti c n�Tmisejour

<sup>c</sup> is the acceleration of the considered region center at time tf . aN represents the

maximal acceleration that a target can reach in the image. GaN,<sup>σ</sup><sup>a</sup> ðxÞ is a Gaussian distribution,

Finally, the mass assignment m<sup>5</sup> (distance) is computed from the distance between the

<sup>m</sup><sup>5</sup> <sup>¼</sup> <sup>j</sup>PD � <sup>0</sup>:5<sup>j</sup>

where <sup>D</sup>tf is the the distance between the predicted position ptf of the considered region center at time tf and the current ideal trajectory. GDN,<sup>σ</sup><sup>D</sup> ðxÞ is a Gaussian distribution, with a mean

Once the degrees of dangerousness are computed for the five sources, Dempster-Shafer [16] combination is used to determine the degree of danger related to the considered region center:

To determine the degree of danger of the target, we take simply the maximum value among

To validate our work, we apply the proposed dangerous situation method on four typical accidental scenarios. These scenarios, registered at a level crossing in the north of France (Mouzon), correspond to real situations occurred in LC accidents (dangerous situations: vehicle zigzagging between the closed half barriers, presence of obstacle in the level crossing and pedestrian crossing level crossing area). Each analyzed scenario includes a sequence with more than 500 frames. Table 1 presents the datasets and materials used in the analysis of our method.

In the framework of this chapter, we determine a pure quantitative degree of dangerousness from different scenarios identified at level crossing (see Eq. (18)). This system is able to detect potentially dangerous situations occurring at the LC both in the two cases (states of the

, m<sup>2</sup> , m<sup>3</sup> , m<sup>4</sup> , m<sup>5</sup>

Danger <sup>¼</sup> Dempster � Shaf erðm<sup>1</sup>

<sup>0</sup>:<sup>5</sup> <sup>ð</sup>16<sup>Þ</sup>

Þ ð18Þ

GDN,<sup>σ</sup><sup>D</sup> ðxÞ dx σ<sup>D</sup> ¼ 2�Vn DN ¼ 0 ð17Þ

at time instant tf and the current ideal trajectory:

ð15Þ

In the case of closed barriers, when train approaching, we provide a level of dangerousness (see Eq. (18)), between 0 and 100%, which could be transmitted all the time to the drivers approaching the level crossing (LC). We could also send additional security advices when the level is greater than a threshold (for instance 75%). In any situation, the presence of any kind of moving objects between the barriers is not allowed. So, the velocity of cars or moving objects should decrease to zero when approaching the LC.

In the case of opened barriers, the surveillance system is working as well. Vehicles or moving objects could traverse the level crossing zone but they couldn't stop on the LC. In case of detection of dangerous situations, we can send information like: barriers open with a pedestrian, a vehicle or an object stopped on the rails.

Table 2 shows different situations that are taken into account by the system to measure the level of dangerousness. In both cases, it all depends on the way of the rail transport operator wants to monitor the LC. For the moment, the final system was not integrated in the daily management of a level crossing. So, when the system will be integrated in a rail network, we


Table 2. Situations allowed for open and closed barriers.

could imagine that the measure of the level of dangerousness will be validated qualitatively by the rail safety experts.

#### 5.2. Scenarios of accident analyzed by the system

Vehicle zigzagging between two closed barriers of LC (Figure 9): In this scenario, a vehicle is approaching the LC, while the barriers are closed. The vehicle crosses the LC, zigzagging between the closed barriers (Figure 9). The purple lines in Figure 9 represent the current ideal trajectory of the center of each extracted region from the object. The white points in the figure represent the instantly predicted displacement of the center of the extracted regions. As shown in Figure 9, if a detected vehicle is approaching the LC and using an abnormal trajectory, the degree of danger is going to increase gradually to reach 70%. Then, when the vehicle enters the LC, this degree continues to grow until reaching 100%. The level of danger begins to decrease when the vehicle is moving away from the LC (Degree of danger DV1 ¼ 40%).

Vehicle stopped (Figure 10): In this scenario, a vehicle crosses the level crossing while the barriers are open (Figure 10). Suddenly, the vehicle stops inside the dangerous zone and becomes a fixed obstacle. After a while, the vehicle moves and leaves the LC. Concerning danger evaluation, the degree of dangerousness related to the detected vehicle increases when it moves toward the level crossing. It reaches 46% during the crossing of the zone of danger. When the vehicle stops in the zone of danger, the stationary is detected and the degree of dangerousness takes a value of 100%. When the vehicle begins to leave the LC, the level of danger decreases progressively.

Queuing across the rail level crossing (Figure 11): In this scenario, a first vehicle stops just after the dangerous zone. Sometime later, two other vehicles find themselves blocked behind the first vehicle, which is motionless. This situation leads to a queue of cars inside the LC (Figure 11). When the two vehicles detected inside the LC are stopped inside the zone of danger, their degree of dangerousness increases progressively and reach their maximum (100%). When the two vehicles restart moving, the degree of dangerousness drops to 46% and decreases gradually, as the vehicles leave away the level crossing.

Pedestrians' scenario (Figure 12): In this scenario, three pedestrians (P1, P2 and P3) are walking around the level crossing zone as the barriers are closed. Pedestrian P1 is moving toward the zone of danger (Degree of danger DP1 ¼ 26%), while pedestrian P2 is crossing is crossing the level crossing area (DP2 ¼ 100%), and Pedestrian P3 is stopped on the middle part of the level crossing near from the rails (DP3 ¼ 100%). After a moment, pedestrian P1 arrives near pedestrian P2, and they are stopped on the rails of the LC, taking into account that the stationary inside the level crossing is always detected by the system. So, the degree of dangerousness related to the pedestrian P1 increases progressively from DP1 ¼ 26% and reaches their maximum DP1 ¼ 100% on the rails. At the end of the scenario, pedestrian P2 is leaving the level crossing zone (Degree of danger DP2 decreases to 11%), when pedestrian P1 is moving toward the stopped pedestrian P3. A vehicle passing near from the LC is also detected in this scenario (DV ¼ 13%).

As a conclusion of these tests, the measure of the prediction system that calculates the level of dangerousness for each moving or stopped object around the LC is able to detect different kind of dangerous scenarios in the case of closed or opened barriers (vehicle zigzagging, stopped Motion Tracking and Potentially Dangerous Situations Recognition in Complex Environment http://dx.doi.org/10.5772/68141 87

could imagine that the measure of the level of dangerousness will be validated qualitatively by

Vehicle zigzagging between two closed barriers of LC (Figure 9): In this scenario, a vehicle is approaching the LC, while the barriers are closed. The vehicle crosses the LC, zigzagging between the closed barriers (Figure 9). The purple lines in Figure 9 represent the current ideal trajectory of the center of each extracted region from the object. The white points in the figure represent the instantly predicted displacement of the center of the extracted regions. As shown in Figure 9, if a detected vehicle is approaching the LC and using an abnormal trajectory, the degree of danger is going to increase gradually to reach 70%. Then, when the vehicle enters the LC, this degree continues to grow until reaching 100%. The level of danger begins to decrease

Vehicle stopped (Figure 10): In this scenario, a vehicle crosses the level crossing while the barriers are open (Figure 10). Suddenly, the vehicle stops inside the dangerous zone and becomes a fixed obstacle. After a while, the vehicle moves and leaves the LC. Concerning danger evaluation, the degree of dangerousness related to the detected vehicle increases when it moves toward the level crossing. It reaches 46% during the crossing of the zone of danger. When the vehicle stops in the zone of danger, the stationary is detected and the degree of dangerousness takes a value of 100%. When the vehicle begins to leave the LC, the level of

Queuing across the rail level crossing (Figure 11): In this scenario, a first vehicle stops just after the dangerous zone. Sometime later, two other vehicles find themselves blocked behind the first vehicle, which is motionless. This situation leads to a queue of cars inside the LC (Figure 11). When the two vehicles detected inside the LC are stopped inside the zone of danger, their degree of dangerousness increases progressively and reach their maximum (100%). When the two vehicles restart moving, the degree of dangerousness drops to 46% and

Pedestrians' scenario (Figure 12): In this scenario, three pedestrians (P1, P2 and P3) are walking around the level crossing zone as the barriers are closed. Pedestrian P1 is moving toward the zone of danger (Degree of danger DP1 ¼ 26%), while pedestrian P2 is crossing is crossing the level crossing area (DP2 ¼ 100%), and Pedestrian P3 is stopped on the middle part of the level crossing near from the rails (DP3 ¼ 100%). After a moment, pedestrian P1 arrives near pedestrian P2, and they are stopped on the rails of the LC, taking into account that the stationary inside the level crossing is always detected by the system. So, the degree of dangerousness related to the pedestrian P1 increases progressively from DP1 ¼ 26% and reaches their maximum DP1 ¼ 100% on the rails. At the end of the scenario, pedestrian P2 is leaving the level crossing zone (Degree of danger DP2 decreases to 11%), when pedestrian P1 is moving toward the stopped pedestrian P3.

As a conclusion of these tests, the measure of the prediction system that calculates the level of dangerousness for each moving or stopped object around the LC is able to detect different kind of dangerous scenarios in the case of closed or opened barriers (vehicle zigzagging, stopped

when the vehicle is moving away from the LC (Degree of danger DV1 ¼ 40%).

decreases gradually, as the vehicles leave away the level crossing.

A vehicle passing near from the LC is also detected in this scenario (DV ¼ 13%).

the rail safety experts.

86 Motion Tracking and Gesture Recognition

danger decreases progressively.

5.2. Scenarios of accident analyzed by the system

Figure 9. Vehicle zigzagging. DV1 represents the degree of danger associated with the vehicle.

Figure 10. The presence of obstacle (vehicle) in the level crossing. DV1 represents the degree of danger associated with the vehicle.

Motion Tracking and Potentially Dangerous Situations Recognition in Complex Environment http://dx.doi.org/10.5772/68141 89

Figure 11. The presence of stopped vehicles line on the LC. DVi represents the degree of danger associated with the vehicle number i. DP represents the degree of danger associated with a pedestrian outside the LC zone.

Figure 10. The presence of obstacle (vehicle) in the level crossing. DV1 represents the degree of danger associated with

the vehicle.

88 Motion Tracking and Gesture Recognition

Figure 12. Three pedestrians walking around the level crossing area.

vehicles, pedestrians around the LC). Then, this system proves his effectiveness as a measure of the level of dangerousness but it also requires to be validated qualitatively after installing this system definitely on a rail transport.

#### 6. Conclusion

Detection of moving objects is an important and basic task for video surveillance systems, for which you can define the initial position of the moving objects in a surveillance scene. However, the detection and separation of the moving objects process become difficult when the objects are close to each other in the scene. In our approach, we propose a method to completely separate the corresponding pixels of each defined target. One of the other objectives of this project is to develop a video surveillance system that will be able to detect and recognize potential dangerous situation around level crossings. Different typical LC accident scenarios (e.g., presence of obstacles, zigzagging between the barriers, stopped cars line) acquired in real conditions are experimentally evaluated by applying the proposed dangerous situation recognition system.

#### Acknowledgements

This work is developed within the framework of PANsafer project (Towards a safer level crossing), supported by the ANR French work program.

#### Author details

Houssam Salmane<sup>1</sup> \*, Yassine Ruichek<sup>2</sup> and Louahdi Khoudour<sup>3</sup>


2 Le2i FRE2005, CNRS, Arts et Métiers, University of Bourgogne Franche-Comté, UTBM, Belfort, France

3 CEREMA, Centre for Studies and Expertise on Risks, Environment, Mobility, and, Urban and Country Planning, Toulouse, France

#### References

Figure 12. Three pedestrians walking around the level crossing area.

90 Motion Tracking and Gesture Recognition

[1] Schnieder E, Slovak R, El Koursi EM, Tordai L, Woods M. A European contribution to level crossing safety. Proceedings of FOVUS, Stuttgart, Germany. September 2008:222– 228


[16] Hong X, Huang Y, Ma W, Miller P, Liu W, Zhou H. Video event recognition by Dempster-Shafer theory. Proceedings of the Twenty-first European Conference on Artificial Intelligence. ECAI'14. Prague, Czech Republic: IOS Press, 2014, pp. 1031–1032. ISBN: 978-1- 61499-418-3

[2] US DOT FRA web site. Intelligent Grade Crossings: IGC. Available from: https://www.

[3] Selectra Vision Company. Level Crossing Surveillance [Internet]. Available from: http://

[4] Fakhfakh N, Khoudour L, et al. Background Subtraction and 3D Localization of Moving and Stationary Obstacles at Level Crossings. International Conference on Image

[5] Oreifej O, Liu Z. Histogram of oriented 4d normals for activity recognition from depth sequences. Proceedings of the IEEE Conference on Computer Vision and Pattern Recog-

[6] Prakash O, Khare M, Binh NT, Khare A. Human object detection in images using shiftinvariant stationary wavelet transform. Proceedings of the International Conference on Recent Cognizance in Wireless Communication & Image Processing. 2016;481–487 [7] Vosters L, Shan C, Gritti T. Real-time robust background subtraction under rapidly changing illumination conditions. Image and Vision Computing, Elseiver. 2012;30:1004–

[8] Hare S, Golodetz S, Saffari A, Vineet V. Cheng MM, Hicks SL, Torr PH. Structured output tracking with kernels. IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9] Blake A, and Isard M. Active Contours: The Application of Techniques from Graphics, Vision,Control Theory and Statistics to Visual Tracking of Shapes in Motion. 1998, 1st

[10] Arvind BC, Nagaraj SK, Seelamantula CS, Gorthi SS. Active-disc-based Kalman filter technique for tracking of blood cells in microfluidic channels. Phoenix, Arizona, USA:

[11] Fukui S, Hayakawa S, Iwahori Y, Nakamura T, Bhuyan MK. Particle Filter Based Tracking with Image-based Localization. Procedia Computer Science. 2016;96:977–986

[12] Klitzke L, Koch C. Robust object detection for video surveillance using stereo vision and Gaussian mixture model. Journal of WSCG (World Society for Computer Graphics).

[13] Vojir T, Matas J, Noskova J. Online adaptive hidden markov model for multi-tracker

[14] Karaman S, Benois-Pineau J, Dovgalecs V, Mégret R, Pinquier J, André-Obrecht R, Gaëstel Y, Dartigues JF. Hierarchical Hidden Markov Model in detecting activities of daily living in wearable videos for studies of dementia.Multimedia Tools and Applications. 2014;69:743–771

[15] Natarajan P, Nevatia R. Coupled Hidden Semi Markov Models for Activity Recognition, Proceedings of IEEE Workshop on Motion and Video Computing. 2007. WMVC '07. IEEE

fusion. Computer Vision and Image Understanding. 2016;153:109–119

edn. Springer-Verlag New York, Inc. Secaucus, NJ, USA, ISBN:3540762175

IEEE International Conference on Image Processing (ICIP). 2016;3394–3398

Václav Skala-UNION Agency. 2016;24(1):9–17

Workshop on p. 1010. doi: 10.1109/WMVC.2007.12

fra.dot.gov/Page/P0309

92 Motion Tracking and Gesture Recognition

nition. 2013:716–723

2016;38:2096–2109

1015

www.selectravision.com/obstacle-lxs.php

Processing Theory, Tools and Applications. 2010:72–78

