*3.3.1 Initialization of intrinsic and extrinsic calibration*

For the color camera, the initial estimation of *Ic* and *T*ð Þ*<sup>i</sup> <sup>c</sup>* for all calibration images is carried out as described in Bouguet's toolbox. The intrinsic parameters for the depth camera are defined as *I* 0 *<sup>d</sup>* ¼ f g f*d;* c*d;* k*d;c*0*;c*<sup>1</sup> , since the depth distortion terms are not considered. They are initialized using preset values, which are publicly

#### **Figure 3.**

*Checkerboard RGB (top) images and the corresponding IR (bottom) calibration images. From the case study roads, a database of 10,540 color and depth test image frames has been acquired and being processed.*


### **Table 2.**

*Intrinsic, distortion and extrinsic calibration matrix parameters.*

*On the Use of Low-Cost RGB-D Sensors for Autonomous Pothole Detection with Spatial… DOI: http://dx.doi.org/10.5772/intechopen.88877*

available for the Kinect, online. For each input disparity map *i*, the plane corners are extracted, defining a polygon. For each point x*<sup>d</sup>* inside the polygon, the corresponding disparity *<sup>d</sup>* is used for computing a depth value z*<sup>d</sup>* using z <sup>¼</sup> <sup>1</sup> *c*1*du*þ*c*<sup>0</sup> , where *d* ¼ *du* since the measured disparities are used, and *c*<sup>0</sup> and *c*<sup>1</sup> are part of the depth camera's intrinsics. The correspondences *xd; yd; zd* � � are used for computing 3D X*<sup>c</sup>* points originating a 3D point cloud. To each 3D point cloud, a plane is fitted using a standard total least squares algorithm.

#### **3.4 Pothole search engine**

checkerboard, with 30 mm square fields, a set of close-up RGB/IR images of the checkerboard placed in different positions and orientations (**Figure 3(a)**), can be collected and used for calibration. The Bouguet's Camera Calibration Toolbox [44] in MATLAB can be used for the identification of RGB and IR camera parameters, utilizing the two versions of Herrera's method [45]. IR camera calibration, the IR emitter should be disabled during imaging so as to achieve appropriate light conditions. The output matrices for the intrinsic, distortion and extrinsic calibration

is carried out as described in Bouguet's toolbox. The intrinsic parameters for the

are not considered. They are initialized using preset values, which are publicly

*Checkerboard RGB (top) images and the corresponding IR (bottom) calibration images. From the case study roads, a database of 10,540 color and depth test image frames has been acquired and being processed.*

**Intrinsic calibration matrix**

**Distortion calibration matrix** 0.243645 �0.572745 �0.008210 0.000119 **Extrinsic calibration matrix** 0.999987 �0.004894 �0.001283 110.506445 �0.004661 �0.989735 0.142836 �133.830468 �0.001969 �0.142828 �0.989746 867.124291 0.000000 0.000000 0.000000 1.000000

536.782668 0.000000 319.133028 0.000000 536.889190 258.356500 0.000000 0.000000 1.000000

*Intrinsic, distortion and extrinsic calibration matrix parameters.*

*<sup>c</sup>* for all calibration images

*<sup>d</sup>* ¼ f g f*d;* c*d;* k*d;c*0*;c*<sup>1</sup> , since the depth distortion terms

parameters are presented in **Table 2**.

depth camera are defined as *I*

**Figure 3.**

**Table 2.**

**156**

*3.3.1 Initialization of intrinsic and extrinsic calibration*

*Geographic Information Systems in Geospatial Intelligence*

For the color camera, the initial estimation of *Ic* and *T*ð Þ*<sup>i</sup>*

0

As a pre-processing step and prior to the segmentation and clustering of the RGB and depth data, pothole search engine (PSE) is necessary. It is then possible to extract potholes-only images for further autonomous processing. This can be accomplished by using a 2-class *k*-means clustering of the candidate RGB image frames, and is confirmed using ellipsoidal fitting on the classified binary image frame.

#### *3.4.1* k*-means clustering and edge ellipse fitting for pothole search*

Since the data collected comprises of pothole and non-pothole pavement defect image frames, the first preprocessing step after the calibration is to eliminate the non-pothole images from the database. Using unsupervised classification on the acquired RGB data frames, images with potential potholes are selected based on *k*means clustering [46], and adaptive median filtering. From the candidate potholes images, edge lines are estimated and the corresponding ellipse(s) are fitted using least squares optimization. This algorithm is applied in a batch processing mode, and the efficiency of the approach is then confirmed by using visual inspection and comparison.

#### *3.4.2 Horizontal and vertical integral projection (HVIP)*

Integral projection (IP) has the discriminative to accumulate and resolve the pixel histograms into pothole and non-potholes pixels, by analyzing the horizontal and vertical (HV) pixel distributions within an image, represented by horizontal and vertical projections. Given a grayscale image *I*(*x, y*), the horizontal and vertical IPs are defined as follows in Eqs. (15) and (16).

$$\text{HP}\ (\mathcal{y}) = \sum\_{i \in \mathbf{x}\_2} I(i, j) \tag{15}$$

$$\text{VP}\left(\mathbf{x}\right) = \sum\_{j \in \mathcal{Y}\_{\mathbf{x}}} I(i, \mathcal{y}) \tag{16}$$

where HP and VP are the horizontal and vertical IP, respectively. *xy* and *yx* denote the set of horizontal pixels at the vertical pixel *y* and the set of vertical pixels at the horizontal pixel *x*, respectively.

#### *3.4.3 Database search for candidate pothole image frames using ellipse fitting and HVIP*

With a visual comparison of 99% efficiency for the pothole database search, **Table 3** shows the results using the pothole search engine (PSE). The ellipse detection indicates the presence of defect or no-defect within the image, and also defines the orientation of the pothole with respect to the longitudinal profile of the road.

 *data.* The results of horizontal and vertical IP (HVIP) analysis for several pavement images with varied sized pixels are presented in **Table 3**. As observed from the test results, a structurally healthy pavement image with non-potholes (e.g., test image #2) is generally characterized by recognizably stable signals of both horizontal and vertical integral projections. On the other hand, the integral projections of images containing potholes (e.g., test images #1, #3 and #4), has peak(s) in either the vertical or horizontal or both IPs, depending on the strength or the severity of the pothole and lighting conditions. Where both the horizontal and vertical signals are strong, the locations of the two peaks tend to be relatively close to each other. Thus in addition to the ellipsoidal fitting, HVIP can effectively be used in the extraction of pothole and non-pothole image frames in a pothole database search engine system. In the PSE search system, data acquired under varied illumination conditions were tested, to ensure the effectiveness of the system with data of different

*On the Use of Low-Cost RGB-D Sensors for Autonomous Pothole Detection with Spatial*

**Figure 4** illustrates the conceptual approximation of a pothole with dimensional parameters that define the pothole metrology as: width, depth, surface area and volume. Assuming the potholes have the shape of a circular paraboloid, then in 2D

The depth-image plane (**Figure 4**) is one of the noise factors, whereby the plane is not necessarily parallel to the pavement surface. The noise points, which are the non-defect points between the pavement-pothole plane and the camera, have to be filtered out for the accurate depth detection and the subsequent 2D-pothole detection from the depth image. The general principle of removing the outlier points (noise), is by determining the local minimum of each column and then subtracting from the column itself in order to extract the pothole from the rest of data [47]. The minimum of each column defines the depth below which the pothole starts on the road pavement surface, and is referred to as the depth-image plane. Using this

*<sup>i</sup>* including the maximum depth

*Representation and approximation of pothole metrology elements: depth, width, surface area and volume.*

*<sup>i</sup>* for a given pothole is also computed.

*x* 2 þ *y* 2 .

*d*

*<sup>i</sup>*max can be quantified, and

*…*

resolutions.

approach, the depths

the mean depth

**Figure 4.**

**159**

**4. Pothole metrology data parametrization**

*DOI: http://dx.doi.org/10.5772/intechopen.88877*

they can be represented by the function *f x*ð Þ¼ *; y*

*d*

*d*

**4.1 Pothole depth determination using depth image**

*PSE*  *On the Use of Low-Cost RGB-D Sensors for Autonomous Pothole Detection with Spatial… DOI: http://dx.doi.org/10.5772/intechopen.88877*

The results of horizontal and vertical IP (HVIP) analysis for several pavement images with varied sized pixels are presented in **Table 3**. As observed from the test results, a structurally healthy pavement image with non-potholes (e.g., test image #2) is generally characterized by recognizably stable signals of both horizontal and vertical integral projections. On the other hand, the integral projections of images containing potholes (e.g., test images #1, #3 and #4), has peak(s) in either the vertical or horizontal or both IPs, depending on the strength or the severity of the pothole and lighting conditions. Where both the horizontal and vertical signals are strong, the locations of the two peaks tend to be relatively close to each other. Thus in addition to the ellipsoidal fitting, HVIP can effectively be used in the extraction of pothole and non-pothole image frames in a pothole database search engine system. In the PSE search system, data acquired under varied illumination conditions were tested, to ensure the effectiveness of the system with data of different resolutions.
