Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint Descriptors and Artificial Neural Network

Archana Harsing Sable and Haricharan A. Dhirbasi

## Abstract

This chapter presents a new technique called entropy volume-based scaleinvariant feature transform for correct face recognition post cosmetic surgery. The comparable features taken are the key points and volume of the Difference of Gaussian (DOG) structure for those points the information rate is confirmed. The information extracted has a minimum effect on uncertain changes in the face since the entropy is the higher-order statistical feature. Then the extracted corresponding entropy volume-based scale-invariant feature transform features are applied and provided to the support vector machine for classification. The normal scaleinvariant feature transform feature extracts the key points based on dissimilarity which is also known as the contrast of the image, and the volume-based scaleinvariant feature transform (V-SIFT) feature extracts the key points based on the volume of the structure. However, the EV-SIFT method provides both the contrast and volume information. Thus, EV-SIFT provides better performance when compared with principal component analysis (PCA), normal scale-invariant feature transform (SIFT), and V-SIFT-based feature extraction. Since it is well known that the artificial neural network (ANN) with Levenberg-Marquardt (LM) is a powerful computation tool for accurate classification, it is further used in this technique for better classification results.

Keywords: face recognition, plastic surgery, scale-invariant feature transform, (SIFT) feature, EV-SIFT feature, Levenberg-Marquardt-based neural network classifier (LM-NN)

## 1. Introduction

Human faces are multidimensional and complex visual stimuli, which contain useful information about the uniqueness of a person. Recognizing their faces used for security and authentication purposes has taken a new turn in the current era of computer image and vision analysis, for example, in monitoring applications, image recovery, man-machine interaction, and biometric authentication. Normally, the facial recognition system does not have the sense of touch or human interaction to

complete the recognition process. This is one of the benefits of face recognition in relation to other recognition methods. Facial recognition can designate the verification phase [1] or the identification phase [2]. In the verification phase, the correspondence between two faces is resolved. There are many methods available to achieve facial recognition [3–8]. But the accuracy of recognition is not always high. This is due to variations in lighting levels, facial expressions, poses, aging, lowresolution input images, or facial markings [9, 10]. Several investigators have implemented several methods of face recognition to treat the effects of imposition [11] of illumination [12], low resolution [13], aging [14], or a combination thereof [15]. However, these uncertainties could be overcome, and, in the face of plastic surgery, recognition will intensify with the identification of the person. The fact that face recognition in plastic surgery is due to the lack or variation of facial components, the texture of the skin, the general appearance of the face, and the geometric relationship between facial features or variation of the facial components [16–18]. Plastic surgery, both economic and sophisticated, has attracted people from all over the world. However, only a few contributions or research methodologies have been reported in the literature to address the problem of face recognition of plastic surgery. Few of them include recognition by local region analysis [19], a local form of cascade texture function (SLBT) with periocular features [20]. A review was also carried out in [21] to illustrate the use of multimodal features in the recognition of plastic surgery on the basis of contributions.

#### 1.1 Related works

De Marsico et al. [22] have made perfect recognition of the face, undergone cosmetic surgery, with region-based approach on a multimodal supervised architecture, also named as Split Face Architecture (SFA). Author proved dominance of their method by the application of supervised SFA to conventional PCA as well as FDA, toward LBP in the multiscale, rotation-invariant version with uniform patterns, face analysis for commercial entities (FACE), as well as face recognition against occlusions and expression variations (FARO).

2. Granular approach for recognizing surgically altered face images

Block diagram of the proposed granular approach for recognizing plastic surgery faces.

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint…

DOI: http://dx.doi.org/10.5772/intechopen.85550

The surgical face recognition is developed, which is based on the granular approach and Laplacian sharpening since it is identified that the sharpening of images will automatically enhance the cornerness and contrast of the image granules. Further, the key point elimination is done in this technique with entropy threshold, because entropy is the effective selection criterion that is used to eliminate the unreliable interest points. Since it is well known that the artificial neural network (ANN) with Levenberg-Marquardt (LM) is a powerful computation tool for accurate classification, it is further used in this technique for better classification results. The architecture diagram of the proposed face recognition technique is

gets cropped, resized, and formulated granularly. Then the local extrema of the

cropped, and formulated. Two types of preprocessing are carried out, namely,

posed recognition technique, EV-SIFT descriptor is used to extract the features. The

<sup>T</sup> is initially preprocessed, in such a way that the image I

<sup>p</sup> is detected using DOG scale space. Moreover, in this pro-

T

<sup>T</sup>, where the image gets resized,

using EV-SIFT and LM trained NN

diagrammatically illustrated in Figure 1.

T

NN classifier with LM is also adopted for better classification.

3. Preprocessing: granular and Laplacian sharpening

This is the initial process with the input image I

Laplacian sharpening and granular processing.

The testing image I

preprocessed image I

177

Figure 1.

Kohli et al. [23] enclose layout of multiple projective dictionary learning framework (MPDL) that never needs to figure norms to recognize usual faces, which have undergone modification via cosmetic surgery. Several projective dictionaries as well as compact binary face descriptors have been used to understand local and global plastic surgery face representations, in order to facilitate the distinction between plastic surgery faces and their original faces. The tests performed on the plastic surgery database resulted in an accuracy of about 97.96%.

Chude-Olisah et al. [24] has overcome the degradation of facial recognition performance; they have found that the approach had gone beyond the facial recognition approaches of cosmetic surgery before accessible, regardless of changes in lighting, facial expressions, and other changes resulting from cosmetic surgery. Ouanan [25] has introduced HOG feature-based facial recognition approach, which uses HOG as a substitute of DOG in the scale-invariant feature transform. Ouloul [26] introduces a perfect recognition approach for face using SIFT feature in RGBD images which depend on RGBD images produced by Kinect; this kind of cameras are low price, as well as it can be utilized in every setting and in several situations. Bhatt et al. [27] have proposed a multi-objective granular evolutionary method, which provides the pairing of images taken before and after in cosmetic surgery. Primarily, the algorithm generates superimposed face granules in three levels of granularity. Facial recognition in plastic surgery has undergone several developments in recent years. Contributions to the research were reported in the literature, either in the feature extraction phase, in the classification phase, or in both phases.

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint… DOI: http://dx.doi.org/10.5772/intechopen.85550

Figure 1. Block diagram of the proposed granular approach for recognizing plastic surgery faces.

## 2. Granular approach for recognizing surgically altered face images using EV-SIFT and LM trained NN

The surgical face recognition is developed, which is based on the granular approach and Laplacian sharpening since it is identified that the sharpening of images will automatically enhance the cornerness and contrast of the image granules. Further, the key point elimination is done in this technique with entropy threshold, because entropy is the effective selection criterion that is used to eliminate the unreliable interest points. Since it is well known that the artificial neural network (ANN) with Levenberg-Marquardt (LM) is a powerful computation tool for accurate classification, it is further used in this technique for better classification results. The architecture diagram of the proposed face recognition technique is diagrammatically illustrated in Figure 1.

The testing image I <sup>T</sup> is initially preprocessed, in such a way that the image I T gets cropped, resized, and formulated granularly. Then the local extrema of the preprocessed image I T <sup>p</sup> is detected using DOG scale space. Moreover, in this proposed recognition technique, EV-SIFT descriptor is used to extract the features. The NN classifier with LM is also adopted for better classification.

## 3. Preprocessing: granular and Laplacian sharpening

This is the initial process with the input image I <sup>T</sup>, where the image gets resized, cropped, and formulated. Two types of preprocessing are carried out, namely, Laplacian sharpening and granular processing.

complete the recognition process. This is one of the benefits of face recognition in relation to other recognition methods. Facial recognition can designate the verification phase [1] or the identification phase [2]. In the verification phase, the correspondence between two faces is resolved. There are many methods available to achieve facial recognition [3–8]. But the accuracy of recognition is not always high. This is due to variations in lighting levels, facial expressions, poses, aging, lowresolution input images, or facial markings [9, 10]. Several investigators have implemented several methods of face recognition to treat the effects of imposition [11] of illumination [12], low resolution [13], aging [14], or a combination thereof [15]. However, these uncertainties could be overcome, and, in the face of plastic surgery, recognition will intensify with the identification of the person. The fact that face recognition in plastic surgery is due to the lack or variation of facial components, the texture of the skin, the general appearance of the face, and the geometric relationship between facial features or variation of the facial components [16–18]. Plastic surgery, both economic and sophisticated, has attracted people from all over the world. However, only a few contributions or research methodologies have been reported in the literature to address the problem of face recognition of plastic surgery. Few of them include recognition by local region analysis [19], a local form of cascade texture function (SLBT) with periocular features [20]. A review was also carried out in [21] to illustrate the use of multimodal features in the

De Marsico et al. [22] have made perfect recognition of the face, undergone cosmetic surgery, with region-based approach on a multimodal supervised architecture, also named as Split Face Architecture (SFA). Author proved dominance of their method by the application of supervised SFA to conventional PCA as well as FDA, toward LBP in the multiscale, rotation-invariant version with uniform patterns, face analysis for commercial entities (FACE), as well as face recognition

Kohli et al. [23] enclose layout of multiple projective dictionary learning framework (MPDL) that never needs to figure norms to recognize usual faces, which have undergone modification via cosmetic surgery. Several projective dictionaries as well as compact binary face descriptors have been used to understand local and global plastic surgery face representations, in order to facilitate the distinction between plastic surgery faces and their original faces. The tests performed on the

Chude-Olisah et al. [24] has overcome the degradation of facial recognition performance; they have found that the approach had gone beyond the facial recognition approaches of cosmetic surgery before accessible, regardless of changes in lighting, facial expressions, and other changes resulting from cosmetic surgery. Ouanan [25] has introduced HOG feature-based facial recognition approach, which uses HOG as a substitute of DOG in the scale-invariant feature transform. Ouloul [26] introduces a perfect recognition approach for face using SIFT feature in RGBD images which depend on RGBD images produced by Kinect; this kind of cameras are low price, as well as it can be utilized in every setting and in several situations. Bhatt et al. [27] have proposed a multi-objective granular evolutionary method, which provides the pairing of images taken before and after in cosmetic surgery. Primarily, the algorithm generates superimposed face granules in three levels of granularity. Facial recognition in plastic surgery has undergone several developments in recent years. Contributions to the research were reported in the literature, either in the feature extraction phase,

recognition of plastic surgery on the basis of contributions.

Visual Object Tracking with Deep Neural Networks

against occlusions and expression variations (FARO).

in the classification phase, or in both phases.

176

plastic surgery database resulted in an accuracy of about 97.96%.

1.1 Related works

## 3.1 Preprocessing-I

The image I <sup>T</sup> from database is cropped and resized to 150 150.

Laplacian operator: This operator is also called derivative operator that is used to identify the edges in an image. The foremost difference among Laplacian and other operators such as Sobel, Prewitt, Kirsch, and Robinson is that all the mentioned operators are first-order derivative masks, whereas the Laplacian is the second-order derivative mask. Further, two classifications are there in this mask:

Laplacian operator is applied to the image, then the resultant image is added to the

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint…

This is the foremost process of the developed model. Consider I as the nominated plastic surgery face image of n � m size. The face granules are formed with the consideration of three levels of granularity. The initial level outputs the information, namely, global information at multiple determinations. The inner and outer information from the face are resulted from the second level of granularity. Normally, features termed "local facial features" play a leading role in the recognition of face and therefore in the third-level extracts of the local facial features. The brief

First level of granularity: In this level, the face granules are generated by applying the Gaussian and Laplacian operators. In accordance with this, Gaussian operator gives the series of low-pass filtered image along 2D Gaussian kernel, whereas the

ules that are resultant from Gaussian as well as Laplacian operators, where g denotes the granule number. If the face image is of size 196 � 224, the output image might be

G <sup>g</sup><sup>1</sup> to I G

G <sup>g</sup><sup>7</sup> to I G

The size of the first three granules is n � m=3. From the size of the next three

determination. From the generated six granules, the facial features are separated at varied determination for providing blurriness, smoothness, edge information, and noise, which presents in I. Hence, the variations are compensated in this level with the alteration of face textures like skin resurfacing, dermabrasion, and facelift.

Second level of granularity: In this level, the face image I is divided into varied

granules. In this way, the second level grants the variations in both the inner and the outer facial regions. The variations that are present in the chin, cheek, ears, and forehead are denoted with the aid of relations among vertical and horizontal granules. Third level of granularity: In general, humans classify individuals by identifying their local face regions like the eyes, mouth, and nose. This property is accomplished in this level, which extracts the local facial regions and is used as the granules. In eye coordinate, with the use of golden ratio face template, it is probable to extract 16 local facial regions. Every region is determined as the local information, in which it denotes the deviations due to the plastic surgery. This granularity preprocessing grants flexibility to deviations in both the inner and outer facial sections. It uses the relation among horizontal and vertical granules to view the deviations in the cheeks, chins,

<sup>g</sup><sup>12</sup> is n � ð Þ m=3 � ∈ , and the size of I

G

G <sup>g</sup><sup>16</sup> to I G <sup>g</sup>24.

G <sup>g</sup><sup>11</sup> is

G <sup>g</sup>15, and

<sup>i</sup> , where i ¼ 1, 2:……ND, which must

<sup>g</sup>6, and it may be either higher or lower

<sup>g</sup><sup>15</sup> and the vertical granules I

G <sup>g</sup><sup>13</sup> and I

<sup>g</sup>16. In the same manner, it generates the vertical

<sup>g</sup> as the gran-

original image for the sharpened image.

DOI: http://dx.doi.org/10.5772/intechopen.85550

in the pyramid view with six granules I

regions to get the horizontal granules I

G <sup>g</sup><sup>10</sup> and I

G

G

forehead, and ears that changed due to plastic surgery processes.

4. EV-SIFT, local binary pattern (LBP), and center-symmetric local

D

<sup>i</sup> and j ¼ 1, 2::…NS, and the database size is given as

n � ð Þ m=3 þ 2∈ . Further, n � ð Þ m=3 þ ∈ is the size of I

granules, the size of I

n � ð Þ m=3 � 2∈ is the size of I

binary pattern (CSLBP)

satisfy the condition Fj ⊂I

Consider the face image Fj and database I

D

4.1 EV-SIFT

179

explanation of the three granularities is explained below:

Laplacian operator gives the sequence of band-pass images. Consider I

3.2 Preprocessing II


Moreover, one of the differences among the operators is that Laplacian will not use any corresponding direction. However, it uses edges in two classifications:


Positive Laplacian operator: This category has the standard mask, the center element of the mask is the negative element, and the elements that present in the corner of the mask must be zero, which is utilized to take the outward edges in the image, which is illustrated in Figure 2.

Negative Laplacian operator: This operator also has a standard mask, in which the center element must be positive; all the elements that exist in the corner must be zero, and the remaining mask elements must be 1. This operator is utilized to take the inward edges in the image, which is illustrated in Figure 2.

Working strategy of Laplacian: This operator deemphasizes the region in image by using gray-level discontinuities, and it is happened by slowly varying gray levels. The operation results in the image that has grayish edge line with dark background, which grants both the inward and outward edges in image. The filter application basically emphasizes two major strategies: it is impossible to apply both the operators (positive and negative); rather only one operator can be applied. If the positive operator is applied to the image, then the resultant image is subtracted from the original image to get the sharpened image. Same as this, if the negative


Laplacian operator is applied to the image, then the resultant image is added to the original image for the sharpened image.

### 3.2 Preprocessing II

3.1 Preprocessing-I

The image I

• Inward edges

• Outward edges

Figure 2.

178

• Positive Laplacian operator

Visual Object Tracking with Deep Neural Networks

• Negative Laplacian operator

image, which is illustrated in Figure 2.

<sup>T</sup> from database is cropped and resized to 150 150. Laplacian operator: This operator is also called derivative operator that is used to identify the edges in an image. The foremost difference among Laplacian and other operators such as Sobel, Prewitt, Kirsch, and Robinson is that all the mentioned operators are first-order derivative masks, whereas the Laplacian is the second-order derivative mask. Further, two classifications are there in this mask:

Moreover, one of the differences among the operators is that Laplacian will not

Positive Laplacian operator: This category has the standard mask, the center element of the mask is the negative element, and the elements that present in the corner of the mask must be zero, which is utilized to take the outward edges in the

Negative Laplacian operator: This operator also has a standard mask, in which the center element must be positive; all the elements that exist in the corner must be zero, and the remaining mask elements must be 1. This operator is utilized to take

Working strategy of Laplacian: This operator deemphasizes the region in image by using gray-level discontinuities, and it is happened by slowly varying gray levels. The operation results in the image that has grayish edge line with dark background, which grants both the inward and outward edges in image. The filter application basically emphasizes two major strategies: it is impossible to apply both the operators (positive and negative); rather only one operator can be applied. If the positive operator is applied to the image, then the resultant image is subtracted from the original image to get the sharpened image. Same as this, if the negative

the inward edges in the image, which is illustrated in Figure 2.

Standard mask of (a) positive Laplacian operator and (b) negative Laplacian operator.

use any corresponding direction. However, it uses edges in two classifications:

This is the foremost process of the developed model. Consider I as the nominated plastic surgery face image of n � m size. The face granules are formed with the consideration of three levels of granularity. The initial level outputs the information, namely, global information at multiple determinations. The inner and outer information from the face are resulted from the second level of granularity. Normally, features termed "local facial features" play a leading role in the recognition of face and therefore in the third-level extracts of the local facial features. The brief explanation of the three granularities is explained below:

First level of granularity: In this level, the face granules are generated by applying the Gaussian and Laplacian operators. In accordance with this, Gaussian operator gives the series of low-pass filtered image along 2D Gaussian kernel, whereas the Laplacian operator gives the sequence of band-pass images. Consider I G <sup>g</sup> as the granules that are resultant from Gaussian as well as Laplacian operators, where g denotes the granule number. If the face image is of size 196 � 224, the output image might be in the pyramid view with six granules I G <sup>g</sup><sup>1</sup> to I G <sup>g</sup>6, and it may be either higher or lower determination. From the generated six granules, the facial features are separated at varied determination for providing blurriness, smoothness, edge information, and noise, which presents in I. Hence, the variations are compensated in this level with the alteration of face textures like skin resurfacing, dermabrasion, and facelift.

Second level of granularity: In this level, the face image I is divided into varied regions to get the horizontal granules I G <sup>g</sup><sup>7</sup> to I G <sup>g</sup><sup>15</sup> and the vertical granules I G <sup>g</sup><sup>16</sup> to I G <sup>g</sup>24. The size of the first three granules is n � m=3. From the size of the next three granules, the size of I G <sup>g</sup><sup>10</sup> and I G <sup>g</sup><sup>12</sup> is n � ð Þ m=3 � ∈ , and the size of I G <sup>g</sup><sup>11</sup> is n � ð Þ m=3 þ 2∈ . Further, n � ð Þ m=3 þ ∈ is the size of I G <sup>g</sup><sup>13</sup> and I G <sup>g</sup>15, and n � ð Þ m=3 � 2∈ is the size of I G <sup>g</sup>16. In the same manner, it generates the vertical granules. In this way, the second level grants the variations in both the inner and the outer facial regions. The variations that are present in the chin, cheek, ears, and forehead are denoted with the aid of relations among vertical and horizontal granules.

Third level of granularity: In general, humans classify individuals by identifying their local face regions like the eyes, mouth, and nose. This property is accomplished in this level, which extracts the local facial regions and is used as the granules. In eye coordinate, with the use of golden ratio face template, it is probable to extract 16 local facial regions. Every region is determined as the local information, in which it denotes the deviations due to the plastic surgery. This granularity preprocessing grants flexibility to deviations in both the inner and outer facial sections. It uses the relation among horizontal and vertical granules to view the deviations in the cheeks, chins, forehead, and ears that changed due to plastic surgery processes.

## 4. EV-SIFT, local binary pattern (LBP), and center-symmetric local binary pattern (CSLBP)

#### 4.1 EV-SIFT

Consider the face image Fj and database I D <sup>i</sup> , where i ¼ 1, 2:……ND, which must satisfy the condition Fj ⊂I D <sup>i</sup> and j ¼ 1, 2::…NS, and the database size is given as

ð Þ M � N . The preprocessing phase initiates with resizing of image. The resizing model of the image is defined in Eq. (1), where SM and SN denote the scaled number of columns and rows:

$$I(\mathbf{x}, \mathbf{y}) = I\_i(m\_r, n\_r) = \frac{\mathbf{1}}{\mathbf{S}\_M \ast \mathbf{S}\_N} \sum\_{\substack{\mathbf{u} = (m\_r - 1)\mathbf{S}\_M \ \mathbf{v} = (n\_r - 1)\mathbf{S}\_N}}^{m\_r \mathbf{S}\_M} I\_i(\mathbf{u}, \mathbf{v}) \tag{1}$$

In Eq. (1), u∈ ½ � 1; M and v∈½ � 1; N , 0≤ mr ≤ Mr � 1 and 0 ≤nr ≤ Nr � 1, ð Þ Mr � Nr is the size of the resized image, and ½ �� denotes the round-off function of the nearest integer:

$$\mathbf{S}\_M = \begin{bmatrix} \mathbf{M} \\ \overline{\mathbf{M}}\_r \end{bmatrix} \tag{2}$$

Step 2: The information basis is both memory less and static. The volume of the structure in EV-SIFT analysis is defined in Eq. (7), which is the probability function:

<sup>∑</sup><sup>i</sup> <sup>∑</sup><sup>j</sup> V ið Þ ; <sup>j</sup> (7)

E Vð Þ¼�Vpð Þ i; j logVpð Þ i; j (8)

<sup>T</sup>ð Þ :; : is defined as given in Eq. (9), where Yi, i <sup>¼</sup> <sup>1</sup>, :::8 is

<sup>T</sup>ð Þ <sup>Y</sup><sup>0</sup> <sup>≤</sup>threshold

<sup>T</sup>ð Þ <sup>Y</sup><sup>0</sup> . threshold , i <sup>¼</sup> <sup>1</sup>, <sup>2</sup>, <sup>3</sup>::……, <sup>8</sup>

i

(9)

Vpð Þ¼ <sup>i</sup>; <sup>j</sup> V ið Þ ; <sup>j</sup>

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint…

DOI: http://dx.doi.org/10.5772/intechopen.85550

describes the entire database that achieved the final EV-SIFT descriptor:

4.2 Local binary pattern (LBP)

thresholding function f

<sup>T</sup>ð Þ <sup>Y</sup><sup>0</sup> ;<sup>I</sup>

f <sup>T</sup> I

Figure 3.

181

known as "micro pattern":

<sup>T</sup>ð Þ Yi

Example of eight neighborhoods around Y0.

� � <sup>¼</sup> <sup>0</sup>, if I<sup>T</sup>ð Þ� Yi <sup>I</sup>

(

Step 3: The computation of entropy is done from the volume of the structure. The entropy calculation for EV-SIFT process is determined in Eq. (8), which states that if E Vð Þ is high entropy, then the volume is from the unvarying direction, and if it is low entropy, then it means that the volume is a varied distribution. Thus, F<sup>D</sup>

The level of Gaussian blur of the image is selected by orientation and gradient magnitude with entropy, and also the volume of the image is also sampled in terms of scale of key points at particular key point location. The sample is an 8 � 8 neighbor window, which is centered on the key point and splits the neighbor into 4 � 4 child window. Hence, the formulation of gradient orientation histogram is done along with eight bins with the aid of each child window. In such a way that within each key point, each descriptor intends the 4 � 4 array of histograms that comprises eight bins. The feature vector attained is the size of 4 � 4 � 8 = 128 dimension.

LBP [1] operator is designed for texture description. It encodes the pixel-wise data in texture images, in such a way that a label is assigned to every pixel of the image. This is done by thresholding the 3 � 3 neighborhood of all pixel value with

the eight neighborhood point around Y0, which is shown in Figure 3. LBP in other words is termed as the concatenation of binary gradient direction, which is also

the center pixel, and the result must be a binary number. The basic LBP

<sup>1</sup>, if I<sup>T</sup>ð Þ� Yi <sup>I</sup>

$$\mathcal{S}\_N = \begin{bmatrix} N \\ \overline{N\_r} \end{bmatrix} \tag{3}$$

#### 4.1.1 Acquisition of the EV-SIFT key points

Choosing the key points in the variation of the Gaussian function is the vital role to be considered. The parameters of the key point are purely depending on distribution property of the gradient operation of the image. Thus, the formulation of both the orientation and gradient modules is done, which registers the invariance toward the rotation of the image. The computation of orientation and gradient module is defined in Eqs. (4) and (5), where θð Þ x; y denotes the orientation of key points and the gradient magnitude and L xð Þ ; y refers to the image sample:

$$\theta(\mathbf{x}, \mathbf{y}) = \tan^{-1} \left( \frac{\left( L(\mathbf{x}, \mathbf{y} + \mathbf{1}) - L(\mathbf{x}, \mathbf{y} - \mathbf{1}) \right)^2}{\left( L(\mathbf{x} + \mathbf{1}, \mathbf{y}) - L(\mathbf{x} - \mathbf{1}, \mathbf{y}) \right)^2} \right) \tag{4}$$

$$m(\mathbf{x}, \boldsymbol{\chi}) = \begin{pmatrix} \overbrace{\left(L(\mathbf{x} + \mathbf{1}, \boldsymbol{\chi}) - L(\mathbf{x} - \mathbf{1}, \boldsymbol{\chi})\right)^2}^{} \\ \sqrt{\begin{pmatrix} L(\mathbf{x}, \boldsymbol{\chi} + \mathbf{1}) - L(\mathbf{x} - \mathbf{1}, \boldsymbol{\chi} - \mathbf{1})\end{pmatrix}^2} \end{pmatrix} \tag{5}$$

The scales used by L are the respective scale for each key point. Further, an orientation histogram is achieved as a result of gradient operation of sample points.

#### 4.1.2 Entropy-based feature descriptor

The Changeable information is measured using entropy. It basically defines the statistical measure of randomness, which determines the texture of the input image. Only the least effect remains in the higher-order statistical feature due to the entropy on uncertain deviations in the face. The following steps show the entropybased feature descriptor:

Step 1: The volume of the image is evaluated with the aid of V-SIFT formulation, which is determined in the matrix form as defined in Eq. (6):

$$V(i,j) = \begin{bmatrix} v\left(i\_1, j\_1\right) & v\left(i\_1, j\_2\right) & \cdot & v\left(i\_1, j\_n\right) \\ v\left(i\_2, j\_1\right) & v\left(i\_2, j\_2\right) & \cdot & v\left(i\_2, j\_n\right) \\ \cdot & \cdot & \cdot & \cdot \\ v\left(i\_m, j\_1\right) & v\left(i\_m, j\_2\right) & \cdot & v\left(i\_m, j\_n\right) \end{bmatrix} \tag{6}$$

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint… DOI: http://dx.doi.org/10.5772/intechopen.85550

Step 2: The information basis is both memory less and static. The volume of the structure in EV-SIFT analysis is defined in Eq. (7), which is the probability function:

$$V\_p(i,j) = \frac{V(i,j)}{\sum\_i \sum\_j V(i,j)}\tag{7}$$

Step 3: The computation of entropy is done from the volume of the structure. The entropy calculation for EV-SIFT process is determined in Eq. (8), which states that if E Vð Þ is high entropy, then the volume is from the unvarying direction, and if it is low entropy, then it means that the volume is a varied distribution. Thus, F<sup>D</sup> i describes the entire database that achieved the final EV-SIFT descriptor:

$$E(V) = -V\_p(i,j)\log V\_p(i,j) \tag{8}$$

The level of Gaussian blur of the image is selected by orientation and gradient magnitude with entropy, and also the volume of the image is also sampled in terms of scale of key points at particular key point location. The sample is an 8 � 8 neighbor window, which is centered on the key point and splits the neighbor into 4 � 4 child window. Hence, the formulation of gradient orientation histogram is done along with eight bins with the aid of each child window. In such a way that within each key point, each descriptor intends the 4 � 4 array of histograms that comprises eight bins. The feature vector attained is the size of 4 � 4 � 8 = 128 dimension.

#### 4.2 Local binary pattern (LBP)

ð Þ M � N . The preprocessing phase initiates with resizing of image. The resizing model of the image is defined in Eq. (1), where SM and SN denote the scaled number

> 1 SM ∗ SN

In Eq. (1), u∈ ½ � 1; M and v∈½ � 1; N , 0≤ mr ≤ Mr � 1 and 0 ≤nr ≤ Nr � 1, ð Þ Mr � Nr is the size of the resized image, and ½ �� denotes the round-off function of

> SM <sup>¼</sup> <sup>M</sup> Mr � �

SN <sup>¼</sup> <sup>N</sup> Nr � �

Choosing the key points in the variation of the Gaussian function is the vital role to be considered. The parameters of the key point are purely depending on distribution property of the gradient operation of the image. Thus, the formulation of both the orientation and gradient modules is done, which registers the invariance toward the rotation of the image. The computation of orientation and gradient module is defined in Eqs. (4) and (5), where θð Þ x; y denotes the orientation of key

points and the gradient magnitude and L xð Þ ; y refers to the image sample:

<sup>θ</sup>ð Þ¼ <sup>x</sup>; <sup>y</sup> tan �<sup>1</sup> ð Þ L xð Þ� ; <sup>y</sup> <sup>þ</sup> <sup>1</sup> L xð Þ ; <sup>y</sup> � <sup>1</sup> <sup>2</sup>

The scales used by L are the respective scale for each key point. Further, an orientation histogram is achieved as a result of gradient operation of sample points.

The Changeable information is measured using entropy. It basically defines the statistical measure of randomness, which determines the texture of the input image. Only the least effect remains in the higher-order statistical feature due to the entropy on uncertain deviations in the face. The following steps show the entropy-

Step 1: The volume of the image is evaluated with the aid of V-SIFT formulation,

2 � � : v i1; j

2 � � : v i2; j

2 � � : v im; j

: :::

ð Þ L xð Þ� þ 1; y L xð Þ � 1; y

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð Þ L xð Þ� þ 1; y L xð Þ � 1; y

þð Þ L xð Þ� ; <sup>y</sup> <sup>þ</sup> <sup>1</sup> L xð Þ � <sup>1</sup>; <sup>y</sup> � <sup>1</sup> <sup>2</sup>

!

2

1

A (5)

2

n � �

n � �

n � �

∑ mrSM u¼ð Þ mr�1 SM

∑ nrSN v¼ð Þ nr�1 SN

Iið Þ u; v (1)

(2)

(3)

(4)

(6)

of columns and rows:

the nearest integer:

I xð Þ¼ ; y Ii mr ð Þ¼ ; nr

Visual Object Tracking with Deep Neural Networks

4.1.1 Acquisition of the EV-SIFT key points

m xð Þ¼ ; y

4.1.2 Entropy-based feature descriptor

based feature descriptor:

180

vuut 0 @

which is determined in the matrix form as defined in Eq. (6):

v i1; j 1 � � v i1; j

v i2; j 1 � � v i2; j

v im; j 1 � � v im; j

V ið Þ¼ ; j

LBP [1] operator is designed for texture description. It encodes the pixel-wise data in texture images, in such a way that a label is assigned to every pixel of the image. This is done by thresholding the 3 � 3 neighborhood of all pixel value with the center pixel, and the result must be a binary number. The basic LBP thresholding function f <sup>T</sup>ð Þ :; : is defined as given in Eq. (9), where Yi, i <sup>¼</sup> <sup>1</sup>, :::8 is the eight neighborhood point around Y0, which is shown in Figure 3. LBP in other words is termed as the concatenation of binary gradient direction, which is also known as "micro pattern":

$$f^T\left(I^T(Y\_0), I^T(Y\_i)\right) = \begin{cases} \mathbf{0}, & \text{if } I^T(Y\_i) - I^T(Y\_0) \le \text{threshold} \\ \mathbf{1}, & \text{if } I^T(Y\_i) - I^T(Y\_0) > \text{threshold} \end{cases}, i = \mathbf{1}, 2, 3, \dots, 8 \\ \text{(9)}$$


Figure 3. Example of eight neighborhoods around Y0.

Figure 4. An example for LBP micro pattern for a given region.

Figure 4 illustrates the sample of attaining an LBP micro pattern when the threshold is set to 0. Further, the resultant histogram of the micro pattern presents the data related to the distribution of edges, spots, and more local features that present in the image. It is observed that the LBP is a great tool for face recognition. Despite a number of static learning approaches that tune with more parameters, LBP is more effective since it has an "easy-to-formulate" feature extraction process, and also the matching strategy is also very simple.

the bias weight to nth hidden neurons, and wð Þ <sup>h</sup>

nonlinear function is represented as Fð Þ• :

DOI: http://dx.doi.org/10.5772/intechopen.85550

model that is defined in Eq. (12), where wð Þ<sup>o</sup>

ik is the output weight from i

and Nð Þ<sup>o</sup> is the number of output neurons:

layer, wð Þ<sup>o</sup>

183

Figure 5.

CSLBP establishment.

weight. The input is the dimensional reduced features from PCA, which is denoted as f, Að Þ <sup>h</sup> is the output of the hidden layer that is defined in Eq. (11), and the

> ð Þ bn <sup>þ</sup> <sup>∑</sup> Nð Þ<sup>I</sup>

where Nð Þ<sup>I</sup> denotes the count of input neurons. B^ is the output of the network

count of hidden neurons. The weight w<sup>∗</sup> is optimally chosen by reducing the objective function, which is defined in Eq. (13), where B denotes the actual output

> bk þ ∑ Nð Þ <sup>h</sup>

i¼1

!

wð Þ<sup>o</sup> ik <sup>A</sup>ð Þ <sup>h</sup> i

> ∑ Nð Þ<sup>o</sup>

k¼1

j¼1 wð Þ <sup>h</sup> jn f

!

<sup>A</sup>ð Þ <sup>h</sup> <sup>¼</sup> F wð Þ <sup>h</sup>

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint…

<sup>B</sup>^ <sup>¼</sup> F wð Þ<sup>o</sup>

<sup>w</sup><sup>∗</sup> <sup>¼</sup> arg min wð Þ <sup>h</sup> ð Þ bi ;wð Þ<sup>h</sup> ji ;wð Þ<sup>o</sup> bk ;w<sup>o</sup> ik h i

Here, the LM algorithm is used for training the NN model. The error

functionEFð Þ <sup>w</sup> to be reduced is represented as the sum of squared errors among the target output BT and the network model output B^, which is defined in Eq. (14):

whereW ¼ W1,W2, ………WN, which presents all the weights of the network and v is the error vector which includes the error of all the training samples. While training with LM model, the growth of weight ΔW is obtained, and it is defined in Eq. (15):

jn represents the hidden neuron's

bk is the weight of the output bias to kth

<sup>k</sup><sup>B</sup> � <sup>B</sup>^<sup>k</sup> (13)

th hidden neuron to kth layer, and Nð Þ <sup>h</sup> is the

EFð Þ <sup>W</sup> <sup>¼</sup> vTv (14)

(11)

(12)

#### 4.3 Center-symmetric local binary pattern (CSLBP)

CSLBP [1] is established for interest region description. It purposes for least LBP labels to generate smaller histograms, which are well suited to utilize in region descriptors. Moreover, it is designed for better stability, especially in regions that include the face image. Here, the comparison of pixel values are not done between the pixels and center pixels; rather the opposing pixels are symmetrically compared in correspondence to the center pixel, which is defined in Eq. (10):

$$\text{CSLBP}\_{\mathbb{S},T}(\mathfrak{u},\mathfrak{v}) = \sum\_{i=0}^{(\mathbb{S}/2)-1} t\left(\mathfrak{s}\_{i} - \mathfrak{s}\_{i+(\mathbb{S}/2)}\right), t(\mathfrak{u}) = \begin{cases} \mathfrak{1}, & \mathfrak{u} \ge T \\ \mathbf{0}, & \text{otherwise} \end{cases} \tag{10}$$

where si and siþð Þ <sup>S</sup>=<sup>2</sup> refer to the gray values of center-symmetric pairs of S similarly space out pixels.

In this work, the value of T threshold is 1% of pixel value. T is set to 0.01 since the data lies among 0 and 1. The size of the neighborhood is eight as illustrated in Figure 5. From the CSLBP formulation, it is evident that CSLBP is related to gradient operator, and also it considers the gray level G differences among pairs of contrary pixels in neighborhood. Thus, the CSLBP features show the advantage of both the LBP parameters and gradient features. CSLBP generates 16 varied binary patterns. Feature vector of every key point is generated by concatenating 128-dimensional descriptor as well as LBP [256-dimensional descriptor]/CSLBP [16-dimensional descriptor]. The feature vectors' dimensions are diminished to 25 dimensions by evaluating the covariance matrix for PCA, from which the highest 25 eigenvectors are chosen for description.

## 5. Recognition system: Levenberg-Marquardt-based neural network classifier (LM-NN)

In this work, LM-NN classifier is used for recognition purpose. The NN model is represented in Eqs. (11)–(13), where n denotes the hidden neurons, wð Þ <sup>h</sup> ð Þ bn refers to

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint… DOI: http://dx.doi.org/10.5772/intechopen.85550

Figure 4 illustrates the sample of attaining an LBP micro pattern when the threshold is set to 0. Further, the resultant histogram of the micro pattern presents the data related to the distribution of edges, spots, and more local features that present in the image. It is observed that the LBP is a great tool for face recognition. Despite a number of static learning approaches that tune with more parameters, LBP is more effective since it has an "easy-to-formulate" feature extraction process,

CSLBP [1] is established for interest region description. It purposes for least LBP

,t uð Þ¼ <sup>1</sup>, u . <sup>T</sup>

0, otherwise

(10)

ð Þ bn refers to

labels to generate smaller histograms, which are well suited to utilize in region descriptors. Moreover, it is designed for better stability, especially in regions that include the face image. Here, the comparison of pixel values are not done between the pixels and center pixels; rather the opposing pixels are symmetrically compared

t si � siþð Þ <sup>S</sup>=<sup>2</sup>

where si and siþð Þ <sup>S</sup>=<sup>2</sup> refer to the gray values of center-symmetric pairs of S

5. Recognition system: Levenberg-Marquardt-based neural network

represented in Eqs. (11)–(13), where n denotes the hidden neurons, wð Þ <sup>h</sup>

In this work, LM-NN classifier is used for recognition purpose. The NN model is

In this work, the value of T threshold is 1% of pixel value. T is set to 0.01 since the data lies among 0 and 1. The size of the neighborhood is eight as illustrated in Figure 5. From the CSLBP formulation, it is evident that CSLBP is related to gradient operator, and also it considers the gray level G differences among pairs of contrary pixels in neighborhood. Thus, the CSLBP features show the advantage of both the LBP parameters and gradient features. CSLBP generates 16 varied binary patterns. Feature vector of every key point is generated by concatenating 128-dimensional descriptor as well as LBP [256-dimensional descriptor]/CSLBP [16-dimensional descriptor]. The feature vectors' dimensions are diminished to 25 dimensions by evaluating the covariance matrix for PCA, from which the highest 25 eigenvectors are chosen for description.

in correspondence to the center pixel, which is defined in Eq. (10):

ð Þ� S=2 1 i¼0

and also the matching strategy is also very simple.

An example for LBP micro pattern for a given region.

Visual Object Tracking with Deep Neural Networks

CSLBPS,Tð Þ¼ u; v ∑

similarly space out pixels.

Figure 4.

classifier (LM-NN)

182

4.3 Center-symmetric local binary pattern (CSLBP)

the bias weight to nth hidden neurons, and wð Þ <sup>h</sup> jn represents the hidden neuron's weight. The input is the dimensional reduced features from PCA, which is denoted as f, Að Þ <sup>h</sup> is the output of the hidden layer that is defined in Eq. (11), and the nonlinear function is represented as Fð Þ• :

$$\mathbf{A}^{(h)} = F\left(\boldsymbol{w}\_{(bn)}^{(h)} + \sum\_{j=1}^{N^{(l)}} \boldsymbol{w}\_{jn}^{(h)} \boldsymbol{f}\right) \tag{11}$$

where Nð Þ<sup>I</sup> denotes the count of input neurons. B^ is the output of the network model that is defined in Eq. (12), where wð Þ<sup>o</sup> bk is the weight of the output bias to kth layer, wð Þ<sup>o</sup> ik is the output weight from i th hidden neuron to kth layer, and Nð Þ <sup>h</sup> is the count of hidden neurons. The weight w<sup>∗</sup> is optimally chosen by reducing the objective function, which is defined in Eq. (13), where B denotes the actual output and Nð Þ<sup>o</sup> is the number of output neurons:

$$\hat{B} = F\left(w\_{bk}^{(o)} + \sum\_{i=1}^{N^{(h)}} w\_{ik}^{(o)} A\_i^{(h)}\right) \tag{12}$$

$$\boldsymbol{w}^\* = \underset{\left[w^{(h)}\_{(h)}, w^{(h)}\_{ji}, w^{(s)}\_{hk}, w^o\_{ik}\right]}{\text{arg min}} \sum\_{k=1}^{N^{(\sigma)}} ||\boldsymbol{B} - \boldsymbol{\hat{B}}|| \tag{13}$$

Here, the LM algorithm is used for training the NN model. The error functionEFð Þ <sup>w</sup> to be reduced is represented as the sum of squared errors among the target output BT and the network model output B^, which is defined in Eq. (14):

$$EF^{(W)} = v^T v \tag{14}$$

whereW ¼ W1,W2, ………WN, which presents all the weights of the network and v is the error vector which includes the error of all the training samples. While training with LM model, the growth of weight ΔW is obtained, and it is defined in Eq. (15):

Visual Object Tracking with Deep Neural Networks

$$
\Delta \mathbf{W} = \left(\mathbf{M}^T \mathbf{M} + \eta\right)^{-1} \mathbf{M}^T v \tag{15}
$$

6.2 Granularity preprocessing

DOI: http://dx.doi.org/10.5772/intechopen.85550

6.3 Analysis on EV-SIFT

Figure 7.

Figure 8.

Figure 9.

185

By dividing the face image into varied regions, we get the vertical as well as horizontal face granules as illustrated in Figure 6. The horizontal granules are represented as R1, R2, and R3, and the size is 150 150/3. Similarly, the vertical

In this work, EV-SIFT descriptor is used for the feature extraction. Figure 7 illustrates the original images. For each original image, the corresponding vertical edge and horizontal edge of the image were evaluated, and it is illustrated in Figures 8 and 9. The gradient magnitude of the images is also shown in Figure 10. Similarly, the theta images of the given input images are illustrated in Figure 11.

granules are denoted as R4, R5, and R6, which is of 150/3 150 size.

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint…

Original images: (a) image 1, (b) image 2, (c) image 3, (d) image 4, and (e) image 5.

Vertical edge of the given images: (a) image 1, (b) image 2, (c) image 3, (d) image 4, and (e) image 5.

Horizontal edge of the given images: (a) image 1, (b) image 2, (c) image 3, (d) image 4, and (e) image 5.

where M is the Jacobian matrix and the learning rate to be updated is represented as η. The updation of η is done using α, which depends to the outcome. Particularly, η is multiplied by <sup>α</sup>,ð Þ <sup>0</sup> , <sup>α</sup> , <sup>1</sup> decay rate when EFð Þ <sup>W</sup> minimizes, whereas when EFð Þ <sup>W</sup> increases, η is divided by α. The given pseudo-code shows the training process of LM.


```
Step 4 Recomputing EFð Þ W
```

```
Step 5 Use W + ΔW as the trail Wand evaluate
```

```
If trail EFð Þ W , EFð Þ W in step 2 then
          W = W + ΔW
          η = η � α αð Þ ¼ 0:01
          Back to step 2
else
          η ¼ η
               α
          Back to step 4
End if
```
## 6. Results and discussion

## 6.1 Experimental setup

The cosmetic surgery face recognition experimentation is conducted in MATLAB 2015a. The database including presurgery faces and postsurgery faces are downloaded from http://www.locateadoc.com/pictures/. The experimentation is performed for different plastic surgery faces. The total number of plastic surgery faces in the database is 460, where it comprises 68 images from blepharoplasty (eyelid surgery), 51 images from brow lift (forehead surgery), 51 images from liposhaving (facial sculpturing), 17 images from malar augmentation (cheek implant), 18 images from mentoplasty (chin surgery), 54 images from otoplasty (ear surgery), 75 images from rhinoplasty (nose surgery), 74 images from rhytidectomy (facelift), and 52 images from skin peeling (skin resurfacing).

Figure 6. Computation of granular images 1, 2, and 3. (a) Horizontal granules and (b) vertical granules.

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint… DOI: http://dx.doi.org/10.5772/intechopen.85550

## 6.2 Granularity preprocessing

<sup>Δ</sup><sup>W</sup> <sup>¼</sup> <sup>M</sup>TM <sup>þ</sup> <sup>η</sup> �'<sup>1</sup>

Step 1 Initializing the weights W and parameter η (η=.01 (approx.)) Step 2 Sum of the squared errors is formulated on the entire EFð Þ <sup>W</sup> inputs.

The cosmetic surgery face recognition experimentation is conducted in MATLAB 2015a. The database including presurgery faces and postsurgery faces are downloaded from http://www.locateadoc.com/pictures/. The experimentation is performed for different plastic surgery faces. The total number of plastic surgery faces in the database is 460, where it comprises 68 images from blepharoplasty (eyelid surgery), 51 images from brow lift (forehead surgery), 51 images from liposhaving (facial sculpturing), 17 images from malar augmentation (cheek implant), 18 images from mentoplasty (chin surgery), 54 images from otoplasty (ear surgery), 75 images from rhinoplasty (nose surgery), 74 images from rhytidectomy (facelift), and 52 images from skin peeling (skin resurfacing).

Computation of granular images 1, 2, and 3. (a) Horizontal granules and (b) vertical granules.

Step 3 Increment of weights ΔW is computed using Eq. (14)

Step 5 Use W + ΔW as the trail Wand evaluate If trail EFð Þ <sup>W</sup> , EFð Þ <sup>W</sup> in step 2 then W = W + ΔW η = η � α αð Þ ¼ 0:01 Back to step 2

> <sup>η</sup> <sup>¼</sup> <sup>η</sup> α Back to step 4

of LM.

Step 4 Recomputing EFð Þ <sup>W</sup>

Visual Object Tracking with Deep Neural Networks

else

End if

6. Results and discussion

6.1 Experimental setup

Figure 6.

184

where M is the Jacobian matrix and the learning rate to be updated is represented as η. The updation of η is done using α, which depends to the outcome. Particularly, η is multiplied by <sup>α</sup>,ð Þ <sup>0</sup> , <sup>α</sup> , <sup>1</sup> decay rate when EFð Þ <sup>W</sup> minimizes, whereas when EFð Þ <sup>W</sup> increases, η is divided by α. The given pseudo-code shows the training process

MTv (15)

By dividing the face image into varied regions, we get the vertical as well as horizontal face granules as illustrated in Figure 6. The horizontal granules are represented as R1, R2, and R3, and the size is 150 150/3. Similarly, the vertical granules are denoted as R4, R5, and R6, which is of 150/3 150 size.

## 6.3 Analysis on EV-SIFT

In this work, EV-SIFT descriptor is used for the feature extraction. Figure 7 illustrates the original images. For each original image, the corresponding vertical edge and horizontal edge of the image were evaluated, and it is illustrated in Figures 8 and 9. The gradient magnitude of the images is also shown in Figure 10. Similarly, the theta images of the given input images are illustrated in Figure 11.

Figure 7.

Original images: (a) image 1, (b) image 2, (c) image 3, (d) image 4, and (e) image 5.

Figure 8.

Vertical edge of the given images: (a) image 1, (b) image 2, (c) image 3, (d) image 4, and (e) image 5.


#### Figure 9.

Horizontal edge of the given images: (a) image 1, (b) image 2, (c) image 3, (d) image 4, and (e) image 5.

#### Figure 10.

Gradient magnitude of the images: (a) image 1, (b) image 2, (c) image 3, (d) image 4, and (e) image 5.

Figure 11. Theta representation of the images: (a) image 1, (b) image 2, (c) image 3, (d) image 4, and (e) image 5.

One of the important processes is the evaluation of image orientation of the eight angles such as 0, 45, 90, 135, 180, 225, 270, and 315° in each image, which is shown in Figures 12–16. The resultant EV-SIFT contour of the input images is illustrated in Figure 17.

#### 6.4 Learning performance of LM-NN

The performance of the LM-NN classifier is illustrated in Figure 18. It is observed that the best performance of the classifier is attained at the epoch 7, where the training performance is 0.00022204, gradient is 7.0363e-08, Mu is 1e-10, and the validation fail is 0 since there is no validation attained.

## 6.5 Comparative performance analysis of best-performing methods of proposed approaches

While analyzing the first research technique, in the evaluation on LM-NN, it is observed that the EV-SIFT proposed technique attained better results in all the measures like accuracy, sensitivity, specificity, precision, false-positive rate (FPR), false-negative rate (FNR), net present value (NPV), false discovery rate (FDR), and F1score (also F-score or F-measure) which is a measure of a test's accuracy and Matthews correlation coefficient (MCC), respectively. The evaluation is summarized in Tables 1–3.

It is observed that the proposed V-SIFT with LM-NN has achieved more over the conventional methods for various plastic surgeries, which is summarized in

Figure 13.

Figure 12.

(h) 315° .

Image orientation of eight angles (image 1): (a) 0°

DOI: http://dx.doi.org/10.5772/intechopen.85550

(h) 315° .

187

Image orientation of eight angles (image 2): (a) 0°

, (b) 45°

, (b) 45°

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint…

, (c) 90°

, (d) 135°

, (e) 180°

, (f) 225°

, (g) 270°

, and

, (c) 90°

, (d) 135°

, (e) 180°

, (f) 225°

, (g) 270°

, and

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint… DOI: http://dx.doi.org/10.5772/intechopen.85550

#### Figure 12.

One of the important processes is the evaluation of image orientation of the eight angles such as 0, 45, 90, 135, 180, 225, 270, and 315° in each image, which is shown in Figures 12–16. The resultant EV-SIFT contour of the input images is illustrated in

Theta representation of the images: (a) image 1, (b) image 2, (c) image 3, (d) image 4, and (e) image 5.

Gradient magnitude of the images: (a) image 1, (b) image 2, (c) image 3, (d) image 4, and (e) image 5.

The performance of the LM-NN classifier is illustrated in Figure 18. It is observed that the best performance of the classifier is attained at the epoch 7, where the training performance is 0.00022204, gradient is 7.0363e-08, Mu is 1e-10, and

6.5 Comparative performance analysis of best-performing methods of

While analyzing the first research technique, in the evaluation on LM-NN, it is observed that the EV-SIFT proposed technique attained better results in all the measures like accuracy, sensitivity, specificity, precision, false-positive rate (FPR), false-negative rate (FNR), net present value (NPV), false discovery rate (FDR), and F1score (also F-score or F-measure) which is a measure of a test's accuracy and Matthews correlation coefficient (MCC), respectively. The evaluation is summa-

It is observed that the proposed V-SIFT with LM-NN has achieved more over the

conventional methods for various plastic surgeries, which is summarized in

the validation fail is 0 since there is no validation attained.

Figure 17.

Figure 11.

Figure 10.

6.4 Learning performance of LM-NN

Visual Object Tracking with Deep Neural Networks

proposed approaches

rized in Tables 1–3.

186

Image orientation of eight angles (image 1): (a) 0° , (b) 45° , (c) 90° , (d) 135° , (e) 180° , (f) 225° , (g) 270° , and (h) 315° .

#### Figure 13.

Image orientation of eight angles (image 2): (a) 0° , (b) 45° , (c) 90° , (d) 135° , (e) 180° , (f) 225° , (g) 270° , and (h) 315° .

Figure 16.

Figure 17.

189

(h) 315° .

Image orientation of eight angles (image 5): (a) 0°

DOI: http://dx.doi.org/10.5772/intechopen.85550

, (b) 45°

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint…

EV-SIFT contour of images: (a) image 1, (b) image 2, (c) image 3, (d) image 4, and (e) image 5.

, (c) 90°

, (d) 135°

, (e) 180°

, (f) 225°

, (g) 270°

, and

#### Figure 14.

Image orientation of eight angles (image 3): (a) 0° , (b) 45° , (c) 90° , (d) 135° , (e) 180° , (f) 225° , (g) 270° , and (h) 315° .

#### Figure 15.

Image orientation of eight angles (image 4): (a) 0° , (b) 45° , (c) 90° , (d) 135° , (e) 180° , (f) 225° , (g) 270° , and (h) 315° .

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint… DOI: http://dx.doi.org/10.5772/intechopen.85550

Figure 16.

Figure 14.

Figure 15.

(h) 315° .

188

Image orientation of eight angles (image 4): (a) 0°

(h) 315° .

Image orientation of eight angles (image 3): (a) 0°

Visual Object Tracking with Deep Neural Networks

, (b) 45°

, (b) 45°

, (c) 90°

, (d) 135°

, (e) 180°

, (f) 225°

, (g) 270°

, and

, (c) 90°

, (d) 135°

, (e) 180°

, (f) 225°

, (g) 270°

, and

Image orientation of eight angles (image 5): (a) 0° , (b) 45° , (c) 90° , (d) 135° , (e) 180° , (f) 225° , (g) 270° , and (h) 315° .

Figure 17. EV-SIFT contour of images: (a) image 1, (b) image 2, (c) image 3, (d) image 4, and (e) image 5.

#### Figure 18.

Performance of LM-NN classifier in correspondence with (a) validation performance and (b) gradient, Mu, and validation fails.


Table 2. It is observed that for all the measures, the method has attained better

LM-NN Measures Surgery Attained result Accuracy Rhytidectomy 0.93 Sensitivity Mentoplasty 0.19 Specificity Rhytidectomy 0.97 Precision Skin peeling 0.03 FPR Rhytidectomy 0.03 FNR Mentoplasty 0.81 NPV Rhytidectomy 0.97 FDR Skin peeling 0.97 F1score Skin peeling 0.05 MCC Skin peeling 0.03

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint…

LM-NN Accuracy Rhinoplasty 0.984 Sensitivity Brow lift 0.17 Specificity Rhinoplasty 0.97 Precision Skin peeling 0.04 FPR Rhinoplasty 0.03 FNR Malar augmentation 0.83 NPV Rhinoplasty 0.97 FDR Skin peeling 0.96 F1score Skin peeling 0.06 MCC Skin peeling 0.04

From the second technique, it is observed that the proposed EV-SIFT with LM-NN are achieved more over the conventional methods for various plastic surgeries, which is summarized in Table 3. It is observed that for all the measures, the method

This chapter gives the detailed description of the second research technique. The

feature descriptor EV-SIFT that is used for feature extraction is well explained. Further, the LM-based NN classifier is defined in this chapter, and the performance of both the EV-SIFT and LM-NN classifiers is shown in the Result section. The better work of EV-SIFT is effectively demonstrated in this section, which shows

results, which also leads to the other types of plastic surgery.

Proposed EV-SIFT with LM-NN of different plastic surgery faces.

Proposed V-SIFT with LM-NN of different plastic surgery faces.

DOI: http://dx.doi.org/10.5772/intechopen.85550

has attained better results.

7. Conclusions

191

Table 2.

Table 3.

#### Table 1.

Proposed SIFT with LM-NN of different plastic surgery faces.

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint… DOI: http://dx.doi.org/10.5772/intechopen.85550


#### Table 2.

Proposed V-SIFT with LM-NN of different plastic surgery faces.


#### Table 3.

Figure 18.

Table 1.

190

Proposed SIFT with LM-NN of different plastic surgery faces.

Visual Object Tracking with Deep Neural Networks

and validation fails.

Performance of LM-NN classifier in correspondence with (a) validation performance and (b) gradient, Mu,

LM-NN Accuracy Rhinoplasty 0.92 Sensitivity Malar augmentation 0.24 Specificity Rhinoplasty 0.97 Precision Skin peeling 0.04 FPR Rhinoplasty 0.03 FNR Malar augmentation 0.76 NPV Rhinoplasty 0.97 FDR Skin peeling 0.96 F1score Skin peeling 0.06 MCC Skin peeling 0.04

Proposed EV-SIFT with LM-NN of different plastic surgery faces.

Table 2. It is observed that for all the measures, the method has attained better results, which also leads to the other types of plastic surgery.

From the second technique, it is observed that the proposed EV-SIFT with LM-NN are achieved more over the conventional methods for various plastic surgeries, which is summarized in Table 3. It is observed that for all the measures, the method has attained better results.

#### 7. Conclusions

This chapter gives the detailed description of the second research technique. The feature descriptor EV-SIFT that is used for feature extraction is well explained. Further, the LM-based NN classifier is defined in this chapter, and the performance of both the EV-SIFT and LM-NN classifiers is shown in the Result section. The better work of EV-SIFT is effectively demonstrated in this section, which shows

how the images are distinguished between them. The analysis of the LM-NN classifier is also more satisfactory with better performance.

References

[1] Sao AK, Yegnanarayana B. Face verification using template matching. IEEE Transactions on Information

[3] Turk MA, Pentland AP. Face recognition using eigenfaces. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition; Maui, HI;

[4] Ruiz-del-Solar J, Navarrete P. Eigenspace-based face recognition: A comparative study of different approaches. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews). 2005;35(3):

[5] Lawrence S, Giles CL, Chung Tsoi A,

convolutional neural-network approach. IEEE Transactions on Neural Networks.

[6] He X, Yan S, Yuxiao H, Niyogi P, Zhang H-J. Face recognition using Laplacianfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2005;27(3):328-340

[7] Inan T, Halici U. 3-D face recognition with local shape descriptors. IEEE Transactions on Information Forensics and Security. 2012;7(2):577-587

[9] Xu Y, Fang X, Li X, Yang J, You J, Liu

[8] Lu Z, Jiang X, Kot AC. A color channel fusion approach for face recognition. IEEE Signal Processing Letters. 2015;22(11):1839-1843

H, et al. Data uncertainty in face recognition. IEEE Transactions on Cybernetics. 2014;44(10):1950-1961

193

Back AD. Face recognition: A

1991. pp. 586-591

315-325

1997;8(1):98-113

Forensics and Security. 2007;2(3):636-641

DOI: http://dx.doi.org/10.5772/intechopen.85550

[10] Jain AK, Klare B, Park U. Face recognition: Some challenges in forensics. In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011); Santa Barbara; 2011.

[11] Sharma P, Yadav RN, Arya KV. Pose-invariant face recognition using

curvelet neural network. IET Biometrics. 2014;3(3):128-138

[12] Lee S-H, Kim D-J, Cho J-H. Illumination-robust face recognition

[13] Zou WWW, Yuen PC. Very low resolution face recognition problem. IEEE Transactions on Image Processing.

[14] Park U, Tong Y, Jain AK. Ageinvariant face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2010;32(5):

[15] Mudunuri SP, Biswas S. Low resolution face recognition across variations in pose and illumination. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2016;38(5):

[16] Singh R, Vatsa M, Noore A. Effect of plastic surgery on face recognition: A preliminary study. In: 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops; Miami, FL;

[17] Singh R, Vatsa M, Bhatt HS,

Bharadwaj S, Noore A, Nooreyezdan SS. Plastic surgery: A new dimension to face recognition. IEEE Transactions on

system based on differential components. IEEE Transactions on Consumer Electronics. 2012;58(3):

pp. 726-733

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint…

963-970

947-954

1034-1040

2009. pp. 72-77

2012;21(1):327-340

[2] Schwartz WR, Guo H, Choi J, Davis LS. Face identification using large feature sets. IEEE Transactions on Image Processing. 2012;21(4):2245-2255

## Acknowledgements

To begin with, I express gratitude to God Shri Gajanan Maharaj, who provided me the potency as well as the ability to bring out this research. Additionally, I express my gratefulness to supervisor, Prof. S.N. Talbar, for his inspiration, priceless suggestion, and support along with concentration during exploration of research. I express thanks to every friend for giving out their experience and information. I also want to give an exceptional thanks to my companion for his honest advice and steady support to do a high-quality research.

## Author details

Archana Harsing Sable<sup>1</sup> \* and Haricharan A. Dhirbasi<sup>2</sup>

1 School of Computational Sciences, Swami Ramanand Teerth Marathwada University, Nanded, Maharashtra, India

2 Department of Mathematics, N.E.S. Science College, S.R.T.M. University, Nanded, Maharashtra, India

\*Address all correspondence to: helloarchu27@gmail.com

© 2019 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/ by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Granular Approach for Recognizing Surgically Altered Face Images Using Keypoint… DOI: http://dx.doi.org/10.5772/intechopen.85550

## References

how the images are distinguished between them. The analysis of the LM-NN classi-

To begin with, I express gratitude to God Shri Gajanan Maharaj, who provided

me the potency as well as the ability to bring out this research. Additionally, I express my gratefulness to supervisor, Prof. S.N. Talbar, for his inspiration, priceless suggestion, and support along with concentration during exploration of research. I express thanks to every friend for giving out their experience and information. I also want to give an exceptional thanks to my companion for his

\* and Haricharan A. Dhirbasi<sup>2</sup>

1 School of Computational Sciences, Swami Ramanand Teerth Marathwada

2 Department of Mathematics, N.E.S. Science College, S.R.T.M. University,

© 2019 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/ by/3.0), which permits unrestricted use, distribution, and reproduction in any medium,

\*Address all correspondence to: helloarchu27@gmail.com

honest advice and steady support to do a high-quality research.

fier is also more satisfactory with better performance.

Visual Object Tracking with Deep Neural Networks

Acknowledgements

Author details

192

Archana Harsing Sable<sup>1</sup>

Nanded, Maharashtra, India

University, Nanded, Maharashtra, India

provided the original work is properly cited.

[1] Sao AK, Yegnanarayana B. Face verification using template matching. IEEE Transactions on Information Forensics and Security. 2007;2(3):636-641

[2] Schwartz WR, Guo H, Choi J, Davis LS. Face identification using large feature sets. IEEE Transactions on Image Processing. 2012;21(4):2245-2255

[3] Turk MA, Pentland AP. Face recognition using eigenfaces. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition; Maui, HI; 1991. pp. 586-591

[4] Ruiz-del-Solar J, Navarrete P. Eigenspace-based face recognition: A comparative study of different approaches. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews). 2005;35(3): 315-325

[5] Lawrence S, Giles CL, Chung Tsoi A, Back AD. Face recognition: A convolutional neural-network approach. IEEE Transactions on Neural Networks. 1997;8(1):98-113

[6] He X, Yan S, Yuxiao H, Niyogi P, Zhang H-J. Face recognition using Laplacianfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2005;27(3):328-340

[7] Inan T, Halici U. 3-D face recognition with local shape descriptors. IEEE Transactions on Information Forensics and Security. 2012;7(2):577-587

[8] Lu Z, Jiang X, Kot AC. A color channel fusion approach for face recognition. IEEE Signal Processing Letters. 2015;22(11):1839-1843

[9] Xu Y, Fang X, Li X, Yang J, You J, Liu H, et al. Data uncertainty in face recognition. IEEE Transactions on Cybernetics. 2014;44(10):1950-1961

[10] Jain AK, Klare B, Park U. Face recognition: Some challenges in forensics. In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011); Santa Barbara; 2011. pp. 726-733

[11] Sharma P, Yadav RN, Arya KV. Pose-invariant face recognition using curvelet neural network. IET Biometrics. 2014;3(3):128-138

[12] Lee S-H, Kim D-J, Cho J-H. Illumination-robust face recognition system based on differential components. IEEE Transactions on Consumer Electronics. 2012;58(3): 963-970

[13] Zou WWW, Yuen PC. Very low resolution face recognition problem. IEEE Transactions on Image Processing. 2012;21(1):327-340

[14] Park U, Tong Y, Jain AK. Ageinvariant face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2010;32(5): 947-954

[15] Mudunuri SP, Biswas S. Low resolution face recognition across variations in pose and illumination. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2016;38(5): 1034-1040

[16] Singh R, Vatsa M, Noore A. Effect of plastic surgery on face recognition: A preliminary study. In: 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops; Miami, FL; 2009. pp. 72-77

[17] Singh R, Vatsa M, Bhatt HS, Bharadwaj S, Noore A, Nooreyezdan SS. Plastic surgery: A new dimension to face recognition. IEEE Transactions on

Information Forensics and Security. 2010;5(3):441-448

[18] Liu X, Shan S, Chen X. Face recognition after plastic surgery: A comprehensive study. In: Computer Vision – ACCV 2012, Lecture Notes in Computer Science. Vol. 7725. Springer. pp. 565-576

[19] De Marsico M, Nappi M, Riccio D, Wechsler H. Robust face recognition after plastic surgery using local region analysis. In: Image Analysis and Recognition, Lecture Notes in Computer Science. Vol. 6754. Springer. pp. 191-200

[20] Lakshmiprabha NS, Majumder S. Face recognition system invariant to plastic surgery. In: 2012 12th International Conference on Intelligent Systems Design and Applications (ISDA); Kochi; 2012. pp. 258-263

[21] Gulhane AR, Ladhake SA, Kasturiwala SB. A review on surgically altered face images recognition using multimodal bio-metric features. In: Proceedings of the 2015 2nd International Conference on Electronics and Communication Systems (ICECS); Coimbatore; 2015. pp. 1168-1171

[22] De Marsico M, Nappi M, Riccio D, Wechsler H. Robust face recognition after plastic surgery using region-based approaches. Pattern Recognition. 2015; 48(4):1261-1276

[23] Kohli N, Yadav D, Noore A. Multiple projective dictionary learning to detect plastic surgery for face verification. IEEE Access. 2015;3: 2572-2580

[24] Chude-Olisah CC, Sulong GB, Chude-Okonkwo UAK, Hashim SZM. Edge-based representation and recognition for surgically altered face images. In: Proceedings of the 2013 7th International Conference on Signal

Processing and Communication Systems (ICSPCS); Carrara, VIC; 2013. pp. 1-7

[25] Ouanan H, Ouanan M. GaborHOG features based face recognition scheme. TELKOMNIKA Indonesian Journal of Electrical Engineering. 2015;15(2): 331-335

[26] Ouloul MI, Moutakki Z, Afdel K, Amghar A. An Efficient Face Recognition using SIFT Descriptor in RGBD Images. International Journal of Electrical and Computer Engineering (IJECE). 2015;5(6)

[27] Bhatt HS, Bharadwaj S, Singh R, Vatsa M. Recognizing surgically altered face images using multiobjective evolutionary algorithm. IEEE Transactions on Information Forensics and Security. 2013;8(1):89-100

Information Forensics and Security.

Visual Object Tracking with Deep Neural Networks

Processing and Communication Systems (ICSPCS); Carrara, VIC; 2013. pp. 1-7

[25] Ouanan H, Ouanan M. GaborHOG features based face recognition scheme. TELKOMNIKA Indonesian Journal of Electrical Engineering. 2015;15(2):

[26] Ouloul MI, Moutakki Z, Afdel K,

Recognition using SIFT Descriptor in RGBD Images. International Journal of Electrical and Computer Engineering

[27] Bhatt HS, Bharadwaj S, Singh R, Vatsa M. Recognizing surgically altered face images using multiobjective evolutionary algorithm. IEEE

Transactions on Information Forensics

and Security. 2013;8(1):89-100

Amghar A. An Efficient Face

(IJECE). 2015;5(6)

331-335

[19] De Marsico M, Nappi M, Riccio D, Wechsler H. Robust face recognition after plastic surgery using local region analysis. In: Image Analysis and

Recognition, Lecture Notes in Computer

[20] Lakshmiprabha NS, Majumder S. Face recognition system invariant to

International Conference on Intelligent Systems Design and Applications (ISDA); Kochi; 2012. pp. 258-263

Kasturiwala SB. A review on surgically altered face images recognition using multimodal bio-metric features. In: Proceedings of the 2015 2nd

International Conference on Electronics and Communication Systems (ICECS); Coimbatore; 2015. pp. 1168-1171

[22] De Marsico M, Nappi M, Riccio D, Wechsler H. Robust face recognition after plastic surgery using region-based approaches. Pattern Recognition. 2015;

[23] Kohli N, Yadav D, Noore A. Multiple projective dictionary learning to detect plastic surgery for face verification. IEEE Access. 2015;3:

[24] Chude-Olisah CC, Sulong GB, Chude-Okonkwo UAK, Hashim SZM. Edge-based representation and recognition for surgically altered face images. In: Proceedings of the 2013 7th International Conference on Signal

48(4):1261-1276

2572-2580

194

Science. Vol. 6754. Springer.

plastic surgery. In: 2012 12th

[21] Gulhane AR, Ladhake SA,

[18] Liu X, Shan S, Chen X. Face recognition after plastic surgery: A comprehensive study. In: Computer Vision – ACCV 2012, Lecture Notes in Computer Science. Vol. 7725. Springer.

2010;5(3):441-448

pp. 565-576

pp. 191-200

## *Edited by Pier Luigi Mazzeo, Srinivasan Ramakrishnan and Paolo Spagnolo*

Visual object tracking (VOT) and face recognition (FR) are essential tasks in computer vision with various real-world applications including human-computer interaction, autonomous vehicles, robotics, motion-based recognition, video indexing, surveillance and security. This book presents the state-of-the-art and new algorithms, methods, and systems of these research fields by using deep learning. It is organized into nine chapters across three sections. Section I discusses object detection and tracking ideas and algorithms; Section II examines applications based on re-identification challenges; and Section III presents applications based on FR research.

Visual Object Tracking with

Deep Neural Networks

*Edited by Pier Luigi Mazzeo,* 

*Srinivasan Ramakrishnan and Paolo Spagnolo*

ISBN 978-1-78985-157-1

ISBN 978-1-78985-142-7

Visual Object Tracking with Deep Neural Networks