**3. The method**

determining the palm orientation and width, establishing local coordinate system,

In [17, 44], 25 hand landmarks are selected to form a shape, including 10 end points and 15 landmarks of the finger valleys and palm boundary. This shape convers the finger roots and the interdigital regions of the palm. By AAM algorithm, both the hand shape and the palm texture information are utilized, the shape and corresponding landmark points can automatically reshape itself to fit the real hand contour. To evaluate the localization performance, the authors proposed a modified point-to-curve distance and a margin width metric. Since the initial position of the shape model is critical to the regression performance, the fitting process is divided into two stages. At first five rotations and five scale factors are used to generate 25 initial shapes. After regression, only the shape models, which obtain the 15 optimal reconstruction errors, are passed to the second stage for fine-grained regression. In [41], the authors proposed a CNN framework based on LeNet [51] to detect the finger valley points. The proposed network involves convolutional layers and fully connected layers; the output is a six-dimensional vector corresponding to the three valley points between fingers excluding the thumb. In their work, two neural networks are designed: one is for identifying whether the hand is left or right, and another is for landmark localization. According to their experiments, the first network can perfectly identify the hand being a left or right hand, and the landmark localization performance is better than the classical method which is based on Otsu

In [24], based on VGG-16 [52], the authors designed an end-to-end neural network to localize the hand landmarks, generate the aligned ROI, and do feature extraction and recognition tasks at the same time. The hand region is extracted from the original Internet image, and then it is resized to 227 227 pixels. The normalized palm image will be put into the designed CNN for aligned ROI localization, feature extraction, and classification. The proposed network consists of two subnetworks, ROI Localization and alignment network (ROI-LAnet) and feature extraction and recognition network (FERnet). More than three landmarks are determined by the

In [47], at first the palm position is detected by techniques of sliding window, histogram of oriented gradient (HOG) [53], and support vector machine (SVM). In the training set, 14 landmark points are determined and labeled manually. After landmark point regression, the reference line is established by the two valley points. The position of the center point and the side length of the ROI both are determined

However, there still exist some challenging problems in ROI localization waiting

5.Palm scale (palm width) determination under various palm and finger poses in

Compared with the new-generation methods which need to label the landmarks manually and train the regression model, the classical methods based on hardware

and computing the ROI locations.

*Biometric Systems*

by the palm width.

for better solutions:

touchless scenarios

**56**

segmentation and Zhang's ROI localization algorithm [4].

author in order to be able to parametrize non-rigid transformations.

1.Palm region segmentation under complex backgrounds

3.Keypoint detection on palms with big rotations

2.Keypoint detection on palms with closed or incomplete fingers

4.Left and right hand detection on palms having long thumbs

### **3.1 The ROI localization method**

As discussed above, palmprint localization involves four main stages: (1) palm region segmentation, (2) palm contour and finger valley landmark detection, (3) ROI coordinate computation, and (4) abnormal detection. The method used in this chapter is modified from [4, 30]. For palm segmentation, Otsu-based methods can achieve good results in IR image, but for visible light image, the segmentation results will be interfered by the shadow regions on the palm surface. To achieve high success rates of ROI localization, the skin-color based classifier is utilized to separate the palm region. The main work of landmark localization is detecting the finger valleys of the index-middle fingers and the ring-little fingers (as is shown in **Figure 2(a)** and **(d)**). For contact-based palmprint image, the palm position is restricted by the pegs, so the finger valleys can be easily localized by line-scan-based methods (as is shown in **Figure 2(c)**). The pixels are tested from up to down, once the value changes from **white** to **black**, point *p*<sup>1</sup> is detected. Keep testing, once the pixel value changes from **black** to **white**, point *p*<sup>2</sup> is detected. The other keypoints (*p*<sup>3</sup> � *p*6) also are searched in this way. If *p*<sup>1</sup> � *p*<sup>6</sup> cannot be detected in the current column, the scan line *ls* will move from left to right by a predefined step to test the new column. This process will be iteratively conducted before *p*<sup>1</sup> � *p*<sup>6</sup> are detected. Then, finger valleys *v*1 and *v*2 can be obtained by edge tracking. *vp*<sup>1</sup> and *vp*<sup>2</sup> are the detected tangent points. This scan-line-based method is effective for contact-based palm images. However, in touchless environment, most palms are captured with obvious rotations, and the space between fingers also varies a lot. Hence, the linescan-based methods are not always workable. To deal with palm rotations, hand direction should be detected first. Hand orientation can be represented by the direction of the principal axis of the palm region pixels after principal component analysis or by the direction of the line which passes through the palm center point *c* and the fingertip of the middle finger (as is shown in **Figure 2(a)**). The PCA-based method can be combined with the line-scan-based method to achieve simple detection for rotated palms. But we prefer the second strategy, because if someone's thumb is as long as the little finger, it may decrease the performance of the linescan-based method. Based on the methods proposed in [30], after palm segmentation and contour detection, the reference point *rp* is determined. Connecting *rp* with each point on the palm contour in anti-clockwise direction, for each pair of points, we could calculate their distance, and then a distance list is obtained after all the points are traversed. The distance curve is shown in **Figure 2(b)**. The extremum points of this curve correspond to fingertips and valley points. Then the finger valleys can be obtained on the palm contour. After *v*<sup>1</sup> and *v*<sup>2</sup> are obtained, the tangent line can be detected by the method proposed in [4]. Its length is denoted as *lt*. Based on the tangent line, the palm coordinate system could be established. *d* stands for the distance between the ROI and the tangent line; *d* can be determined by length *lt*. The ROI side length *s* also can be determined according to *lt*. Let *d* ¼ *α* � *lt*, and *s* ¼ *β* � *lt*. Different values of *α*, *β* may result in different recognition performance.

**Figure 3.** *Abnormal ROIs caused by complex background objects, difficult hand poses, and bad illuminations.*

#### **3.2 Abnormal detection and iterative localization**

The keypoint detection method described above is based on local vision. The algorithms know whether the predefined keypoints, *p*<sup>1</sup> *p*<sup>6</sup> and *vp*<sup>1</sup> *vp*2, are obtained. But they cannot tell whether the detected points are the correct ones. Background noise and abnormal palm poses will cause error localizations and thus generate abnormal ROIs. Those ROI images should be removed to avoid security risks. For example, during the process of sample registration, if a ROI falsely located in the background region, the black ROI image will be extracted and registered. This may cause big risk in real-world applications, since everyone can pass the system by a black image. Many deep learning-based image denoising methods have been proposed [54, 55], but, to some extent, they are too time-consuming to the target of palmprint image preprocessing. Hence, a high-speed abnormal ROI detection method is required. Here, the angle and scale of the ROI, the ratio of the background region (if there exist background regions in the located ROI), and the ratio of the width of the two finger valleys are selected as features for abnormal detection. They are denoted as *θ*, *lt*, *rbg*, and *rh*, respectively. The area of the ROI stands for the scale information, so we use the tangent line length *lt* instead. Then, for each time ROI localization, the feature vector *θ*, *lt*,*rbg*,*rh* can be obtained. To train a SVM-based abnormal detector, first, conduct the simple localization algorithm described above on the training set to generate different kinds of ROIs (as is shown in **Figure 3**). Then, separate the ROIs into normal and abnormal subsets. Last, a binary classifier can be trained by them. According to our experiments, all the false ROIs in **Figure 3** can be successfully detected. With the abnormal detector, for linescan-based method, once the current localized ROI is refused by the detector, it can move to the next position to iteratively detect the ROI. If the terminal condition is triggered, it means this image sample is unprocessable.

difficult to segment the palm region. The strong light source and the small enclosure lead to light reflections and ray occlusions, which generate many bright regions in the background and dark regions on the palm surface. Hence, the brightness information is not sufficient for segmenting the palm region. The color information should be utilized. As is shown in **Figure 5**, we randomly cropped some palm skin and background image patches to build a training set for segmentation. A SVMbased binary classifier is learned from the training dataset; the segmented palm

*Image patches cropped from the palm skin and the black box. (a) Patches of the palm surface; (b) patches of the*

**Hard samples**

0271, 0273, 0287, 0293, 0307, 0311, 0328, 0379

IITD Left 0.38 0037\_0006, 0107\_0002, 0152\_0003, 0181\_0001, 0209\_0003 Right 0.31 0137\_0001, 0140\_0004, 0204\_0003, 0204\_0004

PolyU 0.54 0004, 0039, 0073, 0109, 0127, 0187, 0223, 0224, 0245, 0246, 0259,

COEP 0.31 0103\_0004, 0145\_0003, 0159\_0006, 0164\_0002

*Region of Interest Localization Methods for Publicly Available Palmprint Databases*

**Database Error**

**Table 2.**

**Figure 4.**

**Figure 5.**

*black box.*

**59**

**rate (%)**

*DOI: http://dx.doi.org/10.5772/intechopen.93131*

*ROI localization results of different databases.*

*ROI localization results on IITD. (a) Left hand; (b) right hand.*

#### **4. Experiments on different databases**

In this section, the performance of the designed method is tested on different palmprint databases. More details and further updates can be found at [56]. The error rates of ROI localization are enumerated in **Table 2**. For IITD and COEP, the numbers in hard samples stand for PalmID\_SampleID; for PolyU, the numbers stand for PalmID. After each experiment, the error cases are analyzed in detail.

#### **4.1 ROI extraction for IITD Touchless Palmprint Database**

**Figure 4** shows the ROIs localized by the proposed method. Although the palm images in IITD Touchless Palmprint Database are captured in a black box, it is still


*Region of Interest Localization Methods for Publicly Available Palmprint Databases DOI: http://dx.doi.org/10.5772/intechopen.93131*

#### **Table 2.**

**3.2 Abnormal detection and iterative localization**

**Figure 3.**

*Biometric Systems*

**58**

time ROI localization, the feature vector *θ*, *lt*,*rbg*,*rh*

triggered, it means this image sample is unprocessable.

**4.1 ROI extraction for IITD Touchless Palmprint Database**

**4. Experiments on different databases**

The keypoint detection method described above is based on local vision. The algorithms know whether the predefined keypoints, *p*<sup>1</sup> *p*<sup>6</sup> and *vp*<sup>1</sup> *vp*2, are obtained. But they cannot tell whether the detected points are the correct ones. Background noise and abnormal palm poses will cause error localizations and thus generate abnormal ROIs. Those ROI images should be removed to avoid security risks. For example, during the process of sample registration, if a ROI falsely located in the background region, the black ROI image will be extracted and registered. This may cause big risk in real-world applications, since everyone can pass the system by a black image. Many deep learning-based image denoising methods have been proposed [54, 55], but, to some extent, they are too time-consuming to the target of palmprint image preprocessing. Hence, a high-speed abnormal ROI detection method is required. Here, the angle and scale of the ROI, the ratio of the background region (if there exist background regions in the located ROI), and the ratio of the width of the two finger valleys are selected as features for abnormal detection. They are denoted as *θ*, *lt*, *rbg*, and *rh*, respectively. The area of the ROI stands for the scale information, so we use the tangent line length *lt* instead. Then, for each

*Abnormal ROIs caused by complex background objects, difficult hand poses, and bad illuminations.*

SVM-based abnormal detector, first, conduct the simple localization algorithm described above on the training set to generate different kinds of ROIs (as is shown in **Figure 3**). Then, separate the ROIs into normal and abnormal subsets. Last, a binary classifier can be trained by them. According to our experiments, all the false ROIs in **Figure 3** can be successfully detected. With the abnormal detector, for linescan-based method, once the current localized ROI is refused by the detector, it can move to the next position to iteratively detect the ROI. If the terminal condition is

In this section, the performance of the designed method is tested on different palmprint databases. More details and further updates can be found at [56]. The error rates of ROI localization are enumerated in **Table 2**. For IITD and COEP, the numbers in hard samples stand for PalmID\_SampleID; for PolyU, the numbers stand for PalmID. After each experiment, the error cases are analyzed in detail.

**Figure 4** shows the ROIs localized by the proposed method. Although the palm images in IITD Touchless Palmprint Database are captured in a black box, it is still

can be obtained. To train a

*ROI localization results of different databases.*

**Figure 4.** *ROI localization results on IITD. (a) Left hand; (b) right hand.*

difficult to segment the palm region. The strong light source and the small enclosure lead to light reflections and ray occlusions, which generate many bright regions in the background and dark regions on the palm surface. Hence, the brightness information is not sufficient for segmenting the palm region. The color information should be utilized. As is shown in **Figure 5**, we randomly cropped some palm skin and background image patches to build a training set for segmentation. A SVMbased binary classifier is learned from the training dataset; the segmented palm

**Figure 5.** *Image patches cropped from the palm skin and the black box. (a) Patches of the palm surface; (b) patches of the black box.*

them first. The pegs' colors are green, blue, and yellow. After removing the bright yellow pixels in the image, we extract the red channel from the original RGB image to conduct ROI localization algorithm. In this way, the green and blue pegs can be automatically removed (as is shown in **Figure 7**). **Results:** as is shown in **Figure 8**, after ROI localization, four images failed to be correctly localized. All of the four

*Region of Interest Localization Methods for Publicly Available Palmprint Databases*

For PolyU database, which contains 7752 images, the line-scan-based method is utilized to localize the ROI. At last, 42 samples failed to be localized. As is shown in **Figure 9**, most of them are caused by small finger valleys (palm pose) and unideal palm region segmentations (only grayscale information can be utilized). In **Table 2**,

*Images cannot be localized of COEP database. (a)–(d), (e)–(h), and (i)–(l) are the original, binary, and ROI localization images, respectively. The image ID of (a), (e), and (i) is 0103\_0004; the image ID of (b), (f), and (j) is 0145\_0003; the image ID of (c), (g), and (k) is 0159\_0006; and the image ID of (d), (h), and (l) is*

error cases are caused by closed fingers.

*DOI: http://dx.doi.org/10.5772/intechopen.93131*

**4.3 ROI extraction for PolyU Palmprint Database**

only the user ID is listed for the PolyU database.

**Figure 8.**

*0164\_0002.*

**Figure 9.**

**61**

*Hard samples of PolyU.*

#### **Figure 6.**

*Images cannot be localized in IITD database. (a) and (c) are the finger detection failed samples; (b) and (d) are the ROI localization error samples.*

region can be seen in **Figure 6**. For palm skin patches, both the bright and dark regions are selected to learn a precise classification plane. In the color space, the palm can be easily segmented from the unicolor background. After palm region segmentation, *vp*<sup>1</sup> and *vp*<sup>2</sup> are detected by the local-extremum-based method. **Results: Figure 6** shows the IITD samples that are hard to process. If we cannot detect five fingertips and four finger valley points after palm segmentation, the system will directly return by giving an error code (as is shown in **Figure 6(a)** and **(c)**). If false keypoints are detected due to difficult palm poses, the finally extracted ROI images are abnormal images which should be discarded in real-world applications (as is shown in **Figure 6(b)** and **(d)**).

### **4.2 ROI extraction for COEP Palmprint Database**

The line-scan-based keypoint detection method is used for COEP. **Figure 7** shows the ROI localization results on COEP database. Since the pegs used in their imaging setup may interfere the keypoint detection algorithm, we should delete

**Figure 7.** *ROI localization result of the COEP database.*

### *Region of Interest Localization Methods for Publicly Available Palmprint Databases DOI: http://dx.doi.org/10.5772/intechopen.93131*

them first. The pegs' colors are green, blue, and yellow. After removing the bright yellow pixels in the image, we extract the red channel from the original RGB image to conduct ROI localization algorithm. In this way, the green and blue pegs can be automatically removed (as is shown in **Figure 7**). **Results:** as is shown in **Figure 8**, after ROI localization, four images failed to be correctly localized. All of the four error cases are caused by closed fingers.
