**5.2 Results of pose classification and adaptation**

The system uses 10 hand poses of 7 persons for evaluating pose classification and adaptation method. All the training and test images are 60x60 pixels gray images. This system is first trained using 200 images of 10 poses of person\_1 (20 images of each pose). It automatically clusters the images into 13 clusters. Figure 12 shows the sample outputs of hand poses learning method for person\_1. If the user uses two hands to make the same pose then it forms two different clusters for the same pose. Different clusters can also be formed for the variation of orientation even the pose is same.

Fig. 17. Cluster member distributions for hand poses

If the person is change, then it may form different clusters for the same hand poses (gestures) due to the variation of hand shape and color. After trained with 10 hand poses of person\_1, 200 images of 10 hand poses of person\_2 are feed to the system. The system developed 9 more clusters for the person\_2 corresponding to 8 hand poses. For, the 'LEFTHAND' and 'RIGHTHAND' palm it did not develop new clusters, rather inserted new members in those clusters. Table 1 shows the 22 clusters developed for 10 hand poses of 2 persons and their associations with hand poses. Figure 17 shows the distributions of the clusters of 10 hand poses for two persons.

User, Gesture and Robot Behaviour Adaptation for Human-Robot Interaction 249

**Comparison of Accuracy**

B\_Adp\_Ac

A\_Adp\_Ac

1 3 5 7 9 11 13

Fig. 18. Comparison of pose classification accuracy before after adaptation

**6. Implementation of human-robot interaction** 

has been implemented for the following scenario:

Robot: "Hi Person\_1, How are you?" (speech)

Robot: " Oh Good! Do you want to play now?" (speech)

Robot: imitates user's gesture "Raise Two Arms" as shown in Figure 19.

Person\_1: uses the gesture "Ok"

Person\_1: uses the gesture "YES" Robot: "Oh Thanks" (speech)

Person\_1: uses the gesture "TwoHand"

**ASL Characters**

Figure 18 depicts the graphical representations of 14-ASL characters classification accuracy using adaptation method (after adaptation) and without adaptation method (before adaptation). The comparison curves shows that if we include adaptation method then pose classification performance will be better, but needs user interaction that is bottleneck of this

The real-time gesture based human-robot interaction is implemented as an application of this system. This approach has been implemented on a humanoid robot, name"Robovie". Since the same gestures can mean different tasks for different persons, we need to maintain the gesture with person-to-task knowledge. The robot and the gesture recognition PC are connected to SPAK knowledge server. From the image analysis and recognition PC, person identity and pose names are sent to the SPAK for decisions making and the robot activation. According to gesture and user identity, the knowledge module generates executable codes for robot actions. The robot then follows speech and body action commands. This method

User: "Person\_1" comes in front of Robovie eyes camera and robot recognizes the user as

**Accuracy (%)**

method.

"Person\_1".


Table 1. List of hand poses and associated clusters


Table 2. Comparison of pose classification accuracy (before and after adaptation) for 14 ASL Characters

In this study we have also compared pose classification accuracy (%) using two methods: one is multi-cluster based approach without adaptation, the other is multi-cluster based approach with adaptation. In this experiment we have used total 840 training images, (20 images of each pose of each person and 1660 test images of 14 ASL characters [ASL, 2004] of three persons. Table 2 presents the comparisons of two methods (B\_Adap=before adaptation and A\_Adap=after adaptation). This table shows that the accuracy of pose classification method which includes the adaptation or learning approach is better, because the learning function increments the clusters members or forms new clusters if necessary to classify the new images.

248 The Future of Humanoid Robots – Research and Applications

A 120 81 119 67.50 99.16 B 123 82 109 66.66 88.61 C 110 73 103 66.36 93.63 D 120 78 106 65 88.33 E 127 80 96 62.99 75.59 F 120 81 114 67.50 95 G 120 118 120 98.33 100 I 100 56 100 56 100 K 120 107 116 89.16 96.66 L 120 100 119 83.33 99.16 P 120 79 119 65.83 99.16 V 120 75 86 62.50 71.66 W 120 74 101 61.66 84.16 Y 120 85 98 70.83 81.66

Table 2. Comparison of pose classification accuracy (before and after adaptation) for 14 ASL

In this study we have also compared pose classification accuracy (%) using two methods: one is multi-cluster based approach without adaptation, the other is multi-cluster based approach with adaptation. In this experiment we have used total 840 training images, (20 images of each pose of each person and 1660 test images of 14 ASL characters [ASL, 2004] of three persons. Table 2 presents the comparisons of two methods (B\_Adap=before adaptation and A\_Adap=after adaptation). This table shows that the accuracy of pose classification method which includes the adaptation or learning approach is better, because the learning function increments the clusters members or forms new clusters if necessary to classify the

**Correct Recognition Accuracy (%)**  *Before\_Adap After\_Adap B\_Adap A\_Adap* 

**Pose Name Associated Clusters** 

ONE (Raise Index Finger) PC1, PC15 FIST (Fist Up) PC2, PC3, PC19 OK (Make circle using thumb and index fingers) PC4, PC20 TWO (V sign using index and middle fingers) PC5, PC6, PC16 THREE (Raise index, middle and ring fingers) PC7, PC17, PC18

LEFTHAND (Left hand palm) PC8 RIGHTHAND (Right hand palm) PC9

Table 1. List of hand poses and associated clusters

**Number of Image** 

**Input Images (ASL Char)** 

Characters

new images.

THUMB (Thumb Up) PC10, PC11, PC14 POINTL (Point Left) PC12, PC21 POINTL (Point Right) PC13, PC22

Fig. 18. Comparison of pose classification accuracy before after adaptation

Figure 18 depicts the graphical representations of 14-ASL characters classification accuracy using adaptation method (after adaptation) and without adaptation method (before adaptation). The comparison curves shows that if we include adaptation method then pose classification performance will be better, but needs user interaction that is bottleneck of this method.
