**5.1 Results of user recognition and adaptation**

Seven individuals were asked to act for the predefined face poses in front of the camera and all the sequence of face images were saved as individual image frame. All the training and test images are 60x60 pixels gray face images. The adaptation algorithm is tested for 7 persons frontal face or normal face (NF) images, and five directional face images (normal face, left directed face, right directed face, up directed face, down directed face). Figure 13 244 The Future of Humanoid Robots – Research and Applications

index finger), 'Two' (form V sign using index and middle fingers), 'Three' (raise index, middle and ring fingers), 'ThumbUp' (thumb up), 'Ok' (make circle using thumb and index finger), 'FistUp' (fist up), 'PointLeft' (point left by index finger), 'PointRight' (point right by

It is possible to recognize more gestures including new poses and new rules for the gesture using this system. New poses can be included in the training image database using the interactive learning method and corresponding frame can be defined in the knowledge base to interpret the gesture. To teach the robot a new poses, the user should perform the poses several times (example 10 image frame times.). Then the learning method detects it as a new pose and creates cluster/clusters for that pose. Sequentially, it updates the knowledge base

Robot behaviours or actions can be programmed or learned through experience. But it is difficult to perceive human or user intention to acts robot with his/her gestures. This system has proposed the experience based and user-interactive robot behaviour learning or adaptation method. In this method the history of the similar gesture-action map is stored in the knowledge base. According to maximum user desires action will be select for the gesture and ask for the user acknowledgement. If user consents or uses "YES" gesture (or types

"Yes") the corresponding frame will be store permanently. For Example scenario:

Person\_n: shows "OK" hand gesture (make circle using thumb and index fingure).

Suppose there are two users already use 'OK' gesture to mean Yes, so Robovie adds

This system uses a standard video camera and 'Robovie' eye's camera for data acquisition. Each captured image is digitized into a matrix of 320 240 pixels with 24-bit color. User and hand pose adaptation method is verified using real-time captured images as well as static images. The algorithm has also been tested with a real world human-robot interaction

Seven individuals were asked to act for the predefined face poses in front of the camera and all the sequence of face images were saved as individual image frame. All the training and test images are 60x60 pixels gray face images. The adaptation algorithm is tested for 7 persons frontal face or normal face (NF) images, and five directional face images (normal face, left directed face, right directed face, up directed face, down directed face). Figure 13

index finger).

for the cluster information.

**4.5 Robot behaviours adaptation** 

Person\_n: comes in front of Robovie.

Robovie: asks "Do you mean Yes"? Person\_n: again shows "OK" gesture.

Robovie: cannot recognize, and asks, "Who are you?" Person\_n: types "Person\_n" (or says "Person\_n").

"OK=Yes for everybody" into his knowledge base.

**5. Experimental results and discussions** 

**5.1 Results of user recognition and adaptation** 

Robovie: Says, "Hello Person\_n, do you want to play with me?"

Robovie: add "Ok ="Yes, for Person\_n" into his knowledge base.

system using a humanoid robot, 'Robovie' developed by ATR .

shows the sample result of the user adaptation method for normal faces. In the first step, the system is trained using 60 face images of three persons and developed three clusters (top 3 rows of Figure 13) corresponding to three persons. The cluster information table contents are [1, 11, 23, 29]. For example, in this situation if any input face image matches with the known face image member between 1 and 10 then the person is identified as person 1.

Fig. 13. Sample outputs of the clustering method for frontal faces

In the second step, 20-face image sequences of another person are fed to the system as input. The minimum Euclidian distances (ED) from three known persons face images are shown using upper line graph (B\_adap) in Figure 14. The system identifies these faces as unknown person based on predefined threshold value for the Euclidian distance and activates the user learning function. The user learning function developed new cluster (4th row of Figure 13) and updated the cluster information table as [1, 11, 23, 30, 37]. After adaptation, the minimum Euclidian distance distribution line (A\_adap) in Figure 7.21 shows that, for 8 images, minimum ED is zero and those are included in the new cluster so that the system can recognize the person. This method is tested for 7 persons including 2 females, and as a result of learning, 7 clusters with different length (number of images per cluster) for different persons (as shown in Figure 13) were formed.

The users adaptation method is also tested for 700 five directional face images of 7 persons (sample output in Figure 11). Figure 15 shows the distribution of 41 clusters for the 700 face images of 7 persons. In the first step, the system is trained using 100 face images of person\_1 and it formed 5 clusters based on 5-directional faces. At this time, the contents of the cluster information table (that holds staring pointer of each cluster in the training database) are [1, 12, 17, 27, 44, 52]. After learning person\_2, the cluster information table contents are [1, 12, 17, 27, 44, 53, 68, 82, 86, 96, 106, 116]. Similarly, other persons are adapted. Figure 16 shows the example of errors in clustering process. In the cluster 26 up directed faces of person\_6 and frontal face person\_5 are overlapped and treated as one cluster (Figure 16 (a)). In the case of cluster 31, up directed faces of person\_5 and normal (frontal) faces of person\_6 are overlapped and grouped in the same cluster (Figure 16 (b)). This problem can be solved using narrow threshold, but in that case the number of iteration as well as discard rates of the images classification method will be increased.

User, Gesture and Robot Behaviour Adaptation for Human-Robot Interaction 247

(a) Cluster 26: Person\_5 frontal face and person \_6 up directed face overlapping (231-261)

(b) Cluster 31: Person\_5 up face and person\_6 (from right 2 image) frontal face overlapping

The system uses 10 hand poses of 7 persons for evaluating pose classification and adaptation method. All the training and test images are 60x60 pixels gray images. This system is first trained using 200 images of 10 poses of person\_1 (20 images of each pose). It automatically clusters the images into 13 clusters. Figure 12 shows the sample outputs of hand poses learning method for person\_1. If the user uses two hands to make the same pose then it forms two different clusters for the same pose. Different clusters can also be formed

**Clusters Length for Hand Poses**

Pose\_CL

1 3 5 7 9 11 13 15 17 19 21

**Number of Clusters**

If the person is change, then it may form different clusters for the same hand poses (gestures) due to the variation of hand shape and color. After trained with 10 hand poses of person\_1, 200 images of 10 hand poses of person\_2 are feed to the system. The system developed 9 more clusters for the person\_2 corresponding to 8 hand poses. For, the 'LEFTHAND' and 'RIGHTHAND' palm it did not develop new clusters, rather inserted new members in those clusters. Table 1 shows the 22 clusters developed for 10 hand poses of 2 persons and their associations with hand poses. Figure 17 shows the distributions of the

Fig. 16. Example of errors in clustering process

**5.2 Results of pose classification and adaptation** 

for the variation of orientation even the pose is same.

Fig. 17. Cluster member distributions for hand poses

clusters of 10 hand poses for two persons.

0

10

20

**Length of Cluster**

30

40

Fig. 14. Euclidian distances distribution of 20 frontal faces (before and after adaptation)

Fig. 15. Cluster member distributions for five directed face poses

(a) Cluster 26: Person\_5 frontal face and person \_6 up directed face overlapping (231-261)

(b) Cluster 31: Person\_5 up face and person\_6 (from right 2 image) frontal face overlapping

Fig. 16. Example of errors in clustering process

246 The Future of Humanoid Robots – Research and Applications

**Euclidian Distance Comparison**

B\_adap. A\_adap.

Cluster

1 3 5 7 9 11 13 15 17 19

**Number of Images**

Fig. 14. Euclidian distances distribution of 20 frontal faces (before and after adaptation)

**Cluster Member Distribution**

1 4 7 10 13 16 19 22 25 28 31 34 37 40

**Number of Clusters**

Fig. 15. Cluster member distributions for five directed face poses

0

**Number of Member**

1

2

3

**Minimum Euclidian** 

**Distance (1.e\*10^7)**

4

5
