**3. Frame-based knowledge-representation system for gesture-based HRI**

The 'frame-based approach' is a knowledge-based problem solving approach based on the so called, 'Frame theory', first proposed by Marvin Minsky [Minsky, 1974]. A frame is a data-structure for representing a stereotyped unit of human memory including definitive and procedural knowledge. Attached to each frame there are several kinds of information about the particular object or concept it describes such as name and a set of attributes called slots. Collections of related frames are linked together into frame systems. Framed-based approach has been used successfully in many robotic applications [Ueno, 2002]. Ueno presented the concepts and methodology of knowledge modeling based on Cognitive Science for realizing the autonomous humanoid service robotics arm and hand system HARIS [Ueno, 2000]. A knowledge-based software platform called SPAK (Software Platform for Agent and Knowledge management) has developed for intelligent service robots under the internet-based distributed environment [Ampornaramveth, 2004]. SPAK has been developed to be a platform on which various software components for different robotic tasks can be integrated over a networked environment. SPAK works as a knowledge and data management system, communication channel, intelligent recognizer, intelligent scheduler, and so on. Zhang et. al. [Zhang, 2004b] have developed an Industrial Robot Arm control system using SPAK. In that system SPAK works as a communication channel and intelligent robot actions scheduler. Kiatisevi et. al. [Kiatisevi, 2004] has proposed a

User, Gesture and Robot Behaviour Adaptation for Human-Robot Interaction 237

Fig. 2. Knowledge Editor showing the example gesture-based human-robot interaction

class-frame 'Robot' defined in SPAK.

Fig. 3. Example of a robot action-frame 'RaiseTwoArms' in SPAK

The frame system can be created in Graphical User Interface (GUI) and interpreted by the SPAK inference engine. Figure 2 shows the example knowledge Editor (part of the system) with the example of 'Robot', 'User', 'Gesture', 'Behaviour' (robot actions) and 'Pose' frames. The knowledge Editor Window displays the current frame hierarchy. Each frame is represented as click-able button. The buttons are connected with lines indicating IS\_A relationships among the frames. Clicking on the frame button brings up its slot editor. Figure 3 shows an example of slot editor for the robot (Robovie) action-frame "RaiseTwoArms". The attributes for a Slot are defined by slot name, type, value, condition, argument, required, shared, etc. Figure 4 shows an example instance-frame 'Hasan' of the class-frame 'User' defined in SPAK. Figure 5 shows an example instance-frame 'Robovie' of

distributed architecture for knowledge-based interactive robots and through SPAK they have implemented dialogue-based human-robot ('Robovie') interaction for greeting scenarios. This system employs SPAK, a frame-based knowledge engine, connecting to a group of network software agents such as 'Face recognizers', 'Gesture recognizers', 'Voice recognizers', 'Robot Controller', etc. Using information received from these agents, and based on the predefined frame knowledge hierarchy, SPAK inference engine determines the actions to be taken and submit corresponding commands to the target robot control agents.

Fig. 1. Frame hierarchy for gesture-based human-robot interaction system (SG=Static Gesture, DG=Dynamic Gesture)

Figure 1 shows the frame hierarchy of the knowledge model for the gesture-based humanrobot interaction system, organized by the IS\_A relationship indicated by arrows connecting upper and lower frame boxes. Necessary frames are defined for the users, robots, poses, gestures and robot behaviors (actions). The user frame includes instances of all known users (instance "Hasan", "Cho", …); robot frame includes instances of all the robots ("Aibo", "Robovie",….) used in the system. The behavior frame can be sub-classed further into "AiboAct" and "RobovieAct", where "AiboAct" frame includes instances of all the predefined 'Aibo' actions and "RobovieAct" frame includes instances of all the predefined 'Robovie' actions. The gesture fame is sub-classed into "Static" and "Dynamic" for static and dynamic gestures respectively. Examples of static frame instances are, "TwoHand", "One", etc. Examples of dynamic frame instances are "YES", "NO", etc. The pose frame includes all recognizable poses such as "LEFTHAND", "RIGHTHAND", "FACE", "ONE", etc. Gesture and user frames are activated when SPAK receives information from a network agent indicating a gesture and a face has been recognized. Behavior frames are activated when the predefined conditions are met. In this model each recognizable pose is treated as an instance-frame under the class-frame "Pose". All the known poses are defined as frames. If a predefined pose is classified by vision-based pose classification module, then corresponding pose-frame will be activated. The static gestures are defined using the combination of face and hand poses. These gesture frames has three slots for the gesture components.

Each robot behaviour, includes a command or series of commands for a particular task. Each robot behavior is mapped with user and gesturer (user-gesture-action) as we proposed person-centric interpretation of gesture. In this model same gesture can be used to activate different actions of a robot for different persons even the robot is same.

236 The Future of Humanoid Robots – Research and Applications

distributed architecture for knowledge-based interactive robots and through SPAK they have implemented dialogue-based human-robot ('Robovie') interaction for greeting scenarios. This system employs SPAK, a frame-based knowledge engine, connecting to a group of network software agents such as 'Face recognizers', 'Gesture recognizers', 'Voice recognizers', 'Robot Controller', etc. Using information received from these agents, and based on the predefined frame knowledge hierarchy, SPAK inference engine determines the actions to be taken and submit corresponding commands to the target robot control agents.

Fig. 1. Frame hierarchy for gesture-based human-robot interaction system (SG=Static

and hand poses. These gesture frames has three slots for the gesture components.

different actions of a robot for different persons even the robot is same.

Each robot behaviour, includes a command or series of commands for a particular task. Each robot behavior is mapped with user and gesturer (user-gesture-action) as we proposed person-centric interpretation of gesture. In this model same gesture can be used to activate

Figure 1 shows the frame hierarchy of the knowledge model for the gesture-based humanrobot interaction system, organized by the IS\_A relationship indicated by arrows connecting upper and lower frame boxes. Necessary frames are defined for the users, robots, poses, gestures and robot behaviors (actions). The user frame includes instances of all known users (instance "Hasan", "Cho", …); robot frame includes instances of all the robots ("Aibo", "Robovie",….) used in the system. The behavior frame can be sub-classed further into "AiboAct" and "RobovieAct", where "AiboAct" frame includes instances of all the predefined 'Aibo' actions and "RobovieAct" frame includes instances of all the predefined 'Robovie' actions. The gesture fame is sub-classed into "Static" and "Dynamic" for static and dynamic gestures respectively. Examples of static frame instances are, "TwoHand", "One", etc. Examples of dynamic frame instances are "YES", "NO", etc. The pose frame includes all recognizable poses such as "LEFTHAND", "RIGHTHAND", "FACE", "ONE", etc. Gesture and user frames are activated when SPAK receives information from a network agent indicating a gesture and a face has been recognized. Behavior frames are activated when the predefined conditions are met. In this model each recognizable pose is treated as an instance-frame under the class-frame "Pose". All the known poses are defined as frames. If a predefined pose is classified by vision-based pose classification module, then corresponding pose-frame will be activated. The static gestures are defined using the combination of face

Gesture, DG=Dynamic Gesture)

Fig. 2. Knowledge Editor showing the example gesture-based human-robot interaction

The frame system can be created in Graphical User Interface (GUI) and interpreted by the SPAK inference engine. Figure 2 shows the example knowledge Editor (part of the system) with the example of 'Robot', 'User', 'Gesture', 'Behaviour' (robot actions) and 'Pose' frames. The knowledge Editor Window displays the current frame hierarchy. Each frame is represented as click-able button. The buttons are connected with lines indicating IS\_A relationships among the frames. Clicking on the frame button brings up its slot editor. Figure 3 shows an example of slot editor for the robot (Robovie) action-frame "RaiseTwoArms". The attributes for a Slot are defined by slot name, type, value, condition, argument, required, shared, etc. Figure 4 shows an example instance-frame 'Hasan' of the class-frame 'User' defined in SPAK. Figure 5 shows an example instance-frame 'Robovie' of class-frame 'Robot' defined in SPAK.


Fig. 3. Example of a robot action-frame 'RaiseTwoArms' in SPAK

User, Gesture and Robot Behaviour Adaptation for Human-Robot Interaction 239

activated. Using the received gesture and user information, SPAK processes the facts and activates the corresponding robot action frames to carry out predefined robot actions, which may include body movement and speech. The robot behaviour is user dependent and it is mapped based on user and gesture relationship (user-gesturer-robot-action) in the knowledge base. Figure 3 shows an example of 'Robovie' robot action-frame for the action "Raise Two Arms". This frame only activated if the identified user is 'Hasan', recognized gesture is "TwoHand" and the selected robot is 'Robovie'. User can add or edit the necessary knowledge frames for the users, face and hand poses, gestures and robot behaviors using SPAK knowledge Editor. The new robot behaviour frame can be included in the knowledge base according to generalization of multiple occurrences of the same

Fig. 8. Example of instance-frame 'RIGHTHAND' of class-frame 'Pose' in SPAK

Fig. 9. Example of instance-frame 'TwoHand' of class-frame 'Gesture' in SPAK

This section describes new users, poses, gestures and robot behaviours adaptation methods for implementing human-robot interaction. Suppose, the robot fixed with a same room with same lighting condition, in that case only the user skin color dominates on the color-based face and hands segmentation method. It is essential for the system to cope with the different persons. The new user may not be included in the system during training phase, so the person should be included using on-line registration process. The user may want to perform new gestures that is ever been used by others person or himself. In that case the system should include the new poses with minimal user interaction. The system learns new users, new poses using multi-clustering approach with minimum user interaction. To adapt to new users and new hand poses the system must be able to perceive and extract relevant properties from the unknown faces and hand poses, find common patterns among them and formulate discrimination criteria consistent with the goals of the recognition process. This form of learning is known as clustering and it is the first steps in any recognition process where discriminating features of the objects are not know in advance [Patterson, 1990].

**4. Users, poses, gesture and robot behavior adaptation** 

Subsection 4.1 describes multi-clustering based learning method.

gesture with user consent (first time only).


Fig. 4. Example of instance-frame 'Hasan' of class-frame 'User' in SPAK


Fig. 5. Example of instance-frame 'Robovie' of class-frame 'Robot' in SPAK


Fig. 6. Example of instance-frame 'FACE' of class-frame 'Pose' in SPAK


Fig. 7. Example of instance-frame 'LEFTHAND' of class-frame 'Pose' in SPAK

Image analysis module classifies the hand and face poses and identifies the user. Image analysis module, sends user name (hasan, cho, etc.) and pose names (face, lefthand, righthand, etc.) to the SPAK knowledge module. According to pose name and user name corresponding pose frame and user frame will be activated. Figure 6, Figure 7, and Figure 8 shows the instance-frames 'FACE', 'LEFTHAND', 'RIGHTHNAD' of the class-frame 'Pose' respectively. If the required combination of the pose components is found then the corresponding gesture frame will be activated. Figure 9 shows the gesture frame 'TwoHand' in SPAK. It will be activated if pose fames 'FACE', 'LEFTHAND' and 'RIGHTHNAD' are 238 The Future of Humanoid Robots – Research and Applications

Fig. 4. Example of instance-frame 'Hasan' of class-frame 'User' in SPAK

Fig. 5. Example of instance-frame 'Robovie' of class-frame 'Robot' in SPAK

Fig. 6. Example of instance-frame 'FACE' of class-frame 'Pose' in SPAK

Fig. 7. Example of instance-frame 'LEFTHAND' of class-frame 'Pose' in SPAK

Image analysis module classifies the hand and face poses and identifies the user. Image analysis module, sends user name (hasan, cho, etc.) and pose names (face, lefthand, righthand, etc.) to the SPAK knowledge module. According to pose name and user name corresponding pose frame and user frame will be activated. Figure 6, Figure 7, and Figure 8 shows the instance-frames 'FACE', 'LEFTHAND', 'RIGHTHNAD' of the class-frame 'Pose' respectively. If the required combination of the pose components is found then the corresponding gesture frame will be activated. Figure 9 shows the gesture frame 'TwoHand' in SPAK. It will be activated if pose fames 'FACE', 'LEFTHAND' and 'RIGHTHNAD' are activated. Using the received gesture and user information, SPAK processes the facts and activates the corresponding robot action frames to carry out predefined robot actions, which may include body movement and speech. The robot behaviour is user dependent and it is mapped based on user and gesture relationship (user-gesturer-robot-action) in the knowledge base. Figure 3 shows an example of 'Robovie' robot action-frame for the action "Raise Two Arms". This frame only activated if the identified user is 'Hasan', recognized gesture is "TwoHand" and the selected robot is 'Robovie'. User can add or edit the necessary knowledge frames for the users, face and hand poses, gestures and robot behaviors using SPAK knowledge Editor. The new robot behaviour frame can be included in the knowledge base according to generalization of multiple occurrences of the same gesture with user consent (first time only).


Fig. 8. Example of instance-frame 'RIGHTHAND' of class-frame 'Pose' in SPAK


Fig. 9. Example of instance-frame 'TwoHand' of class-frame 'Gesture' in SPAK
