**1. Introduction**

24 Will-be-set-by-IN-TECH

188 The Future of Humanoid Robots – Research and Applications

A.Symeonidisa, I.Athanasiadisb and P.Mitkasa: A Retraining Methodology for Enhancing Agent Intelligence, Knowledge-Based Systems, vol.20, issue.4, pp. 388–396 (2007)

R.Brooks: Intelligence Without Representation, Artificial Intelligence, vol.47, pp. 139–159

L.Kaelbling: A Situated-Automata Approach to the Design of Embedded Agents, ACM

B.Yartsev, G.Korneev, A.Shalyto, V.Kotov: Automata-Based Programming of the Reactive

T.Kanda, H.Ishiguro, T.Ono, M.Imai, R.Nakatsu: Development and Evaluation of an

T.Kanda, M.Shiomi, Z.Miyashita, H.Ishiguro, and N.Hagita: An Affective Guide Robot in

F.Huang, J.Yang, A.Waibel: Dialogue Management for Multimodal User Registration, In Proc.

M.Denecke: Informational Characterization of Dialogue States, In Proc. Int'l Conf. on Spoken

Voice Extensible Markup Language (VoiceXML) Version 2.0:

N.Ando, T.Suehiro, K.Kitagaki, T.Kotoku, W.Yoon: RT-Middleware: Distributed Component

T. Kawahara, A. Lee, T. Kobayashi, K. Takeda, N. Minematsu, S. Sagayama, K. Itou, A. Ito, M.

I.Hara, F.Asano, H.Asoh, J.Ogata, N.Ichimura, Y.Kawai, F.Kanehiro, H.Hirukawa,

F.Asano, K.Yamamoto, I.Hara, J.Ogata, T.Yoshimura, Y.Motomura, N.Ichimura, H.Asoh:

Middleware for RT (Robot Technology), Proc. IEEE/RSJ International Conference on

Yamamoto, A. Yamada, T. Utsuro and K. Shikano: Free Software Toolkit for Japanese Large Vocabulary Continuous Speech Recognition, In Proc. Int'l Conf. on Spoken

K.Yamamoto: Robust Speech Interface Based on Audio and Video Information Fusion for Humanoid HRP-2, Proc. IEEE/RSJ International Conference on Intelligent

Detection and Separation of Speech Event Using Audio and Video Information Fusion and Its Application to Robust Speech Interface, EURASIP Journal on Applied

Int'l Conf. on Spoken Language Processing, Vol.3, pp. 37–40 (2000)

Multi-Agent Control Systems, Proc. IEEE International Conference on Integration of

Interactive Humanoid Robot Robovie, Proc. IEEE International Conference on

a Shopping Mall", Proc. ACM/IEEE International Conference on Human-Robot

T.Winograd: Understanding Natural Language, Academic Press (1972)

Knowledge Intensive Multiagent Systems, pp. 449–453 (2005)

SIGART Bulletin, vol.2 , issue.4, pp. 85–88 (1991)

Robotics and Automation, pp. 1848–1855 (2002)

Language Processing, Vol.2, pp. 114–117 (2000)

Intelligent Robots and Systems, pp. 3555–3560 (2005)

Signal Processing, vol.2004, issue.11, pp. 1727-1738 (2004)

Language Processing, Vol. 4, pp. 476–479, (2000)

Robots and Systems, pp. 2404–2410 (2004)

http://www.w3.org/TR/voicexml20/

Interaction, pp. 173–180 (2009)

(1991)

Robots are designed to assist human beings to complete tasks. As described in many research journals and scientific fictions, researchers and the general public expect that one day robots can complete certain tasks independently and autonomously in our world.

A typical implementation of Artificial Intelligence (AI) in robotics research is expected to incorporate Knowledge Representation, Automated Reasoning, Machine Learning, and Computer Vision [Russell and Norvig, 2010].

From the beginning of this century, the importance of cognitive sciences is highlighted by the researchers in both robotics and artificial intelligences[Russell and Norvig, 2010]. Real Cognitive Science is based on the testing and experiments on animals and humans to obtain the understanding and knowledge of the cognition, and AI research is based on designing and testing algorithms on a computer based artificial system. In the development, research in both Cognitive Sciences and AI benefit each other. Cognitive Science provides theoretical foundation and solutions to the problems in AI, and AI enhances the research in Cognitive Science and provides possible research directions for Cognitive Science.

Recently, an emerging field, named Cognitive Robotics, is developed for robots to generate human-like behaviors and intelligences, which integrates perception, action, learning, decision-making, and communication[Kawamura and Browne, 2009]. Cognitive Robotics largely incorporates the concepts in the research of cognitive sciences and tries to simulate the architecture of the information processing of human cognition. Currently, some long term requirements are proposed for Cognitive Robotics, which put concentration on the embodiment of cognitive concepts [Anderson, 2003; Sloman et al., 2006]. Increasing achievements in Cognitive Robotics began to grab the attention of the robotics research community. However, currently, it is still difficult to design truly cognition, especially human-like cognition, on a robotic platform due to the limitations in mechanism, computation, architecture, etc. [Brooks, 1991]. Therefore, on one side, researchers try to place robots in a human-existing dynamic environment to complete tasks[Brooks, 1991]; on the other side, researchers still do not obtain a general architecture to generate complex behaviors in such area for robots[Sloman, 2001]. In recent research, researchers try to place robots in robotic aid area, where robots can assist humans to complete tasks[Tan et al., 2005].

Implementation of a Framework for Imitation

Knight two steps forward and subsequently one step left.

Fig. 1. ISAC Robot (peneumatic driven and stationery).

The demonstration by a human teacher is shown in the left picture of Fig. 2. In the middle picture of Fig.1, if the Knight is moved not too far away from the place in the demonstration, ISAC can use current imitation methods to complete the task. But if the Knight is moved far away from the place in the demonstration, ISAC fails to grasp and to move this piece in an expected style as shown in the right picture of Fig.2. That is because current learning

methods.

Learning on a Humanoid Robot Using a Cognitive Architecture 191

concentration on reproducing new movements with dynamics similar to one single behavior in a behavior sequence[Ijspeert et al., 2002] or generating a behavior which has dynamics similar to the whole behavior sequence[Dillmann et al., 2000]. Therefore, the problem is if robots are required to generate new behaviors in a similar situation but the task constraints have been changed a lot, sometime robots may fail to complete this task using current

In a Chess-Moving-Task, a humanoid robot, named ISAC, is required to grasp and move the

An important feature of robotics research is to design human-like mechanism for robots [Brooks et al., 1999]. Humanoid robots, which are designed human-like, have been gradually selected as the platform for experimentally implementing the conceptual design of Cognitive Robotics. An increasing number of humanoid robots have been designed to fulfill the dream of the researchers (such as NASA Robonaut, Aldebaran's NAO and ISAC)[Tan and Kawamura, 2011]. The human-like mechanism provides a possible platform for researchers to design human-like behaviors [Tan and Liao, 2007b]and it is expected that one day such human-like artificial creatures can live in our real world. However, there is no defying the fact that the essential part of designing a truly human-like robot is to provide robots abilities and skills to generate human-like behaviors.

Even a robot with six degrees-of-freedom (DOF) is not easy to plan the motion in humanexisting environment [Wang et al., 2006; Yang et al., 2006]. Futher more, the mechanism of a humanoid robot is much more complicated than an industrial robot, because the degree-offreedom is much higher. Therefore, it is difficult for researchers to use traditional methods to plan the motions for humanoid robots. When humanoid robots are asked to carry out some human-like behaviors, they are much more difficult to be programmed or controlled because such behaviors are more complex. Additionally, as stated in former paragraphs, it is expected that humanoid robots should be placed in a dynamic and human-existing environment to complete tasks. Therefore, researchers began to find solutions from cognitive sciences.

Recently, imitation learning has been considered as a powerful tool for rapidly transferring skills between humans and robots [Uchiyama, 1978; Atkeson and Schaal, 1997; Schaal, 1999]. Human beings often try to imitate the behaviors of other people in the environment. This kind of ability is especially important for children. Concepts of imitation learning can be found from the investigation of the psychology research and biology research [Billard, 2001]. Therefore, it is reasonable to implement imitation learning for robots since we are struggling to develop cognitive skills for robots.

In a typical imitation learning, first, human teachers demonstrate a behavior sequence to complete certain task in a specified situation. Second, this behavior sequence is learned by recording the behavior either by using the encoders on the robot (if the demonstration is given by manually moving arms of robots) or by recording the trajectory in the task space (if the demonstration is given by showing the robot the task in the task space). Third, the robot will generate similar behavior in similar but different situations to complete tasks.

Imitation learning aims to train robot the behaviors and skills from the demonstrations and let the robots adapt them to similar situations, using which robots reproduce similar movements to complete similar tasks [Billard et al., 2007]. Imitation learning algorithms can be divided into two categories [Calinon et al., 2007]: one is trying to train robots to extract and learn the motion dynamics [Schaal and Atkeson, 2002; Ijspeert et al., 2003; Calinon and Billard, 2007; Tan and Kawamura, 2011], and the other is trying to let robots to learn higherlevel behaviors and action primitives [Dillmann et al., 2000; Bentivegna and Atkeson, 2001] by imitation. Both methods require a set of predefined basis behaviors to ensure convergence.

For most tasks, a human teacher shows robots several behaviors in a behavior sequence to complete a task. That means the behavior sequence is composed of several behaviors and each behavior has its own parameters. Recently, imitation learning methods mostly put 190 The Future of Humanoid Robots – Research and Applications

An important feature of robotics research is to design human-like mechanism for robots [Brooks et al., 1999]. Humanoid robots, which are designed human-like, have been gradually selected as the platform for experimentally implementing the conceptual design of Cognitive Robotics. An increasing number of humanoid robots have been designed to fulfill the dream of the researchers (such as NASA Robonaut, Aldebaran's NAO and ISAC)[Tan and Kawamura, 2011]. The human-like mechanism provides a possible platform for researchers to design human-like behaviors [Tan and Liao, 2007b]and it is expected that one day such human-like artificial creatures can live in our real world. However, there is no defying the fact that the essential part of designing a truly human-like robot is to provide

Even a robot with six degrees-of-freedom (DOF) is not easy to plan the motion in humanexisting environment [Wang et al., 2006; Yang et al., 2006]. Futher more, the mechanism of a humanoid robot is much more complicated than an industrial robot, because the degree-offreedom is much higher. Therefore, it is difficult for researchers to use traditional methods to plan the motions for humanoid robots. When humanoid robots are asked to carry out some human-like behaviors, they are much more difficult to be programmed or controlled because such behaviors are more complex. Additionally, as stated in former paragraphs, it is expected that humanoid robots should be placed in a dynamic and human-existing environment to complete tasks. Therefore, researchers began to find solutions from

Recently, imitation learning has been considered as a powerful tool for rapidly transferring skills between humans and robots [Uchiyama, 1978; Atkeson and Schaal, 1997; Schaal, 1999]. Human beings often try to imitate the behaviors of other people in the environment. This kind of ability is especially important for children. Concepts of imitation learning can be found from the investigation of the psychology research and biology research [Billard, 2001]. Therefore, it is reasonable to implement imitation learning for robots since we are struggling

In a typical imitation learning, first, human teachers demonstrate a behavior sequence to complete certain task in a specified situation. Second, this behavior sequence is learned by recording the behavior either by using the encoders on the robot (if the demonstration is given by manually moving arms of robots) or by recording the trajectory in the task space (if the demonstration is given by showing the robot the task in the task space). Third, the robot

Imitation learning aims to train robot the behaviors and skills from the demonstrations and let the robots adapt them to similar situations, using which robots reproduce similar movements to complete similar tasks [Billard et al., 2007]. Imitation learning algorithms can be divided into two categories [Calinon et al., 2007]: one is trying to train robots to extract and learn the motion dynamics [Schaal and Atkeson, 2002; Ijspeert et al., 2003; Calinon and Billard, 2007; Tan and Kawamura, 2011], and the other is trying to let robots to learn higherlevel behaviors and action primitives [Dillmann et al., 2000; Bentivegna and Atkeson, 2001] by imitation. Both methods require a set of predefined basis behaviors to ensure

For most tasks, a human teacher shows robots several behaviors in a behavior sequence to complete a task. That means the behavior sequence is composed of several behaviors and each behavior has its own parameters. Recently, imitation learning methods mostly put

will generate similar behavior in similar but different situations to complete tasks.

robots abilities and skills to generate human-like behaviors.

cognitive sciences.

convergence.

to develop cognitive skills for robots.

concentration on reproducing new movements with dynamics similar to one single behavior in a behavior sequence[Ijspeert et al., 2002] or generating a behavior which has dynamics similar to the whole behavior sequence[Dillmann et al., 2000]. Therefore, the problem is if robots are required to generate new behaviors in a similar situation but the task constraints have been changed a lot, sometime robots may fail to complete this task using current methods.

In a Chess-Moving-Task, a humanoid robot, named ISAC, is required to grasp and move the Knight two steps forward and subsequently one step left.

Fig. 1. ISAC Robot (peneumatic driven and stationery).

The demonstration by a human teacher is shown in the left picture of Fig. 2. In the middle picture of Fig.1, if the Knight is moved not too far away from the place in the demonstration, ISAC can use current imitation methods to complete the task. But if the Knight is moved far away from the place in the demonstration, ISAC fails to grasp and to move this piece in an expected style as shown in the right picture of Fig.2. That is because current learning

Implementation of a Framework for Imitation

Liang, 2011].

work in this paper.

**2. System design** 

Learning on a Humanoid Robot Using a Cognitive Architecture 193

general imitation learning method to generate behaviors in a large number of different situations. Researchers starts to find the solution from the cognitive sciences, because the research of cognitive sciences investigate the stable learning processes in human or animal brains and possibly it can provide solution to current behavior based robotics research,

Recently, the cognitive architecture received broad attention from the robotics research community, because it provides a kind of methods of using cognitive processes[Tan and

Current cognitive architectures can be divided into four categories: Symbolic, Connectionist, Reactive (Behavior Based Connectionist), and Hybrid. Some well-known Symbolic architectures include: ACT-R [Anderson et al., 1997], SOAR [Laird et al., 1987], EPIC [Keiras and Meyer, 1997], Chrest[Gobet et al., 2001], and Clarion [Sun, 2003], in which symbolic knowledge are stored for automated reasoning. For Connectionist, BICS[Haikonen, 2007], Darwinism[Krichmar and Reeke, 2005], and CAP2[Schneider, 1999] are developed, in which connectionism method are implemented to generate behaviors or decisions. A typical Reactive architecture is Subsumption[Brooks, 1986; Brooks, 1991], which directly couples the sensory information and the behavior primitives. Some researchers have begun to recognize the need of both deliberative interaction and reactive interaction for cognitive robots, which motivates the research on hybrid architectures[Kawamura et al., 2004]. Such integration offers the promise of robots which are both fluent in routine operations and capable of adjusting their behavior in the face of unexpected situations or demands. Typical Hybrid architectures include: RCS [Albus and Barbera, 2005], and JACK[Winikoff, 2005]. In our lab, we developed a hybrid cognitive architecture, named ISAC Cognitive Architecture [Kawamura et al., 2004]. In this paper, we propose to use ISAC cognitive architecture to

The rest of the papers are organized as follows: Section II introduces the design of the proposed system, Section III describes the experimental setup, procedure and the results, Section IV discuss the proposed system and the future work, and Section V summarizes the

For ISAC, demonstrations are segmented into behaviors in sequences. The segmented behaviors are recognized based on the pre-defined behavior categorizes. The recognized behaviors are modeled and stored in a behavior sequence. When new task constraints are given to the robot, ISAC generates same behavior sequences with new parameters on each behavior, the dynamics of which are similar to the behaviors in demonstration. These behaviors are assembled into a behavior sequence and sent to the low-level robotic control

Fig.1 is the system diagram of ISAC Cognitive Architecture, which is a multi-agents hybrid architecture. This cognitive architecture provides three control loops for cognitive control of robots: *Reactive*, *Routine* and *Deliberative.* Behaviors can be generated through this cognitive architecture. Imitation learning basically should be involved in the Deliberative control loop. Three memory components are implemented in this architecture, including: Working Memory System (WMS), Short Term Sensory Memory (STM), Long Term Memory (LTM).

system to move the arm and control the end-effectors to complete a task.

especially for the research on humanoid robots[Tan and Liang, 2011].

implement the imitation learning for a humanoid robot.

methods can guarantee the convergence of achieving the global goal, but the local goals have not been taken into consideration.

Therefore, in a biologically inspired way, robots should understand the task as a behavior sequence. The reason for choosing the segmentation of a behavior sequence is: first, behavior based cognitive control method provides a robust method for manipulating the object and complete a task through learning, because behavior based methods can train robots to understand the situation and the task related information; second, the segmentation provides a more robust approach for robots to handle completed tasks. As in Fig.2, if ISAC knows that this task is composed of several parts, the grasping sub-task can be guaranteed by adapting the first reaching behavior. However, robots do not have knowledge on how to segment the behavior sequence in a reasonable way. Former research methods segment behaviors based on its dynamics. However, they are not applicable to some complex tasks. For complex tasks, a global goal is achieved by achieving several local goals.

Fig. 2. Chess-moving task

Fuzzy methods[Dillmann et al., 1995] and the Hidden Markov Model (HMM) [Yang et al., 1997] are normally used for segmentation. Another type of segmentation methods is to detect change-points on the trajectory[Konidaris et al., 2010]. Kulic and Nakamura [Kulic et al., 2008]proposed a segmentation method based on the optical flow in the environment, which is a cluster based method. These methods are not robust, because they rely on the analysis of the dynamics of the demonstration and pre-defined behavior primitives, which are not suitable for extension. Readers can simply image a new situation where the pre-defined behavior primitives cannot be used, and a noises existing environment, where the sensory information of the demonstration can be largely affected.

In this paper, we propose a cognitive segmentation method.

Imitation learning provides a possible solution for behavior generation for humanoid robots. However, researchers gradually found that it is possible to design algorithms and architectures for a specific task in a specific situation, but it seems difficult to design a 192 The Future of Humanoid Robots – Research and Applications

methods can guarantee the convergence of achieving the global goal, but the local goals

Therefore, in a biologically inspired way, robots should understand the task as a behavior sequence. The reason for choosing the segmentation of a behavior sequence is: first, behavior based cognitive control method provides a robust method for manipulating the object and complete a task through learning, because behavior based methods can train robots to understand the situation and the task related information; second, the segmentation provides a more robust approach for robots to handle completed tasks. As in Fig.2, if ISAC knows that this task is composed of several parts, the grasping sub-task can be guaranteed by adapting the first reaching behavior. However, robots do not have knowledge on how to segment the behavior sequence in a reasonable way. Former research methods segment behaviors based on its dynamics. However, they are not applicable to some complex tasks. For complex tasks, a global goal is achieved by achieving several local

Fuzzy methods[Dillmann et al., 1995] and the Hidden Markov Model (HMM) [Yang et al., 1997] are normally used for segmentation. Another type of segmentation methods is to detect change-points on the trajectory[Konidaris et al., 2010]. Kulic and Nakamura [Kulic et al., 2008]proposed a segmentation method based on the optical flow in the environment, which is a cluster based method. These methods are not robust, because they rely on the analysis of the dynamics of the demonstration and pre-defined behavior primitives, which are not suitable for extension. Readers can simply image a new situation where the pre-defined behavior primitives cannot be used, and a noises existing environment, where the sensory information of the demonstration can be

Imitation learning provides a possible solution for behavior generation for humanoid robots. However, researchers gradually found that it is possible to design algorithms and architectures for a specific task in a specific situation, but it seems difficult to design a

have not been taken into consideration.

goals.

Fig. 2. Chess-moving task

largely affected.

In this paper, we propose a cognitive segmentation method.

general imitation learning method to generate behaviors in a large number of different situations. Researchers starts to find the solution from the cognitive sciences, because the research of cognitive sciences investigate the stable learning processes in human or animal brains and possibly it can provide solution to current behavior based robotics research, especially for the research on humanoid robots[Tan and Liang, 2011].

Recently, the cognitive architecture received broad attention from the robotics research community, because it provides a kind of methods of using cognitive processes[Tan and Liang, 2011].

Current cognitive architectures can be divided into four categories: Symbolic, Connectionist, Reactive (Behavior Based Connectionist), and Hybrid. Some well-known Symbolic architectures include: ACT-R [Anderson et al., 1997], SOAR [Laird et al., 1987], EPIC [Keiras and Meyer, 1997], Chrest[Gobet et al., 2001], and Clarion [Sun, 2003], in which symbolic knowledge are stored for automated reasoning. For Connectionist, BICS[Haikonen, 2007], Darwinism[Krichmar and Reeke, 2005], and CAP2[Schneider, 1999] are developed, in which connectionism method are implemented to generate behaviors or decisions. A typical Reactive architecture is Subsumption[Brooks, 1986; Brooks, 1991], which directly couples the sensory information and the behavior primitives. Some researchers have begun to recognize the need of both deliberative interaction and reactive interaction for cognitive robots, which motivates the research on hybrid architectures[Kawamura et al., 2004]. Such integration offers the promise of robots which are both fluent in routine operations and capable of adjusting their behavior in the face of unexpected situations or demands. Typical Hybrid architectures include: RCS [Albus and Barbera, 2005], and JACK[Winikoff, 2005]. In our lab, we developed a hybrid cognitive architecture, named ISAC Cognitive Architecture [Kawamura et al., 2004]. In this paper, we propose to use ISAC cognitive architecture to implement the imitation learning for a humanoid robot.

The rest of the papers are organized as follows: Section II introduces the design of the proposed system, Section III describes the experimental setup, procedure and the results, Section IV discuss the proposed system and the future work, and Section V summarizes the work in this paper.
