**9. References**


http://commtechlab.msu.edu/sites/aslweb/browser.htm, visited on April 2004.


User, Gesture and Robot Behaviour Adaptation for Human-Robot Interaction 253

[Hasanuzzaman, 2007b ] Md. Hasanuzzaman, S. M. Tareeq, Tao Zhang, V. Ampornaramveth,

[Hasanuzzaman, 2006] Md. Hasanuzzaman, T. Zhang, V. Ampornaramveth, and H. Ueno,

[Hasanuzzaman, 2004b] M. Hasanuzzaman, T. Zhang, V. Ampornaramveth, M.A. Bhuiyan, Y.

[Hasanuzzaman, 2004c] M. Hasanuzzaman, T. Zhang, V. Ampornaramveth, P. Kiatisevi, Y.

[Kanda, 2002] T. Kanda, H. Ishiguro, T. Ono, M. Imai, and R. Nakatsu, "Development and

[Kanade, 1977] T. Kanade, "Computer Recognition of Human Faces", Brikhauser Verlag, Basel

[Kiatisevi, 2004] P. Kiatisevi, V. Ampornaramveth and H. Ueno, "A Ditributed Architecture for

on Information Technology for Application (ICITA' 2004), pp. 256-261, 2004. [King, 1990] S. King and C. Weirman, "Helpmate Autonomous Mobile Robot Navigation System", in Proceeding of SPIE Conference on Mobile Robots, pp. 190-198,1990. [Kjeldsen, 1996] R. Kjeldsen, and K. Kender, "Finding Skin in Color Images", in Proceedings of

[Kortenkamp, 1996] D. Kortenkamp, E. Hubber, and P. Bonasso, "Recognizing and

[Kotropoulos, 1997] C. Kotropoulos and I. Pitas, "Rule-based Face Detection in Frontal Views",

[Lin, 2002] J. Lin, Y. Wu, and T. S Huang, "Capturing Human Hand Motion in Image

Biomimetics (ROBIO'2004), China, pp. 379-384, 2004.

and Cybernetics (IEEE SMC'2004), Netherland, 2004.

and Stuttgart, ISR-47, pp. 1-106, 1977.

(AFGR'96), pp. 312-317, 1996.

Florida, December, 2002.

Processing, Vol. 4, pp. 2537-2540, 1997.

Verlag Berlin Heidelberg), Vol. 1, pp. 5300-537, Portugal, 2004.

20(1), pp. 23-34, 2007.

2002.

1996.

H. Gotoda, Y. Shirai, H. Ueno, "Adaptive Visual Gesture Recognition for Human-Robot Interaction", Malaysian Journal of Computer Science, ISSN-0127-9084, Vol.

"Gesture-Based Human-Robot Interaction Using a Knowledge-Based Software Platform", International Journal of Industrial Robot, Vol. 33 (1), pp. 37-49, 2006. [Hasanuzzaman, 2004a] M. Hasanuzzaman, V. Ampornaramveth, T. Zhang, M. A. Bhuiyan, Y.

Shirai, H. Ueno, "Real-time Vision-based Gesture Recognition for Human-Robot Interaction", in Proceeding of IEEE International Conference on Robotics and

Shirai, H. Ueno, "Gesture Recognition for Human-Robot Interaction Through a Knowledge Based Software Platform", in Proceeding of IEEE International Conference on Image Analysis and Recognition (ICIAR 2004), LNCS 3211 (Springer-

Shirai, H. Ueno, "Gesture-based Human-Robot Interaction Using a Frame-based Software Platform", in Proceeding of IEEE International Conference on Systems Man

Evaluation of an Interactive Humanoid Robot: Robovie", in Proceeding of IEEE International Conference on Robotics and Automation (ICRA 2002), pp. 1848-1855,

Knowledge-Based Interactive Robots", in Proceeding of 2nd International Conference

2nd International Conference on Automatic Face and Gesture Recognition

Interpreting Gestures on a Mobile robot", in Proceeding of AAAI'96, pp. 915-921,

in Proceeding of International Conference on Acoustics, Speech and Signal

Sequences", in Proceeding of Workshop on Motion and Video Computing, Orlando,


252 The Future of Humanoid Robots – Research and Applications

[Bartlett, 1998] M. S Bartlett, H. M. Lades, and, T. Sejnowski, "Independent Component

[Belhumeur, 1997] P.N. Belhumeur, J. P. Hespanha, and D. J. Kriegman, " Eigenfaces vs.

Pattern Analysis and Machine Intelligence (PAMI), Vol. 19, pp. 711-720, 1997. [Bhuiyan, 2004] M. A. Bhuiyan, V. Ampornaramveth, S. Muto, and H. Ueno, "On Tracking of

[Bhuiyan, 2003] M. A. Bhuiyan, V. Ampornaramveth, S. Muto, H. Ueno, "Face Detection and

[Birk, 1997] H. Birk, T. B. Moeslund, and C. B. Madsen, "Real-time Recognition of Hand

[Chellappa, 1995] R. Chellappa, C. L. Wilson, and S. Sirohey, "Human and Machine

[Crowley, 1997] J. L. Crowley and F. Berard, "Multi Modal Tracking of Faces for Video

[Cutler, 1998] R. Cutler, M. Turk, "View-based Interpretation of Real-time Optical Flow for

[Darrel, 1993] T. Darrel and A. Pentland, "Space-time Gestures", in Proceedings of IEEE

[Dai, 1996] Y. Dai and Y. Nakano, "Face-Texture Model Based on SGLD and Its Application in

[Endres, 1998] H. Endres, W. Feiten, and G. Lawitzky, "Field Test of a Navigation System:

[Festival, 1999] "The Festival Speech Synthesis System developed by CSTR", University of

[Fong, 2003] T. Fong, I. Nourbakhsh and K. Dautenhahn, "A Survey of Socially Interactive Robots", Robotics and Autonomous System, Vol. 42(3-4), pp.143-166, 2003. [Freeman, 1996] W.T. Freeman, K. Tanaka, J. Ohta, and K. Kyuma, "Computer Vision for

[Hasanuzzaman, 2007a ] Md. Hasanuzzaman, T. Zhang, V. Ampornaramveth, H. Gotoda, Y.

Autonomous Systems (RAS), Elsevier, Vol. 55(8), pp. 643-657, 2007.

Imaging (SPEI): Science and Technology, pp. 528-539, 1998.

Scandinavian Conference on Image Analysis, Finland, 1997.

Face and Gesture Recognition (AFGR'98), pp. 416-421, 1998.

Conference on Robotics & Automation (ICRA '98), 1998.

Edinburgh, http://www.cstr.ed.ac.uk/project/festival

and Gesture Recognition (AFGR'96), pp. 100-105, 1996.

Pattern Recognition (CVPR'97), pp. 640-645, 1997.

Vol. 19, No. 1, pp. 42-54, 2004.

pp. 25-39, 2003.

1995.

335-340, 1993.

1996.

Representation for Face Recognition" in Proceedings of Symposium on Electronic

Fisherfaces: Recognition Using Class Specific Linear Projection" IEEE Transaction on

Eye For Human-Robot Interface", International Journal of Robotics and Automation,

Facial Feature Localization for Human-machine Interface", NII Journal, Vol.5, No. 1,

Alphabet Gesture Using Principal Component Analysis", in Proceeding of 10th

Recognition of faces: A survey", in Proceeding of IEEE, Vol. 83, No. 5, pp. 705-740,

Communications", In proceedings of IEEE Conference on Computer Vision and

Gesture Recognition", in Proceedings of 3rd International Conference on Automatic

International Conference on Computer Vision and Pattern recognition (CVPR'93), pp.

Face Detection in a Color Scene", Pattern Recognition, Vol. 29, No. 6, pp.1007-1017,

Autonomous Cleaning in Supermarkets", in Proceeding of IEEE International

Computer Games", in Proceedings of International Conference on Automatic Face

Shirai, H. Ueno, "Adaptive Visual Gesture Recognition for Human-Robot Interaction Using Knowledge-based Software Platform", International Journal of Robotics and


User, Gesture and Robot Behaviour Adaptation for Human-Robot Interaction 255

[Siegwart, 2003] R. Siegwart et. al., "Robox at Expo.02: A large-scale Installation of Personal Robots", Robotics and Autonomous Systems, Vol. 42, pp. 203-222, 2003. [Sirohey, 1993] S. A. Sirohey, "Human Face Segmentation and Identification", *Technical* 

[Sturman, 1994] D.J. Sturman and D. Zetler, "A Survey of Glove-Based Input" IEEE Computer

[Tsukamoto, 1994] A. Tsukamoto, C.W. Lee, and S. Tsuji, "Detection and Pose Estimation of

[Torras, 1995] C. Torras, "Robot Adaptivity", Robotics and Automation Systems, Vol. 15,

[Torrance, 1994] M. C. Torrance, "Natural Communication with Robots" Master's thesis, MIT,

[Triesch, 2002] J. Triesch and C. V. Malsburg, "Classification of Hand Postures Against

[Turk, 1991] M. Turk and A. Pentland, "Eigenface for Recognition" Journal of Cognitive

[Ueno, 2002] H. Ueno, "A Knowledge-Based Information Modeling for Autonomous

[Ueno, 2000] H. Ueno, "A Cognitive Science-Based Knowledge Modeling for Autonomous

[Waldherr, 2000] S. Waldherr, R. Romero, S. Thrun, "A Gesture Based Interface for Human-

[Weimer, 1989] D. Weimer and S. K. Ganapathy, "A Synthetic Virtual Environment with Hand

[Wiskott, 1997] L. Wiskott, J. M. Fellous, N. Kruger, and C. V. Malsburg, "Face Recognition by

[Yang, 2002] M. H. Yang, D. J. Kriegman and N. Ahuja, "Detection Faces in Images: A survey",

[Yang, 2000] M. H. Yang, "Hand Gesture Recognition and Face Detection in Images", Ph.D

[Yang, 1998] J. Yang, R. Stiefelhagen, U. Meier and A. Waibel, "Visual Tracking for Multimodal

Human Face with Synthesized Image Models," in Proceeding of International

Department of Electrical Engineering and Computer Science, Cambridge, MA,

Complex Backgrounds Using Elastic Graph Matching", Image and Vision

Humanoid Service Robot", IEICE Transactions on Information & Systems, Vol. E85-

Humanoid Service Robot-Towards a Human-Robot Symbiosis", in Proceeding of 10th European-Japanese Conference on Modeling and Knowledge Base, pp. 82-95,

Robot Interaction", Journal of Autonomous Robots, Kluwer Academic Publishers, pp.

Gesturing and Voice Input", in Proceedings of ACM CHI'89 Human Factors in

Elastic Bunch Graph Matching", IEEE Transactions on Pattern Analysis and Machine

IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 24, No.

Human Computer Interaction", in Proceedings of ACM CHI'98 Human Factors in

*Report* CS-TR-3176,University of Maryland, pp. 1-33, 1993.

Graphics and Applications, Vol. 14, pp-30-39, 1994.

Conference of Pattern Recognition, pp. 754-757,1994.

Computing, Vol. 20, pp. 937–943, 2002.

Computing Systems, pp. 235-240, 1989.

Computing Systems, pp. 140-147, 1998.

Intelligence (PAMI), Vol. 19, No.7, pp. 775-779, 1997.

Thesis, University of Illinois, Urbana-Champaign, 2000.

D, No. 4, pp. 657-665, 2002.

Neuroscience, Vol. 3, No.1, pp. 71-86, 1991.

pp.11-23, 1995.

January 1994.

2000.

151-173, 2000.

1, pp. 34-58, 2002.


254 The Future of Humanoid Robots – Research and Applications

[Miao, 1999] J. Miao, B. Yin, K. Wang, L. Shen, and X. Chen, "A Hierarchical Multiscale and

[Moghaddam, 1995] B. Moghaddam and A. Pentland, "Probabilistic Visual Learning for Object

[Nam, 1996] Y. Nam and K. Y. Wohn, "Recognition of Space-Time Hand-Gestures Using

[Oka, 2002] K. Oka, Y. Sato, and H. Koike, "Real-Time Tracking of Multiple Finger-trips and

[Patterson, 1990] D. W. Patterson, "Introduction to Artificial Intelligence and Expert Systems",

[Pavlovic, 1997] V. I. Pavlovic, R. Sharma and T. S. Huang, "Visual Interpretation of Hand

Analysis and Machine Intelligence (PAMI), Vol. 19, No. 7, pp. 677-695, 1997. [Perzanowski, 2001] D. Perzanowski, A. C. Schultz, W. Adams, A. Marsh, and M. Bugajska,

[Pineau, 2003] J. Pineau, M. Montemerlo, M. Pollack, N. Roy, S. Thrun, "Towards Robotic

[Rehg, 1994] J. M. Rehg and T. Kanade, "Digiteyes: Vision-based Hand Tracking for Human-

[Rowley, 1998] H. A. Rowley, S. Baluja and T. Kanade, "Neural Network-Based Face

[Saber, 1998] E. Saber and A. M. Tekalp, "Frontal-view Face Detection and Facial Feature

[Sakai, 1996] T. Sakai. M. Nagao and S. Fujibayashi, "Line Extraction and Pattern Detection in a

[Severinson-Eklundh, 2003] K. Severinson-Eklundh, A. Green, H. Huttenrauch, "Social and

[Shimada, 1996] N. Shimada, and Y. Shirai, "3-D Hand Pose Estimation and Shape Model

Memo 306, 1974.

786-793, 1995.

Software and Technology, pp. 51-58, 1996.

423-428, Washington D.C, USA, 2002.

Systems, Vol. 42, pp. 271-281, 2003.

Articulated Bodies, pp. 16-94, 1994.

Recognition Letters, Vol. 17(8) pp.669-680, 1998.

Autonomous Systems, Vol. 42, pp.223-234, 2003.

Photograph", Pattern Recognition, Vol. 1, pp.233-248, 1996.

Vol. 23, No. 1, pp. 23-38, 1998.

GIFU, pp.23-428, 1996.

16(1), pp. 16-21, 2001.

Prentice-Hall Inc., Englewood Cliffs, N.J, USA, 1990.

Multiangle System for Human Face Detection in a Complex Background Using Gravity-Centre Template", Pattern Recognition, Vol. 32, No. 7, pp. 1237-1248, 1999. [Minsky, 1974] M. Minsky "A Framework for Representing Knowledge", MIT-AI Laboratory

Detection", in Proceeding of 5th International Conference on Computer Vision, pp.

Hidden Markov Model", in Proceedings of ACM Symposium on Virtual Reality

Gesture Recognition for Augmented Desk Interface Systems", in Proceeding of International Conference in Automatic Face and Gesture Recognition (AFGR'02), pp.

Gestures for Human-Computer Interaction: A Review", IEEE Transactions on Pattern

"Building a Multimodal Human-Robot Interface", IEEE Intelligent Systems, Vol.

Assistants in Nursing Homes: Challenges and Results", Robotics and Autonomous

Computer Interaction", in Proceeding of Workshop on Motion of Non-Rigid and

Detection" IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI),

Extraction Using Color, Shape and Symmetry Based Cost Functions", Pattern

Collaborative Aspects of Interaction with a Service Robot", Robotics and

Refinement from a Monocular Image Sequence", in Proceedings of VSMM'96 in


**0**

**13**

*Japan*

**Service Robots**

Komei Sugiura2 and Naoto Iwahashi2 <sup>1</sup>*The University of Electro-Communications*

**Learning Novel Objects for Domestic**

<sup>2</sup>*National Institute of Information and Communications Technology*

Muhammad Attamimi1, Tomoaki Nakamura1, Takayuki Nagai1,

It is fair to say that robots which can interact with and serve humans especially in the domestic environment will spread widely in near future. A fundamental task called mobile manipulation is required for such domestic service robots. Therefore, many humanoid robots have been developed with the ability of mobile manipulation (1–5). Recently, competitions such as RoboCup@Home (6), Mobile Manipulation Challenge (7), and Semantic Robot Vision

Since the tasks are implemented on domestic service robots, it stands to reason that natural interaction such as speech instruction should be used for the mobile manipulation. Here, we focus on the mobile manipulation using natural speech instruction such as "Bring me X" (X is an out-of-vocabulary (OOV) word). In order to realize this task, the integration of navigation,

Image and speech recognition are difficult especially when novel objects are involved in the system. For example, there are objects specific to each home and new products can be brought into the home. It is impossible to register the names and images of all these objects with the robot in advance. Hence, we propose a method for learning novel objects with a simple

The robot, on which the proposed learning method is implemented, is intended to be used in a private domestic environment. Therefore, the procedure of teaching objects to the robot must be simple. For example, the user says, "This object is X" (X is the name of the object) and shows the object to the robot (Fig.1: Left). It is easy for a user to teach a robot many objects with this procedure. Then the user orders the robot to bring him/her something. For example, the user says, "Bring me X" (Fig.1: Right). As we mentioned earlier, such extended manipulation tasks are necessary for domestic service robots. However, there are three problems in teaching novel objects to the robots. The first problem is speech recognition of an object's name. In usual methods, phonemes of names must be registered in an internal dictionary. However, it is impossible to register all objects in advance. The second problem is the speech synthesis. A robot must utter the name of the recognized object for interaction with humans such as "Is it X?" However, conventional robot utterance systems cannot utter a word which is not registered in the dictionary. Even if the phoneme sequence of an OOV word can be recognized,

Challenge (8), have been proposed to evaluate such robots.

manipulation, speech recognition, and image recognition is required.

**1. Introduction**

procedure.

