**5. Agent tutor**

In our MITS, an agent tutor "Alice" can adjust her behavior in response to learner's requests and inferred learner's needs. The agent is "eye-aware" and "affect-aware", and provides consistent empathy using facial expression and synthetic emotional speech. Its emotional response depends on the learner's action. For instance, an agent shows a happy emotion if the learner concentrates on the current study topic. In contrast, if the learner seems to lose

Multimodal Intelligent Tutoring Systems 95

vector *EX* can be obtained via fuzzy mapping, as seen in 1 2 (, ,) *E E R ex ex ex X S <sup>n</sup>* ,where *<sup>i</sup> ex* is the membership of the mode of facial expression *<sup>i</sup> y* to the fuzzy facial expression *EX*

means the compositional operation of the fuzzy relations. Once the fuzzy facial expression *EX*

is determined, its intensity will also be computed. The intensity of selected emotion *<sup>i</sup> x* is fuzzified to the linguistic value, which is then mapped to the linguistic value of related facial expressions according to fuzzy control rule. The intensity of facial expression *<sup>i</sup> y* is obtained by defuzzifying its linguistic value. The emotion intensity and facial expression intensity also have fuzzy characteristics. The fuzzy linguistic values of emotion and facial expression are listed as very low, low, middle, high and very high. According to the emotion-expression intensity mapping, the mapping from linguistic value of emotion intensity to linguistic value of facial expression intensity was realized through fuzzy control. An example of fuzzy control rule is shown in Table 5. Emotion *x* (surprise) can be fuzzily expressed by facial expression *y*1 or *y*2. The very low intensity of *x* can be expressed by small intensity of *y*1 or very small intensity of *y*2.

> Emotion *x*(surprise) facial expression *y*<sup>1</sup> facial expression *y*<sup>2</sup> Very low small Very small low middle small middle large middle high Very large large Very high —— Very large

The facial expression generation model is the module that accepts input of the fuzzy facial

adopted Xface, an MPEG-4 based open source toolkit for 3D facial animation, to generate multiple facial expressions of emotions mentioned above. Figure 7 are some keyframes of

with its intensity and output the agent's facial expression. In this paper, we

Table 5. Fuzzy control rule of fuzzy emotion-expression intensity mapping.

**5.1.2 Facial expression generation model** 

Fig. 7. Some keyframes of facial expressions.

expression *EX*

facial expressions.

,

concentration, the agent will show mild anger or alert the learner. The agent also shows empathy when the learner is sad. In general, the agent tutor interacts between the educational content and the learner. Other tasks of an agent tutor include explaining the study material and providing hints when necessary, moving around the screen to get or direct user attention, and to highlight information. The detailed tutoring strategies will be given latter. In this section, we focus on the facial expression generation and emotional speech synthesis of the agent. The famous agent "Alice" is employed as the tutor. Other agent systems can be used with appropriate diver programs.

#### **5.1 Facial expression generation**

Facial expression plays an important role in human's daily life, as indicated by Mehrabian, in face-to-face human communication 55% of the communicative message is transferred by facial expressions (Mehrabian, 1968). However, the limit in the existence researches is that facial expression generation is mostly monotone, or in the "Invariable View". They usually correlate one model of facial expression to one emotion, and generate facial animation based on that. Whereas, human tend to act more complicated to express one emotion. For example, human display kinds of facial expressions to express happiness, such as smile with mouth open or closed, symmetrically or asymmetrically, even with head wobbled. In this paper, we aim at generating humanoid and expressive facial expressions of agent to achieve natural, harmonious and believable student-agent interaction. Based on the cues of sources and characteristics of facial expression, we propose a novel model of fuzzy facial expression generation, as seen in figure 6.

Fig. 6. Work flow of the textual affect sensing.

#### **5.1.1 Fuzzy emotion-expression mapping**

Fuzzy is one common characteristic of emotion and facial expression. There is also fuzzy relationship between emotion and facial expression. One emotion can be fuzzily expressed by multiple modes of facial expression. Here, we give the model of fuzzy emotionexpression mapping. The mapping of emotion to expression is one-to-many.

Based on the correlation of multiple facial expressions of emotion, fuzzy emotion-expression mapping is proposed, in which emotion and facial expression are supposed to be fuzzy vectors, and a fuzzy matrix consisting of degrees of membership maps the fuzzy motion vector to the fuzzy facial expression vector. Define the emotion space as 1 2 *X xx x* {, } *<sup>m</sup>* , where *<sup>i</sup> x* is any emotion, such as surprise, disgust. Define the facial expression space as 1 2 {, } *Y yy y <sup>n</sup>* , where *<sup>i</sup> <sup>y</sup>* indicates any mode of facial expression. The fuzzy relation *R* from the emotion space X to the facial expression space Y is ( ) *R r ij m n* where ( , ) [0,1] *ij i j r Rx y* indicates the correlation degree of ( , ) *<sup>i</sup> <sup>j</sup> <sup>x</sup> <sup>y</sup>* to *R* . Given the input emotional fuzzy vector *ES* , the fuzzy facial

concentration, the agent will show mild anger or alert the learner. The agent also shows empathy when the learner is sad. In general, the agent tutor interacts between the educational content and the learner. Other tasks of an agent tutor include explaining the study material and providing hints when necessary, moving around the screen to get or direct user attention, and to highlight information. The detailed tutoring strategies will be given latter. In this section, we focus on the facial expression generation and emotional speech synthesis of the agent. The famous agent "Alice" is employed as the tutor. Other

Facial expression plays an important role in human's daily life, as indicated by Mehrabian, in face-to-face human communication 55% of the communicative message is transferred by facial expressions (Mehrabian, 1968). However, the limit in the existence researches is that facial expression generation is mostly monotone, or in the "Invariable View". They usually correlate one model of facial expression to one emotion, and generate facial animation based on that. Whereas, human tend to act more complicated to express one emotion. For example, human display kinds of facial expressions to express happiness, such as smile with mouth open or closed, symmetrically or asymmetrically, even with head wobbled. In this paper, we aim at generating humanoid and expressive facial expressions of agent to achieve natural, harmonious and believable student-agent interaction. Based on the cues of sources and characteristics of facial expression, we propose a novel model of fuzzy facial expression

Fuzzy is one common characteristic of emotion and facial expression. There is also fuzzy relationship between emotion and facial expression. One emotion can be fuzzily expressed by multiple modes of facial expression. Here, we give the model of fuzzy emotion-

Based on the correlation of multiple facial expressions of emotion, fuzzy emotion-expression mapping is proposed, in which emotion and facial expression are supposed to be fuzzy vectors, and a fuzzy matrix consisting of degrees of membership maps the fuzzy motion vector to the fuzzy facial expression vector. Define the emotion space as 1 2 *X xx x* {, } *<sup>m</sup>* , where *<sup>i</sup> x* is any emotion, such as surprise, disgust. Define the facial expression space as 1 2 {, } *Y yy y <sup>n</sup>* , where *<sup>i</sup> <sup>y</sup>* indicates any mode of facial expression. The fuzzy relation *R* from the emotion space X to the facial expression space Y is ( ) *R r ij m n* where ( , ) [0,1] *ij i j r Rx y* indicates the correlation degree of ( , ) *<sup>i</sup> <sup>j</sup> <sup>x</sup> <sup>y</sup>* to *R* . Given the input emotional fuzzy vector *ES* , the fuzzy facial

expression mapping. The mapping of emotion to expression is one-to-many.

agent systems can be used with appropriate diver programs.

**5.1 Facial expression generation** 

generation, as seen in figure 6.

Fig. 6. Work flow of the textual affect sensing.

**5.1.1 Fuzzy emotion-expression mapping** 

vector *EX* can be obtained via fuzzy mapping, as seen in 1 2 (, ,) *E E R ex ex ex X S <sup>n</sup>* ,where *<sup>i</sup> ex* is the membership of the mode of facial expression *<sup>i</sup> y* to the fuzzy facial expression *EX* , means the compositional operation of the fuzzy relations. Once the fuzzy facial expression *EX* is determined, its intensity will also be computed. The intensity of selected emotion *<sup>i</sup> x* is fuzzified to the linguistic value, which is then mapped to the linguistic value of related facial expressions according to fuzzy control rule. The intensity of facial expression *<sup>i</sup> y* is obtained by defuzzifying its linguistic value. The emotion intensity and facial expression intensity also have fuzzy characteristics. The fuzzy linguistic values of emotion and facial expression are listed as very low, low, middle, high and very high. According to the emotion-expression intensity mapping, the mapping from linguistic value of emotion intensity to linguistic value of facial expression intensity was realized through fuzzy control. An example of fuzzy control rule is shown in Table 5. Emotion *x* (surprise) can be fuzzily expressed by facial expression *y*1 or *y*2. The very low intensity of *x* can be expressed by small intensity of *y*1 or very small intensity of *y*2.


Table 5. Fuzzy control rule of fuzzy emotion-expression intensity mapping.
