**4.3 Text**

Text is an important modality for learner-tutor interaction, many of the ITS have the function enabling the tutor to chat with the student or assist the student in theoretical questions. so studying the relationship between natural language and affective information as well as assessing the underpinned affective qualities of natural language is also

Multimodal Intelligent Tutoring Systems 93

Besides the affective database, the learning commonsense database is also constructed. Our idea relies on having broad knowledge about student's common affective attitudes toward learning process. For instance, if the student input "The content is too difficult to understand", it implies that the student is not happy, as for the input "I got a high score in the test" indicates the student is happy. The structure of the learning commonsense database is based on the affect models generated from Open Mind Common Sense (OMCS) (Liu & Lieberman, 2003).

Firstly, the multiple-sentence input by the student is spited into single sentences. Each sentence is estimating the emotion separately. The sentence is tested for occurrences of emoticons, abbreviations, acronyms, interjections. If there is an emoticon, abbreviation, acronym or interjection related to an emotional state, no further analysis of affect in sentence is performed based on the assumption that the emoticon, abbreviation, acronym or abbreviation dominates the affective meaning of the entire sentence. If there are no emotion-relevant emoticons, abbreviations, acronym or interjection in a sentence, we prepare the sentence for the next processing: we use deep syntactical parser, Connexor Machinese Syntax, returns exhaustive information for analyzed sentences. From the parser output in XML style, we can read off the characteristics of each token and the relations between them in a sentence, such as subject, verb, object, and their attributes. Then, we use the word spotting technique to estimate emotion of word based on the affective database. However, the word spotting method is too simple to deal with sentences without any affective word. We hence perform the following steps on sentence-level processing. In this stage, we search the learning commonsense database to get the emotion effect of the verb. Finally, we detect "negation" in sentences. Since negatively prefixed words such as "unhappy" are already included in the emotion database, they do not have to be considered. On the other hand, negative verb forms such as "was not",

When student inputs sentences, the function of textual affect sensing is called firstly. Then the AIML Retrieval Mechanism (www.alicebot.org/aiml.html) starts in order to generate an appropriate reply using the pattern and template from the AIML database. For instance, if the student input "What is Affective Computing? It sounds really interest!", the pattern with happy is mapped. While the question is "What is Affective Computing? It is really too abstract to understand! Can you help me?", the pattern with sad takes effect. Different

*<template>* Affective Computing is a very interesting topic! It is computing that relates to, arises from, or deliberately influences emotion or other affective phenomena. *</template>*

*<template>* Oh, you seem a little unhappy. Be patient and it is easy to understand! Affective Computing is computing that relates to, arises from, or deliberately influences emotion or

In our MITS, an agent tutor "Alice" can adjust her behavior in response to learner's requests and inferred learner's needs. The agent is "eye-aware" and "affect-aware", and provides consistent empathy using facial expression and synthetic emotional speech. Its emotional response depends on the learner's action. For instance, an agent shows a happy emotion if the learner concentrates on the current study topic. In contrast, if the learner seems to lose

"did not" are detected and flip the polarity of the emotion word.

*<pattern>*What is Affective Computing HAPPY *</pattern>*

*<pattern>* What is Affective Computing SAD *</pattern>*

other affective phenomena. *</template>* 

**5. Agent tutor** 

answers are retrieved for the two patterns, as shown in the examples below:

**4.3.2 Textual emotion sensing**


Table 4. Results Using Relative Feature Vector.

important. The Artificial Intelligence Markup Language (AIML) is used to represent the tutor's conversational knowledge, employing a mechanism of stimulus-response. The stimuli (sentences and fragments which may be used to question the tutor) are stored and used to search for pre-defined replies. When the learner poses a question, the tutor starts the AIML Retrieval Mechanism in order to build an appropriate reply using the information, patterns and templates from the AIML database. AIML is an Extensible Markup Language (XML) derivative, which power lies in three basic aspects: AIML syntax enables the semantic content of a question to be extracted easily so that the appropriate answer can be given quickly. The use of labels to combine answers lends greater variety to the answers and increases the number of questions to which an answer can be given. The use of recursivity enables answers to be provided to inputs for which, in theory, there is no direct answer.

Reategui have adopted the AIML in their ITS (Reategui & Boff, 2008), however, it can not sense the affective information conveyed by text automatically. In this paper, we integrate the textual affect sensing algorithm into the AIML Retrieval Mechanism. Figure 5 shows the work flow of the textual affect sensing. The approach for providing emotional estimations from the sentence input by the student is based on a keyword spotting technique and sentence-level processing technique.

Fig. 5. Work flow of the textual affect sensing.

#### **4.3.1 Affective database and learning commonsense database**

In order to support the handling of abbreviated language and the interpretation of affective features of emoticons, abbreviations, interjections and words, an affective database was created using XML. We collected emoticons (such as, ":-)" for happiness and "QQ" for sadness), the most popular emotional acronyms and abbreviations (for instance, "LOL" (laughing out loud) for happiness) and emotional interjections (for example: "damn" for anger and "wow" for happiness). We also have taken emotional adjectives, nouns, verbs, and adverbs words into our database.

anger 14.3 19.1 19.1 19.1 14.3 happiness 31.8 45.5 50.0 40.9 100 sadness 8.7 47.8 73.9 47.8 60.9 disgust 92.3 45.5 50.0 40.9 100 surprise 38.9 33.3 33.3 33.3 83.3 average 30.2 37.1 40.5 36.2 57.8

important. The Artificial Intelligence Markup Language (AIML) is used to represent the tutor's conversational knowledge, employing a mechanism of stimulus-response. The stimuli (sentences and fragments which may be used to question the tutor) are stored and used to search for pre-defined replies. When the learner poses a question, the tutor starts the AIML Retrieval Mechanism in order to build an appropriate reply using the information, patterns and templates from the AIML database. AIML is an Extensible Markup Language (XML) derivative, which power lies in three basic aspects: AIML syntax enables the semantic content of a question to be extracted easily so that the appropriate answer can be given quickly. The use of labels to combine answers lends greater variety to the answers and increases the number of questions to which an answer can be given. The use of recursivity enables answers to be provided to inputs for which, in theory, there is no direct answer. Reategui have adopted the AIML in their ITS (Reategui & Boff, 2008), however, it can not sense the affective information conveyed by text automatically. In this paper, we integrate the textual affect sensing algorithm into the AIML Retrieval Mechanism. Figure 5 shows the work flow of the textual affect sensing. The approach for providing emotional estimations from the sentence input by the student is based on a keyword spotting technique and

1 2 3 4 fusion

Emotion Classifiers

Table 4. Results Using Relative Feature Vector.

sentence-level processing technique.

Fig. 5. Work flow of the textual affect sensing.

and adverbs words into our database.

**4.3.1 Affective database and learning commonsense database** 

In order to support the handling of abbreviated language and the interpretation of affective features of emoticons, abbreviations, interjections and words, an affective database was created using XML. We collected emoticons (such as, ":-)" for happiness and "QQ" for sadness), the most popular emotional acronyms and abbreviations (for instance, "LOL" (laughing out loud) for happiness) and emotional interjections (for example: "damn" for anger and "wow" for happiness). We also have taken emotional adjectives, nouns, verbs, Besides the affective database, the learning commonsense database is also constructed. Our idea relies on having broad knowledge about student's common affective attitudes toward learning process. For instance, if the student input "The content is too difficult to understand", it implies that the student is not happy, as for the input "I got a high score in the test" indicates the student is happy. The structure of the learning commonsense database is based on the affect models generated from Open Mind Common Sense (OMCS) (Liu & Lieberman, 2003).

## **4.3.2 Textual emotion sensing**

Firstly, the multiple-sentence input by the student is spited into single sentences. Each sentence is estimating the emotion separately. The sentence is tested for occurrences of emoticons, abbreviations, acronyms, interjections. If there is an emoticon, abbreviation, acronym or interjection related to an emotional state, no further analysis of affect in sentence is performed based on the assumption that the emoticon, abbreviation, acronym or abbreviation dominates the affective meaning of the entire sentence. If there are no emotion-relevant emoticons, abbreviations, acronym or interjection in a sentence, we prepare the sentence for the next processing: we use deep syntactical parser, Connexor Machinese Syntax, returns exhaustive information for analyzed sentences. From the parser output in XML style, we can read off the characteristics of each token and the relations between them in a sentence, such as subject, verb, object, and their attributes. Then, we use the word spotting technique to estimate emotion of word based on the affective database. However, the word spotting method is too simple to deal with sentences without any affective word. We hence perform the following steps on sentence-level processing. In this stage, we search the learning commonsense database to get the emotion effect of the verb. Finally, we detect "negation" in sentences. Since negatively prefixed words such as "unhappy" are already included in the emotion database, they do not have to be considered. On the other hand, negative verb forms such as "was not", "did not" are detected and flip the polarity of the emotion word.

When student inputs sentences, the function of textual affect sensing is called firstly. Then the AIML Retrieval Mechanism (www.alicebot.org/aiml.html) starts in order to generate an appropriate reply using the pattern and template from the AIML database. For instance, if the student input "What is Affective Computing? It sounds really interest!", the pattern with happy is mapped. While the question is "What is Affective Computing? It is really too abstract to understand! Can you help me?", the pattern with sad takes effect. Different answers are retrieved for the two patterns, as shown in the examples below:

*<pattern>*What is Affective Computing HAPPY *</pattern>*

*<template>* Affective Computing is a very interesting topic! It is computing that relates to, arises from, or deliberately influences emotion or other affective phenomena. *</template>*

*<pattern>* What is Affective Computing SAD *</pattern>*

*<template>* Oh, you seem a little unhappy. Be patient and it is easy to understand! Affective Computing is computing that relates to, arises from, or deliberately influences emotion or other affective phenomena. *</template>* 
