**2. Nonverbal interaction in CVEs**

72 Virtual Reality and Environments

module. The ITS architecture adapted for an Intelligent Collaborative Virtual Environment

Student Module Expert Module

Tutoring Module *(Scaffolding)*

*Environment (CVE)*  Communication Module

Team

The systems for training developed to date may be classified depending on the issue they emphasize as: simulation processes, generation of believable environments, and collaboration processes; three aspects that can be integrated in a single system, an ICVE for

The component in charge of the domain of the application has the capacity to execute the simulations of the task to be trained as they are executed in the present systems. VR technology allows recreating environments with believable features (Youngblut, 1998) offering the possibility of being trained in tasks that may be expensive, risky, or even

Regarding the collaboration process, in an ICVE by including virtual agents to replace team members (Rickel, 2001), the trainees can have a team training experience when the complete human team is not available. It is also possible to integrate into the environment personified PVAs to offer them support, in the same way an instructor or a human peer would do.

Within VR technology, a PVA may assume a 3D representation similar to that used by a human in the environment; they can be embodied through an avatar. The PVAs personification seems to generate positive effects in the perception of the apprentices during their learning experiences (Lester et al., 1997), with two key advantages over earlier work: they increase the bandwidth of communication between students and computers; and they increase the computer's ability to engage and motivate students. Some early empirical results on PVAs embodied effectiveness are on the topics of (W. L. Johnson, Rickel, & Lester, 2000): interactive demonstrations, navigational guidance, gaze and gesture as attentional guides, nonverbal feedback, conversational signals, conveying and eliciting emotion, and virtual teammates.

A PVA can be defined as an intelligent agent that makes decisions about how to maximize the student learning process. PVAs as a result of its goal can function within the ICVE as tutors,

(ICVE) (Aguilar et al., 2010) is shown in Figure 1.

Fig. 1. ICVE Architecture

impossible to reproduce in the reality.

learning.

Broadly defined, nonverbal behavior might include most of what we do; it even includes certain characteristics of verbal behavior by distinguishing the content, or meaning of speech, from paralinguistic cues such as loudness, tempo, pitch or intonation (Patterson, 1983). Moreover, the use of certain objects like our decided outfit, or the physical environment when used to communicate something, without saying it, has traditionally being considered as NVC. Nonverbal behavior can be used to substitute, complement, accent, regulate or even contradict the spoken message (Knapp & Hall, 2007).

In real life, nonverbal interaction involves three factors (Knapp & Hall, 2007): *environmental conditions, physical characteristics of the communicators,* and *behaviors of communicators*, all of them clearly restricted to the computer scenario conditions.

The *environmental conditions* that will probably affect the most during interaction in a computer scenario are given by the architecture and virtual objects around, what Hall (1968) defined as fixed-features, space organized by unmoving boundaries such as a room, and semi-fixed features, the arrangement of moveable objects such as a chair.

In a computerized environment, the *physical characteristics* of the interactants will be given by the users' avatar both appearance and body movements. While the appearance typically is given by a set of characteristics for the user to choose, like male or female avatar, and maybe a set of different cloths, skin or hair colors for example. As of body movements, Kujanpää & Manninen (2003) presented a considerable set of possible elements an avatar can include to manage the transmitting of NVC.

The avatar's body movements are usually restricted mainly due to their associated technology cost. Typically, in CVEs the users' avatars are naturalistic (Salem & Earle, 2000), with a low-level details approach and humanoid-like, they can display some basic humans' actions or expressions.

Other important consideration is that the means offered by the CVE to the user, in order to transmit NVC to his/her avatar, interfere with its spontaneity and therefore its revealing. The three different approaches to transmit nonverbal behavior from the users to his/her avatar in a VE (Capin et al., 1997) are:


The *behaviors of communicators* relay on the context that in a CVE will be given by its purpose. For example, in a video game, the users' interaction will be controlled by their intention on getting the goals of the game, while in a social CVE the participants interaction will be more likely to be directed to those they feel socially attracted −see Table 2.

The Users' Avatars Nonverbal Interaction in Collaborative Virtual Environments for Learning 75

of a sequence of related events. At the foundation of this model is the distinction made between the interaction behavior on itself and the functions served by them. Distinguishing the function served by the interaction behavior means to recognize that the same behavioral

In the service-task function proposed by Patterson (1983), the service component refers to interaction determined by a service relationship between individuals, for example, a physician-patient interaction. While the task function, influential for a CVE for learning, identifies focused or unfocused interactions that require people to relate others through a

According to Patterson (1983), the necessity for variable involvement in task-oriented focused interactions, such as when people collaborate to accomplish a task, seems relatively straightforward. Understanding this type of nonverbal interaction keeps the interpretation of nonverbal behavior to an acceptable extent from cultural and personality influences, since the service-task function identifies determinants of nonverbal involvement that are generally independent of the quality of interpersonal relationships. Accordingly, the

Environmental conditions an scenario according to the domain to be taught

allowed body movements

In order to make use of a nonverbal communication cue to monitor collaboration, it needs to have the faculty of being transmittable to the CVE and recognizable by the computer system. With this in mind, the nonverbal communication cues suggested for the interaction

*Talking turns* - the paralinguistic branch that studies, not what or how people talk but amounts and patterns of talk and that have been use for the comprehension of interaction in

*Proxemics* – to understand the users' position within the environment and related to others. *Facial expressions* – in real life, they might be difficult for interpretation, but when transmitted to a VE not directly controlled by the user, their intention is usually predefined

*Body movements* - such as gaze direction, deictic gestures, head movements and some body

In the next section the analysis of nonverbal behavior from the participants in a collaborative task within a CVE are discussed. Afterwards, a model for an intelligent tutor based on nonverbal behavior with the intent to facilitate collaborative sessions is presented.

**Their conditions in CVEs for learning** 

operable objects for the learning purpose

nonverbal interaction conditions for a CVE for learning are presented in Table 3.

Physical characteristics the users' avatars appearance

*Artifacts manipulation* – when they are part of the collaborative interaction.

analysis as described in Peña & de Antonio (2010) are:

different ways as in (Bales, 1970; Jaffe & Feldstein, 1970).

by the system as in the case of the emoticons.

postures.

Behaviors of communicators consistent with the service-task function Table 3. Conditions of the nonverbal interaction factors in CVEs for learning

patterns can serve very different functions in an interaction.

particular task or activity.

**Nonverbal interaction influential factors** 


Table 2. Conditions of the nonverbal interaction factors in a CVE

Specifically *for a CVE for learning*, the *environmental conditions* will most likely to be constrained by the domain to be taught and the selected pedagogical strategy. The pedagogical strategy will determine the session configuration, like a theme of discussion, solving a problem or accomplishing a task.

Consistent with Collaborative Learning theories the participants' interaction should be implied in the CVE, and recommendable for learning purposes can be to solve a problem through the accomplishment of a task; considering that one of their main advantages is the spacial space with shared objects they offer. Within the CVE, the entities on it with the faculty of being manipulated by the users, the semi-fixed features, will take part of their verbal and nonverbal interchange on being the means to the accomplishment of the task.

As mentioned, the *physical characteristics* of the communicators in a computer-generated environment are determined by his/her avatar's stipulations. The avatar appearance in a learning scenario may represent the apprentices' role. For example, if a firefighter will be trained, his/her avatar will be most likely to use the firefighter's uniform to distinguish him/her from other users.

In order to accomplish the learning purpose in the CVE, the apprentices will control their avatars to communicate, navigate and modify the environment. For that, the mainly related areas of NVC are:

**Paralinguistics** that comprises all non-linguistic characteristics related to speech like the selected language, the tone of voice, or the voice inflexions, among others.

**Proxemics,** the analyses of the chosen body distance and angle during interaction (Guye-Vuillème et al., 1998).

And **Kinesics,** the study of what is called "body language", all body movements except physical contact, which includes gestures, postural shifts and movements of some parts of the body like hands, head or trunk (Argyle, 1990).

As of the *behaviors of communicators* in a virtual learning scenario, of special interest should be those related to collaborative interaction, that is, those behaviors that transmit something about how the group members collaborate in order to achieve the common goal; and they will be consistent with the nonverbal behavior carried out for the service-task function (Patterson, 1983).

The nonverbal behavior in collaborative interaction is expected to be mainly intended for the accomplishment of the task. Following the Miles L. Patterson (1983) Sequential Functional Model for nonverbal exchange, people's interaction behavior is the consequence

Specifically *for a CVE for learning*, the *environmental conditions* will most likely to be constrained by the domain to be taught and the selected pedagogical strategy. The pedagogical strategy will determine the session configuration, like a theme of discussion,

Consistent with Collaborative Learning theories the participants' interaction should be implied in the CVE, and recommendable for learning purposes can be to solve a problem through the accomplishment of a task; considering that one of their main advantages is the spacial space with shared objects they offer. Within the CVE, the entities on it with the faculty of being manipulated by the users, the semi-fixed features, will take part of their verbal and nonverbal interchange on being the means to the accomplishment of the task.

As mentioned, the *physical characteristics* of the communicators in a computer-generated environment are determined by his/her avatar's stipulations. The avatar appearance in a learning scenario may represent the apprentices' role. For example, if a firefighter will be trained, his/her avatar will be most likely to use the firefighter's uniform to distinguish

In order to accomplish the learning purpose in the CVE, the apprentices will control their avatars to communicate, navigate and modify the environment. For that, the mainly related

**Paralinguistics** that comprises all non-linguistic characteristics related to speech like the

**Proxemics,** the analyses of the chosen body distance and angle during interaction (Guye-

And **Kinesics,** the study of what is called "body language", all body movements except physical contact, which includes gestures, postural shifts and movements of some parts of

As of the *behaviors of communicators* in a virtual learning scenario, of special interest should be those related to collaborative interaction, that is, those behaviors that transmit something about how the group members collaborate in order to achieve the common goal; and they will be consistent with the nonverbal behavior carried out for the service-task function

The nonverbal behavior in collaborative interaction is expected to be mainly intended for the accomplishment of the task. Following the Miles L. Patterson (1983) Sequential Functional Model for nonverbal exchange, people's interaction behavior is the consequence

selected language, the tone of voice, or the voice inflexions, among others.

the body like hands, head or trunk (Argyle, 1990).

Environmental conditions the fixed-features of the scenario

Physical characteristics the users' avatars appearance body movements

Behaviors of communicators according to the CVE purpose

Table 2. Conditions of the nonverbal interaction factors in a CVE

**Constrained in a CVE to** 

the semi-fixed features of the scenario

**Nonverbal interaction influential** 

solving a problem or accomplishing a task.

him/her from other users.

areas of NVC are:

Vuillème et al., 1998).

(Patterson, 1983).

**factors** 

of a sequence of related events. At the foundation of this model is the distinction made between the interaction behavior on itself and the functions served by them. Distinguishing the function served by the interaction behavior means to recognize that the same behavioral patterns can serve very different functions in an interaction.

In the service-task function proposed by Patterson (1983), the service component refers to interaction determined by a service relationship between individuals, for example, a physician-patient interaction. While the task function, influential for a CVE for learning, identifies focused or unfocused interactions that require people to relate others through a particular task or activity.

According to Patterson (1983), the necessity for variable involvement in task-oriented focused interactions, such as when people collaborate to accomplish a task, seems relatively straightforward. Understanding this type of nonverbal interaction keeps the interpretation of nonverbal behavior to an acceptable extent from cultural and personality influences, since the service-task function identifies determinants of nonverbal involvement that are generally independent of the quality of interpersonal relationships. Accordingly, the nonverbal interaction conditions for a CVE for learning are presented in Table 3.


Table 3. Conditions of the nonverbal interaction factors in CVEs for learning

In order to make use of a nonverbal communication cue to monitor collaboration, it needs to have the faculty of being transmittable to the CVE and recognizable by the computer system. With this in mind, the nonverbal communication cues suggested for the interaction analysis as described in Peña & de Antonio (2010) are:

*Talking turns* - the paralinguistic branch that studies, not what or how people talk but amounts and patterns of talk and that have been use for the comprehension of interaction in different ways as in (Bales, 1970; Jaffe & Feldstein, 1970).

*Proxemics* – to understand the users' position within the environment and related to others.

*Facial expressions* – in real life, they might be difficult for interpretation, but when transmitted to a VE not directly controlled by the user, their intention is usually predefined by the system as in the case of the emoticons.

*Artifacts manipulation* – when they are part of the collaborative interaction.

*Body movements* - such as gaze direction, deictic gestures, head movements and some body postures.

In the next section the analysis of nonverbal behavior from the participants in a collaborative task within a CVE are discussed. Afterwards, a model for an intelligent tutor based on nonverbal behavior with the intent to facilitate collaborative sessions is presented.

The Users' Avatars Nonverbal Interaction in Collaborative Virtual Environments for Learning 77

*Talking turns and amount of talk.* The idea of taking the time that group members speak to understand group process is not new. In 1949, Eliot Chapple created the chronograph interaction; a device to measure persons' amount of talk with the intention of analyzing talkturns structure (Chapple, 1949). Since then, frequency and duration of speech have been useful tools for the analysis of group interaction in a number of ways, for example to create regulatory tools for meetings as in (Bergstrom & Karahalios, 2007). The students' rates of speech will help to determine if they are participating during discussion periods and to

*Artifacts manipulation and implementation.* When the group's common goal implies implementation, it is desirable a maintained balance between dialogue and action (Jermann, 2004). Artifacts manipulation is an object form of nonverbal behavior, as it can be part of the answer to an expression. The amount of work a student realizes, aside of its quality, is a

*Gazes*. The eyes direction is a reliable indicative of a persons' focus of attention (Bailenson et al., 2003). Via the students' gazes, it can be determined to what they are paying attention.

*Deictic Gestures*. Deictic terms such as "here, there, that", are interpreted resulting from the communication context, and when the conversation is focused on objects and their identities, they are crucial to identify them quickly and securely (Clark & Brennan, 1991). Consequently, deictic gestures directed to the shared objects or the workspace should be

In the application, the user does not see his/her own avatar −see Figure 2. The users' avatars do not have a natural behavior; they are just seated representations of the user that need a

The significant entities associated to the avatars actions are: colored arrows coupled to their hair color (yellow, red, or brown) that take the place of their hands, and can be used to point the objects or grab them to be moved; by a mouse click, the arrow is activated. To move the

objects once they have being selected, the WASD keys can be used to direct them.

good indicative of that student's interest and participation on the task.

useful to determine whether students are talking about the task.

metaphorical representation of their actions in the environment.

Fig. 2. Experimental application

what extent.
