**3. Abstract feelings in interpersonal haptic communication**

In social life, nonverbal communication is crucial for conveying abstract feelings (i.e., "Kansei" information), which is too ambiguous and abstract to be described verbally, and for sharing them with others. When we communicate with others in our daily lives, we use various nonverbal channels such as gestures, facial expressions, or gaze. Among them, haptic communication is the most primitive channel, and it is widely seen in human infants and animals (Knapp & Hall, 1992). In particular, haptic communication plays a dominant role in directly conveying abstract feelings such as emotions (Richmond et al., 2007). Although nonverbal haptic communication has been less understood as compared to verbal communication, some researchers have challenged the kinetic taxonomy of physical contact or categorization based on the meaning of physical contact (Morris, 1971).

The sense of touch or haptics is complex (Katz, 1989), bidirectional, and interactive (e.g., active touch (Gibson, 1962)). Therefore, what is conveyed in interpersonal haptic communication cannot be determined straightforwardly. Instead, the perceptual process for haptic communication itself, the cognitive process for understanding the intention of others, and the context (e.g., the social position of others) would interactively affect what is conveyed. The abstract feelings conveyed in haptic communication are loosely categorized on the basis of the types of motion. For example, stroking someone's head may express affection, and shaking hands may express amity. However, the abstract feeling in haptic communication is also subject to the context of the communication (Richmond et al., 2007). Let us consider a handshake between politicians, that between a mother and a baby, and that between close friends at the time of farewell. It is almost certain that the feelings of the people are not identical in these contexts, although the motions involved in the haptic communication are rather similar.

Given the importance of haptic communication in conveying and sharing abstract feelings, we expect that the quality of experience shared with others could be altered by adding haptic communication. Some innovative studies have already tried to develop haptic devices enabling interpersonal communication and have empirically examined their effect on task performance and subjective impressions (Fujita & Hashimoto, 1999; Reiner, 2004; Sallnäs et al., 2000; Wang et al., 2004). For example, Fujita & Hashimoto (1999) showed that users could feel the presence of another person through haptic bidirectional communication even without seeing each other. Sallnäs et al. (2000) also reported that those who were performing collaborative tasks in a virtual environment felt increased reality or presence and showed an improvement in their task performance due to the force feedback from the virtual environment.

In this chapter, we discuss abstract feelings in interpersonal haptic communication. One question concerns how people experience interpersonal haptic communication in their daily activity. Takahashi, Mitsuhashi, Norieda, Sendoda, Murata & Watanabe (2010) and Takahashi et al. (2011c) conducted a psychological survey to investigate this question. Another question is whether haptic communication could change the subjective quality of an experience, and if 8 Will-be-set-by-IN-TECH

animacy perception may emerge as a result of the frequent experience of touching animate objects such as animals. While the stimulation from these animate objects would involve more complex patterns than those noted in Takahashi et al. (2011a), the frequency of stimulation might be the major factor that induces the feeling of animacy. Therefore, it would be intriguing to survey the frequencies of haptic stimulation from these animate objects in the real world and to compare them to the results of the experimental study in Takahashi et al. (2011a).

In social life, nonverbal communication is crucial for conveying abstract feelings (i.e., "Kansei" information), which is too ambiguous and abstract to be described verbally, and for sharing them with others. When we communicate with others in our daily lives, we use various nonverbal channels such as gestures, facial expressions, or gaze. Among them, haptic communication is the most primitive channel, and it is widely seen in human infants and animals (Knapp & Hall, 1992). In particular, haptic communication plays a dominant role in directly conveying abstract feelings such as emotions (Richmond et al., 2007). Although nonverbal haptic communication has been less understood as compared to verbal communication, some researchers have challenged the kinetic taxonomy of physical contact

The sense of touch or haptics is complex (Katz, 1989), bidirectional, and interactive (e.g., active touch (Gibson, 1962)). Therefore, what is conveyed in interpersonal haptic communication cannot be determined straightforwardly. Instead, the perceptual process for haptic communication itself, the cognitive process for understanding the intention of others, and the context (e.g., the social position of others) would interactively affect what is conveyed. The abstract feelings conveyed in haptic communication are loosely categorized on the basis of the types of motion. For example, stroking someone's head may express affection, and shaking hands may express amity. However, the abstract feeling in haptic communication is also subject to the context of the communication (Richmond et al., 2007). Let us consider a handshake between politicians, that between a mother and a baby, and that between close friends at the time of farewell. It is almost certain that the feelings of the people are not identical in these contexts, although the motions involved in the haptic communication are

Given the importance of haptic communication in conveying and sharing abstract feelings, we expect that the quality of experience shared with others could be altered by adding haptic communication. Some innovative studies have already tried to develop haptic devices enabling interpersonal communication and have empirically examined their effect on task performance and subjective impressions (Fujita & Hashimoto, 1999; Reiner, 2004; Sallnäs et al., 2000; Wang et al., 2004). For example, Fujita & Hashimoto (1999) showed that users could feel the presence of another person through haptic bidirectional communication even without seeing each other. Sallnäs et al. (2000) also reported that those who were performing collaborative tasks in a virtual environment felt increased reality or presence and showed an improvement in their task performance due to the force feedback from the virtual

In this chapter, we discuss abstract feelings in interpersonal haptic communication. One question concerns how people experience interpersonal haptic communication in their daily activity. Takahashi, Mitsuhashi, Norieda, Sendoda, Murata & Watanabe (2010) and Takahashi et al. (2011c) conducted a psychological survey to investigate this question. Another question is whether haptic communication could change the subjective quality of an experience, and if

**3. Abstract feelings in interpersonal haptic communication**

or categorization based on the meaning of physical contact (Morris, 1971).

rather similar.

environment.

so, how. To examine this question, Takahashi et al. (2011b) conducted an experimental study investigating the effects of haptic communication on the quality of participants' experiences, as well as on their impression of the person with whom they communicated.

#### **3.1 Psychological survey on interpersonal haptic communication**

In a series of psychological surveys, Takahashi, Mitsuhashi, Norieda, Sendoda, Murata & Watanabe (2010) and Takahashi et al. (2011c) investigated how people experience interpersonal haptic communication in their daily activity.

Takahashi, Mitsuhashi, Norieda, Sendoda, Murata & Watanabe (2010) presented observers with pictures depicting two persons performing either positive (e.g., hugging, shaking hands) or negative (e.g., grappling) haptic communication. The observers were asked to make a subjective evaluation of the depicted communication concerning its desirability, frequency of experience, and impressiveness. All these evaluations had a positive correlation (R > 0.59, p < .001), suggesting that the haptic communication that was more frequently experienced and that made a stronger impression tended to be evaluated as more desirable. Although these correlations cannot reveal a causal relationship, one possibility may be related to the widely known psychological phenomenon of "mere exposure effect" (Zajonc, 1968), and the other is related to "directional forgetting" (Golding & Macleod, 1998).

The observers were also asked to identify the person who was imagined to be involved in the communication depicted in the picture. The persons associated with the desirable communication were a spouse, a child, a mother, a son or a daughter, a father, and a grandparent. Thus, haptic communications that recall family members are more desirable.

Takahashi et al. (2011c) asked their observers to evaluate two types of haptic communication, tapping and padding, by using adjectives, and examined what factors reside in haptic communication. Table 1 shows the results of the factorial analysis. The results suggested that two factors represent interpersonal haptic communication. One factor affected the evaluation of lightsome, pleasant, comfortable, and warm feelings, and the other factor affected the evaluation of strong, rough, and active feelings. Thus, the first factor was associated with the abstract feelings of haptic communication, while the second factor was relevant to the characteristics of motion. These structures were common between the two types of haptic communications. Furthermore, the first factor strongly affected the desirability of the haptic communications when compared with the second factor, suggesting that the abstract feeling emerging from the communication, not the motion itself, strongly influenced the desirability of the haptic communication.


Table 1. First and second component score of each adjective for describing two types (tapping and patting) of haptic interpersonal communication. Data were from Takahashi et al. (2011c).

Q1 The movie was hilarious.

strongly disagree–7: strongly agree).

3.0

3.5

4.0

4.5

5.0

5.5

6.0

Q5 I was concerned about the other.

Q2 It felt as if the movie was more hilarious due to watching it with the other. Q3 It seemed as if I wanted to convey the hilarious feeling to the other. Q4 It seemed as if I could convey the hilarious feeling to the other.

Abstract Feelings Emerging from Haptic Stimulation 107

Q6 It seemed as if the other's hilarious feeling could be convey to me.

Table 2. Questions on the questionnaire. Each question was rated on a 7-point scale (1:

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8

Uninterrupted Interrupted

experience and the impression of the other person. Fig. 5 shows the ratings for each question (Table 2). A higher score indicates a stronger agreement with the question. The statistical test (t-test) showed that the rating scores of three questions (Q2, Q3, and Q8) were significantly higher in the uninterrupted group than in the interrupted group (all t(38) > 2.66, p < .05). Q2 asked whether the participants felt that the experience with the other person made the movie more hilarious. The participants' impressions may be attributed more to the sharing of the experience with the other person if the haptic communication was uninterrupted. Q3 asked if the participants felt eager to convey their feelings to the other person, and Q8 asked if the participants felt they would like to watch a movie with the other person again in the future. These questions concerned the participants' feelings toward the other person, and the interruption of haptic communication decreased such responses toward the other person. On the other hand, as for Q1, where the participants were asked to rate the hilariousness of the movie itself, the rating score of the uninterrupted group was slightly higher than that

Fig. 5. Mean rating score for each question of the questionnaire. Error bars indicate the standard errors of means. The figure was modified from Takahashi et al. (2011b).

Q8 It felt as if I wanted to watch the movie again with the other.

Q7 It felt as if I agreed with the other's hilarious feeling.

These psychological surveys in Takahashi, Mitsuhashi, Norieda, Sendoda, Murata & Watanabe (2010) and Takahashi et al. (2011c) suggested that desirable haptic communication is experienced with closer persons in daily life (Knapp & Hall, 1992; Morris, 1971; Richmond et al., 2007), and that the abstract feelings emerging from the haptic communication are essential for determining the desirability of the communication.

#### **3.2 Experimental study on shared experience via interpersonal haptic telecommunication**

Takahashi et al. (2011b) conducted an experimental study to investigate how interpersonal haptic communication affects shared experience and found that haptic communication could improve the quality of experience as well as the impression of the other person. They examined the effects of haptic communication on the quality of the experience shared with another person along with the impression of the other person with whom they communicated. In the experiment, two persons watched a movie at the same time. All forms of communication except haptic communication were completely removed. The persons could communicate by triggering a haptic stimulation in the other person.

Forty graduate and undergraduate students participated in the experiment in 20 pairs, which were divided into two groups of 10 pairs each. Each pair in the groups was assigned to one of two experimental conditions. In the experiment, two LCD TVs were placed on two desks. The participants were seated at the desks and simultaneously watched a movie on the TV with headphones. The desks were separated by a partition; hence, the participants could not see or hear each other. Each participant held a push button with his or her left hand and touched the haptic device (Fig. 2.) with his or her right hand. The button sent a signal to the operator PC when it was pressed and released. The haptic stimulator generated 2-Hz of vibratory pressure stimulation when receiving a signal from the operator PC. The operator PC sent a signal to the haptic stimulator of one person when the other person pressed his or her button. The timing of the button press/release was recorded by the operator PC. Takahashi et al. (2011b) chose a Japanese comedy movie clip of approximately 220 s duration as the visual stimulus. The movie was played on two temporally synchronized screens.

In an experimental session, the participants were instructed to watch the movie and to press the button while they found it hilarious. They were also told explicitly that pressing their button sent a haptic stimulation to the other person and that the other person's button press triggered a haptic stimulation in their sensor. They experienced this haptic communication prior to the experimental session. Thus, the participants were aware that they could perceive the other person's response through the haptic stimulation. The participants were explicitly instructed to decide on their own whether to press the button and to avoid following the lead of the other person.

One of the two experimental groups was an uninterrupted group, in which the button press always triggered the haptic stimulation to the other person throughout the experimental session. The other experimental group was an interrupted group, wherein the haptic stimulation was interrupted for 90 s, between 75 s and 165 s after the movie started. The participants in the interruption group were not told of the interruption of the haptic stimulation prior to the experimental session; hence, they believed that the absence of haptic stimulation indicated that the other person did not find the movie hilarious. After watching the movie, the participants answered a questionnaire consisting of 8 questions and using a 7-point scale (Table 2).

Using a post-experiment questionnaire survey, Takahashi et al. (2011b) found that the interruption of haptic communication affected the subjective evaluation of the quality of the 10 Will-be-set-by-IN-TECH

These psychological surveys in Takahashi, Mitsuhashi, Norieda, Sendoda, Murata & Watanabe (2010) and Takahashi et al. (2011c) suggested that desirable haptic communication is experienced with closer persons in daily life (Knapp & Hall, 1992; Morris, 1971; Richmond et al., 2007), and that the abstract feelings emerging from the haptic communication are

**3.2 Experimental study on shared experience via interpersonal haptic telecommunication** Takahashi et al. (2011b) conducted an experimental study to investigate how interpersonal haptic communication affects shared experience and found that haptic communication could improve the quality of experience as well as the impression of the other person. They examined the effects of haptic communication on the quality of the experience shared with another person along with the impression of the other person with whom they communicated. In the experiment, two persons watched a movie at the same time. All forms of communication except haptic communication were completely removed. The persons

Forty graduate and undergraduate students participated in the experiment in 20 pairs, which were divided into two groups of 10 pairs each. Each pair in the groups was assigned to one of two experimental conditions. In the experiment, two LCD TVs were placed on two desks. The participants were seated at the desks and simultaneously watched a movie on the TV with headphones. The desks were separated by a partition; hence, the participants could not see or hear each other. Each participant held a push button with his or her left hand and touched the haptic device (Fig. 2.) with his or her right hand. The button sent a signal to the operator PC when it was pressed and released. The haptic stimulator generated 2-Hz of vibratory pressure stimulation when receiving a signal from the operator PC. The operator PC sent a signal to the haptic stimulator of one person when the other person pressed his or her button. The timing of the button press/release was recorded by the operator PC. Takahashi et al. (2011b) chose a Japanese comedy movie clip of approximately 220 s duration as the visual stimulus. The

In an experimental session, the participants were instructed to watch the movie and to press the button while they found it hilarious. They were also told explicitly that pressing their button sent a haptic stimulation to the other person and that the other person's button press triggered a haptic stimulation in their sensor. They experienced this haptic communication prior to the experimental session. Thus, the participants were aware that they could perceive the other person's response through the haptic stimulation. The participants were explicitly instructed to decide on their own whether to press the button and to avoid following the lead

One of the two experimental groups was an uninterrupted group, in which the button press always triggered the haptic stimulation to the other person throughout the experimental session. The other experimental group was an interrupted group, wherein the haptic stimulation was interrupted for 90 s, between 75 s and 165 s after the movie started. The participants in the interruption group were not told of the interruption of the haptic stimulation prior to the experimental session; hence, they believed that the absence of haptic stimulation indicated that the other person did not find the movie hilarious. After watching the movie, the participants answered a questionnaire consisting of 8 questions and using a

Using a post-experiment questionnaire survey, Takahashi et al. (2011b) found that the interruption of haptic communication affected the subjective evaluation of the quality of the

essential for determining the desirability of the communication.

could communicate by triggering a haptic stimulation in the other person.

movie was played on two temporally synchronized screens.

of the other person.

7-point scale (Table 2).


Table 2. Questions on the questionnaire. Each question was rated on a 7-point scale (1: strongly disagree–7: strongly agree).

Fig. 5. Mean rating score for each question of the questionnaire. Error bars indicate the standard errors of means. The figure was modified from Takahashi et al. (2011b).

experience and the impression of the other person. Fig. 5 shows the ratings for each question (Table 2). A higher score indicates a stronger agreement with the question. The statistical test (t-test) showed that the rating scores of three questions (Q2, Q3, and Q8) were significantly higher in the uninterrupted group than in the interrupted group (all t(38) > 2.66, p < .05). Q2 asked whether the participants felt that the experience with the other person made the movie more hilarious. The participants' impressions may be attributed more to the sharing of the experience with the other person if the haptic communication was uninterrupted. Q3 asked if the participants felt eager to convey their feelings to the other person, and Q8 asked if the participants felt they would like to watch a movie with the other person again in the future. These questions concerned the participants' feelings toward the other person, and the interruption of haptic communication decreased such responses toward the other person.

On the other hand, as for Q1, where the participants were asked to rate the hilariousness of the movie itself, the rating score of the uninterrupted group was slightly higher than that

In addition to the subjective evaluation using the post-experiment questionnaire survey, the interruption of haptic communication affected the participants' behavior while watching the movie—namely, the number of button presses during the movie presentation. Fig. 6. shows the total number of button presses for three different periods. The first one was the period before the interruption of haptic stimulation; hence, there was inherently no difference between the two groups. The numbers of button presses did not differ between the two groups in the first period (t(18) = 0.62, p = 0.54). In the second period, the haptic stimulation was cut off between 75 and 165 s for the interrupted group. In this period, the number of button presses in the uninterrupted group was significantly larger than that in the interrupted group (t(18) = 2.65, p < 0.05). The third period was the one after the interruption. Although the number of actions was slightly larger in the uninterrupted group, the difference was not

Abstract Feelings Emerging from Haptic Stimulation 109

In the questionnaire survey, we found that the retrospective evaluation of how hilarious the movie was did not significantly differ between the two groups (Q1). In contrast, the behavioral performance clearly showed that the number of button presses—namely, how often the participants felt the movie was hilarious—was decreased by the interruption of the haptic communication. These results suggest that the haptic communication implicitly modulated the quality of the experience, although the participants could not be explicitly

Both psychological surveys (Takahashi et al., 2011c; Takahashi, Mitsuhashi, Norieda, Sendoda, Murata & Watanabe, 2010) and the experimental study (Takahashi et al., 2011b) showed the importance of haptic stimulation in interpersonal communication. The desirability of haptic communication was associated with the factor relevant to abstract feelings (lightsome, pleasant, comfortable, and warm) (Takahashi et al., 2011c; Takahashi, Mitsuhashi, Norieda, Sendoda, Murata & Watanabe, 2010). The presence of haptic communication improved the quality of experience and the impression of the other person as well (Takahashi et al., 2011b). As the vibratory haptic stimulation could affect the quality of experience, the presence of haptic communication, not the contents of the stimulation, may be essential for influencing the quality of experience. Abstract feelings emerging from the haptic stimulation in interpersonal communication would be important for persons who share a high-quality experience and

Humans have multiple sensory modalities for accessing the external world, such as vision, audition, and, of course, haptics. Although inputs into different sensory modalities are processed in different brain regions at early sensory levels, different sensory modalities are no doubt interactively processed at the perceptual level to build up unified perceptions of the external world, which is referred to as cross-modal interaction (Shams & Kim, 2010; Shimojo & Shams, 2001). For example, the ventriloquism effect (Alais & Burr, 2004) and McGurk effect McGurk & MacDonald (1976) are well known phenomena demonstrating visual modulation

Although cross-modal interaction has mainly been examined using vision and audition, cross-modal interaction at the perceptual level is not limited between them. Inputs into haptics can affect perceptual processes in the sensory modalities other than haptics. For example, the dominant interpretation for an ambiguous visual figure (e.g., the Necker cube)

significant (t(18) = 1.59, p = 0.13).

have a good impression of each other (Morris, 1971).

**4. Cross-modal effect of haptic stimulation in abstract feelings**

aware of that.

**3.3 Summary**

on auditory perception.

of the interrupted group, but the difference was not statistically significant (t(38) = 1.64, p = 0.11). It seemed that the subjective evaluation of the quality of the experience itself was less susceptible to the absence or presence of haptic communication.

As for the other questions, Q4, Q5, Q6, and Q7, the rating scores were comparable between the two experimental groups. Q4 and Q6 asked whether feelings were conveyed efficiently between the participants during the experiment. Q5 and Q7 asked about their involvement and agreement with the other person during the experiment. These questions were partially similar to Q3 and Q8 in that all the questions were related to the communication with the other person. However, while Q4, Q5, Q6, and Q7 focused on the feelings of the other person during the experiment, Q3 and Q8 referred to the strength of the intentions toward the other person in present and future communication. In summary, the interruption of haptic communication had little effect on the subjective evaluation of the shared experience with others. However, it modulated the impression of the other person, especially the intention to build a relationship with him or her.

The results of an exploratory factor analysis of the subjective evaluation were interesting, since one factor (Factor 1) affected Q2, Q3, and Q8, which showed the group difference in the rating score, and another factor (Factor 2) affected Q6 and Q7 (the loadings of Factor 2 for Q1 and Q4 were also comparatively high). The straightforward interpretation would be that the questions affected by Factor 1 were relevant to the attitude toward the other person, and that the questions affected by Factor 2 were relevant to the feeling of the experience itself. The questions related to Factor 2 did not show any group difference in the rating score. Taken together, the interruption of haptic communication did not degrade the retrospective impression of the experience itself but did reduce the participants' concern toward the other person with whom they communicated.

Fig. 6. Number of button presses. Error bars indicate the standard errors of means. The figure was modified from Takahashi et al. (2011b).

In addition to the subjective evaluation using the post-experiment questionnaire survey, the interruption of haptic communication affected the participants' behavior while watching the movie—namely, the number of button presses during the movie presentation. Fig. 6. shows the total number of button presses for three different periods. The first one was the period before the interruption of haptic stimulation; hence, there was inherently no difference between the two groups. The numbers of button presses did not differ between the two groups in the first period (t(18) = 0.62, p = 0.54). In the second period, the haptic stimulation was cut off between 75 and 165 s for the interrupted group. In this period, the number of button presses in the uninterrupted group was significantly larger than that in the interrupted group (t(18) = 2.65, p < 0.05). The third period was the one after the interruption. Although the number of actions was slightly larger in the uninterrupted group, the difference was not significant (t(18) = 1.59, p = 0.13).

In the questionnaire survey, we found that the retrospective evaluation of how hilarious the movie was did not significantly differ between the two groups (Q1). In contrast, the behavioral performance clearly showed that the number of button presses—namely, how often the participants felt the movie was hilarious—was decreased by the interruption of the haptic communication. These results suggest that the haptic communication implicitly modulated the quality of the experience, although the participants could not be explicitly aware of that.

### **3.3 Summary**

12 Will-be-set-by-IN-TECH

of the interrupted group, but the difference was not statistically significant (t(38) = 1.64, p = 0.11). It seemed that the subjective evaluation of the quality of the experience itself was less

As for the other questions, Q4, Q5, Q6, and Q7, the rating scores were comparable between the two experimental groups. Q4 and Q6 asked whether feelings were conveyed efficiently between the participants during the experiment. Q5 and Q7 asked about their involvement and agreement with the other person during the experiment. These questions were partially similar to Q3 and Q8 in that all the questions were related to the communication with the other person. However, while Q4, Q5, Q6, and Q7 focused on the feelings of the other person during the experiment, Q3 and Q8 referred to the strength of the intentions toward the other person in present and future communication. In summary, the interruption of haptic communication had little effect on the subjective evaluation of the shared experience with others. However, it modulated the impression of the other person, especially the intention to build a relationship

The results of an exploratory factor analysis of the subjective evaluation were interesting, since one factor (Factor 1) affected Q2, Q3, and Q8, which showed the group difference in the rating score, and another factor (Factor 2) affected Q6 and Q7 (the loadings of Factor 2 for Q1 and Q4 were also comparatively high). The straightforward interpretation would be that the questions affected by Factor 1 were relevant to the attitude toward the other person, and that the questions affected by Factor 2 were relevant to the feeling of the experience itself. The questions related to Factor 2 did not show any group difference in the rating score. Taken together, the interruption of haptic communication did not degrade the retrospective impression of the experience itself but did reduce the participants' concern toward the other

Period

During interruption (75 ~ 165 s)

Fig. 6. Number of button presses. Error bars indicate the standard errors of means. The

After interruption (165 s ~)

Uninterrupted Interrupted

susceptible to the absence or presence of haptic communication.

with him or her.

Count of button press

0

Before interruption (~ 75 s)

figure was modified from Takahashi et al. (2011b).

2

4

6

8

10

12

14

person with whom they communicated.

Both psychological surveys (Takahashi et al., 2011c; Takahashi, Mitsuhashi, Norieda, Sendoda, Murata & Watanabe, 2010) and the experimental study (Takahashi et al., 2011b) showed the importance of haptic stimulation in interpersonal communication. The desirability of haptic communication was associated with the factor relevant to abstract feelings (lightsome, pleasant, comfortable, and warm) (Takahashi et al., 2011c; Takahashi, Mitsuhashi, Norieda, Sendoda, Murata & Watanabe, 2010). The presence of haptic communication improved the quality of experience and the impression of the other person as well (Takahashi et al., 2011b). As the vibratory haptic stimulation could affect the quality of experience, the presence of haptic communication, not the contents of the stimulation, may be essential for influencing the quality of experience. Abstract feelings emerging from the haptic stimulation in interpersonal communication would be important for persons who share a high-quality experience and have a good impression of each other (Morris, 1971).

#### **4. Cross-modal effect of haptic stimulation in abstract feelings**

Humans have multiple sensory modalities for accessing the external world, such as vision, audition, and, of course, haptics. Although inputs into different sensory modalities are processed in different brain regions at early sensory levels, different sensory modalities are no doubt interactively processed at the perceptual level to build up unified perceptions of the external world, which is referred to as cross-modal interaction (Shams & Kim, 2010; Shimojo & Shams, 2001). For example, the ventriloquism effect (Alais & Burr, 2004) and McGurk effect McGurk & MacDonald (1976) are well known phenomena demonstrating visual modulation on auditory perception.

Although cross-modal interaction has mainly been examined using vision and audition, cross-modal interaction at the perceptual level is not limited between them. Inputs into haptics can affect perceptual processes in the sensory modalities other than haptics. For example, the dominant interpretation for an ambiguous visual figure (e.g., the Necker cube)

and two trials without haptic stimulus. The order of trials and the contents of the conversation

Abstract Feelings Emerging from Haptic Stimulation 111

Fig. 7. Rating scores of abstract feelings for voice with and without haptic stimulus. The error

Fig. 7. shows the mean rating scores of 5 questions. The feeling of animacy was significantly stronger when the voice was accompanied by the haptic stimulus than when the voice was presented alone (one-tailed paired t-test, t(23) = 1.87, p < .05). As for the other questions, although the means rating score was slightly higher when the haptic stimulus was present, the difference between the haptic-present and the haptic-absent trial was not statistically significant (Pleasant: t(23) = 0.85, p = .20, Warm: t(23) = 0.50, p = .31, Familiar: t(23) = 0.82, p

Thus, the results of the experiment clearly showed that the feeling of voice animacy increased when haptic vibratory stimulation was added to the voice stimuli. Even when the haptic stimulation was task irrelevant and the observers were explicitly told to ignore it, the haptic stimulation modulated the abstract feelings for voice stimuli. As described in Section 2, Takahashi et al. (2011a) and Takahashi, Mitsuhashi, Norieda, Murata & Watanabe (2010) showed that a strong feeling of animacy emerged from 1–2Hz of haptic vibratory stimulation. The fact that only the feeling of animacy, that was expected to emerge from haptic stimulatin, was manifest, out of 5 abstract feelings, seems to demonstrate that the effect was not due to

We investigated if haptic stimulation can modulate abstract feelings for voice stimuli. The results showed that haptic stimulation that is associated with feelings of animacy can increase the felt animacy for voice stimuli. Perhaps the abstract feeling emerging from haptic

These results suggested that the processes concerning abstract feelings, as well as perceptual (Shams & Kim, 2010; Shimojo & Shams, 2001) and emotional (Bertelson & de Gelder, 2005; Tanaka et al., 2010) processes, work interactively among different sensory modalities. It may be plausible to argue that higher levels of cognitive processes have vaguer boundaries

general bias in rating higher when the voice was accompanied by haptic stimulus.

stimulation interacts with inputs into sensory modalities other than haptics.

**Haptic Stimulus** Absent Present

Animate Pleasant Warm Familiar Attractive

were randomized.

Rating Score

3.5

**4.2 Summary**

bars indicate standard errors of means.

= .21, Attractive: t(23) = 0.33, p = .37).

between different sensory modalities (Fig. 1.).

4.0

4.5

5.0

5.5

is biased by haptic inputs to make visual and haptic perception consistent (Blake et al., 2004; Bruno et al., 2007). Furthermore, inputs into haptics and vision are integrated in a statistically optimal fashion (Ernst & Banks, 2002; Ernst & Bülthoff, 2004). Thus, a good amount of experimental evidence has shown that inputs into haptic modality can affect perceptual processes in the modalities other than haptics beyond the boundary of sensory modalities.

Consequently, can abstract feelings emerging from different sensory modalities also interact with or modulate from each other? Recent experimental studies suggest that cross-modal interaction takes place at quite high levels of the perceptual process, such as emotion recognition (Bertelson & de Gelder, 2005; Tanaka et al., 2010). For example, face emotion and voice emotion are strongly related. Emotion recognition of facial expression is biased when the voice expresses emotion different from that of the face, and vice versa (Bertelson & de Gelder, 2005).

Although it is true that perceptual and emotional processes interact with each other between different sensory modalities, it is still open if abstract feelings such as animacy can be shared among multiple sensory modalities. However, in daily experience, it seems that such abstract feelings simultaneously emerge from multiple sensory modalities. For example, a mother and a baby communicate through eyes, ears, and hands to feel presence or animacy from each other. Therefore, it would be possible to expect that cross-modal interaction also appears for the abstract feelings. In the following section, we introduce an original research recently conducted in our laboratory, which examined if haptic stimulation can modulate abstract feelings for voice stimuli.

#### **4.1 Voice perception modified by haptic sensation**

In order to investigate if the abstract feeling emerging from haptic sensation can affect the cognitive process of sensory modalities other than haptics, we conducted an experiment wherein observers evaluated abstract feelings for another person's voice. In the experiment, the voice stimuli either were or were not accompanied by haptic stimulation.

Twenty-four paid volunteers took part in the experiment. All of the participants reported normal hearing and haptic sensitivity. They were naive as to the purpose of the experiment.

The haptic stimulus was 2Hz sinusoidal vibratory stimulation presented by the custom-made haptic display device shown in Fig. 2. Voice stimuli were presented though headphones. The voice stimuli were 5 Japanese greeting phrases from daily conversation ("Ohayou," "Irasshai-mase," "Arigatou-gozaimashita," "Oyasumi-nasai," and "Otsukare-sama-deshita," which correspond to "Good morning," "Can I help you?" "Thank you," "Good night," and "That's wrap," respectively). The voices were spoken by 5 professional voice actors and were emotionally-neutral.

In a trial, the participants touched the haptic device with their right index fingers. After the participants pressed the space bar of an operator PC, the voice stimulus was presented. In half of the trials, the voice stimuli were accompanied by the haptic vibratory stimulation. The haptic stimulus was presented for 3 s and the voice stimuli were presented during the haptic stimulation. In the other half of the trials, the haptic stimulation was not presented and only the voice stimuli were presented.

After the stimulus presentation, the participants were asked to rate how strongly the voice yielded various abstract feelings—animate, pleasant, warm, familiar, or attractive—on a 7-point scale. They were explicitly told to ignore the contents of the conversation as well as the haptic stimulation. One practical trial was followed by two trials with haptic stimulus 14 Will-be-set-by-IN-TECH

is biased by haptic inputs to make visual and haptic perception consistent (Blake et al., 2004; Bruno et al., 2007). Furthermore, inputs into haptics and vision are integrated in a statistically optimal fashion (Ernst & Banks, 2002; Ernst & Bülthoff, 2004). Thus, a good amount of experimental evidence has shown that inputs into haptic modality can affect perceptual processes in the modalities other than haptics beyond the boundary of sensory modalities. Consequently, can abstract feelings emerging from different sensory modalities also interact with or modulate from each other? Recent experimental studies suggest that cross-modal interaction takes place at quite high levels of the perceptual process, such as emotion recognition (Bertelson & de Gelder, 2005; Tanaka et al., 2010). For example, face emotion and voice emotion are strongly related. Emotion recognition of facial expression is biased when the voice expresses emotion different from that of the face, and vice versa (Bertelson &

Although it is true that perceptual and emotional processes interact with each other between different sensory modalities, it is still open if abstract feelings such as animacy can be shared among multiple sensory modalities. However, in daily experience, it seems that such abstract feelings simultaneously emerge from multiple sensory modalities. For example, a mother and a baby communicate through eyes, ears, and hands to feel presence or animacy from each other. Therefore, it would be possible to expect that cross-modal interaction also appears for the abstract feelings. In the following section, we introduce an original research recently conducted in our laboratory, which examined if haptic stimulation can modulate abstract

In order to investigate if the abstract feeling emerging from haptic sensation can affect the cognitive process of sensory modalities other than haptics, we conducted an experiment wherein observers evaluated abstract feelings for another person's voice. In the experiment,

Twenty-four paid volunteers took part in the experiment. All of the participants reported normal hearing and haptic sensitivity. They were naive as to the purpose of the experiment. The haptic stimulus was 2Hz sinusoidal vibratory stimulation presented by the custom-made haptic display device shown in Fig. 2. Voice stimuli were presented though headphones. The voice stimuli were 5 Japanese greeting phrases from daily conversation ("Ohayou," "Irasshai-mase," "Arigatou-gozaimashita," "Oyasumi-nasai," and "Otsukare-sama-deshita," which correspond to "Good morning," "Can I help you?" "Thank you," "Good night," and "That's wrap," respectively). The voices were spoken by 5 professional voice actors and were

In a trial, the participants touched the haptic device with their right index fingers. After the participants pressed the space bar of an operator PC, the voice stimulus was presented. In half of the trials, the voice stimuli were accompanied by the haptic vibratory stimulation. The haptic stimulus was presented for 3 s and the voice stimuli were presented during the haptic stimulation. In the other half of the trials, the haptic stimulation was not presented and only

After the stimulus presentation, the participants were asked to rate how strongly the voice yielded various abstract feelings—animate, pleasant, warm, familiar, or attractive—on a 7-point scale. They were explicitly told to ignore the contents of the conversation as well as the haptic stimulation. One practical trial was followed by two trials with haptic stimulus

the voice stimuli either were or were not accompanied by haptic stimulation.

de Gelder, 2005).

feelings for voice stimuli.

emotionally-neutral.

the voice stimuli were presented.

**4.1 Voice perception modified by haptic sensation**

and two trials without haptic stimulus. The order of trials and the contents of the conversation were randomized.

Fig. 7. Rating scores of abstract feelings for voice with and without haptic stimulus. The error bars indicate standard errors of means.

Fig. 7. shows the mean rating scores of 5 questions. The feeling of animacy was significantly stronger when the voice was accompanied by the haptic stimulus than when the voice was presented alone (one-tailed paired t-test, t(23) = 1.87, p < .05). As for the other questions, although the means rating score was slightly higher when the haptic stimulus was present, the difference between the haptic-present and the haptic-absent trial was not statistically significant (Pleasant: t(23) = 0.85, p = .20, Warm: t(23) = 0.50, p = .31, Familiar: t(23) = 0.82, p = .21, Attractive: t(23) = 0.33, p = .37).

Thus, the results of the experiment clearly showed that the feeling of voice animacy increased when haptic vibratory stimulation was added to the voice stimuli. Even when the haptic stimulation was task irrelevant and the observers were explicitly told to ignore it, the haptic stimulation modulated the abstract feelings for voice stimuli. As described in Section 2, Takahashi et al. (2011a) and Takahashi, Mitsuhashi, Norieda, Murata & Watanabe (2010) showed that a strong feeling of animacy emerged from 1–2Hz of haptic vibratory stimulation. The fact that only the feeling of animacy, that was expected to emerge from haptic stimulatin, was manifest, out of 5 abstract feelings, seems to demonstrate that the effect was not due to general bias in rating higher when the voice was accompanied by haptic stimulus.

#### **4.2 Summary**

We investigated if haptic stimulation can modulate abstract feelings for voice stimuli. The results showed that haptic stimulation that is associated with feelings of animacy can increase the felt animacy for voice stimuli. Perhaps the abstract feeling emerging from haptic stimulation interacts with inputs into sensory modalities other than haptics.

These results suggested that the processes concerning abstract feelings, as well as perceptual (Shams & Kim, 2010; Shimojo & Shams, 2001) and emotional (Bertelson & de Gelder, 2005; Tanaka et al., 2010) processes, work interactively among different sensory modalities. It may be plausible to argue that higher levels of cognitive processes have vaguer boundaries between different sensory modalities (Fig. 1.).

**6. References**

pp. 151–177.

pp. 334–337.

and Winston.

Alais, D. & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal

Abstract Feelings Emerging from Haptic Stimulation 113

Bertelson, P. & de Gelder, B. (2005). Psychology of multimodal perception, *in* C. Spence &

Blake, R., Sobel, K. V. & James, T. W. (2004). Neural synergy between kinetic vision and touch,

Blakemore, S.-J., Boyer, P., Pachot-Clouard, M., Meltzoff, A., Segebarth, C. & Decety, J. (2003).

Bruno, N., Jacomuzzi, A., Bertamini, M. & Meyer, G. (2007). A visual-haptic Necker cube

Ernst, M. O. & Banks, M. S. (2002). Humans integrate visual and haptic information in a

Ernst, M. O. & Bülthoff, H. H. (2004). Merging the senses into a robust percept, *Trends in*

Fujita, Y. & Hashimoto, S. (1999). Experiments of haptic and tactile display for human

Fukuda, H. & Ueda, K. (2010). Interaction with a Moving Object Affects One's Perception of

Gallese, V. & Goldman, A. (1998). Mirror neurons and the simulation theory of mind-reading,

Gao, T., McCarthy, G. & Scholl, B. J. (2010). The wolfpack effect. Perception of animacy irresistibly influences interactive behavior., *Psychological science* 21(12): 1845–1853. Gao, T., Newman, G. E. & Scholl, B. J. (2009). The psychophysics of chasing: A case study in

Gao, T. & Scholl, B. J. (2011). Chasing vs. stalking: Interrupting the perception of animacy., *Journal of experimental psychology: Human perception and performance* 37(3): 669–684.

Golding, J. & Macleod, C. (1998). Intentional Forgetting: Interdisciplinary Approaches,

Heider, F. & Simmel, M. (1944). An Experimental Study of Apparent Behavior, *The American*

Knapp, L. M. & Hall, A. J. (1992). *Nonverbal communication in human interaction*, Holt Rinehart

McGurk, H. & MacDonald, J. (1976). Hearing lips and seeing voices, *Nature* 264(5588): 746–748. Morito, Y., Tanabe, H. C., Kochiyama, T. & Sadato, N. (2009). Neural representation of animacy

Reiner, M. (2004). The role of haptics in immersive telecommunication environments, *IEEE Transactions on Circuits and Systems for Video Technology* 14(3): 392–401. Richmond, V. P., McCroskey, J. C. & Hickson, M. L. (2007). *Nonverbal Behavior in Interpersonal*

in the early visual areas: a functional MRI study, *Brain research bulletin* 79(5): 271–280.

Its Animacy, *International Journal of Social Robotics* 2(2): 187–193.

the perception of animacy, *Cognitive psychology* 59(2): 154–179.

Gibson, J. J. (1962). Observations on active touch., *Psychological Review* 69: 477–491.

brain, *Cerebral cortex (New York, NY : 1991)* 13(8): 837–844.

statistically optimal fashion, *Nature* 415(6870): 429–433.

J. Driver (eds), *Crossmodal space and crossmodal attention*, Oxford University Press,

The detection of contingency and animacy from simple animations in the human

reveals temporal constraints on intersensory merging during perceptual exploration,

telecommunication, *8th IEEE International Workshop on Robot and Human Interaction*,

integration, *Current biology* 14(3): 257–262.

*Psychological science* 15(6): 397–402.

*Neuropsychologia* 45(3): 469–475.

*cognitive sciences* 8(4): 162–169.

Lawrence Erlbaum Assoc Inc.

*Journal of Psychology* 57(2): 243–259. Katz, D. (1989). *The World of Touch*, Psychology Press.

*Relations*, 6 edn, Allyn & Bacon.

Morris, D. (1971). *Intimate Behavior*, 1st us edition edn, Triad Books.

*Trends in cognitive sciences* 2(12): 493–501.
