**4. Modulation of brain responses toward vocal expression by other nonverbal expressions**

One of the key questions in emotional communication is how decoding vocal information is aided by other nonverbal cues. The neurophysiological studies have focused on emotional processing when voice is paired with other nonverbal social cues (such as face). In a task when participants were asked to evaluate the actor's identity (e.g., monkey or not) rather than the emotion, the simultaneous presentation of vocal and facial expressions revealed some similar ERP correlates of emotional information as the vocal expression did [18]. The bimodal emotional cues elicited a larger P200 and P300 for happy and angry expressions and a larger N250 for neutral expression, suggesting that an implicit affective processing of audiovisual information emerges as early as 200 ms. Using a priming paradigm in which a face was fol‐ lowed by a vocal expression of words either congruent or incongruent with the emotion of the facial expression (happiness vs. anger), Diamond and Zhang [19] revealed that the mismatch elicited an increased N400 followed by a late positivity. Further, source localization of these two effects revealed activations in the superior temporal gyrus and inferior parietal gyrus dominated in the right hemisphere.

insincere one. This ERP effect was localized in the left insula which is associated with the

Recent growing evidence has been accumulated in the field of decoding of speaker's feeling of (un)knowing using event‐related potentials. In Jiang and Pell [15], vocal expression of con‐ fidence was manipulated such that statements which sounded very confident, somewhat con‐ fident, and unconfident and those which sounded neutral were presented to native English speakers. At the onset of the vocal expression, the confident expression elicited an increased positive response than the other two types of expressions. The unconfident expression elicited an increased P300 as compared with the confident and the neutral expression. The neutral voice produced a more‐delayed positivity as compared with all confidence‐intending expressions. Two follow‐up experiments further evaluated how the decoding of vocal confidence expres‐ sion is impacted by the presence of additional linguistic cues which either congruent [16] or incongruent [17] with the tone of voice in statements which followed the linguistic cues. Different from the statements with no lexical cues, statements with congruent cues (e.g., I'm sure; Maybe) elicited an increased N1 and P2 for confident than for unconfident and close‐to‐ confident expressions, and an enhanced delayed positivity in unconfident and close‐to‐con‐ fident expression than confident one. Moreover, the direct comparison between statements with and without a preceding lexical phrase elicited a reduced N1, P2, and N400 in those without a phrase [16]. The incongruent cues elicited different ERP effects at the onset of the main statement of confident and unconfident tones. The unconfident statement elicited an increased N400 or late positivity (depending on the listener's gender). The confident state‐ ment elicited a more delayed, sustained positivity effect. Source localization of these ERP effects revealed pre‐SMA for N400, suggesting a difficulty in accessing the speaker mean‐ ing, and SFG, STG and insula underlying the late positivity effect, suggesting an increased demand of executive control to implement the attentional resources and socioevaluative pro‐ cesses [17]. These studies extended the neurocognitive model for basic vocal emotion and argued for a perspective of studying the neurophysiological mechanisms underlying decod‐

action of lying and concealment.

50 Emotion and Attention Recognition Based on Biological Signals and Images

ing interpersonal and sociointeractive affective voice.

**nonverbal expressions**

**4. Modulation of brain responses toward vocal expression by other** 

One of the key questions in emotional communication is how decoding vocal information is aided by other nonverbal cues. The neurophysiological studies have focused on emotional processing when voice is paired with other nonverbal social cues (such as face). In a task when participants were asked to evaluate the actor's identity (e.g., monkey or not) rather than the emotion, the simultaneous presentation of vocal and facial expressions revealed some similar ERP correlates of emotional information as the vocal expression did [18]. The bimodal emotional cues elicited a larger P200 and P300 for happy and angry expressions and a larger N250 for neutral expression, suggesting that an implicit affective processing of audiovisual The interaction between vocal and other nonverbal emotional information was also exam‐ ined in detection of emotional change. In a study in which participants were presented with simultaneously presented vocal and facial expressions while being asked to detect the change of emotion from neutral to anger or happiness conveyed in voice or in face [20]. The P3 associ‐ ated with the detection of the emotional categorical change in both voice and face was larger than the sum of the change in single channel (see also [21]). The N1 associated with the detec‐ tion of early acoustic change was dependent on whether their attention was guided to the voice or the face, with the attention to the voice yielding to a N1 in bimodal change larger than the sum of the two single modal change conditions. These findings suggest the modulation of selection attention on voice‐face integration during emotional change perception in early sensory processing.
