Preface

After receiving the green lights from the InTech office, the invitations went out to the senior scholars in the field from January 2016. During this 1 year of intensive efforts, all the chapters were reviewed and revised accordingly to meet high-quality standards of InTech and my vision for the whole concept of the chapters. I envision that both neuroscientists and clinical investiga‐ tors will be the primary audience of this book. Moreover, the common interest of these individ‐ uals will be the application of cognitive neuroscience approaches in studies to assess or treat individuals with the related disorders based on emotion and attention.

The main focus of the book is based on emotion and attention recognition. This book is rela‐ tively brief but provides a comprehensive survey of different approaches for emotion recog‐ nition. Apart from this introductory chapter, this book has four more chapters (Chapters #2– 5). The rest of this introductory chapter is given in providing brief chapters and the impor‐ tance of the other proposed chapters.

Chapter 2: Multimodal Affect Recognition: Current Approaches and Challenges

This chapter provides an overview of emotion recognition, different approaches and chal‐ lenges, public multimodal emotional datasets, and applications of emotion recognition. I strongly encourage the young researchers to deeply study this chapter to get a bird's-eye view of emotion and attention recognition systems.

This chapter explains that numerous studies found multimodal methods to perform as good as or better than unimodal ones. However, the improvements of multimodal systems over unimodal ones are modest when affect detection is performed on spontaneous expressions in natural settings #[15]#. Also, multimodal methods introduce new challenges that have not fully been resolved. These challenges are discussed in this chapter.

Chapter 3: Human Automotive Interaction: Affect Recognition for Motor Trend Magazine's Best Driver Car of the Year

This chapter provides two important parts of the facial emotion recognition pipeline: (1) face detection and (2) facial appearance features. This chapter proposes a face detector that uni‐ fies state-of-the-art approaches and provides quality control for face detection results, called Reference-Based Face Detection. This chapter also proposes a method for facial feature ex‐ traction that compactly encodes the spatiotemporal behavior of the face and removes the background texture, called Local Anisotropic-Inhibited Binary Patterns in Three Orthogonal Planes (LAIBP-TOP). Real-world results show promise for the automatic observation of driver inattention and stress.

Chapter 4: Affective Valence Detection from EEG Signals using Wrapper Methods

This chapter provides a valence recognition system based on a wrapper classification algo‐ rithm using EEG signals. The feature extraction in short time intervals is based on measures of the relative energies computed and certain frequency bands of the EEG signals timelocked to the stimulus presentation. These measures represent event-related desynchroniza‐ tion/synchronization of underlying brain neural networks. The subsequent feature selection

and classification steps comprise a wrapper technique based on two different classification approaches: (1) an ensemble classifier and (2) a support vector machine classifier. The fea‐ ture reduction has been used to identify the most relevant features both for intrasubject and for intersubject settings, using single-trial signals and ensemble-averaged signals, respec‐ tively. The proposed approaches allowed to identify the frontal region and beta band as the most relevant characteristics, extracted from the electrical brain activity, in order to deter‐ mine the affective valence elicited by visual stimuli.

Chapter 5: Tracking the Sound of Human Affection: EEG Signals Reveal Online Decoding of Socioemotional Expression in Human Speech and Voice

This chapter provides a perspective from the latest EEG evidence on how the brain signals enlighten the neurophysiological and neurocognitive mechanisms underlying the recogni‐ tion of socioemotional expression conveyed in human speech and voice, drawing upon ERP studies. Human sound can encode emotional meanings by different vocal parameters in words, real- vs. pseudospeeches, and vocalizations. Based on the ERP findings, recent devel‐ opment of the three-stage model in vocal processing has highlighted initial and late-stage processing of vocal emotional stimuli. These processes, depending on which ERP compo‐ nents they were mapped onto, can be divided into the acoustic analysis, relevance and moti‐ vational processing, fine-grained meaning analysis/integration/access, and higher-level social inference, as the unfolding of the time scale. ERP studies on vocal socioemotion, such as happiness, anger, fear, sadness, neutral, sincerity, confidence, and sarcasm in the human voice and speech, have employed different experimental paradigms such as cross-splicing, cross-modality priming, oddball, stroop, etc. Moreover, task demand and listener character‐ istics affect the neural responses underlying the decoding processes, revealing the role of attention deployment and interpersonal sensitivity in the neural decoding of vocal emotion‐ al stimuli. Culture affects our ability to decode emotional meaning in the voice. Neurophy‐ siological patterns were compared between normal and abnormal emotional processing in the vocal expressions, especially schizophrenia and congenital amusia. Future directions will merit the study of human vocal expression aligning with other nonverbal cues, such as facial and body language, and the need to synchronize listener's brain potentials with other peripheral measures.

This book will provide the audiences with most recent evidences from different disciplines in brain studies on the wide range of researches in an integrative way toward *Emotion and Attention Recognition Based on Biological Signals and Images*. The hope is that the information provided in this book will trigger new researches that will help to connect basic cognitive neuroscience to clinical medicine.

### **Acknowledgment**

Seyyed-Abed would like to thank Ms. Iva Simcic for her valuable comments and sugges‐ tions to improve the quality of this book.

> **Dr. Seyyed Abed Hosseini** Islamic Azad University, Mashhad, Iran

### **Introductory Chapter: Emotion and Attention Recognition Based on Biological Signals and Images Introductory Chapter: Emotion and Attention Recognition Based on Biological Signals and Images**

Seyyed Abed Hosseini

and classification steps comprise a wrapper technique based on two different classification approaches: (1) an ensemble classifier and (2) a support vector machine classifier. The fea‐ ture reduction has been used to identify the most relevant features both for intrasubject and for intersubject settings, using single-trial signals and ensemble-averaged signals, respec‐ tively. The proposed approaches allowed to identify the frontal region and beta band as the most relevant characteristics, extracted from the electrical brain activity, in order to deter‐

Chapter 5: Tracking the Sound of Human Affection: EEG Signals Reveal Online Decoding of

This chapter provides a perspective from the latest EEG evidence on how the brain signals enlighten the neurophysiological and neurocognitive mechanisms underlying the recogni‐ tion of socioemotional expression conveyed in human speech and voice, drawing upon ERP studies. Human sound can encode emotional meanings by different vocal parameters in words, real- vs. pseudospeeches, and vocalizations. Based on the ERP findings, recent devel‐ opment of the three-stage model in vocal processing has highlighted initial and late-stage processing of vocal emotional stimuli. These processes, depending on which ERP compo‐ nents they were mapped onto, can be divided into the acoustic analysis, relevance and moti‐ vational processing, fine-grained meaning analysis/integration/access, and higher-level social inference, as the unfolding of the time scale. ERP studies on vocal socioemotion, such as happiness, anger, fear, sadness, neutral, sincerity, confidence, and sarcasm in the human voice and speech, have employed different experimental paradigms such as cross-splicing, cross-modality priming, oddball, stroop, etc. Moreover, task demand and listener character‐ istics affect the neural responses underlying the decoding processes, revealing the role of attention deployment and interpersonal sensitivity in the neural decoding of vocal emotion‐ al stimuli. Culture affects our ability to decode emotional meaning in the voice. Neurophy‐ siological patterns were compared between normal and abnormal emotional processing in the vocal expressions, especially schizophrenia and congenital amusia. Future directions will merit the study of human vocal expression aligning with other nonverbal cues, such as facial and body language, and the need to synchronize listener's brain potentials with other

This book will provide the audiences with most recent evidences from different disciplines in brain studies on the wide range of researches in an integrative way toward *Emotion and Attention Recognition Based on Biological Signals and Images*. The hope is that the information provided in this book will trigger new researches that will help to connect basic cognitive

Seyyed-Abed would like to thank Ms. Iva Simcic for her valuable comments and sugges‐

**Dr. Seyyed Abed Hosseini** Islamic Azad University,

Mashhad, Iran

mine the affective valence elicited by visual stimuli.

peripheral measures.

VIII Preface

**Acknowledgment**

neuroscience to clinical medicine.

tions to improve the quality of this book.

Socioemotional Expression in Human Speech and Voice

Additional information is available at the end of the chapter Seyyed Abed Hosseini

http://dx.doi.org/10.5772/66483 Additional information is available at the end of the chapter

**1. Emotion and attention recognition based on biological signals and images**

This chapter will attempt to introduce the different approaches for recognition of emotional and attentional states, from a historical development, focusing particularly on the recent development of the field and its specialization within psychology, cognitive neuroscience, and engineering. The basic idea of this book is to present a common framework for the neuroscientists from diverse backgrounds in the cognitive neuroscience to illustrate their theoretical and applied research findings in emotion, stress, and attention.

Biological signal processing and medical image processing have helped greatly in understanding the below-mentioned cognitive processes. Up to now, researchers and neuroscientists have studied continuously to improve the performances of the emotion and attention recognition systems (e.g., [1–10]). In spite of all of these efforts, there is still an abundance of scope for the additional researches in emotion and attention recognition based on biological signals and images. In the meantime, interpreting and modeling the notions of the brain activity, especially emotion and attention, through soft computing approaches is a challenging problem.

Emotions and attentions have an important role in our daily lives [11]. They definitely make life more challenging and interesting; however, they provide useful actions and functions that we seldom think about. Emotion and attention, due to its considerable influence on many brain activities, are important topics in the cognitive neurosciences, psychology, and biomedical engineering. These cognitive processes are core to human cognition and accessing it and being able to act have important applications ranging from basic science to applied science.

'Emotion' has many medical applications such as voice intonation, rehabilitation, autism, music therapy, and many engineering applications such as brain-computer interface (BCI),

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2017 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

human-computer interaction (HCI), facial expression, body languages, neurofeedback, marketing, law, and robotics. In addition, 'attention' has many medical applications such as rehabilitation, autism, attention deficit disorder (ADD), attention deficit hyperactivity disorder (ADHD), attention-seeking personality disorder, and many engineering applications such as BCI, neurofeedback, decision-making, learning, and robotics.

Up to now, different definitions have been presented for the emotion and attention. According to most researchers, attention phenomenon and emotion phenomenon are not well-defined words. Kleinginna and her colleagues collected and analyzed 92 different definitions of emotion, then they made a decision that "*emotion is a complex set of interactions among subjective and objective factors*, *mediated by neural or hormonal systems* [12]." In addition, Solso [13] said that attention is "*the concentration of mental effort on sensory*/*mental events*." In another definition, the attention function is defined as "*a cognitive brain mechanism that enables one to process relevant inputs*, *thoughts*, *or actions*, *whilst ignoring irrelevant or distracting ones* [14]."

In different researches, suitable techniques are usually used according to invasive or noninvasive acquisition techniques. Invasive techniques often lead to efficient systems. However, they have inherent technical difficulties such as the risks associated with surgical implantation of electrodes, stricter ethical requirements, and the fact that in humans, this can only be done in patients undergoing surgery. Therefore, noninvasive techniques such as electroencephalography (EEG), magnetoencephalography (MEG), event-related potentials (ERPs), and functional magnetic resonance imaging (fMRI) are generally preferred.
