**Equivalent-Current-Dipole-Source-Localization-Based BCIs with Motor Imagery**

Toshimasa Yamazaki, Maiko Sakamoto, Shino Takata, Hiromi Yamaguchi, Kazufumi Tanaka, Takahiro Shibata, Hiroshi Takayanagi, Ken-ichi Kamijo and Takahiro Yamanoi

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/55805

**1. Introduction**

18 Brain-Computer Interface

University of London, 1991.

*March 18-23 2005, Philadelphia*, volume 1, 2005.

154 Brain-Computer Interface Systems – Recent Progress and Future Prospects

years. *Psychophysiology*, 41(1):30–36, 2004.

[36] Arlene Duncan, Judith H Meek, Matthew Clemence, Clare E Elwell, Lidia Tyszczuk, Mark Cope, and David T Delpy. Optical pathlength measurements on adult head, calf and forearm and the head of the newborn infant using phase resolved optical

[37] Mark Cope. *The development of a near infrared spectroscopy system and its application for non invasive monitoring of cerebral blood and tissue oxygenation in the newborn infant*. Ph.d.,

[38] Lawrence R. Rabiner. A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. *Proceedings of the IEEE*, 77(2):257–286, February 1989.

[39] R. Hu, X. Li, and Y. Zhao. Acoustic model training using greedy EM. In *proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP'05,*

[40] Kevin Murphy. Hidden Markov Model (HMM) Toolbox for Matlab.

[41] S. Luu and T. Chau. Decoding subjective preference from single-trial near-infrared

[42] V. Gumenyuk, O. Korzyukov, K. Alho, C. Escera, and R. Naatanen. Effects of auditory distraction on electrophysiological brain activity and performance in children aged 8-13

spectroscopy. *Physics in Medicine and Biology*, 40:295–304, 1995.

http://www.cs.ubc.ca/∼murphyk/Software/HMM/hmm.html.

spectroscopy signals. *Journal of Neural Engineering*, 6(4), 2009.

This chapter will propose a new paradigm for single-trial-electroencephalogram (EEG)-based Brain-Computer Interfaces (BCIs) with motor imagery (MI) [1] tasks. Among such BCIs, the sensorimotor rhythm (SMR)-based ones, when using common spatial patterns (CSPs), require features over broad frequency bands, such as mu, beta and gamma rhythms [2]. Therefore, very high-dimensional feature vectors and continuous-valued patterns necessary for spatio‐ temporally checking the features [3,4] could yield an enormous amount of data and much computational time [5]. So, various data reduction such as downsampling [6,7] and optimal EEG channel configuration [8,9,10] have been investigated for the BCIs.

The present method consists of 1) the categorization of single-trial EEGs as data reduction, and 2) the classifiers for the categorical data. 1) is realized by equivalent current dipole source localization (ECDL) after independent component analysis (ICA). For 2), we have been applying both Hayashi's second method of quantification (H2MQ) and Bayesian network model (BNM) to the ECDL-based categorical data. For the former, we have obtained the good accuracy, for example, the accuracy average across all the ten subjects for left- and right-hand imageries in each 10-trial validation was more than 90 % [11].

This chapter addresses itself to the single-trial-EEG-based BCI using the BNM and to the generalization to dynamic BNM (DBNM) because of the time-varying functional networks in the brain. For the purposes, two experiments were conducted to obtain single-trial EEGs scalp-

© 2013 Yamazaki et al.; licensee InTech. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2013 Yamazaki et al.; licensee InTech. This is a paper distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

recorded during the MI tasks and movement-related potentials (MRPs) including the Bereit‐ schaftspotential (BP) [12].

Recently, neuroscience has been attempting to take in various methodologies in network science, because the brain could be considered to be a kind of complex systems forming networks of interacting components, and the collective actions of the components, that is, individual neurons, linked by a dense web of intricate connectivity [13]. In addition to the network approach, one of the applied researches in neuroscience, the BCI, has extensive‐ ly received probabilistic approaches whose aims are mainly two. One is to cope with nonstationarities in EEG signals such intertrial and intersubject variations, and the other to incorporate time-varying brain states and uncertainties into BCI design. For the former aim, adaptive classifications were executed by Kalman filtering [14,15], while the DBN ach‐ ieved the latter one [16]. Micheloyannis et al. [17] analyzed multi-channel EEGs using graph theory. However, because the nodes are electrode positions, they have few functional meanings. In addition, all the above methods had been throughout applied to continuousvalued data.

A BN could be one of graphical models in neuroscience, where brain connectivity would be quantified by conditional probabilities. However, the existing graphical models require a large-scale anatomical data [18] and huge quantities of diffusion MRI data [19,20]. In this study, the brain connectivity will be calculated from the neural activity data even less than that in the above graphical models.

Figure 1 shows a typical BN, showing both the topology and the conditional probability tables (CPTs), given the joint probability distribution:

where *X*i(*i*=1,…,5) (nodes) are random variables whose values could be 0 or 1, and *B*<sup>S</sup> represents the BN topology, and Table 1 depicts 20 sample data generated from the BN model (BNM) [21]. The BN construction refers to that the topology of BNMs is estimated from such data in Table 1. In this study, BNM and DBNM consist of functionally distinct sites of the brain as nodes and directed relationships among these sites as edges, each of which is accompanied by conditional probabilities.

In order to predict the tasks which would have been executed by the subjects, in particular to discriminate between left- and right-hand imageries, the conditional probabilities at all the nodes in the BNM must be calculated for each trial. For the purpose, the probabilistic inference is made by the belief propagation [21], where the ECDL results for each trial correspond to the evidences. Hereafter, *node activities* are defined to be the summation of conditional probabilities at each node. Based on the node activities, a rule is proposed to classify into left- and righthand imageries. This classification rule will be examined by the statistical tests.

Moreover, our method will be validated and compared with the existing one with the best performance, called common spatial pattern (CSP) [22]. In the section 3, MRPs will be modeled by the DBNM. Finally, we will mainly mention future perspectives.

**Figure 1.** An example for causal model of Bayesian Network [21].

recorded during the MI tasks and movement-related potentials (MRPs) including the Bereit‐

Recently, neuroscience has been attempting to take in various methodologies in network science, because the brain could be considered to be a kind of complex systems forming networks of interacting components, and the collective actions of the components, that is, individual neurons, linked by a dense web of intricate connectivity [13]. In addition to the network approach, one of the applied researches in neuroscience, the BCI, has extensive‐ ly received probabilistic approaches whose aims are mainly two. One is to cope with nonstationarities in EEG signals such intertrial and intersubject variations, and the other to incorporate time-varying brain states and uncertainties into BCI design. For the former aim, adaptive classifications were executed by Kalman filtering [14,15], while the DBN ach‐ ieved the latter one [16]. Micheloyannis et al. [17] analyzed multi-channel EEGs using graph theory. However, because the nodes are electrode positions, they have few functional meanings. In addition, all the above methods had been throughout applied to continuous-

A BN could be one of graphical models in neuroscience, where brain connectivity would be quantified by conditional probabilities. However, the existing graphical models require a large-scale anatomical data [18] and huge quantities of diffusion MRI data [19,20]. In this study, the brain connectivity will be calculated from the neural activity data even less than that in the

Figure 1 shows a typical BN, showing both the topology and the conditional probability tables

where *X*i(*i*=1,…,5) (nodes) are random variables whose values could be 0 or 1, and *B*<sup>S</sup> represents the BN topology, and Table 1 depicts 20 sample data generated from the BN model (BNM) [21]. The BN construction refers to that the topology of BNMs is estimated from such data in Table 1. In this study, BNM and DBNM consist of functionally distinct sites of the brain as nodes and directed relationships among these sites as edges, each of which is accompanied by conditional

In order to predict the tasks which would have been executed by the subjects, in particular to discriminate between left- and right-hand imageries, the conditional probabilities at all the nodes in the BNM must be calculated for each trial. For the purpose, the probabilistic inference is made by the belief propagation [21], where the ECDL results for each trial correspond to the evidences. Hereafter, *node activities* are defined to be the summation of conditional probabilities at each node. Based on the node activities, a rule is proposed to classify into left- and right-

Moreover, our method will be validated and compared with the existing one with the best performance, called common spatial pattern (CSP) [22]. In the section 3, MRPs will be modeled

hand imageries. This classification rule will be examined by the statistical tests.

by the DBNM. Finally, we will mainly mention future perspectives.

schaftspotential (BP) [12].

156 Brain-Computer Interface Systems – Recent Progress and Future Prospects

valued data.

probabilities.

above graphical models.

(CPTs), given the joint probability distribution:


**Table 1.** Data example [21] generated by the BN shown in Figure 1.

#### **2. Bayesian network models for single-trial-EEG-based BCI**

#### **2.1. Materials and methods**

*Subjects.* Ten healthy subjects (two females and eight males; mean age: 28.4 ± 4.27 years) participated in this experiment. All the subjects were right-handed according to the Edinburgh Inventory [23]. Some of the subjects were paid volunteers and received 2000 Yen (about US 25\$).

*EEG data acquisition.* The subjects were seated inside an electrically shielded room with sound attenuation, and gazed at a monochromatic monitor of an AV tachistscope (IS-701B, IWATSU ISEL) 0.9 m away from their eyes. They were requested to relax their both hands on a table and with their chins on a chinrest (Figure 2A). The present experiment used a visual oddball paradigm, and three kinds of line drawings of hands were presented on the monitor: (1) righthand stimulus to imagine being shaken with the subject's right hand, (2) left-hand one for the subject's left hand imagery and (3) open-right-hand one as control (Fig.2B). These stimuli were sequentially and randomly presented with probabilities of 0.20, 0.20 and 0.60, respectively. That is, (1) and (2) are rare targets or rare non-targets, and (3) is frequent non-target. We tried four conditions consisting of movement imagery of right hand, left one and the actual movements, where each condition was separately carried out. Names of the conditions and the instruction to subjects were as follows: R-MIRP (right-hand-movement-imagery-related potential) condition is to imagine grasping the right-hand stimulus with her or his own right hand as soon as possible when the stimulus was displayed; L-MIRP (left-hand-MIRP) condi‐ tion to image grasping the left-hand stimulus with her or his own when the stimulus was presented; R-MRP (right-hand-movement-related potential) condition to actually grasp and loosen her or his own right palm as soon as possible if right-hand stimuli as the rare targets were displayed; L-MRP (left-hand-MRP) condition to grasp and loosen her or his palm when left-hand stimuli as the rare targets were presented. Both hands were hidden under a black coverlet so that it is easier for the subjects to imagine the hand movement. There was the following training session. At first, each subject was instructed to actually reach the monitor for the line drawing, then to shake (the stimulus) by her or his hand, and to practice the task scores of times. Then, after covering both of the hands with the coverlet, the subject was requested to imagine being shaken and to practice the MI tasks scores of times. One test session includes all the four conditions with a five-minute break between the conditions, where each condition contains 130, 130 and 400 trials of rare targets, rare non-targets and frequent nontargets, respectively. Therefore, it took about 90 minutes to finish one session. Note that different subject had different order of the conditions. This study addressed itself to only the L- and R-MIRP conditions, while the L- and R-MRP ones will be used in our another research in future.

With an electro cap (ECI, Electrocap International), EEG was from 32 electrodes (FP1, FPz, FP3, F7, F3, Fz, F4, F8, FC5, FC1, FC2, FC6, T3, C3, Cz, C4, T4, CP5, CP1, CPz, CP2, CP6, T5, P3, Pz, P4, T6, PO3, POz, PO4, O1, Oz, O2) defined on the basis of the International 10-20 System [24]. All the electrodes were referred to A1, the ground electrode was attached to FPz and their impedances were kept below 5kΩ. Vertical and horizontal eye movements were monitored

**2. Bayesian network models for single-trial-EEG-based BCI**

158 Brain-Computer Interface Systems – Recent Progress and Future Prospects

*Subjects.* Ten healthy subjects (two females and eight males; mean age: 28.4 ± 4.27 years) participated in this experiment. All the subjects were right-handed according to the Edinburgh Inventory [23]. Some of the subjects were paid volunteers and received 2000 Yen (about US

*EEG data acquisition.* The subjects were seated inside an electrically shielded room with sound attenuation, and gazed at a monochromatic monitor of an AV tachistscope (IS-701B, IWATSU ISEL) 0.9 m away from their eyes. They were requested to relax their both hands on a table and with their chins on a chinrest (Figure 2A). The present experiment used a visual oddball paradigm, and three kinds of line drawings of hands were presented on the monitor: (1) righthand stimulus to imagine being shaken with the subject's right hand, (2) left-hand one for the subject's left hand imagery and (3) open-right-hand one as control (Fig.2B). These stimuli were sequentially and randomly presented with probabilities of 0.20, 0.20 and 0.60, respectively. That is, (1) and (2) are rare targets or rare non-targets, and (3) is frequent non-target. We tried four conditions consisting of movement imagery of right hand, left one and the actual movements, where each condition was separately carried out. Names of the conditions and the instruction to subjects were as follows: R-MIRP (right-hand-movement-imagery-related potential) condition is to imagine grasping the right-hand stimulus with her or his own right hand as soon as possible when the stimulus was displayed; L-MIRP (left-hand-MIRP) condi‐ tion to image grasping the left-hand stimulus with her or his own when the stimulus was presented; R-MRP (right-hand-movement-related potential) condition to actually grasp and loosen her or his own right palm as soon as possible if right-hand stimuli as the rare targets were displayed; L-MRP (left-hand-MRP) condition to grasp and loosen her or his palm when left-hand stimuli as the rare targets were presented. Both hands were hidden under a black coverlet so that it is easier for the subjects to imagine the hand movement. There was the following training session. At first, each subject was instructed to actually reach the monitor for the line drawing, then to shake (the stimulus) by her or his hand, and to practice the task scores of times. Then, after covering both of the hands with the coverlet, the subject was requested to imagine being shaken and to practice the MI tasks scores of times. One test session includes all the four conditions with a five-minute break between the conditions, where each condition contains 130, 130 and 400 trials of rare targets, rare non-targets and frequent nontargets, respectively. Therefore, it took about 90 minutes to finish one session. Note that different subject had different order of the conditions. This study addressed itself to only the L- and R-MIRP conditions, while the L- and R-MRP ones will be used in our another research

With an electro cap (ECI, Electrocap International), EEG was from 32 electrodes (FP1, FPz, FP3, F7, F3, Fz, F4, F8, FC5, FC1, FC2, FC6, T3, C3, Cz, C4, T4, CP5, CP1, CPz, CP2, CP6, T5, P3, Pz, P4, T6, PO3, POz, PO4, O1, Oz, O2) defined on the basis of the International 10-20 System [24]. All the electrodes were referred to A1, the ground electrode was attached to FPz and their impedances were kept below 5kΩ. Vertical and horizontal eye movements were monitored

**2.1. Materials and methods**

25\$).

in future.

**Figure 2.** Experimental design: (A) EEG, EOG, EMG and electrode position measurement and stimulus presentation; (B) stimulus contents; (C) time-scheduling of the stimulation and the measurement of EEG, EOG and EMG.

with two electrodes placed directly above the nasion and the outer canthus of the right eye as electrooculogram (EOG). Another two electrodes were placed at both the medial antibrachi‐ ums to record arm electromyogram (EMG) so that EEGs could be excluded when mistakenly grasping during the movement imagery.

The 32 signals of the EEGs were amplified by a Biotop 6R12-4 amplifier (GE Marquette Medical Systems Japan, Ltd.), and filtered a frequency bandwidth of 0.01-100 Hz. The amplified signals were sampled at a rate of 1 kHz during an epoch of 100 ms preceding and 700 ms following the stimulus onset. The inter-stimulus interval (ISI) was 1600 ms (Figure 2C). The on-line A/D converted EEG signals were immediately stored on a hard disk in a PC-9821Xt personal computer (PC) (NEC Corporation). The EOG and EMG data were also amplified by a Poly‐ graph 360 amplifier (GE Marquette Medical Systems Japan, Ltd.), and sent to the same PC.

*Independent component analysis (ICA).* Fast ICA [25] was applied to each single-trial EEG in the L- and R-MIRP conditions, using ICALAB [26]. After ICA for each trial, among 32 ICs, we removed those associated with eye movement and line noise, and having a broader high frequency (50-100 Hz) spectrum that might be likely to be generated by scalp muscles [27], according to the spectra of all the ICs for each trial. Consequently, the following ECDL was applied to about 20 ICs containing only neural activity.

*Equivalent current dipole source localization (ECDL).* Independent EEG sources obtained by ICA are dipolar [28]. ECDL was applied to the reconstructed EEGs, namely the projection of each IC on the scalp surface by the deflation procedure, using "SynaCenterPro" (PC-based commer‐ cial software for multiple ECDL) (NEC Corporation). This software estimates unconstrained dipoles [29] at any timepoint, using the three-layered concentric sphere head model by the nonlinear optimization methods [30]. An unconstrained dipole was estimated at any time‐ point with maximalpeak ortrough in the reconstructedEEGs for each IC. Here, we searchedfor appropriate and reliable dipole solutions that had goodness of fit (GOF) of more than 90 % and the simplified confidence limit [31] of less than 1 mm and had stable localization to the same brain site around the peak or trough. The brain sites, where dipoles were located, were deter‐ mined by inspection with reference to the textbook of neuroanatomy (e.g., [32]). The inspec‐ tors had a good knowledge of neuroanatomy. Thus, one brain site was assigned to each IC.

*Bayesian network model (BNM) construction.* The present BNM consists of functionally distinct sites of the brain as nodes and directed relationships among these sites as edges. Nodes of the BNM are the brain sites where ECDs were located by the ECDL method. The BNM structure was constructed by the conditional independency (CI) test [33]. The BNMs obtained for each subject had fifteen nodes corresponding to the brain sites such as the frontal, temporal, occipital and cingulate gyri, hippocampus, insula, left and right parietal cortices, left and right motor areas, left and right cerebellum, left and right somatosensory areas, and others. The BNM initialization was made so that there are edges from the mesial prefrontal areas to the premotor and primary motor cortices because the early BP begins in the pre-supplementary motor area (preSMA) and the SMA proper and then in the premotor cortex, and the late BP (NS') occurs in the primary motor and premotor cortices [12].

*Probabilistic inference for classification rule.* In order to discriminate between left- and right-hand imageries, the conditional probabilities at all the nodes in the BNM must be calculated for each trial. For the purpose, the probabilistic inference is made by the belief propagation using the clique tree algorithm [34], where the ECDL results for each trial correspond to the evidences. A free software, "MSBNx" (Microsoft Research) [35] enables this inference. On the basis of the *node activities*, a rule was proposed to classify into left- and right-hand imageries.

#### **2.2. Results**

Figure 3 shows an example for a series of our results for one trial by one subject: (A) 32-channel raw EEG data; (B) ICA results (showing the first 10 ICs); (C) deflation for the 10th IC; (D) ECDL result, where when to be estimated is depicted by a black arrow at the 11th reconstructed EEG in (C) and the same time is for the rest EEGs, and one localized dipole is represented by a blue arrow in (D). This figure exemplifies that the dipole was located at the left pre-central gyrus, for the single-trial EEG recorded during the right-hand movement imagery task. Table 2 illustrates a summary for these categorized ECDL results, including the same data as in Figure 3. That is, this table indicates that the 10th IC, after the deflation procedure, from the singletrial EEGs recorded during the 7th right-hand movement imagery task, was localized to the left motor area (the pre-central gyrus), where n1 to n15 correspond to the above 15 brain sites, in particular n1 the left motor area.

#### **2.3. BN structure**

*Equivalent current dipole source localization (ECDL).* Independent EEG sources obtained by ICA are dipolar [28]. ECDL was applied to the reconstructed EEGs, namely the projection of each IC on the scalp surface by the deflation procedure, using "SynaCenterPro" (PC-based commer‐ cial software for multiple ECDL) (NEC Corporation). This software estimates unconstrained dipoles [29] at any timepoint, using the three-layered concentric sphere head model by the nonlinear optimization methods [30]. An unconstrained dipole was estimated at any time‐ point with maximalpeak ortrough in the reconstructedEEGs for each IC. Here, we searchedfor appropriate and reliable dipole solutions that had goodness of fit (GOF) of more than 90 % and the simplified confidence limit [31] of less than 1 mm and had stable localization to the same brain site around the peak or trough. The brain sites, where dipoles were located, were deter‐ mined by inspection with reference to the textbook of neuroanatomy (e.g., [32]). The inspec‐ tors had a good knowledge of neuroanatomy. Thus, one brain site was assigned to each IC.

*Bayesian network model (BNM) construction.* The present BNM consists of functionally distinct sites of the brain as nodes and directed relationships among these sites as edges. Nodes of the BNM are the brain sites where ECDs were located by the ECDL method. The BNM structure was constructed by the conditional independency (CI) test [33]. The BNMs obtained for each subject had fifteen nodes corresponding to the brain sites such as the frontal, temporal, occipital and cingulate gyri, hippocampus, insula, left and right parietal cortices, left and right motor areas, left and right cerebellum, left and right somatosensory areas, and others. The BNM initialization was made so that there are edges from the mesial prefrontal areas to the premotor and primary motor cortices because the early BP begins in the pre-supplementary motor area (preSMA) and the SMA proper and then in the premotor cortex, and the late BP (NS') occurs

*Probabilistic inference for classification rule.* In order to discriminate between left- and right-hand imageries, the conditional probabilities at all the nodes in the BNM must be calculated for each trial. For the purpose, the probabilistic inference is made by the belief propagation using the clique tree algorithm [34], where the ECDL results for each trial correspond to the evidences. A free software, "MSBNx" (Microsoft Research) [35] enables this inference. On the basis of the

Figure 3 shows an example for a series of our results for one trial by one subject: (A) 32-channel raw EEG data; (B) ICA results (showing the first 10 ICs); (C) deflation for the 10th IC; (D) ECDL result, where when to be estimated is depicted by a black arrow at the 11th reconstructed EEG in (C) and the same time is for the rest EEGs, and one localized dipole is represented by a blue arrow in (D). This figure exemplifies that the dipole was located at the left pre-central gyrus, for the single-trial EEG recorded during the right-hand movement imagery task. Table 2 illustrates a summary for these categorized ECDL results, including the same data as in Figure 3. That is, this table indicates that the 10th IC, after the deflation procedure, from the singletrial EEGs recorded during the 7th right-hand movement imagery task, was localized to the left motor area (the pre-central gyrus), where n1 to n15 correspond to the above 15 brain sites, in

*node activities*, a rule was proposed to classify into left- and right-hand imageries.

in the primary motor and premotor cortices [12].

160 Brain-Computer Interface Systems – Recent Progress and Future Prospects

**2.2. Results**

particular n1 the left motor area.

Figure 4 shows a representative BNM constructed for subject 1, who satisfied the classification rule mentioned later. The BNM construction required each about 30 trials of the left-hand and right-hand-movement imageries. This number will be confirmed later. The BNM for subject 1 directs the link from the "occipital" node to the "frontal" one via the cingulate gyrus, and that fromthe "frontal"node todirectly to the "leftmotor area" one and,to the "rightmotor area" one via the somatosensory cortex.The "frontal" node contains the superior andinferiorfrontal gyri. The "motor area" node includes the primary motor and premotor cortices. This BNM might reveal the neural network involving from the visual stimulus input to the movement imageries.

**Figure 3.** A series of the present results for one trial by one subject: (A) raw EEG data; (B) ICA; (C) deflation; (D) ECDL.

#### **2.4. Node activities**

A BN topology could be determined from the ECDL results on all the trials for each subject. For all the nodes, however, the connectivity between any two nodes, that is, the conditional probability cannot be estimated from even all the ECDL results for each subject. Especially, for any one trial, the ECDL results do not necessarily enrich all the nodes in the subject's BN. However, the probabilistic inference could enrich all the nodes for each trial in terms of the conditional probabilities.

Assuming that the summation of these conditional probabilities in each node reflects the neural activity of the node, the node activity was calculated for each trial. Moreover, we focused on the "left and right motor areas" nodes, because there have been many findings about the involvement of the primary and/or premotor cortices during the motor imagery [36-41]. The t-test concerning the mean of the node activities across the trials for each subject was as follows. For the right-hand imagery, there were significant differences in the node activities between the "left and right motor areas" nodes for subjects 1, 2, 3, 4, 5, 6 and 9 (t(51)=2.41, p=0.0193; t(59)=-2.81, p = 0.00672; t(60)=-7.27, p=1.05E-09; t(54)=-2.18, p=0.0336; t(58)=-2.77, p=0.00754; t(57)=5.11, p=3.90E-06; t(50)=2.19, p=0.0330, respectively). On the other hand, for the left-hand imagery, there were no differences for subjects 1, 2, 4, 5, 7, 8 and 10 (p>0.05). These findings might lead us to one possibility of classification rules to discriminate between left and right hand to be imagined for our single-trial-EEG-based BCI. That is, for right-handed subjects, there is a significant difference in the node activities between left and right "motor area" nodes during the right-hand-movement imagery, while no difference during the left-hand-move‐ ment one.

**Figure 4.** Bayesian network model of motor imagery for subject 1


**Table 2.** A summary for categorized ECDL results. "L" and "R" represent the left- and right-hand movement imageries, respectively. "IC"s independent components and n1 to n15 the 15 brain sites (see the text in more details).

#### **2.5. Discussion**

**2.4. Node activities**

162 Brain-Computer Interface Systems – Recent Progress and Future Prospects

conditional probabilities.

ment one.

**Figure 4.** Bayesian network model of motor imagery for subject 1

A BN topology could be determined from the ECDL results on all the trials for each subject. For all the nodes, however, the connectivity between any two nodes, that is, the conditional probability cannot be estimated from even all the ECDL results for each subject. Especially, for any one trial, the ECDL results do not necessarily enrich all the nodes in the subject's BN. However, the probabilistic inference could enrich all the nodes for each trial in terms of the

Assuming that the summation of these conditional probabilities in each node reflects the neural activity of the node, the node activity was calculated for each trial. Moreover, we focused on the "left and right motor areas" nodes, because there have been many findings about the involvement of the primary and/or premotor cortices during the motor imagery [36-41]. The t-test concerning the mean of the node activities across the trials for each subject was as follows. For the right-hand imagery, there were significant differences in the node activities between the "left and right motor areas" nodes for subjects 1, 2, 3, 4, 5, 6 and 9 (t(51)=2.41, p=0.0193; t(59)=-2.81, p = 0.00672; t(60)=-7.27, p=1.05E-09; t(54)=-2.18, p=0.0336; t(58)=-2.77, p=0.00754; t(57)=5.11, p=3.90E-06; t(50)=2.19, p=0.0330, respectively). On the other hand, for the left-hand imagery, there were no differences for subjects 1, 2, 4, 5, 7, 8 and 10 (p>0.05). These findings might lead us to one possibility of classification rules to discriminate between left and right hand to be imagined for our single-trial-EEG-based BCI. That is, for right-handed subjects, there is a significant difference in the node activities between left and right "motor area" nodes during the right-hand-movement imagery, while no difference during the left-hand-move‐

#### *2.5.1. The present BNMs supported by the topological map of human cortical network*

For subjects 1, 3 to 61 , the paths to left and right "motor area" nodes were consistent with the existing topological map of human cortical network. For example, "frontal" → "right soma‐ tosensory" and "occipital" → "right parietal" are exemplified by MFG.R – PoCG.R and SOG.R – ANG.R in [20] (p.530, Fig.4), respectively.

#### *2.5.2. Classification rule*

Subjects 1, 2, 4 and 5 perfectly met the classification rule proposed above. This rule, which might show bilaterally non-symmetrical event-related desynchronization (ERD) patterns [42], contrary to that of Qin et al. [43], is strongly supported by Bai et al. [44].

#### *2.5.3. Number of trials for BNM learning*

Figure 5 shows the number of trials necessary to satisfy with the rule for subject 1, plotting the p-values in the t-test, as functions of the number of trials, for subject 1. The t-test examines difference in node activities between the "left and right motor areas" nodes for the left- and right-hand imageries. Figure 5 depicts the p-values for every 5 trials. This figure demonstrates that the above rule is effective after about 25 trials. That is, the present BNM learning for BCI requires about 25 trials.

<sup>1</sup> The BNMs of subjects 3 to 6 are not here.

**Figure 5.** P-values in the t-test concerning the differences in conditional probabilities between the "left and right mo‐ tor areas" nodes, as a function of number of trials.

#### *2.5.4. Comparison with CSP*

The present BCI based on the classification rule was validated and compared with the common spatial pattern (CSP) method [22]. The test data includes 20 trials with left- and right-hand movement imageries. In our BCI after the probabilistic inference for each trial, on the basis of the classification rule, if "the left motor areas" node activity are significantly different from "the right motor areas" one, the trial was judged to be a right hand imagery, while both of the node activities are not so different, the trial a left hand imagery.

The CSP is an algorithm for obtaining a spatial filter to transform multi-channel EEG data with two conditions into the surrogate space enabling the optimal discrimination of the conditions. This filtering is achieved by solving the generalized eigenvalue problem for the estimates of the covariance matrices of the band-pass filtered EEG signal. For each trial, 1 dimensional feature is calculated after operating the spatial filter on the single-trial EEG. From these features, the threshold is determined so that all the trials for the learning are optimally discriminated between the two conditions as exemplified in Figure 6. Thus, for each of α, μ, β and γ frequency bands, we conducted one CSP classifier with its thresh‐ old, using the same EEG data as in the subject 1's BNM (Figure 4) construction. Finally, for subjects 1 and 2 among ones who satisfied with the proposed classification rule, the accuracy of the present BNM and the 1-CSP classifier was 90 % and 75 %, and 85 % and 70 %, respectively.

**Figure 6.** Principle of 1-CSP classifier. ζ<sup>i</sup> is a feature value for the *i* th trial, and *b* is the threshold.

#### **3. Generalization to dynamic BNM**

#### **3.1. Materials and methods**

**Figure 5.** P-values in the t-test concerning the differences in conditional probabilities between the "left and right mo‐

The present BCI based on the classification rule was validated and compared with the common spatial pattern (CSP) method [22]. The test data includes 20 trials with left- and right-hand movement imageries. In our BCI after the probabilistic inference for each trial, on the basis of the classification rule, if "the left motor areas" node activity are significantly different from "the right motor areas" one, the trial was judged to be a right hand imagery, while both of the

The CSP is an algorithm for obtaining a spatial filter to transform multi-channel EEG data with two conditions into the surrogate space enabling the optimal discrimination of the conditions. This filtering is achieved by solving the generalized eigenvalue problem for the estimates of the covariance matrices of the band-pass filtered EEG signal. For each trial, 1 dimensional feature is calculated after operating the spatial filter on the single-trial EEG. From these features, the threshold is determined so that all the trials for the learning are optimally discriminated between the two conditions as exemplified in Figure 6. Thus, for each of α, μ, β and γ frequency bands, we conducted one CSP classifier with its thresh‐ old, using the same EEG data as in the subject 1's BNM (Figure 4) construction. Finally, for subjects 1 and 2 among ones who satisfied with the proposed classification rule, the accuracy of the present BNM and the 1-CSP classifier was 90 % and 75 %, and 85 % and

node activities are not so different, the trial a left hand imagery.

tor areas" nodes, as a function of number of trials.

164 Brain-Computer Interface Systems – Recent Progress and Future Prospects

*2.5.4. Comparison with CSP*

70 %, respectively.

*Acquisition of single-trial EEGs.* This experiment was carried out to obtain MRPs with BP in term of single-trial EEGs. Subjects reclined in an EEG chair in an electrically shielded room, and gazed at a display 90 cm away from their eyes (Figure 7). For any one trial, on the display, "choice" was presented at first, "+" was presented 3.5 s after "choice" and "report" was presented 2 s after "+" in turn. The duration of "choice" was 2 s, that of "+" was 0.2 s and that of "report" was 2.5 s (Figure 8). In case of "left" or "right" selection in "choice", the subjects were requested to grasp their hand of the same side as the selection as soon as possible when "+" was presented, and then to speak aloud the selection in "report". Otherwise, there was no grasping and no speaking. During this task, 32-channel single-trial EEGs, EMG and EOG were recorded. This section reports results on only the same subject as in the section 2 (subject 1).

*Division of the EEGs.* According to Shibasaki and Hallett [12], the EEG data was divided into three intervals. That is, assuming that the time when the onset of the EMG is 0 ms, the interval between -1700 and -400 ms refers to early-BP (E-BP), that between -400 and 0 ms NS' (negative slope) and that between 0 and 200 ms MP (motor potential).

*ICA, ECDL then BNM and DBNM construction and probabilistic inference.* For each interval, ECDL was applied to single-trial EEGs after ICA. For each interval, using the categorical data concerning the ECDL results on both the "left" and "right" tasks, each including 10 trials, a BNM structure was determined by the CI test. Moreover, connecting with the frontal node on the top of each BN led to a DBNM. Then, the probabilistic inference enriched all the nodes of the DBNM for each trial, in terms of conditional probabilities.

**Figure 7.** EEG measurement and stimulus presentation system.

**Figure 8.** Time-scheduling of the stimulus presentation, the task and the EEG measurement.

**Figure 9.** E-BP BNM, where open rectangles and arrows depict nodes and edges, respectively.

#### **3.2. Results and discussion**

**Figure 7.** EEG measurement and stimulus presentation system.

166 Brain-Computer Interface Systems – Recent Progress and Future Prospects

**Figure 8.** Time-scheduling of the stimulus presentation, the task and the EEG measurement.

Figure 9 shows a BNM obtained for the E-BP. This BNM is also consistent with the topological map of human cortical network (for example, "cingulate gyrus" →"left motor area" was found in ACG.L-SMA.L-PreCG.L [20]), and reflects the previous neurophysiological findings (for example, "hippocampus"→"left/right cerebellum" might be explained by that lesions of the cerebellar nuclei abolish conditioned increases in hippocampal CA1 neural activity evoked by the tone-conditioned stimulus [45]). The DBNM, shown in Figure 10, contains the neural generators for the MRPs. Namely, the neural generator of the E-BP is the pre-SMA at first, next the SMA and then the premotor cortex, that of the NS' the premotor cortex at the first and next the motor cortex and that of the MP the somatosensory cortex [8]. In the E-BP BNM, there is no difference in node activities between the "left- and right-motor area" nodes for the left- (t(14)=-2.0018, p=0.06507) and right-hand (t(16)=-1.0849, p=0.294) movement. There was also the same tendency (left-hand movement: t(14)=-1.0832, p=0.297; right-hand one: t(16)=-0.1059, p=0.917) in the NS' BNM. In the MP-BNM, however, there were significant differences between the two nodes both for the right-hand movement (t(16)=-3.9817, p=0.001072) and for the lefthand one (t(14)=-2.2532, p=0.04081). These findings suggest that there may be differences in neural network connectivity between the MP BNM and the others. On the other hand, also in the present DBNM, there were significant differences between the two nodes both for the lefthand movement (t(46)=-2.9645, p=0.004791) and for the right-hand one (t(52)=-2.6109, p=0.01177). The difference between the DBNM and the BNM obtained in the section 2 might not reflect only that in the neural connectivity but also that in the tasks, that is, the MI and the actual movement.

**Figure 10.** DBNM including (A) E-BP, (B) NS' and (C) MP.

**Figure 11.** An example of a system for automatically specifying the brain sites where estimated ECDs are located.

#### **4. Consideration and conclusion**

**Figure 10.** DBNM including (A) E-BP, (B) NS' and (C) MP.

168 Brain-Computer Interface Systems – Recent Progress and Future Prospects

In this chapter, we have proposed a new framework for single-trial-EEG-based BCIs with the MI tasks. This framework consists of the categorization of the EEGs, which could lead us to data reduction, and the classifiers for the categorical data. The ECDL after ICA enabled us the former. For the latter, the classifier using Hayashi's second method of quantification has yielded the accuracy of more than 90 % [11]. This chapter has concentrated on another classifier for the categorical data, called Bayesian networks. The present BCI learning required 25 trials at least. Although for the results on only two subjects, the accuracy in 20-trial validation was higher than that by the CSP, in addition to exceeding the existing ECDL-based BCIs [43,46]. If any subject met the classification rule, the subject would be expected to achieve the good accuracy.

To our knowledge, Shenoy and Rao [16] made the first report of the application of DBNM to BCIs. Although the DBN allowed continuous tracking and prediction of the brain states over time, it included hidden but ambiguous state variables. Our DBNM has revealed the difference among the E-BP, NS' and MP in MRPs and that between the MI and the actual movement tasks. However, the DBNMs for BCIs are still in the data-processing stage, not in the validation one.

When obtaining ECDL results on 20 ICs for each of 50 (=25x2) trials by one subject, inspection with reference to the textbook of neuroanatomy must be iterated 1000 (=20x50) times, at least. Even if the inspector has the full knowledge of neuroanatomy, different inspectors might yield different ECDL results. In order to cope with this problem, we are developing a computer system for automatically specifying the brain sites for ECDL results on each individual with MR images (Figure 11) [47]. However, we cannot directly utilize the Talairach-Tournoux brain atlas [48], because there are big differences in the brain shape between Westerners and Asians, in particular Japanese. Therefore, we are now constructing the brain atlas for Japanese.

### **Acknowledgements**

This research was partly supported by a Grants-in-Aid for Scientific Research on Scientific Research (B) (20300196) - The Japan Society for the Promotion of Science.

#### **Author details**

Toshimasa Yamazaki1\*, Maiko Sakamoto2 , Shino Takata3 , Hiromi Yamaguchi1 , Kazufumi Tanaka1 , Takahiro Shibata4 , Hiroshi Takayanagi5 , Ken-ichi Kamijo6 and Takahiro Yamanoi7

\*Address all correspondence to: t-ymzk@bio.kyutech.ac.jp

1 Department of Bioscience and Bioinformatics, Kyushu Institute of Technology, Iizuka, Fu‐ kuoka, Japan


#### **References**

[1] Wolpaw J, Birbaumer N, McFarland D, Pfurtscheller G, Vaughan T. Brain-computer interfaces for communication and control. Clinical Neurophysiology 2002; 113 767-791.

[2] Pfurtscheller G, Lopes da Silva FH. Functional meaning of event-related desynchro‐ nization (ERD) and synchronization (ERS). In: Pfurtscheller G., Lopes da Silva FH. (eds) Event-Related Desynchronization. Handbook of Electroencephalography and Clinical Neurophysiology, Revised Series, Vol.6. Elsevier Science B.V.; 1999. p51-65.

Even if the inspector has the full knowledge of neuroanatomy, different inspectors might yield different ECDL results. In order to cope with this problem, we are developing a computer system for automatically specifying the brain sites for ECDL results on each individual with MR images (Figure 11) [47]. However, we cannot directly utilize the Talairach-Tournoux brain atlas [48], because there are big differences in the brain shape between Westerners and Asians, in particular Japanese. Therefore, we are now constructing the brain atlas for Japanese.

This research was partly supported by a Grants-in-Aid for Scientific Research on Scientific

, Shino Takata3

, Hiroshi Takayanagi5

1 Department of Bioscience and Bioinformatics, Kyushu Institute of Technology, Iizuka, Fu‐

[1] Wolpaw J, Birbaumer N, McFarland D, Pfurtscheller G, Vaughan T. Brain-computer interfaces for communication and control. Clinical Neurophysiology 2002; 113

, Hiromi Yamaguchi1

, Ken-ichi Kamijo6

,

and

Research (B) (20300196) - The Japan Society for the Promotion of Science.

**Acknowledgements**

**Author details**

Kazufumi Tanaka1

Takahiro Yamanoi7

kuoka, Japan

**References**

767-791.

Toshimasa Yamazaki1\*, Maiko Sakamoto2

3 Lincrea Corporation, Tokyo, Japan

6 NEC Corporation, Tokyo, Japan

, Takahiro Shibata4

170 Brain-Computer Interface Systems – Recent Progress and Future Prospects

\*Address all correspondence to: t-ymzk@bio.kyutech.ac.jp

2 Hitachi Public System Service Co. Ltd., Tokyo, Japan

5 Information Science Research Center, Tokyo, Japan

7 Hokkai Gakuen University, Sapporo, Japan

4 Olympus Software Technology Corporation, Tokyo, Japan


[29] Mosher JC, Lewis PS, Leahy RM. Multiple dipole modeling and localization from spatio-temporal MEG data. IEEE Transactions on Biomedical Engineering 1992; 39 541-557.

[15] Yoon JW, Roberts SJ, Dyson M, Gan JQ. Adaptive classification for Brain Computer Interface systems using sequential montecarlo sampling. Neural Networks 2009; 22

[16] Shenoy P, Rao RPN. Dynamic Bayesian networks for brain-computer interfaces. Ad‐

[17] Micheloyannis S, Pachou E, Stam CJ, Vourkas M, Erimaki S, Tsirka V. Using graph theoretical analysis of multi channel EEG to evaluate the neural efficiency hypothe‐

[18] Honey CJ, Kötter R, Breakspear M, Sporns O. Network structure of cerebral cortex shapes functional connectivity on multiple time scales. PNAS 2007;104 10240-10245.

[19] Hagmann P, Kurant M, Gigandet X, Thiran P, Wedeen VJ, Meuli R, Thiran J-P. Map‐ ping human whole-brain structural networks with diffusion MRI. PLoS ONE 2007;is‐

[20] Gong G, He Y, Concha L, Lebel C, Gross DW, Evans AC, Beaulieu C. Mapping ana‐ tomical connectivity patterns of human cerebral cortex using in vivo diffusion tensor

[21] Shigemasu K. Ueno M, Motomura Y. Introduction to Bayesian Networks. Tokyo:

[22] Blankertz B, Tomioka R, Lemm S, Kawanabe M, Müller K-R. Optimizing spatial fil‐ ters for robust EEG single-trial analysis. IEEE Signal Processing Magazine 2008; 25

[23] Oldfield RC. The assessment and analysis of handedness: the Edinburgh Inventory.

[24] Soufflet L, Toussaint M, Luthringer R, Gressor J, Minot R, Macher JP. A statistical evaluation of the main interpolation methods applied to 3-dimensional EEG map‐

ping. Electroencephalography and Clinical Neurophysiology 1991; 79 393-402.

[25] Hyvärinen A, Oja E. A fast fixed-point algorithm for independent component analy‐

[26] Cichocki A, Amari S, Siwek K, Tanaka T, Phan AH, Zdunek R,Cruces S, Georgiev P, Washizawa Y, Leonowicz Z, Bakardijan H, Rutkowski T, Choi S, Belouchrani A, Bar‐ ros A, Thawonmas R, Hoya T, Hashimoto W, Terazono Y. ICALAB version 3 tool‐

[27] Makeig S, Bell AJ, Jung TP, Sejnowski TJ. Independent component analysis of electro‐ encephalographic data. In: Touretzky D, Mozer M, Hasselmo M. (eds) Advances in

[28] Delorme A, Palmer J, Onton J, Oostenveld R, Makeig S. Independent EEG sources are

box, RIKEN BSI.http://www.bsp.brain.riken.jp/ICALAB/ (accessed 2007).

Neural Information Processing Systems; 1996. p145-151.

1286-1294.

sue 7:e597.

41-56.

vances in NIPS17; 2005.

sis. Neuroscience Letters 2006;402 273-277.

172 Brain-Computer Interface Systems – Recent Progress and Future Prospects

BAIFUKAN CO., LTD; 2006, in Japanese.

sis. Neural Computation 1997;91483-1492.

Neuropsychologia 1971; 9 97-113.

dipolar. PLoS ONE 2012;7 e30135.

imaging tractography. Cerebral Cortex 2009;19 524-536.


#### **Sources of Electrical Brain Activity Most Relevant to Performance of Brain-Computer Interface Based on Motor Imagery Sources of Electrical Brain Activity Most Relevant to Performance of Brain-Computer Interface Based on Motor Imagery**

Alexander Frolov, Dušan Húsek, Pavel Bobrov, Olesya Mokienko and Jaroslav Tintera Olesya Mokienko and Jaroslav Tintera Additional information is available at the end of the chapter

Alexander Frolov, Dušan Húsek, Pavel Bobrov,

Additional information is available at the end of the chapter 10.5772/55166

http://dx.doi.org/10.5772/55166

#### **1. Introduction**

[41] Ehrsson HH, Geyer S, Naito E. Imagery of voluntary movement of fingers, toes, and tongue activates corresponding body-part-specific motor representations. Journal of

[42] Pfurtscheller G, NeuperCh, Flotzinger D, Pregenzer M. EEG-based discrimination between imagination of right and left hand movement. Electroencephalography and

[43] Qin L, Ding L, He B. Motor imagery classification by means of source analysis for brain–computer interface applications. Journal of Neural Engineering 2004;1 135-141.

[44] Bai O, Maria Z, Vorbacha S, Hallett M. Asymmetric spatiotemporal patterns of eventrelated desynchronization preceding voluntary sequential finger movements: a high-

[45] Clark GA, McCormick DA, Lavond DG, Thompson RF. Effects of lesions of cerebel‐ lar nuclei on conditional behavioral and hippocampal neuronal responses. Brain Re‐

[46] Kamousi B, Liu Z, He B. Classification of motor imagery tasks for Brain-Computer Interface applications by means of two equivalent dipoles analysis. IEEE Transac‐

[47] Tanaka K, Motoi M, Sasaguri Y, Yamazaki T, Takayanagi H, Yamanoi T, Kamijo K. A new single-trial-EEG-based BCI –Validation of quantification method of type II mod‐ elling. Clinical Neurophysiology 2010;121 S161 (Abstracts of ICCN 2010), 29th Inter‐ national Congress of Clinical Neurophysiology, Oct 28 – Nov 1, 2010, Kobe, Japan.

[48] Talairach J, Tournoux P. Co-Planar Sterotaxic Atlas of the Human Brain. New York:

tions on Neural Systems and Rehabilitation Engineering 2005;13(2) 166-171.

resolution EEG study. Clinical Neurophysiology 2005;116 1213-1221.

Neurophysiology 2003;90 3304-3316.

174 Brain-Computer Interface Systems – Recent Progress and Future Prospects

search 1984;291 125-136.

Thieme; 1988.

Clinical Neurophysiology 1997;103 642-651.

A brain-computer interface (BCI) provides a direct functional interaction between the human brain and the external device. Many kinds of signals (from electromagnetic to metabolic [23, 38, 42]) could be used in BCI. However the most widespread BCI systems are based on EEG recordings. BCI consists of a brain signal acquisition system, data processing software for feature extraction and pattern classification, and a system to transfer commands to an external device and, thus, providing feedback to an operator. The most prevalent BCI systems are based on the discrimination of EEG patterns related to execution of different mental tasks [14, 21, 24]. This approach is justified by the presence of correlation between brain signal features and tasks performed, revealed by basic research [24, 28, 30, 45]. By agreement with the BCI operator each mental task is associated with one of the commands to the external device. Then to produce the commands, the operator switches voluntary between corresponding mental tasks. If BCI is dedicated to control device movements then psychologically convenient mental tasks are motor imaginations. For example, when a patient controls by BCI the movement of a wheelchair its movement to the left can be associated with the imagination of the left arm movement and movement to the right - with right arm movement. Another advantage of these mental tasks is that their performance is accompanied by the easily recognizable EEG patterns. Moreover, motor imagination is considered now as an efficient rehabilitation procedure to restore movement after paralysis [4]. Thus, namely the analysis of BCI performance based on motor imagination is the object of the present chapter.

©2012 Frolov et al., licensee InTech. This is an open access chapter distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2013 Frolov et al.; licensee InTech. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2012 Frolov et al.; licensee InTech. This is a paper distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

The most stable electrophysiological phenomenon accompanying motor performance is the decrease of EEG mu-rhythm recording from the central electrodes located over the brain areas representing the involved extremity [29]. This decrease (Event Related Desynchronization, ERD) occurs also when the subject observes the movement of another person [34] and during motor preparation and imagination [30]. In the state of motor relaxation the increase of EEG mu-rhythm is observed [29] which is called Event Related Synchronization (ERS). The exposure of ERD and ERS in specific brain areas during motor imagination of different extremities is the reason of the high efficiency of BCI based on motor imagination [32]. From the other hand, BCI training allows to stabilize and to contrast brain activity corresponding to different mental tasks and hence to facilitate the search of brain areas involved in their performance.

Until now the most widespread technique to localize brain functions is fMRI study which provides high spatial but low temporal resolution. By contrast EEG study provides high temporal but low spatial resolution. The most prospective seems to be the combination of these techniques [9] especially if to take into account the fast progress in methods of solving inverse EEG problem [13, 19] dedicated to localize sources of brain activity by distribution of electric potential over head surface. One of the approaches towards the integration of these techniques was suggested in our previous work [11]. Here we develop the approach and apply it to the analysis of more mental states used for BCI control.

#### **2. Methods**

#### **2.1. Experimental procedure**

Eight subjects (4 male, 4 female) aged from 25 to 65 participated in the study. All subjects were right-handed and had no neurological diseases. The subjects have provided written participation consent. The experimental procedure was approved by the Board of Ethics at the Institute for Higher Nervous Activity and Neurophysiology of the Russian Academy of Sciences.

The experiment with each subject was conducted for 10 experimental days, the one series per day. Each series consisted of training and testing sessions (Fig. 1 A). The first, training, session was designed to train BCI classifier. The following, testing session was designed to provide subjects with the output of the BCI classifier in real time to enhance their efforts to imagine a movement. The subjects had to perform one of the four instructions presented on a screen of a monitor: to relax and to imagine the movement of the right or left hand or feet. The movement which they were asked to imagine was a handgrip or feet pressure.

Subject was sitting in a comfortable chair, one meter from a 17" monitor, and was instructed to fix a gaze on a motionless circle (1 cm in diameter) in the middle of the screen. Four gray markers were placed around the circle to indicate the mental task to be performed. The change of the marker color into green signaled the subject to perform the corresponding mental task. Left and right markers corresponded to left and right hand movement imagining respectively. The lower marker corresponded to feet movement imagining and the upper one corresponded to relaxation. Each command was displayed for 10 seconds. Each clue was preceded by a 4-second warning when the marker color changed into blue.

Four such instructions presented in random order constituted a block, one block constituted a training session and nine blocks a testing session (Fig. 1). Thus each subject received 10 blocks of instructions at each experimental day.

2 Brain-Computer Interface

performance.

**2. Methods**

Sciences.

**2.1. Experimental procedure**

The most stable electrophysiological phenomenon accompanying motor performance is the decrease of EEG mu-rhythm recording from the central electrodes located over the brain areas representing the involved extremity [29]. This decrease (Event Related Desynchronization, ERD) occurs also when the subject observes the movement of another person [34] and during motor preparation and imagination [30]. In the state of motor relaxation the increase of EEG mu-rhythm is observed [29] which is called Event Related Synchronization (ERS). The exposure of ERD and ERS in specific brain areas during motor imagination of different extremities is the reason of the high efficiency of BCI based on motor imagination [32]. From the other hand, BCI training allows to stabilize and to contrast brain activity corresponding to different mental tasks and hence to facilitate the search of brain areas involved in their

Until now the most widespread technique to localize brain functions is fMRI study which provides high spatial but low temporal resolution. By contrast EEG study provides high temporal but low spatial resolution. The most prospective seems to be the combination of these techniques [9] especially if to take into account the fast progress in methods of solving inverse EEG problem [13, 19] dedicated to localize sources of brain activity by distribution of electric potential over head surface. One of the approaches towards the integration of these techniques was suggested in our previous work [11]. Here we develop the approach and

Eight subjects (4 male, 4 female) aged from 25 to 65 participated in the study. All subjects were right-handed and had no neurological diseases. The subjects have provided written participation consent. The experimental procedure was approved by the Board of Ethics at the Institute for Higher Nervous Activity and Neurophysiology of the Russian Academy of

The experiment with each subject was conducted for 10 experimental days, the one series per day. Each series consisted of training and testing sessions (Fig. 1 A). The first, training, session was designed to train BCI classifier. The following, testing session was designed to provide subjects with the output of the BCI classifier in real time to enhance their efforts to imagine a movement. The subjects had to perform one of the four instructions presented on a screen of a monitor: to relax and to imagine the movement of the right or left hand or feet.

Subject was sitting in a comfortable chair, one meter from a 17" monitor, and was instructed to fix a gaze on a motionless circle (1 cm in diameter) in the middle of the screen. Four gray markers were placed around the circle to indicate the mental task to be performed. The change of the marker color into green signaled the subject to perform the corresponding mental task. Left and right markers corresponded to left and right hand movement imagining respectively. The lower marker corresponded to feet movement imagining and the upper one corresponded to relaxation. Each command was displayed for 10 seconds. Each clue was

The movement which they were asked to imagine was a handgrip or feet pressure.

preceded by a 4-second warning when the marker color changed into blue.

apply it to the analysis of more mental states used for BCI control.

**Figure 1.** Schematic representation of the experimental protocol and the stages of each experimental session. A. The sequence of sessions. B. The structure of the experimental block. In each block each instruction was presented once. The sequence of their presentation in each block was random. The duration of instructions is given in seconds. Green areas - instructions for performance, blue areas - warnings

During the training session classifier was switched off and recording was used only for its learning. During the following testing sessions classifier was switched on and the result of classification was presented to a subject by color of the central circle. The circle became green if the result coincided with the instruction and its brightness increased with the increase of classifying confidence. During the instruction to relax the presentation of classifying result was switched off not to attract the subject's attention.

EEG was recorded by 48 active electrodes using g.USBamp and g.USBamp API for MATLAB (g-tec, Graz, Austria) with sampling frequency 256 Hz and filtered by notch filter to suppress supply noise. Electrode positions were Fz, F3, F4, Fcz, Fc3, Fc4, F7, F8, Fcz, Fc3, Fc4, Fc5, Fc6, Fc7, Fc8, Cz, C1, C2, C3, C4, C5, C6, T7, T8, Cpz, Cp1, Cp2, Cp3, Cp4, Cp5, Cp6, Tp7, Tp8, Pz, P1, P2, P3, P4, P5, P6, P7, P8, Poz, Po3, Po4, Po7, Po8, Oz, O1, O2. Afz was a reference.

The data of the last BCI session were used for solving inverse EEG problem, that is to localize the sources of EEG signals inside the brain. We solved it taking into account individual geometry of brain and its covers. To identify the source positions relative to the brain structures, electrodes positions have to be also identified relative to these structure. Since brain structures are given in MRI coordinate system electrode positions have to be also identified in MRI coordinate system. To that end not moving the cap on the subject head we removed electrodes from nodes at the cap and replaced them by small pellets visible on MRI slices (Fig. 2). Just after this procedure the subject was placed in MRI chamber. Since the position of each electrode was marked by the pellet, then identification of the pellet positions on slices allowed to obtain the electrodes coordinates in MRI reference frame.

During fMRI recording instructions to relax or to imagine hand or feet movements were presented to the subject without EEG recording. Since the efficiency of BCI control depends

**Figure 2.** Pellets marking electrode positions on the head surface (A) and their images reconstructed by MRI slices (B).

on the subject's ability to stabilize and to contrast patterns of the brain activity following different instructions, then we believe that the subjects who demonstrate an efficient BCI control are able to produce stable and contrast patterns of the brain activity during both EEG and fMRI experiments. The fMRI examinations were conducted with 3T MR scanner (Siemens Trio Tim, Erlangen, Germany) using 12-channel head coil. Functional MRI data were acquired with a T2\*weighted gradient echo EPI sequence (40 slices, TR=2500 ms, TE=30 ms, 64 × 64 matrix, FOV=192 × 192 mm, slice thickness 3 mm, voxel size= 3×3×3 mm, BW=2232 Hz/pixel, axial orientation). Total of 240 dynamic measurements were acquired during fMRI scans resulting to 10 minutes of acquisition time. A T1-weighted anatomical scan using MP-RAGE sequence (224 slices, TR=2300 ms, TE=4.64 ms, FOV=256×256, slice thickness 1 mm, voxel size of 1×1×1 mm, PAT=2, sagittal orientation) was also acquired for each subject. The measurement lasted of 5 min.

#### **2.2. Classifier**

One crucial part of a BCI system is the EEG pattern classifier which can be designed by many methods [1]. We used in our experiments the simplest Bayesian classifier based on EEG covariance matrices. As shown in [10] it provides classification accuracy comparable with other more sophisticated classifiers.

Suppose that there are *L* different mental tasks to be distinguished and probabilities of each task to be performed are equal to 1/*L*. Let also for each mental task the EEG signal distribution be Gaussian with zero mean. Also, let **C***i*, a covariance matrix of the signal corresponding to execution of the *i*-th mental task (*i* = 1, . . . , *L*), be non-singular. Then probability to obtain EEG signal **X** under the condition that the signal corresponds to performing the *<sup>i</sup>*-th mental task is *<sup>P</sup>* (**<sup>X</sup>** <sup>|</sup> *<sup>i</sup>*) <sup>∝</sup> *<sup>e</sup>*<sup>−</sup> *Vi* <sup>2</sup> , where *Vi* <sup>=</sup> **<sup>X</sup>**T**C**−<sup>1</sup> *<sup>i</sup>* **<sup>X</sup>** + ln (det(**C***i*)). Following the Bayesian approach, the maximum value of *P* (**X** | *i*) over all *i* determines the class to which **X** belongs. Hence, the signal **X** is considered to correspond to execution of the *<sup>k</sup>*-th mental task as soon as *<sup>k</sup>* <sup>=</sup> argmin*<sup>i</sup>* {*Vi*}. The equality **<sup>X</sup>**T**C**−<sup>1</sup> *<sup>i</sup>* **<sup>X</sup>** <sup>=</sup> trace **C**−<sup>1</sup> *<sup>i</sup>* **XX**<sup>T</sup> implies that

4 Brain-Computer Interface

**Figure 2.** Pellets marking electrode positions on the head surface (A) and their images reconstructed by MRI slices (B).

each subject. The measurement lasted of 5 min.

with other more sophisticated classifiers.

performing the *<sup>i</sup>*-th mental task is *<sup>P</sup>* (**<sup>X</sup>** <sup>|</sup> *<sup>i</sup>*) <sup>∝</sup> *<sup>e</sup>*<sup>−</sup> *Vi*

**2.2. Classifier**

on the subject's ability to stabilize and to contrast patterns of the brain activity following different instructions, then we believe that the subjects who demonstrate an efficient BCI control are able to produce stable and contrast patterns of the brain activity during both EEG and fMRI experiments. The fMRI examinations were conducted with 3T MR scanner (Siemens Trio Tim, Erlangen, Germany) using 12-channel head coil. Functional MRI data were acquired with a T2\*weighted gradient echo EPI sequence (40 slices, TR=2500 ms, TE=30 ms, 64 × 64 matrix, FOV=192 × 192 mm, slice thickness 3 mm, voxel size= 3×3×3 mm, BW=2232 Hz/pixel, axial orientation). Total of 240 dynamic measurements were acquired during fMRI scans resulting to 10 minutes of acquisition time. A T1-weighted anatomical scan using MP-RAGE sequence (224 slices, TR=2300 ms, TE=4.64 ms, FOV=256×256, slice thickness 1 mm, voxel size of 1×1×1 mm, PAT=2, sagittal orientation) was also acquired for

One crucial part of a BCI system is the EEG pattern classifier which can be designed by many methods [1]. We used in our experiments the simplest Bayesian classifier based on EEG covariance matrices. As shown in [10] it provides classification accuracy comparable

Suppose that there are *L* different mental tasks to be distinguished and probabilities of each task to be performed are equal to 1/*L*. Let also for each mental task the EEG signal distribution be Gaussian with zero mean. Also, let **C***i*, a covariance matrix of the signal corresponding to execution of the *i*-th mental task (*i* = 1, . . . , *L*), be non-singular. Then probability to obtain EEG signal **X** under the condition that the signal corresponds to

Following the Bayesian approach, the maximum value of *P* (**X** | *i*) over all *i* determines the class to which **X** belongs. Hence, the signal **X** is considered to correspond to execution of

<sup>2</sup> , where *Vi* <sup>=</sup> **<sup>X</sup>**T**C**−<sup>1</sup>

*<sup>i</sup>* **<sup>X</sup>** + ln (det(**C***i*)).

$$V\_i = \text{trace}\left(\mathbf{C}\_i^{-1}\mathbf{X}\mathbf{X}^\mathrm{T}\right) + \ln\left(\det(\mathbf{C}\_i)\right) \tag{1}$$

In practice all *Vi* are rather variable, so it is more beneficial to split signal into epochs and compute average value �*Vi*� for each EEG epoch to be classified, using equation (1)

$$
\langle V\_i \rangle = \text{trace}\left(\mathbf{C} \mathbf{C}\_i^{-1}\right) + \ln\left(\det(\mathbf{C}\_i)\right) \tag{2}
$$

where **C** denotes an epoch data covariance matrix estimated as **XX**<sup>T</sup> . Therefore to perform the classifier training it is sufficient to compute the covariance matrices corresponding to each mental task. It makes BC computationally inexpensive.

During testing sessions covariance matrix **C** was calculated every 250 msec over 1 second sliding window. Respectively, the variable *s* defining the accuracy of brain state recognition was updated each 250 msec. It took 1 if the state recognized corresponded to the instruction and 0 in the opposite case. Then the variable *S* was updated according to the formula *S* = (1 − *γs*)*S* + *γss*. At the beginning of new instruction presentation *S* was set to be zero. The brightness of the central circle which indicated the quality of BCI performance was proportional to *S*. The value of *γs* was taken to be 0.1. So the characteristic time of the biofeedback was about 2.5 sec.

The covariance matrices **C***<sup>i</sup>* were calculated during the training session and then updated during the testing session at the end of each block according to the formula **<sup>C</sup>***<sup>i</sup>* = (<sup>1</sup> − *γc*)**C***<sup>i</sup>* + *γc***c***<sup>i</sup>* where **c***<sup>i</sup>* is a covariance matrix for the *i*-th state calculated over block data. In our experiments we used *γ<sup>c</sup>* = 0.1.

The quality of BCI performance was estimated by the results of on-line classifying during the testing session of each experimental day and offline by the data obtained during both training and testing sessions. For offline analysis the data were additionally filtered within 5-30 Hz bandpass. Then 7 blocks of 10 were randomly chosen for classifier learning, i.e. for calculation of covariance matrices for all mental states. Recordings of the remaining 3 blocks were split into epochs of 1 second length. These epochs were used for classifier testing. 50 such classification trials were made. Averaging over all classification trials resulted in *L* × *L* confusion matrix **P** = *pij* . Here *pij* is an estimate of probability to recognize the *i*-th mental task in case the instruction is to perform the *j*-th mental task.

We chose the mean probability of correct classification *p*, mutual information *g* between states recognized and instructions presented, and Cohen's *κ* as indices of classification efficacy. Given the confusion matrix **P** these indices can be calculated as follows:

$$\begin{aligned} p &= \frac{1}{L} \sum\_{i=1}^{L} p\_{ii} \\ g &= -\sum\_{i,j=1,1}^{L,L} p\_{ij} p\_{0j} \log\_2 \left( p\_{ij} / p\_{i0} \right), \\ \kappa &= \frac{\sum\_{i=1}^{L} p\_{ii} p\_{0i} - \sum\_{i=1}^{L} p\_{0i} p\_{i0}}{1 - \sum\_{i=1}^{L} p\_{0i} p\_{i0}} \end{aligned} \tag{3}$$

where *<sup>p</sup>*0*<sup>j</sup>* <sup>=</sup> 1/*<sup>L</sup>* is probability of the *<sup>j</sup>*-th instruction to be presented and *pi*<sup>0</sup> <sup>=</sup> *<sup>L</sup>* ∑ *j*=1 *pij p*0*<sup>j</sup>* is

probability of the *i*-th mental state to be recognized.

The better classifier performs the more confusion matrix is close to identity matrix. In case *<sup>L</sup>* states are classified perfectly *<sup>p</sup>* = 1, *<sup>g</sup>* = log2 *<sup>L</sup>*, and *<sup>κ</sup>* = 1. If classification is random, i.e. *pij* = *pi*<sup>0</sup> for all *j*, then *p* = 1/*L*, *g* = 0, and *κ* = 0.

Index *p* has an advantage of being evidently interpreted as the percentage of correct classification while its disadvantages are, first, that it does not account for distribution of errors since it does not depend on nondiagonal elements of confusion matrix, second, its lower value depends on the number of states classified. For example the value *p* = 0.5 corresponds to a random recognition when *L* = 2 and twice exceeds the random level when *L* = 4. Thus it is difficult to compare the qualities of BCIs with different states classified. Both indices *g* and *κ* have the advantage of considering the error distribution over nondiagonal elements of confusion matrix and of being normalized to the case of random classifying independently of *L*. In this case both indices are equal to zero. Moreover, *κ* is also normalized to the case of perfect classifying. It takes one in this case, while *g* depends on *L*. Thus, on one hand *κ* is more convenient for comparing qualities of BCI with different *L*, but on another hand *g* gives the estimate of BCI quality directly in the rate of information transfer. Since *g* is calculated every second then it can be measured in bits per second. Thus all three indices are reasonable.

When all probabilities of correct classification are equal, i.e. *pii* = *p* for all *i*, and all probabilities of incorrect classification are equal, i.e. *pij* = (1 − *p*)/(*L* − 1) for all *i* �= *j*, the mutual information between instructions presented and the states classified can be obtained as:

$$g = \log\_2 L + p \log\_2 p + (1 - p) \log\_2 \left(\frac{1 - p}{L - 1}\right) \tag{4}$$

Based on [43], (4) is often used to estimate BCI efficacy ([3], [5] [44]). But if the corresponding assumptions do not hold true, the value of *g*, calculated according to (4), is lower than the actual mutual information. In this study we used the general formula (3).

To estimate the on-line BCI quality we used confusion matrix obtained over the data of testing session based on EEG signal classification every 250 msec as described above.

#### **2.3. ICA**

6 Brain-Computer Interface

are reasonable.

as:

Given the confusion matrix **P** these indices can be calculated as follows:

*L* ∑ *i*=1 *pii*,

*L*,*L* ∑ *i*,*j*=1,1

*pij <sup>p</sup>*0*<sup>j</sup>* log2

*p*0*<sup>i</sup> pi*<sup>0</sup>

The better classifier performs the more confusion matrix is close to identity matrix. In case *<sup>L</sup>* states are classified perfectly *<sup>p</sup>* = 1, *<sup>g</sup>* = log2 *<sup>L</sup>*, and *<sup>κ</sup>* = 1. If classification is random, i.e.

Index *p* has an advantage of being evidently interpreted as the percentage of correct classification while its disadvantages are, first, that it does not account for distribution of errors since it does not depend on nondiagonal elements of confusion matrix, second, its lower value depends on the number of states classified. For example the value *p* = 0.5 corresponds to a random recognition when *L* = 2 and twice exceeds the random level when *L* = 4. Thus it is difficult to compare the qualities of BCIs with different states classified. Both indices *g* and *κ* have the advantage of considering the error distribution over nondiagonal elements of confusion matrix and of being normalized to the case of random classifying independently of *L*. In this case both indices are equal to zero. Moreover, *κ* is also normalized to the case of perfect classifying. It takes one in this case, while *g* depends on *L*. Thus, on one hand *κ* is more convenient for comparing qualities of BCI with different *L*, but on another hand *g* gives the estimate of BCI quality directly in the rate of information transfer. Since *g* is calculated every second then it can be measured in bits per second. Thus all three indices

When all probabilities of correct classification are equal, i.e. *pii* = *p* for all *i*, and all probabilities of incorrect classification are equal, i.e. *pij* = (1 − *p*)/(*L* − 1) for all *i* �= *j*, the mutual information between instructions presented and the states classified can be obtained

Based on [43], (4) is often used to estimate BCI efficacy ([3], [5] [44]). But if the corresponding assumptions do not hold true, the value of *g*, calculated according to (4), is lower than the

 1 − *p L* − 1 

*<sup>g</sup>* <sup>=</sup> log2 *<sup>L</sup>* <sup>+</sup> *<sup>p</sup>* log2 *<sup>p</sup>* + (<sup>1</sup> <sup>−</sup> *<sup>p</sup>*)log2

actual mutual information. In this study we used the general formula (3).

*pii <sup>p</sup>*0*<sup>i</sup>* <sup>−</sup> *<sup>L</sup>* ∑ *i*=1

<sup>1</sup> <sup>−</sup> *<sup>L</sup>* ∑ *i*=1

where *<sup>p</sup>*0*<sup>j</sup>* <sup>=</sup> 1/*<sup>L</sup>* is probability of the *<sup>j</sup>*-th instruction to be presented and *pi*<sup>0</sup> <sup>=</sup> *<sup>L</sup>*

 *pij*/*pi*<sup>0</sup> 

*p*0*<sup>i</sup> pi*<sup>0</sup>

, (3)

∑ *j*=1 *pij p*0*<sup>j</sup>* is

(4)

*<sup>p</sup>* <sup>=</sup> <sup>1</sup> *L*

*g* = −

*L* ∑ *i*=1

*κ* =

probability of the *i*-th mental state to be recognized.

*pij* = *pi*<sup>0</sup> for all *j*, then *p* = 1/*L*, *g* = 0, and *κ* = 0.

To identify the sources of brain activity the most relevant to BCI performance we used Independent Component Analysis (ICA). Last years ICA becomes widely used in EEG processing, particularly in BCI studies [16]. ICA provides representation of a multidimensional EEG signal **X**(*t*) (where components of **X**(*t*) represent electric potentials recording from *N* individual electrodes at the head surface) as a superposition of activities of independent components *ξ*:

$$\mathbf{X}(t) = \mathbf{W}\tilde{\boldsymbol{\zeta}}(t) = \mathbf{W}\_1\tilde{\boldsymbol{\zeta}}\_1 + \mathbf{W}\_2\tilde{\boldsymbol{\zeta}}\_2 + \dots + \mathbf{W}\_N\tilde{\boldsymbol{\zeta}}\_N \tag{5}$$

Columns **W***<sup>i</sup>* of matrix **W** specify the contribution of the corresponding independent component (or source) into each of the electrodes and the components *ξ<sup>i</sup>* of the vector *ξ*(*t*) specify sources intensity in each time point. The combination of active sources is supposed to be specific and individual for each mental task. Thus their activities in many tasks can be treated as independent.

There exist a lot of methods to represent the signal **X** in the form (5). We used algorithm RUNICA (MATLAB toolbox EEGLab,[7]). RUNICA provides the identification of the independent components maximizing distinction of their distributions from normal one in terms of kurtosis [15]. The using of this method is reasonable because it corresponds to the suggestion that activities of sources are different in different mental states: in some states they expose ERD, in others - ERS. Thus in one state the distribution of EEG amplitude should be narrow, in others - wide. So the common distribution would be maximally different from the normal one.

To reveal the sources of brain activity the most significant for BCI performance the quality index *κ* was calculated in dependence on the number *Ncmp* of ICA components used for mental state classifying. For each *Ncmp* we found the optimal combination of components providing the highest *κ*. Since the total number 2*<sup>N</sup>* (where *N* = 48) of possible component combinations is extremely large we used exhaustive search to find the optimal combination of components only for *Ncmp* = 3. To find the optimal combination of components for *Ncmp* > 3 we used a "greedy" algorithm which added components one by one starting from the optimal combination of 3 components. At each step a component was added to the optimal component combination found at the previous step, so that extended combination provided the highest *κ*.

When all ICA components are used for mental states classifying, i.e. *Ncmp* = *N*, then *ξ* = **<sup>W</sup>**−1**<sup>X</sup>** where **<sup>W</sup>**−<sup>1</sup> is non-singular. In this case **<sup>C</sup>***<sup>ξ</sup> <sup>i</sup>* <sup>=</sup> **<sup>W</sup>**−1**C***i*(**W**−1)*<sup>T</sup>* and

$$\begin{split} V\_{i}^{\tilde{\mathbf{c}}} &= \text{trace}((\mathbf{C}\_{i}^{\tilde{\mathbf{c}}})^{-1} \tilde{\mathbf{c}} \tilde{\mathbf{c}}^{T}) + \ln(\det(\mathbf{C}\_{i}^{\tilde{\mathbf{c}}})) = \text{trace}((\mathbf{C}\_{i})^{-1} \mathbf{X} \mathbf{X}^{T}) \\ &+ \ln(\det(\mathbf{C}\_{i})) - 2\ln(\det(\mathbf{W})) = V\_{i} - 2\ln(\det(\mathbf{W})) \end{split}$$

Since ln(det(**W**)) does not depend of the mental state, then *V<sup>ξ</sup> <sup>i</sup>* and *Vi* reach minima for the same mental state and the mental state classified by Bayesian classifier in terms of *ξ* coincides with that classified in terms of **X**. Thus, the case *Ncmp* = *N* directly corresponds to BCI performance based on classifying the original signal *X*.

#### **2.4. Sources localization**

With respect to the EEG analysis, it is reasonable to assume that independent sources of electrical brain activity recorded at the head surface are current dipoles distributed over the neocortex. As shown below our experiments confirm this assumption and at least for the sources the most relevant for BCI performance the distribution of electrical potential over the head produced by each of these sources could be actually interpreted in terms of electrical field produced by single current dipole. Thus, for each of such sources its localization was searched in a single dipole approximation. In other words, position and orientation of a single current dipole were searched which provided maximal matching between patterns of EEG distribution on the head surface given by ICA and by a dipole. The pattern of EEG distribution for dipole with given position and orientation was calculated by solving the direct EEG problem.

It was solved by the finite element method (FEM) which allows to take into account individual geometry of the brain and its covers. In FEM, one critical requirement for the mesh generation is to represent the geometric and electrical properties of the head volume conductors. To generate the FEM meshes from the MRI data, MR images were segmented into five sub-regions: white matter, gray matter, cerebrospinal fluid (CSF), skull and scalp. The segmentation of the different tissues within the head was made by means of SPM8 New Segmentation Tool. To construct the FE models of the whole head the FEM mesh generation was performed using tetrahedral elements with inner-node spacing of 2 mm. Thus, the total number of nodes amounted to about 1.5 millions. Electrical conductivities were assigned to the tissues segmented in accordance with each tissue type: 0.14 S/m for white matter, 0.33 S/m for gray matter, 1.79 S/m for CSF, 0.0132 S/m for skull, and 0.35 S/m for scalp [17, 46]. To solve the EEG forward problems, the FEM mesh along with electrical conductivities were imported into the commercial software ANSYS (ANSYS, Inc., PA, USA).

#### **3. Results**

#### **3.1. The ICA components most relevant for BCI control**

For all subjects three indices averaged over all experimental days are shown in Fig. 3. The subjects are ranged according to mean *κ* computed for on-line classification. Note that subjects ranking according to *p* completely preserves their order and according to *g* changes it only slightly (S5 must be shifted before S3). Thus all indices give good relative estimation of subject ability to control BCI. Mean on-line and offline estimates of BCI performance by all indices are shown to be very close. However on-line estimations are more variable. This is reasonable because on-line estimation is based on one classification trial obtained directly during experiment performance, while offline estimation is based on 50 classification trials as described above. Therefore, confusion matrix computed offline is more confident than that computed on-line.

<sup>182</sup> Brain-Computer Interface Systems – Recent Progress and Future Prospects Sources of Electrical Brain Activity Most Relevant to Performance of Brain-Computer Interface Based on Motor Imagery 9 10.5772/55166 Sources of Electrical Brain Activity Most Relevant to Performance of Brain-Computer Interface Based on Motor Imagery http://dx.doi.org/10.5772/55166 183

8 Brain-Computer Interface

**2.4. Sources localization**

direct EEG problem.

**3. Results**

that computed on-line.

Since ln(det(**W**)) does not depend of the mental state, then *V<sup>ξ</sup>*

BCI performance based on classifying the original signal *X*.

the same mental state and the mental state classified by Bayesian classifier in terms of *ξ* coincides with that classified in terms of **X**. Thus, the case *Ncmp* = *N* directly corresponds to

With respect to the EEG analysis, it is reasonable to assume that independent sources of electrical brain activity recorded at the head surface are current dipoles distributed over the neocortex. As shown below our experiments confirm this assumption and at least for the sources the most relevant for BCI performance the distribution of electrical potential over the head produced by each of these sources could be actually interpreted in terms of electrical field produced by single current dipole. Thus, for each of such sources its localization was searched in a single dipole approximation. In other words, position and orientation of a single current dipole were searched which provided maximal matching between patterns of EEG distribution on the head surface given by ICA and by a dipole. The pattern of EEG distribution for dipole with given position and orientation was calculated by solving the

It was solved by the finite element method (FEM) which allows to take into account individual geometry of the brain and its covers. In FEM, one critical requirement for the mesh generation is to represent the geometric and electrical properties of the head volume conductors. To generate the FEM meshes from the MRI data, MR images were segmented into five sub-regions: white matter, gray matter, cerebrospinal fluid (CSF), skull and scalp. The segmentation of the different tissues within the head was made by means of SPM8 New Segmentation Tool. To construct the FE models of the whole head the FEM mesh generation was performed using tetrahedral elements with inner-node spacing of 2 mm. Thus, the total number of nodes amounted to about 1.5 millions. Electrical conductivities were assigned to the tissues segmented in accordance with each tissue type: 0.14 S/m for white matter, 0.33 S/m for gray matter, 1.79 S/m for CSF, 0.0132 S/m for skull, and 0.35 S/m for scalp [17, 46]. To solve the EEG forward problems, the FEM mesh along with electrical conductivities were

For all subjects three indices averaged over all experimental days are shown in Fig. 3. The subjects are ranged according to mean *κ* computed for on-line classification. Note that subjects ranking according to *p* completely preserves their order and according to *g* changes it only slightly (S5 must be shifted before S3). Thus all indices give good relative estimation of subject ability to control BCI. Mean on-line and offline estimates of BCI performance by all indices are shown to be very close. However on-line estimations are more variable. This is reasonable because on-line estimation is based on one classification trial obtained directly during experiment performance, while offline estimation is based on 50 classification trials as described above. Therefore, confusion matrix computed offline is more confident than

imported into the commercial software ANSYS (ANSYS, Inc., PA, USA).

**3.1. The ICA components most relevant for BCI control**

*<sup>i</sup>* and *Vi* reach minima for

**Figure 3.** Indices *p*, *g* and *κ* of BCI control accuracy for all subjects averaged over all experimental days (means and standard errors). Blue bars - on-line, green bars - offline, red - three the most relevant components, grey - optimal components. The subjects are ranged according to *κ* computed on-line. The most relevant components and optimal components were calculated to maximize *κ*. Then these components were used to compute *p* and *g*.

Fig. 4 demonstrates the dependency of *κ* on the number *Ncmp* of ICA components for all subjects on the last experimental day. For each *Ncmp* index *κ* is shown for individual optimal component combination.

**Figure 4.** Index *κ* of BCI control accuracy in dependence on the number *Ncmp* of ICA components used for mental states classifying. The optimal combination of three components (*Ncmp* = 3) was obtained by the exhausted search. The other optimal combinations were obtained by the "greedy" algorithm. Each curve represents the data for each individual subject obtained for the last day of BCI training.

As shown in Fig. 4 *κ* depends on *Ncmp* not monotonically and reaches maximum when some ICA components are discarded. However the classifying accuracy for *Ncmp* = 3 is very close to that obtained for optimal combination of components providing maximal *κ*. Thus the obtained combinations of 3 components can be considered as good representations of the components which are the most relevant for BCI performance. Indices *p*, *g* and *κ* for three the most relevant components and for optimal components averaged over all experimental days are shown in Fig. 3 for all subjects. As shown, elimination of not relevant (noisy) components improves BCI performance several times. Basing on the most relevant and optimal components BCI classifier provides the accuracy of mental state recognition significantly exceeding the random level for all subjects and all indices (t-test, *P <* 0.001). For Subject 1 index *g* computed for optimal components reached 0.83 that is twice more than *g* computed on-line for this subject by the original EEG signal. This maximal value can be also compared, for example, with the maximal value of *g* = 0.58 obtained for Berlin BCI [5] in experiments with a large group of untrained subjects. Thus, filtering of noisy ICA components is an efficient method to improve BCI performance. Note that although filtering of noisy components essentially improves BCI performance for all subjects it almost preserves the order of their ability to control BCI. Thus the relative subjects skill is mainly determined by their inherent properties but not by the properties of BCI classifier. Note also that the quality of BCI performance is rather variable over experimental days for each subject. Due to this variability the order of subjects ability to control BCI was not the same every day. For example, as shown in Fig. 4 on the last experimental day the quality of BCI control for S3 was higher than for subjects S1 and S2, although on average it was lower.

For each experimental day three the most relevant components and optimal components were chosen to maximize *κ* for the data of namely this day. The components happened to be not identical for all experimental days and all subjects but some of them appeared very repeatedly. Four such components which appeared most often among the most relevant ones over all subjects and all experimental days are shown in Fig. 5. The features of these components are shown in terms of their contribution into EEG electrodes (topoplots) and their spectrograms for four considered mental states: relaxation, right or left hand movement imagination and feet movement imagination. The data relates to the last experimental day of Subject 1 who showed the best BCI control on average.

Three of these components (*µ*1, *µ*2 and *µ*3) demonstrate very well exposed Event Related Desynchronization (ERD) of mu-rhythm. For the component denoted *µ*1 mu-rhythm is suppressed during the left hand motor imagery. Its focus is in the right hemisphere presumably above the primary sensorimotor areas presenting the left hand. Respectively, for the components *µ*2 and *µ*3 mu-rhythm is suppressed during the right hand and feet motor imagery and their foci are presumably above the areas presenting right hand and feet. The fourth component denoted *β* we ascribed to the activity of supplementary motor areas (SMA). First, they are located on the midline surface of the hemispheres and hence their activity can produce the focus corresponding to the topoplot of this component. Second, as shown in [27] by electrocorticographic recordings, during the motor act SMA demonstrates ERD in the spectral band 10 - 40 Hz. Just the same is shown for the component *β*. Remind that for our offline analysis EEG signal was filtered in the band 5 - 30 Hz, so ERD for this component is shown in Fig. 4 up to 30 Hz.

Three of four ICA components discussed here (*µ*1, *µ*2 and *β*) are completely identical to those obtained in [11] (Compare Fig. 5 with the first and last rows in Fig. 3 of [11]). Component *β* was identified but not discussed in [11]. Component *µ*3 is new because only three mental tasks (relaxation and left or right hand motor imagery but not feet motor imagery) were used for BCI control in [11].

As in [11] components *µ*1 and *µ*2 demonstrate also well exposed Event Related Synchronization (ERS) during the motor imagery of the opponent extremities. Especially it is seen for *µ*1. Mu-rhythm of this component essentially increased during the motor imagery of both right hand and feet. It is worth to note that the topoplots of four described components were very stable for all experimental days but the manifestation of ERD and ERS was rather variable. Namely the variability of their manifestation determined the variability of BCI control quality. However in any case ERD and ERS are much better exposed in terms of ICA components than in direct EEG recording. The level of ERD can be estimated as *r* = *Sim*/*Srel* where *Sim* and *Srel* are the maximal spectral densities in the alpha band during the motor imagination and relaxation, respectively. For Subject 1 on average over all experimental days and imagination of both hands *r* amounted to 0.2 in terms of ICA components and to 0.69 for two central electrodes C3 and C4 where ERD was maximally exposed. Thus ICA allowed to rectify ERD and ERS due to excluding the components not exposing these changes of the brain activity.

#### **3.2. Sources localization**

10 Brain-Computer Interface

component combination.

obtained for the last day of BCI training.

Fig. 4 demonstrates the dependency of *κ* on the number *Ncmp* of ICA components for all subjects on the last experimental day. For each *Ncmp* index *κ* is shown for individual optimal

**Figure 4.** Index *κ* of BCI control accuracy in dependence on the number *Ncmp* of ICA components used for mental states classifying. The optimal combination of three components (*Ncmp* = 3) was obtained by the exhausted search. The other optimal combinations were obtained by the "greedy" algorithm. Each curve represents the data for each individual subject

As shown in Fig. 4 *κ* depends on *Ncmp* not monotonically and reaches maximum when some ICA components are discarded. However the classifying accuracy for *Ncmp* = 3 is very close to that obtained for optimal combination of components providing maximal *κ*. Thus the obtained combinations of 3 components can be considered as good representations of the components which are the most relevant for BCI performance. Indices *p*, *g* and *κ* for three the most relevant components and for optimal components averaged over all experimental days are shown in Fig. 3 for all subjects. As shown, elimination of not relevant (noisy) components improves BCI performance several times. Basing on the most relevant and optimal components BCI classifier provides the accuracy of mental state recognition significantly exceeding the random level for all subjects and all indices (t-test, *P <* 0.001). For Subject 1 index *g* computed for optimal components reached 0.83 that is twice more than *g* computed on-line for this subject by the original EEG signal. This maximal value can be also compared, for example, with the maximal value of *g* = 0.58 obtained for Berlin BCI [5] in experiments with a large group of untrained subjects. Thus, filtering of noisy ICA components is an efficient method to improve BCI performance. Note that although filtering of noisy components essentially improves BCI performance for all subjects it almost preserves the order of their ability to control BCI. Thus the relative subjects skill is mainly determined by their inherent properties but not by the properties of BCI classifier. Note also that the quality of BCI performance is rather variable over experimental days for each subject. Due to this variability the order of subjects ability to control BCI was not the same every day. For example, as shown in Fig. 4 on the last experimental day the quality of BCI control for

S3 was higher than for subjects S1 and S2, although on average it was lower.

Although the shown ICA components happened to be rather stable and repeatable over all subjects and all experimental days one could expect that they are only the formal

**Figure 5.** Topoplots and spectrograms for four sources of electrical brain activity the most relevant to BCI performance. *µ*1, *µ*2 and *µ*3 demonstrate ERD of mu-rhythm during imagination of left hand, right hand and feet movement. *β* demonstrates ERD in the band 10 - 30 Hz. Spectral frequency is given in Hz, blue curves - relaxation, red lines - right hand motor imagery, green lines - left hand motor imagery, black lines - feet motor imagery.

results of some mathematical transformations of the actual experimental data and have no physiological sense. To clarify their sense we show, first, that their contribution to EEG recordings can be explained by the current dipole sources of brain activity located in the sensorimotor cortical areas, and, second, that locations of these sources coincide with locations of brain activity identified in fMRI study.

As an example, topoplot of *µ*1 is compared in Fig. 6 with that produced by a current dipole model found by solving inverse EEG problem. To solve it we found the location and orientation of the dipole provided the best fit between its contribution to EEG electrodes obtained by solving the direct EEG problem and the contribution obtained for *µ*1. As shown in Fig. 6 the found dipole provides a good coincidence between both types of contribution. On average over all subject and all components the residual variance of the single dipole approximation amounted only 1%.

Examples of dipole localization along with results of fMRI analysis are shown in Fig. 7. Figure demonstrates voxels for which BOLD level was significantly higher left hand (a), right hand (b), and feet (c) motor imagery compared to relaxation.

Dipole positions obtained for components *µ*1 and *µ*2 happened to be very close to the sensorimotor "hand areas" marked in Fig. 7a,b. Dipole position obtained for component *µ*3 is close to the sensorimotor "feet areas" in the superior regions of post- and precentral gyri (Fig.7c) and for component *β* slightly more anterior, i.e. close to SMA. However, the positions of dipoles corresponding to *µ*1, *µ*2 and *µ*3 happened to be a little deeper than foci of fMRI activity near "hand" and "feet" areas. Small discrepancy (in the limit of 25 <sup>186</sup> Brain-Computer Interface Systems – Recent Progress and Future Prospects Sources of Electrical Brain Activity Most Relevant to Performance of Brain-Computer Interface Based on Motor Imagery 13 10.5772/55166 Sources of Electrical Brain Activity Most Relevant to Performance of Brain-Computer Interface Based on Motor Imagery http://dx.doi.org/10.5772/55166 187

12 Brain-Computer Interface

**μ1 μ2**

**μ3 β**

**Figure 5.** Topoplots and spectrograms for four sources of electrical brain activity the most relevant to BCI performance. *µ*1, *µ*2 and *µ*3 demonstrate ERD of mu-rhythm during imagination of left hand, right hand and feet movement. *β* demonstrates ERD in the band 10 - 30 Hz. Spectral frequency is given in Hz, blue curves - relaxation, red lines - right hand motor imagery,

results of some mathematical transformations of the actual experimental data and have no physiological sense. To clarify their sense we show, first, that their contribution to EEG recordings can be explained by the current dipole sources of brain activity located in the sensorimotor cortical areas, and, second, that locations of these sources coincide with

As an example, topoplot of *µ*1 is compared in Fig. 6 with that produced by a current dipole model found by solving inverse EEG problem. To solve it we found the location and orientation of the dipole provided the best fit between its contribution to EEG electrodes obtained by solving the direct EEG problem and the contribution obtained for *µ*1. As shown in Fig. 6 the found dipole provides a good coincidence between both types of contribution. On average over all subject and all components the residual variance of the single dipole

Examples of dipole localization along with results of fMRI analysis are shown in Fig. 7. Figure demonstrates voxels for which BOLD level was significantly higher left hand (a),

Dipole positions obtained for components *µ*1 and *µ*2 happened to be very close to the sensorimotor "hand areas" marked in Fig. 7a,b. Dipole position obtained for component *µ*3 is close to the sensorimotor "feet areas" in the superior regions of post- and precentral gyri (Fig.7c) and for component *β* slightly more anterior, i.e. close to SMA. However, the positions of dipoles corresponding to *µ*1, *µ*2 and *µ*3 happened to be a little deeper than foci of fMRI activity near "hand" and "feet" areas. Small discrepancy (in the limit of 25

right hand (b), and feet (c) motor imagery compared to relaxation.

green lines - left hand motor imagery, black lines - feet motor imagery.

locations of brain activity identified in fMRI study.

approximation amounted only 1%.

**Figure 6.** Topoplot of component *µ*1 (experiment) compared with topoplot produced by a single dipole model (subject S1)

**Figure 7.** Examples of dipole localization for *µ*1 (a), *µ*2 (b) and *µ*3 (c) components along with the results of fMRI analysis.

mm) between current dipole positions and areas of maximal fMRI activity was also obtained in the most studies with measurements of somatosensory activity as a response to hand electrical stimulation (see, for example, [6, 12, 41]). This may be explained by the fact that different processes are responsible for EEG and fMRI outcomes. EEG is the result of neuronal electric activity while fMRI relates to blood flow activity associated with the energetically dominant processes. If, for example, neuronal electric activity in the depth of central sulcus is less energy-expensive than in the crown of pre- and postcentral gyrus than fMRI activity removes up comparing with EEG activity. Moreover, in our experiments a motor imagination results in the increase of fMRI activity but decrease of EEG activity. Since relation between fMRI activity and neuronal electric activity is rather complex [25], then the brain areas where these two kinds of brain activity are maximally exposed could be slightly different. Hence the small discrepancy of their positions does not evidence against the precision of dipole location. In our experiments the distance between dipole location and the COM of fMRI activity near central sulcus averaged over subjects and components amounted to 9 ± 1.5 mm. The dipoles corresponding to components *µ*1, *µ*2 and *µ*3 were located at the bottom of the central sulcus (Fig. 7), i.e. at the area 3a responsible for proprioceptive sensation. According to reports of the subjects this corresponds to their internal feeling of the imagined movement.

#### **4. Discussion**

Generally, the difficulties in interpreting the original EEG signals are due to the overlapping of activities coming from different brain sources, due to the distortion of the current flows caused by the inhomogeneity in the conductivity of the brain and its covers and due to uncertainty not only in dipole source locations but also in the dipole orientation which determines the relation between its position and the EEG amplitude maxima which it produces at the head surface. These difficulties result in common notion that EEG data provide high temporal but very low spatial resolution comparing to fMRI data. Last years there were many efforts to match these techniques to enhance both resolutions [25]. One of the approaches is presented here in the chapter. It contains the following steps:


This approach allowed us to find the location of the sources of the brain activity which are the most relevant for motor imagination. Three of them denoted as *µ*1, *µ*2 and *µ*3 happened to be localized at the bottom of central sulcus close to the Brodmann area 3a responsible for proprioceptive sensation. This location corresponds to the internal feeling of the imagined hand or feet movement according to the reports of subjects. Thus, the experience of imagery (at least for motor imagery) involves perceptual structures despite the absence of perceptual stimulation.

There is a long story of the debates concerning the brain areas involved into the motor imagination, especially the involvement of the primary sensorimotor cortex. Activated areas in or around primary motor area have been described in PET studies [40] during imagination of arm movement, but not of the grasping movement [8]. Primary sensorimotor cortex fMRI activation during the motor imagination was denied in [35, 37], but was claimed in [9, 20, 36].

There were also many efforts to reveal whether SM1 is active during motor imagery basing on EEG data [2, 26, 30]. Particularly, in [26, 30] the conclusion that it activates was based on the observation that ERD is maximally exposed at the electrodes related to SM1 activity. However as mentioned above it is difficult to prescribe electrical activity to some particular brain area on the base of original EEG data. We believe that our approach allows to do this more substantiated. The reasonable and well interpreted results were obtained here due to solving the inverse EEG problem with the data refined by ICA. The components which are the most relevant to the performance of BCI based on the hand motor imagination were also obtained in [22]. They are very close or may be even identical to the components obtained in the present paper. But in [22] they were not interpreted in terms of dipole sources and consequently the inverse EEG problem was not solved. We tried to solve it with the most realistic head model as a volume conductor taking into account the individual geometry of brain and its covers and the difference in conductivity of white and grey matters, CSF, skull and scalp. The only thing that we ignored is the anisotropy of the white matter (WM). Although as shown in [19] the head model incorporating realistic anisotropic WM conductivity distributions do not substantially improve the accuracy of EEG dipole localization, our next step is to take into account also the anisotropy.

Besides the foci of fMRI activity which were associated with three sources of EEG activity the most relevant for BCI performance (foci in primary sensorimotor areas 3 and 4 shown in Fig. 7) we observed many other foci. Among them are foci in cerebellum, superior temporal area 22, ventral anterior cingulate area 24 and insula. Thus motor imagery involves rather wide brain networks. According to the literature it can involve also superior and inferior parietal lobule, pre-frontal areas, inferior frontal gyrus, secondary somatosensory area and basal ganglia (see, for example [9, 39]). We also obtained many other ICA components which were relevant to motor imagination except four main components *µ*1, *µ*2, *µ*3 and *β*. Since we obtained good relation between these components and fMRI data the natural goal of our future research is to reveal the relations between other fMRI foci and other ICA components.

#### **Acknowledgments**

14 Brain-Computer Interface

**4. Discussion**

stimulation.

location. In our experiments the distance between dipole location and the COM of fMRI activity near central sulcus averaged over subjects and components amounted to 9 ± 1.5 mm. The dipoles corresponding to components *µ*1, *µ*2 and *µ*3 were located at the bottom of the central sulcus (Fig. 7), i.e. at the area 3a responsible for proprioceptive sensation. According to reports of the subjects this corresponds to their internal feeling of the imagined movement.

Generally, the difficulties in interpreting the original EEG signals are due to the overlapping of activities coming from different brain sources, due to the distortion of the current flows caused by the inhomogeneity in the conductivity of the brain and its covers and due to uncertainty not only in dipole source locations but also in the dipole orientation which determines the relation between its position and the EEG amplitude maxima which it produces at the head surface. These difficulties result in common notion that EEG data provide high temporal but very low spatial resolution comparing to fMRI data. Last years there were many efforts to match these techniques to enhance both resolutions [25]. One of

1. Subjects training in BCI control to stabilize and to contrast the patterns of EEG activities

2. When subjects are sufficiently trained, finding the ICA components which are the most relevant for the BCI control. This allows to suppress the influence of not relevant sources

3. Obtaining of individual geometries of the brain and its covers (e.g. by MRI data) - as

4. The inverse EEG problem solving for each individual relevant component. This allows to

5. Verifying the found dipole locations by fMRI data to be sure that the found ICA components are not only the formal results of some mathematical transformations.

This approach allowed us to find the location of the sources of the brain activity which are the most relevant for motor imagination. Three of them denoted as *µ*1, *µ*2 and *µ*3 happened to be localized at the bottom of central sulcus close to the Brodmann area 3a responsible for proprioceptive sensation. This location corresponds to the internal feeling of the imagined hand or feet movement according to the reports of subjects. Thus, the experience of imagery (at least for motor imagery) involves perceptual structures despite the absence of perceptual

There is a long story of the debates concerning the brain areas involved into the motor imagination, especially the involvement of the primary sensorimotor cortex. Activated areas in or around primary motor area have been described in PET studies [40] during imagination of arm movement, but not of the grasping movement [8]. Primary sensorimotor cortex fMRI activation during the motor imagination was denied in [35, 37], but was claimed in [9, 20, 36]. There were also many efforts to reveal whether SM1 is active during motor imagery basing on EEG data [2, 26, 30]. Particularly, in [26, 30] the conclusion that it activates was based on the observation that ERD is maximally exposed at the electrodes related to SM1 activity.

the approaches is presented here in the chapter. It contains the following steps:

result inverse problem can be solved taking geometries into account

related to the mental tasks under consideration.

of brain activity and to refine the most relevant ones.

perform the solution in the single dipole approximation.

This research has been partly funded by project GACR P202/10/0262, by the IT4Innovations Centre of Excellence project, reg. no. CZ.1.05/1.1.00/02.0070 supported by Operational Programme "Research and Development for Innovations" funded by Structural Funds of the European Union and state budget of the Czech Republic and by long-term strategic development financing of the Institute of Computer Science (RVO:67985807).

#### **Author details**

Alexander Frolov1,2,<sup>⋆</sup>, Dušan Húsek3, Pavel Bobrov1,2, Olesya Mokienko1 and Jaroslav Tintera<sup>4</sup>

<sup>⋆</sup> Address all correspondence to: aafrolov@mail.ru

1 Institute of Higher Nervous Activity and Neurophysiology, RAS, Moscow, Russia

2 Faculty of Electronics and Informatics, VŠB-Technical University of Ostrava, Ostrava – Poruba, Czech Republic

3 Institute of Computer Science, Academy of Sciences of the Czech Republic, Prague, Czech Republic

4 Institute for Clinical and Experimental Medicine, Prague, Czech Republic

#### **References**


[14] Haynes J, Rees G (2006) Decoding mental states from brain activity in humans. Nature Reviews Neuroscience 7: 523-534.

16 Brain-Computer Interface

600-602

28(10):1403-12

Neural engineering 4: R32.

Electroecephalogr. Clin. Neurophysiol. 96:183-193.

subjects. Neuroimage. 2007. 37(2): 539U550.

based on motor imagery. NNW 1/12: 21-37.

Neuroimage, 17(3):1373-83.

[1] Bashashati A, Fatourechi M, Ward R, Birch G (2007) A survey of signal processing algorithms in brain-computer interfaces based on electrical brain signals. Journal of

[2] Beisteiner R, Hollinger P, Lindinger G, Lang W, Berthoz A (1995) Mental representations of movements. Brain potentials associated with imagination of hand movements.

[3] Besserve M, Jerbi K, Laurent F, Baillet S, Martinerie J, et al. (2007) Classification methods

[4] Birbaumer N., Cohen L.G. (2007). Brain-computer interfaces: communication and

[5] Blankertz B., Dornhege G., Krauledat M., Muller K.R, Curio G. The non-invasive Berlin Brain Computer Interface: Fast acquisition of effective performance in untrained

[6] Christmann C, Ruf M, Braus D F, Flor H (2002) Simultaneous electroencephalography and functional magnetic resonance imaging of primary and secondary somatosensory

[7] Delorme A., Makeig S., 2004, EEGLAB: an open source toolbox for analysis of single-trial

[8] Decety J, Perani D, Jeannerod M, Bettinardi V, Tadary B, Woods R, Mazziotta JC, Fazio F (1994) Mapping motor representations with positron emission tomography. Nature 371:

[9] Formaggio E, Storti SF, Cerini R, Fiaschi A, Manganotti P (2010) Brain oscillatory activity during motor imagery in EEG-fMRI coregistration. Magnetic Resonance Imaging

[10] Frolov A, Husek D., Bobrov P. (2011) Comparison of four classification methods for

[11] Frolov A, Husek D, Bobrov P, Korshakov A, Chernikova L, Konovalov R, Mokienko O (2012) Sources of EEG activity most relevant to performance of brain-computer interface

[12] Del Gratta C, Della Penna S, Ferretti A, Franciotti R, Pizzella V, Tartaro A, Torquati K, Bonomo L, Romani G L, Rossini PM (2002). Topographic organization of the human primary and secondary somatosensory cortices: comparison of fMRI and MEG findings.

[13] Grech R, Cassar T, Muscat J, Camilleri KP, Fabri SG, Zervakis M, Xanthopoulos P, Sakkalis V, Vanrumste B (2008) Review on solving the inverse problem in EEG source

analysis. Journal of NeuroEngineering and Rehabilitation 5(25): 1-33.

˝

cortex in humans after electric stimulation. Neuroscience Letters 333: 69-73.

for ongoing EEG and MEG signals. Biological research 40: 415-437.

restoration of movement in paralysis, J Physiol., 579, pp. 621-636.

EEG dynamics. Journal of Neuroscience Methods 134:9-21

brain computer interface. Neural Network World, 21(2) 101-115

**References**


[41] Thees S, Blabkenburg F, Taskin B, Curio G, Villringer A (2003) Dipole source localization and fMRI of simultaneously recorded data applied to somatosensory categorization. NeuroImage 18: 707-719

18 Brain-Computer Interface

Applications 16: 293-299.

NeuroImage 31: 153-159.

2311-2318.

Abstr. 18: 1208

clinical Neurophysiology 103: 642-651.

humans. Neuroscience Letters. 239: 65-68.

[28] Pfurtscheller G, Flotzinger D, Kalcher J (1993) Brain-computer Interface–a new communication device for handicapped persons. Journal of Microcomputer

[29] Pfurtscheller, G., Neuper, C. (1994). Event-related synchronization of mu rhythm in the

[30] Pfurtscheller G, Neuper C, Flotzinger D, Pregenzer M (1997) EEG-based discrimination between imagination of right and left hand movement. Electroencephalography and

[31] Pfurtscheller G, Neuper C (1997) Motor imagery activates primary sensorimotor area in

[32] Pfurtscheller G., Neuper C. (2001) Motor imagery and direct brain-computer

[33] Pfurtscheller G., Brunner C., Schlogl A., Lopes da Silva F. (2006) Mu rhythm (de) synchronization and EEG single-trial classification of different motor imagery tasks.

[34] Pineda J.A. (2005) The functional significance of mu rhythm: Translating "seeing" and

[35] Rao SM, Binder JR, Bandettini PA, Hammeke TA, Yetkin FZ, Jesmanowicz A, Lisk LM, Morris GL, Mueller WM, Estkowski LD, Wong EC, Haughton VM, Hyde JS (1993) Functional magnetic resonance imaging of complex human movements. Neurology 43:

[36] Sabbah P, Simond G, Levrier O, Habib M, Trabaud V, Murayama N, Mazoyer BM, Briant JF, Raybaud C, Salamon G (1995) Functional magnetic resonance imaging at 1.5 T during

[37] Sanes JN, Stern CE, Baker JR, Kwong KK, Donoghue JP, Rosen BR (1993) Human frontal motor cortical areas related to motor performance and mental imagery. Soc. Neurosci.

[38] Sitaram R, Zhang H, Guan C, Thulasidas M, Hoshi Y, Ishikawa A, Shimizu K, Birbaumer N (2007) Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain-computer interface. NeuroImage 34.4, 1416-1427

[39] Solodkin A, Hlustik P, Chen EE, Small SL (2004) Fine modulation in network activation during motor execution and motor imagery. Cerebral Cortex 14.11, 1246-1255

[40] Stephan K M., Fink GR, Passingham RE., Silbersweig D., Ceballos-Baumann AO., Frith CD., Frackowiak RSJ (1995) Functional anatomy of the mental representation of upper extremity movements in healthy subjects. Journal of Neurophysiology, 73 (1): 373-386

EEG over the cortical hand area in man. Neurosci. Lett. 174, 93-96.

communication. Proceedings of the IEEE, 82 (7), 1123-1134.

"hearing" into "doing". Brain Research Reviews 50: 57- 68.

sensory motor and cognitive tasks. Eur. Neurol. 35: 131-136/


## **A Review of P300, SSVEP, and Hybrid P300/SSVEP Brain-Computer Interface Systems**

Setare Amiri, Ahmed Rabbi, Leila Azinfar and Reza Fazel-Rezai

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/56135

#### **1. Introduction**

There are several techniques for measuring brain activities such as magnetoencephalogram (MEG), near infrared spectroscopy (NIRS), electrocorticogram (ECoG), functional magnetic resonance imaging (fMRI), and electroencephalography (EEG). Each technique has some advantages and disadvantages compared to other techniques. For example, in EEG the temporal resolution is high but the special resolution is low compared to fMRI. Because of low cost and portability, EEG has been largely used in both clinical and research applications [1][2] [3][4].

One of the EEG research applications is in a brain computer interface (BCI) system. A BCI can provide a new way of communications for special users who cannot communicate via normal pathways. A BCI system can send commands, controlled by brain activity and distinguished by EEG signal processing. There are many features which can be extracted from EEG, for example, six brain rhythms can be distinguished in EEG based on the differences in frequency ranges; delta (1- 4 Hz), theta (4-7 Hz), alpha (8-12 Hz), mu (8-13 Hz), beta (12-30 Hz), and gamma (25-100 Hz). The delta and theta rhythms occur in high emotional conditions or in a sleep stage. The alpha rhythm happens in awake and eyes closed relax condition. The oscilla‐ tion in alpha rhythm has smooth pattern. The beta rhythm pattern is desynchronized and the condition is the normal awake open eyes. The gamma rhythm can be acquired from somato‐ sensory cortex and mu rhythm from sensorimotor cortex.

BCIs are categorized based on the EEG brain activity patterns into four different types: event– related desynchronization/synchronization (ERD/ERS) [5], steady state visual evoke poten‐ tials (SSVEP) [6][7][8], P300 component of event related potentials (ERPs) [9], and slow corti‐

© 2013 Amiri et al.; licensee InTech. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2013 Amiri et al.; licensee InTech. This is a paper distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

cal potentials (SCPs) [6][10]. The focus of this chapter is on P300, SSVEP and hybrid P300- SSVEP BCI systems.

Compared to other modalities for BCI approaches, such as the P300-based and the SCP BCIs, SSVEP-based BCI system has the advantage of having higher accuracy and higher information transfer rate (ITR). In addition, short/no training time and fewer EEG channels are required. However, similar to other BCI modalities, most current SSVEP-based BCI techniques also face some challenges that prevent them from being accepted by the majority of the population. Two important features of each BCI system are information transfer rate and required training time. A general comparison of different BCI approaches is shown in Figure 1.

**Figure 1.** A general comparison of SCP, ERD/ERS, P300, and SSVEP with respect to their training time and information transfer rate.

The process of detecting patterns from EEG is divided into three steps [11]: signal preprocessing, feature extraction and classification. The first step is to remove noise such as artifacts or power line noise which is added to EEG. So filtering is the first step in EEG signal pre-processing. Band pass and notch filters are the most common filters utilized in EEG signal filtering.

In the next step, features that are selected in feature extraction step and the type of classifier should be chosen based on the type of BCI. For example, for P300, time domain or timefrequency domain features such as wavelets are appropriate and for SSVEP BCIs frequen‐ cy domain features are more appropriate. Classifiers such as Fischer's linear discriminant analysis (FLDA), Bayesian linear discriminant analysis (BLDA), stepwise linear discrimi‐ nant analysis (SWLDA), and support vector machine (SVM) are utilized [12][13] for P300 classifications. For SSVEP feature extraction and classification, different methods such as the Fast Fourier transform (FFT), the canonical correlation analysis (CCA), stimuluslocked inter-trace correlation (SLIC), and the common special patterns (CSPs) have been used [14][15] [16].

In recent years, the BCI research projects and the number of publications in this area have been increased rapidly [17]. Different areas of research such as new feature extraction methods, new classification techniques, new BCI paradigms, or new approaches for combining different BCI types have been investigated for improving accuracy, reliability, information transfer rate, and user acceptability. Combining different BCI types called a hybrid BCI is a new trend in BCI research which is the main focus of this chapter. In the next sections, the P300 and SSVEP BCI are explained and then different approaches for building a P300-SSVEP hybrid BCI are discussed.

#### **2. P300-based BCI**

cal potentials (SCPs) [6][10]. The focus of this chapter is on P300, SSVEP and hybrid P300-

Compared to other modalities for BCI approaches, such as the P300-based and the SCP BCIs, SSVEP-based BCI system has the advantage of having higher accuracy and higher information transfer rate (ITR). In addition, short/no training time and fewer EEG channels are required. However, similar to other BCI modalities, most current SSVEP-based BCI techniques also face some challenges that prevent them from being accepted by the majority of the population. Two important features of each BCI system are information transfer rate and required training time.

**Figure 1.** A general comparison of SCP, ERD/ERS, P300, and SSVEP with respect to their training time and information

The process of detecting patterns from EEG is divided into three steps [11]: signal preprocessing, feature extraction and classification. The first step is to remove noise such as

A general comparison of different BCI approaches is shown in Figure 1.

196 Brain-Computer Interface Systems – Recent Progress and Future Prospects

SSVEP BCI systems.

transfer rate.

#### **2.1. The P300 component**

Event related potentials (ERPs) are the measurement of brain responses to specific cognitive, sensory or motor events. One of the main approaches towards BCI is based on ERPs. P300 is a major peak and one of the most used components of an ERP. The presentation of stimulus in an oddball paradigm can produce a positive peak in the EEG, 300 msec after onset of the stimulus. The stimulus can either be visual, auditory or somatosensory. This evoked response in EEG is called P300 component of ERP.

#### **2.2. Properties of P300**

The spatial amplitude distribution is strongest in the occipital region of brain and is symmetric around central location Cz recorded based on the 10-20 international system [18]. The spatial amplitude distribution of 10-20 international system and the electrodes that P300 is typically recorded from are shown in the following Figure 2. In terms of temporal pattern, P300 wave amplitude is typically in the range of 2 to 5 μV with duration of 150 to 200 msec as shown in Figure 3. Considering the P300 low amplitude relative to background activities of the brain (in the rage of 50 μV), it is clear that P300 detection requires special signal processing. One of the simplest approaches is ensemble averaging EEG over multiple responses to enhance P300 amplitude to identify it while suppressing background EEG activities.

**Figure 2.** Recoding of EEG based on 10-20 system and location of the electrodes typically used for P300 detection [18].

P300-based BCI has been used as one of the most widely used BCI systems since 1988 [1]. New advancements in inexpensive and portable hardware made it possible to have real-life application outside of laboratory environment [17][1][20][21][22]. P300-based BCI has been used from controlling a wheelchair for helping disable people to a virtual keyboard for spelling word and interacting with computers. This type of BCI systems possesses the potential to improve the quality of life.

P300-based visual speller paradigms are attracting much attention as they could provide means to communicate letters, words, and simple commands to computer directly from the

**Figure 3.** Temporal pattern of P300 component.

simplest approaches is ensemble averaging EEG over multiple responses to enhance P300

**Figure 2.** Recoding of EEG based on 10-20 system and location of the electrodes typically used for P300 detection [18].

improve the quality of life.

P300-based BCI has been used as one of the most widely used BCI systems since 1988 [1]. New advancements in inexpensive and portable hardware made it possible to have real-life application outside of laboratory environment [17][1][20][21][22]. P300-based BCI has been used from controlling a wheelchair for helping disable people to a virtual keyboard for spelling word and interacting with computers. This type of BCI systems possesses the potential to

P300-based visual speller paradigms are attracting much attention as they could provide means to communicate letters, words, and simple commands to computer directly from the

amplitude to identify it while suppressing background EEG activities.

198 Brain-Computer Interface Systems – Recent Progress and Future Prospects

brain. In the following sections, we will review the classical speller paradigm and discuss current and future trends in this area.

Processing and successful use of P300 wave in a BCI application requires several processing steps. First of all, the recorded EEG data have to be processed to reduce the effect of noise. A feedback mechanism is required where a visible signal is presented in the monitor correlated with the recorded signal. A pattern recognition or classification algorithm has to be developed to identify the P300 wave in the recorded ERP epochs. The algorithm parameters should be adjustable to adapt according to the change of user characteristics [11][17].

Figure 4 shows a typical BCI setting for speller application. Stimulus is presented by random flashing of the characters on the screen. This eventually evokes P300 wave in the recorded EEG. A signal processing technique performs the processing of P300 related information and the classifier contains the pattern recognition algorithm as described earlier [17].

The classical paradigm for P300-based BCI speller was originally introduced by Farwell and Donchin in 1988 [1]. This Row-Column (RC) paradigm is the most popular speller format. It consists of 6 × 6 matrix of characters as shown in Figure 5. This matrix is presented on computer screen and the row and columns are flashed in a random order. The user is instructed to select a character by focusing on it. The flashing row or column evokes P300 response in EEG. The

Figure 4. A typical P300 BCI setup with visual feedback. **Figure 4.** A typical P300 BCI setup with visual feedback.

non-flashing rows and columns do not contribute in generating P300 [1]. Therefore, the computer can determine the desired row and column after averaging several responses. Finally, the desired character is selected. The classical paradigm for P300-based BCI speller was originally introduced by Farwell and Donchin in 1988 [1]. This Row-Column (RC) paradigm is the most popular speller format. It

consists of 6 × 6 matrix of characters as shown in Figure 5. This matrix is presented on


Figure 5. A typical row/column paradigm [1]. **Figure 5.** A typical row/column paradigm [1].

It is interesting to note that P300-based BCI did not receive much attention when it was first proposed. However, recent trend is quite different where P300 BCI has emerged as one of the main BCI approaches. The researchers have focused on identifying the scopes of improvement of the traditional paradigm by introducing new ways of flashing, introducing colors, or investigating other ways to enhance the ERPs. Much focus has put on applying advanced digital signal processing techniques and classification methods in order to improve the classification results. Also, there have been several attempts to introduce new paradigms to evoke P300 potentials. Figure 6 shows such a different approach which is called single character (SC) paradigm that only single character is flashed instead of a row or column. The SC paradigm randomly flashes one character at a time with a delay between flashes [17]. The delay in SC speller is longer than the delay in RC speller. Though SC speller is slower than RC speller, SC speller can produce It is interesting to note that P300-based BCI did not receive much attention when it was first proposed. However, recent trend is quite different where P300 BCI has emerged as one of the main BCI approaches. The researchers have focused on identifying the scopes of improvement of the traditional paradigm by introducing new ways of flashing, introducing colors, or investigating other ways to enhance the ERPs. Much focus has put on applying advanced digital signal processing techniques and classification methods in order to improve the classification results. Also, there have been several attempts to introduce new paradigms to evoke P300 potentials. Figure 6 shows such a different approach which is called single character Figure 5. A typical row/column paradigm[1]. It is interesting to note that P300-based BCI did not receive much attention when it was first proposed. However, recent trend is quite different where P300 BCI has emerged as one of the main BCI approaches. The researchers have focused on identifying the scopes of improvement of the traditional paradigm by introducing new ways of flashing, introducing

Figure 6. Single character paradigm where each character is flashed [14][1].

Checkerboard (CB) speller is another paradigm proposed to overcome a problem associated with RC speller [17]. This drawback is arising from the distraction or inherent noise due to row/column association [17]. CB speller effectively reduces these two

The region-based (RB) paradigm was proposed by Fazel-Rezai et. al*.* in 2009 [21]. It is a two-level speller where the regions have to flash instead of rows and columns. In the first level, characters are placed in several regions (seven groups) as shown in Figure 8 [17][1][20][21]. The users are instructed to focus attention on a specific character in one of the seven regions. After several flashes

limitations as the characters are arranged in a checkerboard style as shown in Figure 7. CB speller also increases ITR [20].

larger P300 amplitude [17].

Figure 7. Checkerboard paradigm [20].

methods in order to improve the classification results. Also, there have been several attempts to introduce new paradigms to evoke P300 potentials. Figure 6 shows such a different approach which is called single character (SC) paradigm that only single character

It is interesting to note that P300-based BCI did not receive much attention when it was first proposed. However, recent trend is

ways to enhance the ERPs. Much focus has put on applying advanced digital signal processing techniques and classification

(SC) paradigm that only single character is flashed instead of a row or column. The SC paradigm randomly flashes one character at a time with a delay between flashes [17]. The delay in SC speller is longer than the delay in RC speller. Though SC speller is slower than RC speller, SC speller can produce larger P300 amplitude [17]. methods in order to improve the classification results. Also, there have been several attempts to introduce new paradigms to evoke P300 potentials. Figure 6 shows such a different approach which is called single character (SC) paradigm that only single character is flashed instead of a row or column. The SC paradigm randomly flashes one character at a time with a delay between flashes [17]. The delay in SC speller is longer than the delay in RC speller. Though SC speller is slower than RC speller, SC speller can produce It is interesting to note that P300-based BCI did not receive much attention when it was first proposed. However, recent trend is quite different where P300 BCI has emerged as one of the main BCI approaches. The researchers have focused on identifying the scopes of improvement of the traditional paradigm by introducing new ways of flashing, introducing colors, or investigating other ways to enhance the ERPs. Much focus has put on applying advanced digital signal processing techniques and classification

Figure 5. A typical row/column paradigm [1].

Figure 6. Single character paradigm where each character is flashed [14][1]. **Figure 6.** Single character paradigm where each character is flashed [14][1].

larger P300 amplitude [17].

Checkerboard (CB) speller is another paradigm proposed to overcome a problem associated with RC speller [17]. This drawback is arising from the distraction or inherent noise due to row/column association [17]. CB speller effectively reduces these two limitations as the characters are arranged in a checkerboard style as shown in Figure 7. CB speller also increases ITR [20]. Checkerboard (CB) speller is another paradigm proposed to overcome a problem associated with RC speller [17]. This drawback is arising from the distraction or inherent noise due to row/column association [17]. CB speller effectively reduces these two limitations as the characters are arranged in a checkerboard style as shown in Figure 7. CB speller also increases ITR [20]. Figure 6. Single character paradigm where each character is flashed [14][1]. Checkerboard (CB) speller is another paradigm proposed to overcome a problem associated with RC speller [17]. This drawback is



The region-based (RB) paradigm was proposed by Fazel-Rezai et. al*.* in 2009 [21]. It is a two-level speller where the regions have to flash instead of rows and columns. In the first level, characters are placed in several regions (seven groups) as shown in Figure 8 [17][1][20][21]. The users are instructed to focus attention on a specific character in one of the seven regions. After several flashes

arising from the distraction or inherent noise due to row/column association [17]. CB speller effectively reduces these two

limitations as the characters are arranged in a checkerboard style as shown in Figure 7. CB speller also increases ITR [20].

The region-based (RB) paradigm was proposed by Fazel-Rezai et. al*.* in 2009 [21]. It is a two-level speller where the regions have to

The delay in SC speller is longer than the delay in RC speller. Though SC speller is slower than RC speller, SC speller can produce flash instead of rows and columns. In the first level, characters are placed in several regions (seven groups) as shown in Figure 8 [17][1][20][21]. The users are instructed to focus attention on a specific character in one of the seven regions. After several flashes **Figure 7.** Checkerboard paradigm [20].

Figure 7. Checkerboard paradigm [20].

non-flashing rows and columns do not contribute in generating P300 [1]. Therefore, the computer can determine the desired row and column after averaging several responses.

The classical paradigm for P300-based BCI speller was originally introduced by Farwell and Donchin in 1988 [1]. This Row-Column (RC) paradigm is the most popular speller format. It consists of 6 × 6 matrix of characters as shown in Figure 5. This matrix is presented on computer screen and the row and columns are flashed in a random order. The user is instructed to select a character by focusing on it. The flashing row or column evokes P300 response in EEG. The non-flashing rows and columns do not contribute in generating P300 [1]. Therefore, the computer can determine the desired row and column after averaging

Figure 4. A typical P300 BCI setup with visual feedback.

**Extraction Classification**

**Feature** 

It is interesting to note that P300-based BCI did not receive much attention when it was first proposed. However, recent trend is quite different where P300 BCI has emerged as one of the main BCI approaches. The researchers have focused on identifying the scopes of improvement of the traditional paradigm by introducing new ways of flashing, introducing colors, or investigating other ways to enhance the ERPs. Much focus has put on applying advanced digital signal processing techniques and classification methods in order to improve the classification results. Also, there have been several attempts to introduce new paradigms to evoke P300 potentials. Figure 6 shows such a different approach which is called single character

It is interesting to note that P300-based BCI did not receive much attention when it was first proposed. However, recent trend is quite different where P300 BCI has emerged as one of the main BCI approaches. The researchers have focused on identifying the scopes of improvement of the traditional paradigm by introducing new ways of flashing, introducing

Figure 6. Single character paradigm where each character is flashed [14][1].

larger P300 amplitude [17].

Figure 7. Checkerboard paradigm [20].

P300 potentials. Figure 6 shows such a different approach which is called single character (SC) paradigm that only single character is flashed instead of a row or column. The SC paradigm randomly flashes one character at a time with a delay between flashes [17].

**EEG Data Acquisition**

**Pre-Processing**

Checkerboard (CB) speller is another paradigm proposed to overcome a problem associated with RC speller [17]. This drawback is arising from the distraction or inherent noise due to row/column association [17]. CB speller effectively reduces these two

The region-based (RB) paradigm was proposed by Fazel-Rezai et. al*.* in 2009 [21]. It is a two-level speller where the regions have to flash instead of rows and columns. In the first level, characters are placed in several regions (seven groups) as shown in Figure 8 [17][1][20][21]. The users are instructed to focus attention on a specific character in one of the seven regions. After several flashes

limitations as the characters are arranged in a checkerboard style as shown in Figure 7. CB speller also increases ITR [20].

Finally, the desired character is selected.

**Figure 4.** A typical P300 BCI setup with visual feedback.

200 Brain-Computer Interface Systems – Recent Progress and Future Prospects

several responses. Finally, the desired character is selected.

**Figure 5.** A typical row/column paradigm [1].

The region-based (RB) paradigm was proposed by Fazel-Rezai et. al. in 2009 [21]. It is a twolevel speller where the regions have to flash instead of rows and columns. In the first level, characters are placed in several regions (seven groups) as shown in Figure 8 [17][1][20][21]. The users are instructed to focus attention on a specific character in one of the seven regions. After several flashes the desired region is selected. In the second level, characters are distrib‐ uted following the same rule used in the first level and each character flashes in similar order. After several flashes, the desired character is identified [21]. the desired region is selected. In the second level, characters are distributed following the same rule used in the first level and each

character flashes in similar order. After several flashes, the desired character is identified [21].

Figure 8. Region based paradigm where a set of characters in level 1 (E) are expanded in level 2 for spelling character "B" (F). **Figure 8.** Region based paradigm where a set of characters in level 1 (E) are expanded in level 2 for spelling character "B" (F).

It is reported that RB speller has decreased the adjacency problem significantly [17][1][20][21]. The RB and CB paradigms show new directions in BCI speller paradigms apart from RC speller. There has been much progress in bringing BCI technology out of lab environment to real-life applications. BCI has widely been It is reported that RB speller has decreased the adjacency problem significantly [17][1][20][21]. The RB and CB paradigms show new directions in BCI speller paradigms apart from RC speller.

studied in helping disable people, for example, enabling controlling a wheel chair using brain signals [22]. The other promising applications are in managing smart home environment, controlling a virtual reality environment, and next generation gaming [12]. **3. SSVEP BCI**  There has been much progress in bringing BCI technology out of lab environment to real-life applications. BCI has widely been studied in helping disable people, for example, enabling controlling a wheel chair using brain signals [22]. The other promising applications are in managing smart home environment, controlling a virtual reality environment, and next generation gaming [12].

Electrophysiological and neurophysiological studies have demonstrated increases in neural activity elicited by gazing at a stimulus [23]. Visual evoked potentials are elicited by sudden visual stimuli and the repetitive visual stimuli would lead to stable voltage

different frequency of the visual stimulus. The flickering stimulus of different frequency with a constant intensity can evoke the SSVEP in verity of amplitudes, ranging from (5-12Hz) as low frequencies, (12-25 Hz) as medium ones and (25-50 Hz) as high frequency bands [26]. This type of stimulus is a powerful indicator in the diagnosis of visual pathway function, visual imperceptions in patients with cerebral lesions, loss of multifocal sensitivity in patients with multiple sclerosis, and neurological

In addition to the usual clinical purpose of diagnosing visual pathway and brain mapping impairments, the SSVEP can serve as a basis for BCI. Recently, SSVEP BCI systems have gained a special place in the BCI paradigms continuum because of having a variety of different possibilities. SSVEP BCIs are useful in different applications, especially the ones that need some major

A typical SSVEP-based BCI system uses a light-emitting diode (LED) for flickering. SSVEP responses can be measured within narrow frequency bands (e.g. around the visual stimulation frequency. Several numbers of stimuli can be implemented by using not necessarily a wide range of flickering frequencies, as the minimum detectable difference between frequencies is 0.2 Hz [27]. The occipital region is the area where this feature is generated more prominently [6]. The most wide-spread signal processing technique

Large number of BCI commands is necessary (in SSVEP BCI limitations are mostly defined only by the design).

High reliability of recognition is necessary (in SSVEP BCI, patterns are clearly distinguishable by frequency).

#### oscillations pattern in EEG that is called SSVEP. **3. SSVEP BCI**

requirements as follows [27]:

Self-paced performance is required.

SSVEP is considered as a concept with two different definitions. Ragan [24] proposed that SSVEP is a direct response in the primary visual cortex. In the other hand, Silberstein *et al.* [25] assumed that the SSVEP includes indirect cortical responses via cortical-loops, from the peripheral retina, while a cognitive task is performed. SSVEP in this model has a complex amplitude and phase topography across the posterior scalp with considerable inter-subject variability. Although the main mechanism of SSVEP still is unknown, generally SSVEP is considered as a continuous visual cortical response evoked by repetitive stimuli with a Electrophysiological and neurophysiological studies have demonstrated increases in neural activity elicited by gazing at a stimulus [23]. Visual evoked potentials are elicited by sudden visual stimuli and the repetitive visual stimuli would lead to stable voltage oscillations pattern in EEG that is called SSVEP.

constant frequency on the central retina. As a nearly sinusoidal oscillatory waveform, the SSVEP usually contains the same fundamental frequency as the stimulus and some harmonics of the fundamental frequency. For example, when the retina is excited by a visual stimulus at presentation rates ranging from 3.5 Hz to 75 Hz, the brain generates an electrical activity at the same and SSVEP is considered as a concept with two different definitions. Ragan [24] proposed that SSVEP is a direct response in the primary visual cortex. On the other hand, Silberstein *et al.* [25]

abnormalities in patients with schizophrenia and other clinical diagnoses [26].

No training (or just a short time training for classifier training) is allowed.

After several flashes, the desired character is identified [21]. the desired region is selected. In the second level, characters are distributed following the same rule used in the first level and each assumed that the SSVEP includes indirect cortical responses via cortical-loops, from the peripheral retina, while a cognitive task is performed. SSVEP in this model has a complex amplitude and phase topography across the posterior scalp with considerable inter-subject variability. Although the main mechanism of SSVEP still is unknown, generally SSVEP is considered as a continuous visual cortical response evoked by repetitive stimuli with a constant frequency on the central retina. As a nearly sinusoidal oscillatory waveform, the SSVEP usually contains the same fundamental frequency as the stimulus and some harmonics of the fundamental frequency. For example, when the retina is excited by a visual stimulus at presentation rates ranging from 3.5 Hz to 75 Hz, the brain generates an electrical activity at the same and different frequency of the visual stimulus. The flickering stimulus of different frequency with a constant intensity can evoke the SSVEP in verity of amplitudes, ranging from (5-12Hz) as low frequencies, (12-25 Hz) as medium ones and (25-50 Hz) as high frequency bands [26]. This type of stimulus is a powerful indicator in the diagnosis of visual pathway function, visual imperceptions in patients with cerebral lesions, loss of multifocal sensitivity in patients with multiple sclerosis, and neurological abnormalities in patients with schizo‐ phrenia and other clinical diagnoses [26].

> In addition to the usual clinical purpose of diagnosing visual pathway and brain mapping impairments, the SSVEP can serve as a basis for BCI. Recently, SSVEP BCI systems have gained a special place in the BCI paradigms continuum because of having a variety of different possibilities. SSVEP BCIs are useful in different applications, especially the ones that need some major requirements as follows [27]:


The region-based (RB) paradigm was proposed by Fazel-Rezai et. al. in 2009 [21]. It is a twolevel speller where the regions have to flash instead of rows and columns. In the first level, characters are placed in several regions (seven groups) as shown in Figure 8 [17][1][20][21]. The users are instructed to focus attention on a specific character in one of the seven regions. After several flashes the desired region is selected. In the second level, characters are distrib‐ uted following the same rule used in the first level and each character flashes in similar order.

202 Brain-Computer Interface Systems – Recent Progress and Future Prospects

character flashes in similar order. After several flashes, the desired character is identified [21].

Figure 8. Region based paradigm where a set of characters in level 1 (E) are expanded in level 2 for spelling character "B" (F).

**Figure 8.** Region based paradigm where a set of characters in level 1 (E) are expanded in level 2 for spelling character

It is reported that RB speller has decreased the adjacency problem significantly [17][1][20][21]. The RB and CB paradigms show new directions in BCI speller paradigms apart from RC

There has been much progress in bringing BCI technology out of lab environment to real-life applications. BCI has widely been studied in helping disable people, for example, enabling controlling a wheel chair using brain signals [22]. The other promising applications are in managing smart home environment, controlling a virtual reality environment, and next

Electrophysiological and neurophysiological studies have demonstrated increases in neural activity elicited by gazing at a stimulus [23]. Visual evoked potentials are elicited by sudden visual stimuli and the repetitive visual stimuli would lead to stable voltage oscillations pattern

SSVEP is considered as a concept with two different definitions. Ragan [24] proposed that SSVEP is a direct response in the primary visual cortex. On the other hand, Silberstein *et al.* [25]

different frequency of the visual stimulus. The flickering stimulus of different frequency with a constant intensity can evoke the SSVEP in verity of amplitudes, ranging from (5-12Hz) as low frequencies, (12-25 Hz) as medium ones and (25-50 Hz) as high frequency bands [26]. This type of stimulus is a powerful indicator in the diagnosis of visual pathway function, visual imperceptions in patients with cerebral lesions, loss of multifocal sensitivity in patients with multiple sclerosis, and neurological

In addition to the usual clinical purpose of diagnosing visual pathway and brain mapping impairments, the SSVEP can serve as a basis for BCI. Recently, SSVEP BCI systems have gained a special place in the BCI paradigms continuum because of having a variety of different possibilities. SSVEP BCIs are useful in different applications, especially the ones that need some major

A typical SSVEP-based BCI system uses a light-emitting diode (LED) for flickering. SSVEP responses can be measured within narrow frequency bands (e.g. around the visual stimulation frequency. Several numbers of stimuli can be implemented by using not necessarily a wide range of flickering frequencies, as the minimum detectable difference between frequencies is 0.2 Hz [27]. The occipital region is the area where this feature is generated more prominently [6]. The most wide-spread signal processing technique

Large number of BCI commands is necessary (in SSVEP BCI limitations are mostly defined only by the design).

High reliability of recognition is necessary (in SSVEP BCI, patterns are clearly distinguishable by frequency).

new directions in BCI speller paradigms apart from RC speller.

abnormalities in patients with schizophrenia and other clinical diagnoses [26].

No training (or just a short time training for classifier training) is allowed.

oscillations pattern in EEG that is called SSVEP.

**3. SSVEP BCI** 

generation gaming [12].

in EEG that is called SSVEP.

**3. SSVEP BCI**

"B" (F).

speller.

requirements as follows [27]:

Self-paced performance is required.

Electrophysiological and neurophysiological studies have demonstrated increases in neural activity elicited by gazing at a stimulus [23]. Visual evoked potentials are elicited by sudden visual stimuli and the repetitive visual stimuli would lead to stable voltage SSVEP is considered as a concept with two different definitions. Ragan [24] proposed that SSVEP is a direct response in the primary visual cortex. In the other hand, Silberstein *et al.* [25] assumed that the SSVEP includes indirect cortical responses via cortical-loops, from the peripheral retina, while a cognitive task is performed. SSVEP in this model has a complex amplitude and phase topography across the posterior scalp with considerable inter-subject variability. Although the main mechanism of SSVEP still is unknown, generally SSVEP is considered as a continuous visual cortical response evoked by repetitive stimuli with a constant frequency on the central retina. As a nearly sinusoidal oscillatory waveform, the SSVEP usually contains the same fundamental frequency as the stimulus and some harmonics of the fundamental frequency. For example, when the retina is excited by a visual stimulus at presentation rates ranging from 3.5 Hz to 75 Hz, the brain generates an electrical activity at the same and A typical SSVEP-based BCI system uses a light-emitting diode (LED) for flickering. SSVEP responses can be measured within narrow frequency bands (e.g. around the visual stimulation frequency. Several numbers of stimuli can be implemented by using not necessarily a wide range of flickering frequencies, as the minimum detectable difference between frequencies is 0.2 Hz [27]. The occipital region is the area where this feature is generated more prominently [6]. The most wide-spread signal processing technique to extract the SSVEP responses of the brain from the raw EEG data is based on power spectral density (PSD) using FFT of a sliding data window with a fixed length. Template matching and recursive outlier rejection have also been used to show the feasibility of SSVEP BCI systems. Other methods which attempt to improve on robustness upon the FFT-based methods are autoregressive spectral analysis, and the frequency stability coefficient (SC) which has been shown to be better than power spectrum for short data windows; although training is necessary for building the SC model. Furthermore, CCA is also an efficient method for online SSVEP-BCI, as the required data window lengths are shorter than those necessary for power spectrum estimation.

Pastor et al.[28] studied the relationship between visual stimulation and SSVEP-evoked amplitudes, showing that the amplitude of SSVEPs peaks at 15 Hz, forms a lower plateau at 27 Hz, and declines further at higher frequencies (>30 Hz) as shown in Figure 9. spectral analysis, and the frequency stability coefficient (SC) which has been shown to be better than power spectrum for short data windows; although training necessary for building the SC model. Furthermore, CCA is also an efficient method for online SSVEP-BCI, as the required data window lengths are shorter than those necessary for power spectrum estimation. Pastor et al.[28] studied the relationship between visual stimulation and SSVEP-evoked amplitudes, showing that the amplitude of

data window with a fixed length. Template matching and recursive outlier rejection have also been used to show the feasibility of SSVEP BCI systems. Other methods which attempt to improve on robustness upon the FFT-based methods are autoregressive

SSVEPs peaks at 15 Hz, forms a lower plateau at 27 Hz, and declines further at higher frequencies (>30 Hz) as shown in Figure 9.

Figure 9. SSVEP amplitude with different flickering frequency [28]. **Figure 9.** SSVEP amplitude with different flickering frequency [28].

In low-frequency stimulation, SSVEP detection is more accurate. In spite of its favorable detection properties, this band presents two major inconveniences [28], In low-frequency stimulation, SSVEP detection is more accurate. In spite of its favorable detection properties, this band presents two major inconveniences [28],


Ding *et al.* [23] demonstrated that a person's attention level modulates his/her SSVEP. Since the SSVEP depends directly on the stimulation frequency of visual flickering, user's attended target can be identified by analyzing the frequency contents in the induced SSVEP. By tagging different flickers with distinct flickering frequencies, subjects can shift their gaze to their desired A simple solution could be in using higher stimulation frequencies. From empirical and subjective evidence, the threshold could be set to 40 Hz for low stimulation [28].

flickers. These gaze targets can then be identified using the Fourier spectrum of the measured SSVEP signals. Middendorf *et al.* [29] designed a flight simulator controlled by two flickering lights that controlled leftwards or rightwards movement with a classification accuracy of 92%. Cheng *et al.* [30] implemented a SSVEP-based virtual keypad that achieved a mean ITR of 27.15 bits/min using twelve frequency-tagged flickering lights. Using two EEG electrodes positioned at the primary visual cortex, Kelly *et al.* [31] developed a method allowing participants to interact with a computer game. Moreover some visual BCIs have been developed as independent from users' eye gaze of users' eye gaze. Allison *et al.* [6] investigated selective attention using overlapping stimulus to induce SSVEPs difference in an online control study. Zhang *et al.* [32] also modulated the SSVEP amplitude and phase response by means of shifting covert attention on two sets of random dots with distinct colors, motion direction and flickering frequencies in the same visual field. Trader *et al.* [33] compared the performance of the Hex-o-Spell and matrix design using covert attention. Their results demonstrate that the Hex-o-Spell is increasing 50% than matrix design with covert attention. This SSVEP-based BCI identifies user's intended targets on calculated Fourier spectra. Nevertheless, the Fourier spectrum requires a time window (e.g., 1 or 2 sec) for computation to achieve sufficient frequency resolution in identifying two distinct gaze targets. Data segment with insufficient length in Fourier spectrum computation usually results in reduction of frequency resolution, which can limit the number of available targets in SSVEP-based BCI. Since BCI Ding *et al.* [23] demonstrated that a person's attention level modulates his/her SSVEP. Since the SSVEP depends directly on the stimulation frequency of visual flickering, user's attended target can be identified by analyzing the frequency contents in the induced SSVEP. By tagging different flickers with distinct flickering frequencies, subjects can shift their gaze to their desired flickers. These gaze targets can then be identified using the Fourier spectrum of the measured SSVEP signals. Middendorf *et al.* [29] designed a flight simulator controlled by two flickering lights that controlled leftwards or rightwards movement with a classification accuracy of 92%. Cheng *et al.* [30] implemented a SSVEP-based virtual keypad that achieved a mean ITR of 27.15 bits/min using twelve frequency-tagged flickering lights. Using two EEG electrodes positioned at the primary visual cortex, Kelly *et al.*[31] developed a method allowing participants to interact with a computer game.

performance depends on accuracy and speed, a reliable method for extracting SSVEPs and recognizing gaze targets in an appropriate data segments is crucial. It has been shown that the refreshing frequency, of a cathode ray tube (CRT) monitor can evoke a clear SSVEP. For SSVEP-based BCI development, the decoding accuracy is the most important factor, and a suitable stimulator is very crucial in this regard [34]. In previous studies, CRT flicker has been the most widely adopted stimulator, the LED flicker has only been reported in a small number of studies, and liquid crystal display (LCD) flicker has not appeared in the literature [23]. Since each of the three kinds of flicker can successfully evoke SSVEP, it is important to investigate the SSVEP differences that result from these different stimulators, and ascertain the type of flicker is most helpful in improving the accuracy of SSVEP-based BCI application. In the selection of the stimulating frequencies in a BCI application, one must ensure that the responses are as unique as possible. Thus, the Moreover, some visual BCIs have been developed as independent from users' eye gaze. Allison *et al.* [6] investigated selective attention using overlapping stimulus to induce SSVEPs differ‐ ence in an online control study. Zhang *et al.* [32] also modulated the SSVEP amplitude and phase response by means of shifting covert attention on two sets of random dots with distinct colors, motion direction and flickering frequencies in the same visual field. Trader *et al.* [33] compared the performance of the Hex-o-Spell and matrix design using covert attention. Their results demonstrated that the Hex-o-Spell is more than 50% better than those with matrix

design with covert attention. This SSVEP-based BCI identifies user's intended targets on calculated Fourier spectra. Nevertheless, the Fourier spectrum requires a time window (e.g., 1 or 2 sec) for computation to achieve sufficient frequency resolution in identifying two distinct gaze targets. Data segment with insufficient length in Fourier spectrum computation usually results in reduction of frequency resolution, which can limit the number of available targets in SSVEP-based BCI. Since BCI performance depends on accuracy and speed, a reliable method for extracting SSVEPs and recognizing gaze targets in an appropriate data segments is crucial.

It has been shown that the refreshing frequency, of a cathode ray tube (CRT) monitor can evoke a clear SSVEP. For SSVEP-based BCI development, the decoding accuracy is the most impor‐ tant factor, and a suitable stimulator is very crucial in this regard [34]. In previous studies, CRT flicker has been the most widely adopted stimulator, the LED flicker has only been reported in a small number of studies, and liquid crystal display (LCD) flicker has not appeared in the literature [23]. Since each of the three kinds of flicker can successfully evoke SSVEP, it is important to investigate the SSVEP differences that result from these different stimulators, and ascertain the type of flicker is most helpful in improving the accuracy of SSVEP-based BCI application. In the selection of the stimulating frequencies in a BCI application, one must ensure that the responses are as unique as possible. Thus, the stimulating frequencies are neither harmonics nor sub–harmonics from each other. From a practical point of view, the advantages of SSVEP BCI systems can be summarized as follows [34]:

**•** User is allowed to have small eye movements.

Pastor et al.[28] studied the relationship between visual stimulation and SSVEP-evoked amplitudes, showing that the amplitude of SSVEPs peaks at 15 Hz, forms a lower plateau at

BCI, as the required data window lengths are shorter than those necessary for power spectrum estimation.

to extract the SSVEP responses of the brain from the raw EEG data is based on power spectral density (PSD) using FFT of a sliding data window with a fixed length. Template matching and recursive outlier rejection have also been used to show the feasibility of SSVEP BCI systems. Other methods which attempt to improve on robustness upon the FFT-based methods are autoregressive spectral analysis, and the frequency stability coefficient (SC) which has been shown to be better than power spectrum for short data windows; although training necessary for building the SC model. Furthermore, CCA is also an efficient method for online SSVEP-

Pastor et al.[28] studied the relationship between visual stimulation and SSVEP-evoked amplitudes, showing that the amplitude of SSVEPs peaks at 15 Hz, forms a lower plateau at 27 Hz, and declines further at higher frequencies (>30 Hz) as shown in Figure 9.

In low-frequency stimulation, SSVEP detection is more accurate. In spite of its favorable detection properties, this band presents

Flickering Frequency (Hz)

According to visual perception studies, stimulation frequencies in this band are rather annoying and tiring for the subject

A simple solution could be in using higher stimulation frequencies. From empirical and subjective evidence, the threshold could be

Ding *et al.* [23] demonstrated that a person's attention level modulates his/her SSVEP. Since the SSVEP depends directly on the stimulation frequency of visual flickering, user's attended target can be identified by analyzing the frequency contents in the induced SSVEP. By tagging different flickers with distinct flickering frequencies, subjects can shift their gaze to their desired flickers. These gaze targets can then be identified using the Fourier spectrum of the measured SSVEP signals. Middendorf *et al.* [29] designed a flight simulator controlled by two flickering lights that controlled leftwards or rightwards movement with a classification accuracy of 92%. Cheng *et al.* [30] implemented a SSVEP-based virtual keypad that achieved a mean ITR of 27.15 bits/min using twelve frequency-tagged flickering lights. Using two EEG electrodes positioned at the primary visual cortex, Kelly *et* 

Moreover some visual BCIs have been developed as independent from users' eye gaze of users' eye gaze. Allison *et al.* [6] investigated selective attention using overlapping stimulus to induce SSVEPs difference in an online control study. Zhang *et al.* [32] also modulated the SSVEP amplitude and phase response by means of shifting covert attention on two sets of random dots with distinct colors, motion direction and flickering frequencies in the same visual field. Trader *et al.* [33] compared the performance of the Hex-o-Spell and matrix design using covert attention. Their results demonstrate that the Hex-o-Spell is increasing 50% than matrix design with covert attention. This SSVEP-based BCI identifies user's intended targets on calculated Fourier spectra. Nevertheless, the Fourier spectrum requires a time window (e.g., 1 or 2 sec) for computation to achieve sufficient frequency resolution in identifying two distinct gaze targets. Data segment with insufficient length in Fourier spectrum computation usually results in reduction of frequency resolution, which can limit the number of available targets in SSVEP-based BCI. Since BCI performance depends on accuracy and speed, a reliable method for extracting SSVEPs and recognizing gaze targets in an

It has been shown that the refreshing frequency, of a cathode ray tube (CRT) monitor can evoke a clear SSVEP. For SSVEP-based BCI development, the decoding accuracy is the most important factor, and a suitable stimulator is very crucial in this regard [34]. In previous studies, CRT flicker has been the most widely adopted stimulator, the LED flicker has only been reported in a small number of studies, and liquid crystal display (LCD) flicker has not appeared in the literature [23]. Since each of the three kinds of flicker can successfully evoke SSVEP, it is important to investigate the SSVEP differences that result from these different stimulators, and ascertain the type of flicker is most helpful in improving the accuracy of SSVEP-based BCI application. In the selection of the stimulating frequencies in a BCI application, one must ensure that the responses are as unique as possible. Thus, the

The risk for inducing photo epileptic seizures is higher for stimulation frequencies in the 15 – 25 Hz.

set to 40 Hz for low stimulation. The low stimulation threshold can be set to 40 Hz [28].

A simple solution could be in using higher stimulation frequencies. From empirical and

Ding *et al.* [23] demonstrated that a person's attention level modulates his/her SSVEP. Since the SSVEP depends directly on the stimulation frequency of visual flickering, user's attended target can be identified by analyzing the frequency contents in the induced SSVEP. By tagging different flickers with distinct flickering frequencies, subjects can shift their gaze to their desired flickers. These gaze targets can then be identified using the Fourier spectrum of the measured SSVEP signals. Middendorf *et al.* [29] designed a flight simulator controlled by two flickering lights that controlled leftwards or rightwards movement with a classification accuracy of 92%. Cheng *et al.* [30] implemented a SSVEP-based virtual keypad that achieved a mean ITR of 27.15 bits/min using twelve frequency-tagged flickering lights. Using two EEG electrodes positioned at the primary visual cortex, Kelly *et al.*[31] developed a method allowing

subjective evidence, the threshold could be set to 40 Hz for low stimulation [28].

**•** The risk for inducing photo epileptic seizures is higher for stimulation frequencies in the 15

In low-frequency stimulation, SSVEP detection is more accurate. In spite of its favorable

**•** According to visual perception studies, stimulation frequencies in this band are rather

*al.* [31] developed a method allowing participants to interact with a computer game.

Moreover, some visual BCIs have been developed as independent from users' eye gaze. Allison *et al.* [6] investigated selective attention using overlapping stimulus to induce SSVEPs differ‐ ence in an online control study. Zhang *et al.* [32] also modulated the SSVEP amplitude and phase response by means of shifting covert attention on two sets of random dots with distinct colors, motion direction and flickering frequencies in the same visual field. Trader *et al.* [33] compared the performance of the Hex-o-Spell and matrix design using covert attention. Their results demonstrated that the Hex-o-Spell is more than 50% better than those with matrix

27 Hz, and declines further at higher frequencies (>30 Hz) as shown in Figure 9.

Figure 9. SSVEP amplitude with different flickering frequency [28].

detection properties, this band presents two major inconveniences [28],

two major inconveniences [28],

annoying and tiring for the subject

– 25 Hz.

**Figure 9.** SSVEP amplitude with different flickering frequency [28].

204 Brain-Computer Interface Systems – Recent Progress and Future Prospects

SSVEP Amplitude (μV)

appropriate data segments is crucial.

participants to interact with a computer game.


#### **4. Hybrid BCIs SSVEP-P300 hybrid BCIs**

There are some obstacles for BCIs to be more applicable, such as reliability, BCI illiteracy [35], low ITR, and no satisfactory accuracy for all different subjects. In recent years, an extensive amount of work in BCI has been invested based on utilizing the combination of different types of BCI systems, or BCI and non-BCI, called hybrid BCI systems. Overcoming the limitations and disadvantages of the conventional BCI systems is the main goal of hybrid BCI. The focus and attraction toward hybrid BCI field has been extended in recent years. This is shown in Figure 10, based on the Scopus search engine [36], and the keyword (("hybrid" AND ("BCI" OR "brain computer interface"))) and ("SSVEP" AND "P300") and limited to "Engineering", "Neuroscience" and "Computer Science" subject areas.

subject areas.

BCI systems can be summarized as follows [34]:

Command delays of 1-3 s are allowed.

**4. Hybrid BCIs SSVEP-P300 hybrid BCIs** 

User is allowed to have small eye movements.

User is capable of mild but sustained attention effort.

User's visual system is not engaged in other activities.

Visual stimulation can be performed by usual equipment like computer display or LED panel.

stimulating frequencies are neither harmonics nor sub–harmonics from each other. From practical view the advantages of SSVEP

There are some obstacles for BCIs to be more applicable, such as reliability, BCI illiteracy [35], low ITR, and no satisfactory accuracy for all different subjects. In recent years, an extensive amount of work in BCI has been invested based on utilizing the combination of different types of BCI systems, or BCI and non-BCI, called hybrid BCI systems. Overcoming the limitations and disadvantages of

"brain computer interface"))) and ("SSVEP" AND "P300") and limited to "Engineering", "Neuroscience" and "computer science"

Figure 10.The increasing research trend in hybrid BCI area. **Figure 10.** The increasing research trend in hybrid BCI area.

In general, the BCI systems can be combined in the way that, each system has separate input signal or the output of one system would be the input of the second system. The systems are called sequentially and simultaneously hybrid BCI, respectively [37]. Figure 11 shows a general block diagram of a sequential and a simultaneous hybrid BCI system. In sequential hybrid BCI, the first system mostly acts as a switch [37]. For this task, one of the appropriate options is SSVEP. SSVEP has high classification accuracy; high information transfer rate does not need training. In general, the BCI systems can be combined in the way that, each system has separate input signal or the output of one system would be the input of the second system. The systems are called sequentially and simultaneously hybrid BCI, respectively [37]. Figure 11 shows a general block diagram of a sequential and a simultaneous hybrid BCI system. In sequential hybrid BCI, the first system mostly acts as a switch [37]. For this task, one of the appropriate options is SSVEP. SSVEP has high classification accuracy; high information transfer rate does not need training.

**Figure 11.** a) Sequential and (b) simultaneous hybrid BCI systems.

One of the main issues in this area is the optimum combination and selection of conventional BCIs. Several combinations of hybrid BCI systems have been introduced [37]. Conventional BCI systems are combined together based on the features of each system and the application of the hybrid BCI. If there are different tasks to be performed by the hybrid BCI, for each task, the more appropriate BCI can be chosen and, depending on the how the tasks are related to each other, the overall system can be combined. Some of the combinations for hybrid BCI that have been studied in recent years are shown in Table 1. Most of the studies in this area are focused on the combinations of BCI systems and few studies are on BCI and other physiological systems or devices.

**Table 1.** The covered cells in each row have been introduced as hybrid BCI systems

stimulating frequencies are neither harmonics nor sub–harmonics from each other. From practical view the advantages of SSVEP

There are some obstacles for BCIs to be more applicable, such as reliability, BCI illiteracy [35], low ITR, and no satisfactory accuracy for all different subjects. In recent years, an extensive amount of work in BCI has been invested based on utilizing the combination of different types of BCI systems, or BCI and non-BCI, called hybrid BCI systems. Overcoming the limitations and disadvantages of the conventional BCI systems is the main goal of hybrid BCI. The focus and attraction toward hybrid BCI field has been extended in recent years. This is shown in Figure 10, based on the Scopus search engine [36], and the keyword (("hybrid" AND ("BCI" OR "brain computer interface"))) and ("SSVEP" AND "P300") and limited to "Engineering", "Neuroscience" and "computer science"

In general, the BCI systems can be combined in the way that, each system has separate input signal or the output of one system would be the input of the second system. The systems are called sequentially and simultaneously hybrid BCI, respectively [37]. Figure 11 shows a general block diagram of a sequential and a simultaneous hybrid BCI system. In sequential hybrid BCI, the first system mostly acts as a switch [37]. For this task, one of the appropriate options is SSVEP. SSVEP has high classification accuracy;

One of the main issues in this area is the optimum combination and selection of conventional BCIs. Several combinations of hybrid BCI systems have been introduced [37]. Conventional BCI systems are combined together based on the features of each system and the application of the hybrid BCI. If there are different tasks to be performed by the hybrid BCI, for each task, the more appropriate BCI can be chosen and, depending on the how the tasks are related to each other, the overall system can be combined. Some of the combinations for hybrid BCI that have been studied in recent years are shown in Table 1. Most of the studies in this area are

**Brain Non-Brain** 

**(Non-Invasive) Non-Invasive Invasive EEG Non-EEG** 

focused on the combinations of BCI systems and few studies are on BCI and other physiological systems or devices.

**BCI System 2** 

Visual stimulation can be performed by usual equipment like computer display or LED panel.

BCI systems can be summarized as follows [34]:

Command delays of 1-3 s are allowed.

206 Brain-Computer Interface Systems – Recent Progress and Future Prospects

subject areas.

training.

**4. Hybrid BCIs SSVEP-P300 hybrid BCIs** 

Number of hybrid BCI publications

Number of P300, SSVEP hybrid BCI publications

Figure 10.The increasing research trend in hybrid BCI area.

**Figure 10.** The increasing research trend in hybrid BCI area.

2000 2002 2004 2005 2006 2007 2008 2009 2010 2011 2012

In general, the BCI systems can be combined in the way that, each system has separate input signal or the output of one system would be the input of the second system. The systems are called sequentially and simultaneously hybrid BCI, respectively [37]. Figure 11 shows a general block diagram of a sequential and a simultaneous hybrid BCI system. In sequential hybrid BCI, the first system mostly acts as a switch [37]. For this task, one of the appropriate options is SSVEP. SSVEP has high classification accuracy; high information transfer rate does not need

> **BCI System 2**

One of the main issues in this area is the optimum combination and selection of conventional BCIs. Several combinations of hybrid BCI systems have been introduced [37]. Conventional BCI systems are combined together based on the features of each system and the application of the hybrid BCI. If there are different tasks to be performed by the hybrid BCI, for each task, the more appropriate BCI can be chosen and, depending on the how the tasks are related to

**BCI System 1** 

high information transfer rate does not need training.

Figure 11.(a) Sequential and (b) simultaneous hybrid BCI systems.

**BCI System 1** 

**Papers Numbers** 

(a)

(b)

**Figure 11.** a) Sequential and (b) simultaneous hybrid BCI systems.

User is allowed to have small eye movements.

User is capable of mild but sustained attention effort.

User's visual system is not engaged in other activities.

In control applications, the BCI systems should be capable to cover multi-tasks. Hybrid BCIs has opened new opportunities for BCI systems to have more intense application in different areas. One of the areas in which hybrid BCI could play an important role is smart home control. There have been several studies done in this field [12][46]. The smart home or virtual envi‐ ronment systems are consisting of several stages and several control command types in each stage. In motor control systems, hybrid BCI shows improvement in accuracy and facilitates control tasks. Control commands, have different characteristics, and are divided to different types. Based on the characteristics, for each type, special types of BCI would be appropriate and for combination of control commands, two or more BCI types would fit. In discrete control commands, the task is the selection of one option from several options. P300 and Mu-Beta are more appropriate for this type of control commands. For series of the commands, continuous commands, ERD and SSVEP are more suitable. Also other features of the conventional BCIs should be considered, e.g., P300 is a slow responded system, but reliable. Mu-beta is fast responded, but not as efficient as P300.

The main enhancement that has been made by hybrid BCI is improvement in the applicability of BCI systems. As presenting two or more BCI type to the user, in the simultaneously combination, the user has the chance to get more efficient respond through utilizing the BCI type that is more appropriate for him or her. It also can decrease the fatigue, as the user can shift to another BCI option. It is shown that the accuracy is improved in the hybrid condition [42][43].

In [39], the hybrid BCI is introduced for Functional Electrical Stimulation (FES) control. Two tasks were considered to be implemented by the BCI systems; selecting one object among three objects and movement imagination to trigger FES. The high true positive rate (TPR) for SSVEP and ERD shows the capability of hybrid BCI for implementing several tasks that can be used in various control fields. SSVEP switch was introduced for smart home control [12]. The selection of control options, displayed on the screen is based on P300 BCI and SSVEP is operated as toggle switch. SSVEP was introduced as a switch for P300-based system [13]

Another BCI that can be introduced as a brain switch is ERS. In [41], post-imagery beta ERSbased brain switch was introduced for activating and deactivating the process of opening and closing the orthosis hand which was operated using SSVEP BCI.

More studies have focused on the simultaneous combinations of the conventional BCIs. As for one task, two or more BCI type are presented at the same time, the difficulty and complexity of performing the task is increased but, on the other hand, the accuracy in most of the cases increases for majority of the users. In addition, the fatigue may decrease, as users can switch between the BCI types that may be more comfortable for them. Another parameter is BCI illiteracy that can be decrease as users have the opportunity of accessing multi approaches [35]. The task of tracking the hint arrow was presented by SSVEP and ERD, and the accuracy was improved in the hybrid condition [38].

In another type of hybrid, more than one source of measurement is presented for one BCI type, for example, EEG and NIRS were acquired simultaneously for the ERD-based BCI [43] EEG and ECG were fused for motor imagery (MI) based BCI system[48]. EEG and EMG were utilized as hybrid in [44]. In this hybrid BCI, improvement in accuracy was shown. In some application areas, the tasks may be divided to two or more parts and each part is implemented by one BCI or non-BCI system. In this way, based on the features of the task, the system is selected. For example, in [42] control commands were divided to two parts and were imple‐ mented by EEG and electrooculography (EOG).

One of the issues in hybrid BCIs is that the system may be feasible but not optimum in all features. The hybrid BCI may improve the performance or accuracy but not compared to each of conventional BCI systems. For example, in the simultaneous combination of SSVEP and ERD, as in the conventional SSVEP, the accuracy is enhanced compared to ERD-based BCI system but not a lot changes compared to SSVEP.

P300 and SSVEP BCI were introduced as hybrid in an asynchronous BCI system in [13]. It seems that P300 and SSVEP combination works well as the stimuli for evoking both patterns can be shown on one screen simultaneously. The P300 paradigm considered in this study is a 6x6 speller matrix based on the original P300 row/column paradigm introduced by Farwell and Donchin [19]. Only one frequency is allocated for SSVEP paradigm. Background color was flashed with the frequency slightly less than 18 Hz. This facilitates the SSVEP detection. During the classification, P300 and SSVEP signals are separated by a band pass filter. The SSVEP is utilized as a control state (CS) detection, in the way that, when the user is gazing at the screen, the SSVEP is detected and it is assumed that the user intends to send a command. The system detects P300 target selection and CS simultaneously.

For SSVEP detection, the mean power spectral density (PSD) in the narrow band near the desired frequency and the PSD in the wider range near the desired frequency were utilized in an objective function (these values were subtracted from each other and divided over the PSD value from the wide band) and the function value was compared to a specified threshold. During the data acquisition, the channels for acquiring EEG signal were not fixed for all subjects. For P300 classification, FLDA or BLDA was utilized [14][15]. The experiment was presented as offline and online test. Ten subjects participated in the experiment. Subjects had training runs. In offline test, forty characters were presented for detection, divided to four groups. For better evaluation of SSVEP effect, two groups with and two groups without SSVEP were presented. In control state, subjects were instructed to count the number of time they distinguish the highlighted character. In non-control state (NCS), subjects were instructed to do a mental task like multiplication of two numbers and relax with closed eyes. For four out of five subjects, the accuracy was improved inconsiderably during the presence of SSVEP and P300 detection was not determinate. Between ten characters detection, there was a break and the time of the break depends on the time subjects pressed a keyboard button, and an auditory cue alerted about the finish of NCS time. The average classification accuracy of 96.5% and control state detection accuracy of 88% with the ITR of 20 bits/min were achieved during the offline test. The online test was presented under the semi synchronous condition. The experi‐ ment was consisted of blocks with 5 rounds, for detecting each character. SSVEP detection for at least three out of five runs showed the control state detection by the subject and P300 was detected during the control state. If the control state was not detected, the '=' character was shown on the screen. The break time and the auditory alert was the same as offline test. The average control state detection accuracy of 88.15%, the classification accuracy of 94.44% and the ITR of 19.05 bits/min were achieved during the online test. P300 and SSVEP combination was also introduced to control smart home environment in [12]. P300-based BCI was used for controlling the virtual smart home environment and SSVEP was implemented as a switch for the P300 BCI operation. Results from this experiment show that P300 is suitable for discrete control commands and SSVEP is suitable for continuous control signals. The hybrid BCI achieved high accuracy and reliability in all subjects. In this chapter, P300, SSVEP and the hybrid P300 and SSVEP BCI systems were reviewed. The new trend and direction in BCI systems is to use new approaches in stimulating brain patterns such as hybrid BCIs while keeping the system complexity low and user acceptability high.

#### **Author details**

shift to another BCI option. It is shown that the accuracy is improved in the hybrid condition

In [39], the hybrid BCI is introduced for Functional Electrical Stimulation (FES) control. Two tasks were considered to be implemented by the BCI systems; selecting one object among three objects and movement imagination to trigger FES. The high true positive rate (TPR) for SSVEP and ERD shows the capability of hybrid BCI for implementing several tasks that can be used in various control fields. SSVEP switch was introduced for smart home control [12]. The selection of control options, displayed on the screen is based on P300 BCI and SSVEP is operated as toggle switch. SSVEP was introduced as a switch for P300-based system [13]

Another BCI that can be introduced as a brain switch is ERS. In [41], post-imagery beta ERSbased brain switch was introduced for activating and deactivating the process of opening and

More studies have focused on the simultaneous combinations of the conventional BCIs. As for one task, two or more BCI type are presented at the same time, the difficulty and complexity of performing the task is increased but, on the other hand, the accuracy in most of the cases increases for majority of the users. In addition, the fatigue may decrease, as users can switch between the BCI types that may be more comfortable for them. Another parameter is BCI illiteracy that can be decrease as users have the opportunity of accessing multi approaches [35]. The task of tracking the hint arrow was presented by SSVEP and ERD, and the accuracy was

In another type of hybrid, more than one source of measurement is presented for one BCI type, for example, EEG and NIRS were acquired simultaneously for the ERD-based BCI [43] EEG and ECG were fused for motor imagery (MI) based BCI system[48]. EEG and EMG were utilized as hybrid in [44]. In this hybrid BCI, improvement in accuracy was shown. In some application areas, the tasks may be divided to two or more parts and each part is implemented by one BCI or non-BCI system. In this way, based on the features of the task, the system is selected. For example, in [42] control commands were divided to two parts and were imple‐

One of the issues in hybrid BCIs is that the system may be feasible but not optimum in all features. The hybrid BCI may improve the performance or accuracy but not compared to each of conventional BCI systems. For example, in the simultaneous combination of SSVEP and ERD, as in the conventional SSVEP, the accuracy is enhanced compared to ERD-based BCI

P300 and SSVEP BCI were introduced as hybrid in an asynchronous BCI system in [13]. It seems that P300 and SSVEP combination works well as the stimuli for evoking both patterns can be shown on one screen simultaneously. The P300 paradigm considered in this study is a 6x6 speller matrix based on the original P300 row/column paradigm introduced by Farwell and Donchin [19]. Only one frequency is allocated for SSVEP paradigm. Background color was flashed with the frequency slightly less than 18 Hz. This facilitates the SSVEP detection. During the classification, P300 and SSVEP signals are separated by a band pass filter. The SSVEP is utilized as a control state (CS) detection, in the way that, when the user is gazing at the screen,

closing the orthosis hand which was operated using SSVEP BCI.

208 Brain-Computer Interface Systems – Recent Progress and Future Prospects

improved in the hybrid condition [38].

mented by EEG and electrooculography (EOG).

system but not a lot changes compared to SSVEP.

[42][43].

Setare Amiri, Ahmed Rabbi, Leila Azinfar and Reza Fazel-Rezai

Biomedical Image and Signal Processing Laboratory, Department of Electrical Engineering, University of North Dakota, Grand Forks, USA

#### **References**


[14] Hoffmann, U, Vesin, J. M, Ebrahimi, T, & Diseren, K. An efficient brain-computer in‐ terface for disabled subjects. Journal of neuroscience methods (2008). , 300.

**References**

113-767.

(2003). , 19-577.

ment (2004).

1842-1857.

science (2008). , 28-1000.

210 Brain-Computer Interface Systems – Recent Progress and Future Prospects

[1] Wolpaw, J, Birbaumer, N, Mcfarland, D, Pfurtscheller, G, & Vaughan, T. Brain-com‐ puter interfaces for communication and control. Clinical neurophysiology (2002). ,

[2] Weiskopf, N, Veit, R, Erb, M, Mathiak, K, Grodd, W, Goebel, R, & Birbaumer, N. Physiological self-regulation of regional brain activity using real-time functional magnetic resonance imaging (fMRI): methodology and exemplary data. Neuroimage

[3] Waldert, S, Preissl, H, Demandt, E, Braun, C, Birbaumer, N, Aertsen, A, & Mehring, C. Hand movement direction decoded from MEG and EEG. The Journal of neuro‐

[4] Coyle, S, Ward, T, Markham, C, & Mcdarby, G. On the suitability of near-infrared (NIR) systems for next-generation brain-computer interfaces. Physiological Measure‐

[5] Pfurtscheller, G. and Lopes Da Silva FH. Event-related EEG/MEG synchronization and desynchronization: basic principles. Clinical Neurophysiology (1999). , 110(11),

[6] Allison, B, Faller, J, & Neuper, C. H. BCIs that use steady state visual evoked poten‐ tials or slow cortical potentials. Brain-Computer Interfaces: Principles and Practice.

[7] Vidal, J. J. Toward direct brain-computer communication. Annual Review of Biophy‐

[8] Vidal, J. J. Real-time detection of brain events in EEG. Proceedings of the IEEE

[9] Sellers, E, Arbel, Y, & Donchin, E. BCIs that uses event related potentials. Brain-Com‐ puter Interfaces: Principles and Practice. J. Wolpaw and E.W. Wolpaw, Eds. Oxford

[10] Birbaumer, N, Ghanayim, N, & Hinterberger, T. A spelling device for the paralysed.

[12] Edlinger, G, Holzner, C, & Guger, C. A hybrid brain-computer interface for smart home control Human-Computer Interaction. Interaction Techniques and Environ‐

[13] Panicker, R, Puthusserypady, S, & Sun, Y. An Asynchronous BCI with SSVEP-Based Control State Detection. IEEE Transactions on Biomedical Engineering (2011). , 300.

[11] Sanei, S, & Chambers, J. A. EEG Signal Processing. West Sussex: Wiley; (2007).

Wolpaw and E. W. Wolpaw, Eds. Oxford University Press; (2012).

sics and Bioengineering (1973). , 2-157.

(1977). , 65(5), 633-64.

ments (2011). , 417-426.

University Press; (2012). , 300.

Nature (1999). , 398(6725), 297-298.


[42] Punsawad, Y, Wongsawat, Y, & Parnichkun, M. Hybrid EEG-EOG brain-computer interface system for practical machine control. IEEE Engineering in Medicine and Bi‐ ology (2010). , 1360-1363.

[30] Cheng, M. Ming Cheng. Design and implementation of a brain-computer interface with high transfer rates. IEEE Transactions on Biomedical Engineering (2002).

[31] Kelly, S. P, Lalor, E. C, Reilly, R. B, & Foxe, J. J. Visual spatial attention tracking using high-density SSVEP data for independent brain-computer communication. Neural Systems and Rehabilitation Engineering, IEEE Transactions on (2005). , 13(2),

[32] Zhang, D, Gao, X, Gao, S, Engel, A. K, & Maye, A. An independent brain-computer interface based on covert shifts of non-spatial visual attention. Engineering in Medi‐ cine and Biology Society, 2009. EMBC (2009). Annual International Conference of the

[33] Treder, M. S, & Blankertz, B. Research (C) overt attention and visual speller design in

[34] Bakardjian, H. Optimization of SSVEP brain responses with application to eight-com‐

[35] Kubler, A, & Muller, K. R. An introduction to brain-computer interfacing Toward Brain-Computer Interfacing, Ed. Dornhedge G, Millan JR, Hinterberger T, McFarland

[37] Pfurtscheller, G, Allison, B, Brunner, C, Bauernfeind, G, Solis-escalante, T, Scherer, T. O, Zander, R, Mueller-putz, G, Neuper, C, & Birbaumer, N. The hybrid BCI Frontiers

[38] Allison, B, Brunner, C, Kaiser, V, Mueller-putz, G, Neuper, C, & Pfurtscheller, G. To‐ ward a hybrid brain-computer interface based on imagined movement and visual at‐

[39] Savic, A, Kisic, U, & Popovic, M. Toward a Hybrid BCI for Grasp Rehabilitation in Proceedings of the 5th European Conference of the International Federation for Med‐

[40] Brunner, C, Allison, B, Altstätter, C, & Neuper, C. A comparison of three brain-com‐ puter interfaces based on event-related desynchronization, steady state visual evoked potentials, or a hybrid approach using both signals. Journal of neural engi‐

[41] Pfurtscheller, G, Solis-escalante, T, Ortner, R, Linortner, P, & Muller-putz, G. Selfpaced operation of an SSVEP-Based orthosis with and without an imagery-based 'brain switch: a feasibility study towards a hybrid BCI IEEE Transactions on Neural

172-178.

IEEE: IEEE; 2009.

[36] wwwscopus.com/scopus/home.url

in Neuroscience (2010).

neering (2011).

an ERP-based brain-computer interface (2010).

212 Brain-Computer Interface Systems – Recent Progress and Future Prospects

tention Journal of Neural Engineering (2010).

mand Brain-Computer Interface. Neurosci Lett (2010).

DJ, and Muller KR (Cambridge, MA: MIT Press) ; , 2007-1.

ical and Biological Engineering Proceedings (2012). , 806-809.

Systems and Rehabilitation Engineering (2010). , 18-409.


### **Review of Wireless Brain-Computer Interface Systems**

Seungchan Lee, Younghak Shin, Soogil Woo, Kiseon Kim and Heung-No Lee

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/56436

#### **1. Introduction**

Over the past two decades, the study of the Brain Computer Interfaces (BCI) has grown dra‐ matically. According to Scopus search engine, a search result with the keyword "Brain Com‐ puter Interface" returns only two papers for year 1991. But, the same query returns 897 journals and conference papers for year 2011 (date of search: March 14, 2013). BCI systems provide a new communication channel to humans who use it. They measure neurophysio‐ logical signals of the human, electroencephalogram (EEG) in particular. EEG based BCI sys‐ tems are designed to decode the intension of the human user and generate commands to control external devices or computer applications. The human can produce these commands by generating the neurophysiological signals intentionally. This process can become more successful – fast and accurate – through training and practice. This technology allows the users with new experiences which enable a direct communication between the human and the computers or external devices such as home appliances, and prosthetic devices.

The BCI systems consist of two parts, signal acquisition and translation (see page 7 Figure 2). The signal acquisition part contains electrodes, analog circuit and digital system for neu‐ rophysiological signal recording and transmission. The translation part is normally comput‐ ing devices which are equipped with high performance processor such as laptops, PDAs, and smart phones. With an application program, this part performs algorithmic processes such as feature extraction and classification to convert the raw neurophysiological signals into computer readable messages. Depending on the type of connection between the two parts, we can divide BCI systems into two kinds, wired versus wireless BCI systems.

Many conventional BCI systems are wired. With just three electrodes positioned at the occi‐ pital lobe, the acquisition part of wired BCI systems generally comes with bulky and heavy amplifiers and preprocessing units. Connection wiring is usually complicated with a large

© 2013 Lee et al.; licensee InTech. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2013 Lee et al.; licensee InTech. This is a paper distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

number of cables between the electrodes and the acquisition part. For these reasons, prepa‐ ration time for measuring EEG signals is typically very long. In addition, user's movement is limited due to cable constraints. Therefore, the application of BCI systems is difficult to es‐ cape from laboratory scale experiments. These restrictions make the types of applications for which BCI systems can be made useful be severely limited. Wireless BCI systems are to eliminate the wire connection, between the signal acquisition and the translation part, with the use of a wireless transmission unit such as Bluetooth and Zigbee modules. Removing wire connections, portability of BCI systems is greatly improved. Postures and movements of users wearing the acquisition part of wireless BCI systems are also unimpeded. These de‐ sirable aspects of wireless BCI systems promote to go beyond a laboratory scale experiments and to develop everyday-life applications.

With portable wireless BCI systems, various real-life applications are under development now. In the early days of BCI researches, cursor control and speller applications were devel‐ oped mainly targeted for helping the disabled people. Recently, with growing interest, wire‐ less BCI systems have been applied in entertainments as well. For example, Emotiv and Neurosky companies have recently released their wireless BCI headsets for entertainment uses such as brain gaming and mind monitoring. Moreover, international research groups have applied wireless BCI systems for interesting new applications such as home automa‐ tion system based on monitoring human physiological states [29], cellular phone dialing [28], and drowsiness detection for drivers [19][20][32].

In this book chapter, we will review recent research trends in wireless BCI systems. In Sec‐ tion 2, we summarize several research topics in wireless BCI systems such as electrodes, em‐ bedded systems, user-friendly designs, and novel applications. We then take a closer look into emerging wireless BCI systems designed by BCI researchers, and discuss general BCI systems recently introduced into the market in Section 3. In Section 4, we discuss current challenges and possible future research directions on wireless BCI systems. Finally, we pro‐ vide concluding remarks in Section 5.

#### **2. Research trends of wireless BCI systems**

Wireless brain computer interface (BCI) systems are neurophysiological signal acquisition and processing systems where acquired physiological signals are wirelessly transmitted to the translation unit. Wireless systems, unlike their traditional wired counterparts, are de‐ signed to provide convenience in monitoring the neurophysiological signals of users.

Compared to conventional wired BCI systems, wireless BCI systems provide enhanced port‐ ability and wearability, facilitated by elimination of the wire connections between the wear‐ able acquisition unit and the translation unit. The translation unit is usually housed in a portable device such as laptops and smart phones. This improvement provides an easy in‐ stallation process and freedom of postures for users. Furthermore, owing to advanced inte‐ grated circuit designs, the components of the wireless BCI systems are small in sizes and efficient in power consumption. Employing these components, the acquisition part of wire‐ less BCI systems can be miniaturized. These advantages allow wireless BCI systems to be shaped in user friendly styles such as baseball caps [14][19][21][29], headsets [16][17][23][25] [27], and headbands [18][20][22][28][30][32]; thus, applying them in various applications such as entertainments and health care becomes easier than ever before.

Even though wireless BCI systems may provide a number of advantages, there are still many issues that need to be resolved including improving signal quality, more compact and stylish system designs, and excavation of useful applications. First, the quality of the meas‐ ured EEG signals has to be improved for more precise classification of user's intensions. The measured EEG signals are easily contaminated by various noise sources such as the pres‐ ence of other physiological signals and the power line noise. Moreover, various impedi‐ ments exist in electrodes-scalp interfaces such as hairs, sweat, and stratum corneum of skin [6]. These obstacles cause deleterious effects in signal measurements with high leakage cur‐ rents and high contact impedances. Due to these difficulties, EEG signals get easily corrupt‐ ed and the quality of measured signals is often undesirable. Consequentially, these factors lead to drop of application accuracy. Second, stylish, miniaturized and light weight wireless BCI systems are necessary for daily life application and long-term wearing. Conventional wired BCI systems are bulky, not user-friendly in system appearances because of their com‐ plicated connection between electrodes and signal acquisition part. Multi-channel electrode installation is also inconvenient and time-consuming, usually taking more than 30 minutes. Thus, the users are easily irritated and long-term monitoring becomes difficult. Third, killer applications are needed. Many researchers and developers introduced various applications related to entertainments and health care. But such contributions are still not enough to make any fundamental change in our life.

To help these issues, research groups are paying attention to the following aspects:


In this section, we aim to analyze the current status of research and development efforts in these directions.

#### **2.1. New electrodes for EEG signal acquisition**

number of cables between the electrodes and the acquisition part. For these reasons, prepa‐ ration time for measuring EEG signals is typically very long. In addition, user's movement is limited due to cable constraints. Therefore, the application of BCI systems is difficult to es‐ cape from laboratory scale experiments. These restrictions make the types of applications for which BCI systems can be made useful be severely limited. Wireless BCI systems are to eliminate the wire connection, between the signal acquisition and the translation part, with the use of a wireless transmission unit such as Bluetooth and Zigbee modules. Removing wire connections, portability of BCI systems is greatly improved. Postures and movements of users wearing the acquisition part of wireless BCI systems are also unimpeded. These de‐ sirable aspects of wireless BCI systems promote to go beyond a laboratory scale experiments

With portable wireless BCI systems, various real-life applications are under development now. In the early days of BCI researches, cursor control and speller applications were devel‐ oped mainly targeted for helping the disabled people. Recently, with growing interest, wire‐ less BCI systems have been applied in entertainments as well. For example, Emotiv and Neurosky companies have recently released their wireless BCI headsets for entertainment uses such as brain gaming and mind monitoring. Moreover, international research groups have applied wireless BCI systems for interesting new applications such as home automa‐ tion system based on monitoring human physiological states [29], cellular phone dialing

In this book chapter, we will review recent research trends in wireless BCI systems. In Sec‐ tion 2, we summarize several research topics in wireless BCI systems such as electrodes, em‐ bedded systems, user-friendly designs, and novel applications. We then take a closer look into emerging wireless BCI systems designed by BCI researchers, and discuss general BCI systems recently introduced into the market in Section 3. In Section 4, we discuss current challenges and possible future research directions on wireless BCI systems. Finally, we pro‐

Wireless brain computer interface (BCI) systems are neurophysiological signal acquisition and processing systems where acquired physiological signals are wirelessly transmitted to the translation unit. Wireless systems, unlike their traditional wired counterparts, are de‐

Compared to conventional wired BCI systems, wireless BCI systems provide enhanced port‐ ability and wearability, facilitated by elimination of the wire connections between the wear‐ able acquisition unit and the translation unit. The translation unit is usually housed in a portable device such as laptops and smart phones. This improvement provides an easy in‐ stallation process and freedom of postures for users. Furthermore, owing to advanced inte‐ grated circuit designs, the components of the wireless BCI systems are small in sizes and efficient in power consumption. Employing these components, the acquisition part of wire‐

signed to provide convenience in monitoring the neurophysiological signals of users.

and to develop everyday-life applications.

216 Brain-Computer Interface Systems – Recent Progress and Future Prospects

vide concluding remarks in Section 5.

[28], and drowsiness detection for drivers [19][20][32].

**2. Research trends of wireless BCI systems**

Among the above research topics, the development of advanced EEG electrodes which measure brain signals precisely with low noise is the most important challenge. Practically, the signal acquisition part of general wireless BCI systems only contains a signal acquisition circuit and a micro-processor based embedded system for transmission of the measured EEG signals. For example, a well-known Emotiv EPOC neuro-headset is composed of 14 channel electrodes and a small integrated embedded system powered by an onboard battery for signal conditioning and transmission. In the translation part of wireless BCI systems, analysis of acquired signals is performed either online or offline on a computer or a mobile device which is equipped with high-performance processors. For this segregated structure to work well, signal acquisition part of a wireless BCI system should be devoted to high fi‐ delity signal recording. To provide clear EEG signal acquisition at the electrode-skin inter‐ face, development of outstanding electrodes becomes a critical issue. For this reason, many research groups have recently been interested in developing advanced electrodes which can provide low-noise recording, convenience in installation, and comfort even in long-term wearing.

In conventional wired BCI systems, passive electrodes are widely used to measure EEG sig‐ nals. Generally, these electrodes are disc or ring shaped and are made of Ag/AgCl alloy [10]. Due to their simple structure, it is easy to make them small. However, they have many dis‐ advantages as well. Extra treatments are essential for recording reliable EEG signals because the scalp potentials are only on the order of several micro-volts and thus very noise-sensi‐ tive. Treatments are needed, including a hair preparation step and the use of conductive gels or glues for better attachment and higher conductivity. These preparations induce dis‐ comfort and require long preparation time. Furthermore, the conductive gels easily desic‐ cate and lose their adhesion. These problems bring about worse contact impedances at electrode-scalp interfaces, causing a large reduction of signal-to-noise ratio. In addition, the quality of recorded signals is sensitive to cable vibrations [17]. For these reasons, the longterm monitoring of EEG signals using passive electrodes is not feasible.

Recently, to overcome the weaknesses of passive electrodes, many researchers have studied advanced electrodes. For examples, research of dry electrodes is active recently. Generally, dry electrodes are defined as those that do not require the use of conductive gels or glues for installation process. Thus, a user can conveniently attach the electrodes to the user's scalp without any hair arrangement. To make dry contact at the electrode-skin interface, research‐ ers employ special materials or shapes in the design of dry electrodes. Extensive research has produced a huge variety of electrode materials and structures, including conductive rubber [8][9], conductive carbon nanotubes [7], micro-tip structures [6], micro-machined structures [14][15][20], non-contact types [4][5], spring-loaded fingers [2][13], bristle struc‐ tures [3], and conductive foams [27]. The most widely used dry electrode design is a set of contact posts which look like fingers [13][16]. This design has an advantage in contact ability because it is easy to penetrate into the scalp through the hair without an extended hair ar‐ rangement. Recently, some research groups have altered this finger design to produce ad‐ vanced mechanical designs, such as spring-loaded fingers [2] or bristle structures [3]. These designs seem to provide flexibility and geometric adaptation between the sensor and the ir‐ regular scalp surface to obtain low interface impedance. To achieve low contact impedance and provide a robust and stable electrical interface, some research groups have employed multi-walled carbon nanotube arrays [7] and micro-tip structures [6], which are able to pen‐ etrate the outer skin layer (which is 5 to10 um thick and called the *stratum corneum*).

While we can reduce the installation time significantly using dry electrodes, the contact im‐ pedance between the scalp and the electrodes is higher than that with gel-based passive electrodes due to the absence of conductive gels. Thus, signal quality of dry electrodes would be not better than that of the gel-based passive electrodes. To overcome this draw‐

device which is equipped with high-performance processors. For this segregated structure to work well, signal acquisition part of a wireless BCI system should be devoted to high fi‐ delity signal recording. To provide clear EEG signal acquisition at the electrode-skin inter‐ face, development of outstanding electrodes becomes a critical issue. For this reason, many research groups have recently been interested in developing advanced electrodes which can provide low-noise recording, convenience in installation, and comfort even in long-term

218 Brain-Computer Interface Systems – Recent Progress and Future Prospects

In conventional wired BCI systems, passive electrodes are widely used to measure EEG sig‐ nals. Generally, these electrodes are disc or ring shaped and are made of Ag/AgCl alloy [10]. Due to their simple structure, it is easy to make them small. However, they have many dis‐ advantages as well. Extra treatments are essential for recording reliable EEG signals because the scalp potentials are only on the order of several micro-volts and thus very noise-sensi‐ tive. Treatments are needed, including a hair preparation step and the use of conductive gels or glues for better attachment and higher conductivity. These preparations induce dis‐ comfort and require long preparation time. Furthermore, the conductive gels easily desic‐ cate and lose their adhesion. These problems bring about worse contact impedances at electrode-scalp interfaces, causing a large reduction of signal-to-noise ratio. In addition, the quality of recorded signals is sensitive to cable vibrations [17]. For these reasons, the long-

Recently, to overcome the weaknesses of passive electrodes, many researchers have studied advanced electrodes. For examples, research of dry electrodes is active recently. Generally, dry electrodes are defined as those that do not require the use of conductive gels or glues for installation process. Thus, a user can conveniently attach the electrodes to the user's scalp without any hair arrangement. To make dry contact at the electrode-skin interface, research‐ ers employ special materials or shapes in the design of dry electrodes. Extensive research has produced a huge variety of electrode materials and structures, including conductive rubber [8][9], conductive carbon nanotubes [7], micro-tip structures [6], micro-machined structures [14][15][20], non-contact types [4][5], spring-loaded fingers [2][13], bristle struc‐ tures [3], and conductive foams [27]. The most widely used dry electrode design is a set of contact posts which look like fingers [13][16]. This design has an advantage in contact ability because it is easy to penetrate into the scalp through the hair without an extended hair ar‐ rangement. Recently, some research groups have altered this finger design to produce ad‐ vanced mechanical designs, such as spring-loaded fingers [2] or bristle structures [3]. These designs seem to provide flexibility and geometric adaptation between the sensor and the ir‐ regular scalp surface to obtain low interface impedance. To achieve low contact impedance and provide a robust and stable electrical interface, some research groups have employed multi-walled carbon nanotube arrays [7] and micro-tip structures [6], which are able to pen‐

etrate the outer skin layer (which is 5 to10 um thick and called the *stratum corneum*).

While we can reduce the installation time significantly using dry electrodes, the contact im‐ pedance between the scalp and the electrodes is higher than that with gel-based passive electrodes due to the absence of conductive gels. Thus, signal quality of dry electrodes would be not better than that of the gel-based passive electrodes. To overcome this draw‐

term monitoring of EEG signals using passive electrodes is not feasible.

wearing.

**Figure 1.** Various EEG electrode types. (a) a miniature passive ring electrode [10] (b) a spring-loaded dry electrode [2] (c) a bristle-type dry electrode [3] (d) the Quasar hybrid EEG biosensor [11] (e) a non-contact-type active dry EEG sen‐ sor [4] (f) Diagram of a micro-tip electrode and the pyramidal shape of a micro-tip [6].

back, many research groups have been interested in active electrodes. Active electrodes con‐ tain amplifier or buffer circuits integrated to the electrodes themselves [1][4][5][7][11][12] [13]. This amplifier or buffer circuits are located between the electrodes and the signal ac‐ quisition frontend. They are aimed at impedance conversion. Providing high input impe‐ dance on the electrode-amplifier interface, active circuits reduce the distortion of the measured signals. This is desired for dry electrodes which do not use conductive fluids. Al‐ so, the low output impedance of the amplifier eliminates artifacts caused by posture changes in mobile environments. Therefore, the quality of measured physiological signals can be re‐ mained in a desirable state by the use of active electrodes.

Recent wireless BCI systems are equipped with active dry electrodes to combine the advan‐ tages of active and dry electrodes such as convenient installation and high fidelity signals. Because these electrodes provide more robust and stable signal quality in mobile environ‐ ments, they are suitable for wireless BCI systems. Researchers working on the development of advanced electrodes have produced a variety of active dry electrodes. Valchinov *et al*. de‐ signed body surface electrodes equipped with a biopotential amplifier using two op-amps [1]. Matthews *et al*. designed an ambulatory wireless EEG system using Quasar hybrid bio‐ sensors [11][12][13]. In this type of sensors, they employ a special circuit which uses the common mode follower (CMF) technology. This technology provides an ultra-high input impedance to ensure low distortion of the biopotential signals. Chi *et al.* designed and built dry and non-contact electrodes [5]. In their dry electrodes, a unity gain buffer is used to re‐ duce the effects of cable artifacts and external interference. Their non-contact electrodes also integrate discrete circuits to achieve high input impedance. To further optimize the size and power consumption of such electrodes, some researchers have employed customized ASIC designs for amplifiers. Xu *et al.* produce a low-power 8-channel active electrode system [17]. In this system, to reduce the power consumption of voltage buffers in dry electrodes, they designed active electrodes including ASICs based on chopper instrumentation amplifiers.

#### **2.2. Wireless BCI system design and structure**

In EEG-based wireless BCI systems, additional signal conditioning is essential to enable the transmission of precise neurophysiological signals. Many noise sources are present such as physiological interferences and power line noise. Physiological interferences are the other biopotential signals such as electromyogram (EMG), electrocardiogram (ECG), and electroo‐ culogram (EOG). They have relatively lager amplitudes around 50uV and up to 20-30mV while the amplitude of EEG signals is much smaller on the scale of roughly 10~100uV. Thus, the EEG signals are easily buried by these physiological signals unavoidably. In the case that the BCI system is connected to a desktop which operate with the electric power outlet, we also have to consider the power line noise as well. The power line noise contaminates the desired EEG signals in the range of 50 or 60Hz. Furthermore, the users of portable wireless BCI systems are usually in an active state making free motions and postures, whereas the users of wired BCI system are asked to stay in a motionless state, while their EEG signals are monitored. Therefore, the measured EEG signals of wireless BCI systems are also subject to heavy motion and vibration artifacts.

To avoid interference from the various noise components and recognize the user's intention correctly, the system must be designed carefully. Figure 2 is the general block diagram of a typical wireless BCI system. In EEG acquisition block of the system, there are two main parts, namely, the analog front end circuit and the digital system.

**Figure 2.** Block diagram of a typical wireless BCI system

In the analog front-end stage, the amplifier and bandwidth limiter circuits are included to make more robust and reliable EEG signals from the sensitive raw signals. Because the am‐ plitude of EEG signals is quiet small, the pre-amplification of the measured EEG signals at the analog front end is extremely important. In this amplification process, many developed wireless BCI systems use operational amplifiers or instrumentation amplifiers. Those ampli‐ fiers normally provide a gain ranging from thousands to hundreds of thousands. Amplifica‐ tion with high gain provides greater robustness against a variety of noise sources. However, we need to determine a suitable amplification gain to maximize the signal resolution in the analog digital converter (ADC) because the ADC has a restricted input dynamic range. Therefore, the amplification gain of the analog front end varies depending on the compo‐ nents of the digital system.

integrate discrete circuits to achieve high input impedance. To further optimize the size and power consumption of such electrodes, some researchers have employed customized ASIC designs for amplifiers. Xu *et al.* produce a low-power 8-channel active electrode system [17]. In this system, to reduce the power consumption of voltage buffers in dry electrodes, they designed active electrodes including ASICs based on chopper instrumentation amplifiers.

In EEG-based wireless BCI systems, additional signal conditioning is essential to enable the transmission of precise neurophysiological signals. Many noise sources are present such as physiological interferences and power line noise. Physiological interferences are the other biopotential signals such as electromyogram (EMG), electrocardiogram (ECG), and electroo‐ culogram (EOG). They have relatively lager amplitudes around 50uV and up to 20-30mV while the amplitude of EEG signals is much smaller on the scale of roughly 10~100uV. Thus, the EEG signals are easily buried by these physiological signals unavoidably. In the case that the BCI system is connected to a desktop which operate with the electric power outlet, we also have to consider the power line noise as well. The power line noise contaminates the desired EEG signals in the range of 50 or 60Hz. Furthermore, the users of portable wireless BCI systems are usually in an active state making free motions and postures, whereas the users of wired BCI system are asked to stay in a motionless state, while their EEG signals are monitored. Therefore, the measured EEG signals of wireless BCI systems are also subject to

To avoid interference from the various noise components and recognize the user's intention correctly, the system must be designed carefully. Figure 2 is the general block diagram of a typical wireless BCI system. In EEG acquisition block of the system, there are two main

(MUX) Microprocessor

In the analog front-end stage, the amplifier and bandwidth limiter circuits are included to make more robust and reliable EEG signals from the sensitive raw signals. Because the am‐ plitude of EEG signals is quiet small, the pre-amplification of the measured EEG signals at the analog front end is extremely important. In this amplification process, many developed wireless BCI systems use operational amplifiers or instrumentation amplifiers. Those ampli‐ fiers normally provide a gain ranging from thousands to hundreds of thousands. Amplifica‐

EEG Acquisition part Translation part

Wireless

Signal Processing (Feature translation)

Receiver

Application Program

parts, namely, the analog front end circuit and the digital system.

Analog Front End Digital system

Analog to Digital Converter (ADC)

Transmitter Amplifier

Multiplexer

**2.2. Wireless BCI system design and structure**

220 Brain-Computer Interface Systems – Recent Progress and Future Prospects

heavy motion and vibration artifacts.

Filters(Bandpass, 50/60Hz notch)

**Figure 2.** Block diagram of a typical wireless BCI system

EEG Sensors

We also need a frequency filtering procedure to remove various noise components. The EEG signals occupy a narrow bandwidth: normally from 0.1Hz to less than 50 Hz. Thus, filtering is helpful for extracting useful signals from the desired frequency bands. To filter out signals from useless frequency bands, the analog front-end of the system takes both a low-pass filter and a high-pass filter. Especially to filter out the power line noise, a notch filter which elimi‐ nates the specific frequency components of signals is also applied in this stage. Those filter‐ ing processes are performed using passive or active filtering circuits [24][26].

In the digital system stage, four integrated circuits are included: a multiplexer, an ADC, a microprocessor, and a wireless transmission unit. Generally, most EEG-based wireless BCI systems support multi-channel recording. To measure multi-channel signals simultaneous‐ ly, a multiplexer is needed to access all of the channels. Because the measured EEG signals are analog signals, an ADC has to be included to process the recorded EEG data on the digi‐ tal circuits. This integrated circuit transforms the EEG analog signals into discrete digitized data with a specific sampling frequency. The sampling frequency is determined by the speed of the microprocessor, wireless transmission, and translated frequencies of EEG fea‐ tures. Formally, researchers and system developers choose the sampling frequency between about 100 Hz and 1000 Hz. The microprocessor makes data packets from the corrected EEG data and hands them over to the wireless transmission unit. The microprocessor also man‐ ages the components of the entire system. Some wireless BCI systems load the feature ex‐ traction algorithm on the microprocessor to process the EEG signals internally [19][20][21] [29][30]. Because the recorded multichannel EEG data is transmitted from the portable EEG acquisition device to the host system, the wireless transmission unit is essential. Regarding the protocol of wireless transmissions, various communication modules are employed for transmission of the measured signals from the signal acquisition unit to the translation unit, such as Bluetooth and IEEE 802.15.4 Zigbee. Bluetooth has many advantages such as suffi‐ cient transmission rates and wide accessibility. Thus, many wireless BCI systems employ this transmission module. Including analog front-end and digital system stage, the acquisi‐ tion unit of wireless BCI systems generally operates onboard power sources such as Li-ion, Li-polymer, and NiMH batteries.

Because the analog front-end and digital system parts have to be loaded in portable and wearable acquisition part of wireless BCI systems, longer operation time and small size are necessary in system specifications. Thus, system developers should choose low-power com‐ ponents with smaller packages. Recently, many semiconductor manufacturers have released low-power microprocessors and integrated analog front end circuits for bio-potential meas‐ urements. For example, Texas Instruments released the ADS129x series integrated circuit solutions [34] for the analog front end of ECG/EEG applications. This series provides up to 8-channel high-resolution ADCs and a built-in programmable gain amplifier (PGA) with low noise and low power consumption features. In the microprocessor area, various ultralow power processors have been released on the market for portable devices. The most widely used microprocessors are the PIC24 microcontroller series [35], the dsPIC digital sig‐ nal controller series [36] (manufactured by Microchip Technology) and the MSP430 micro‐ controller series [37] (manufactured by Texas Instruments). In particular, the dsPIC processor series is applied as the processing unit of the Emotiv EPOC system [38].

Regarding the design approach of system appearance, a variety of designs have been adopt‐ ed depending on the application purpose and target users. Widely used styles of the acquisi‐ tion part of wireless BCI systems include headsets [16][17][23][25][27], head bands [18][20] [22][28][30][32], baseball caps [14][19][21][29], and military helmets [13]. In designing the ap‐ pearance of wireless BCI systems, we need to consider several factors, such as wearability, stability, and convenience of installation. To provide long-term monitoring capability, wear‐ able part of wireless BCI systems has to be light with comfort fitting. Also, convenient instal‐ lation is necessary to save time in the set up process. Appropriate pressures are also needed to maintain stable electrode positions and low impedance characteristics at the sensor-skin interfaces. Additionally, to allow for the diversity of users' head sizes, the materials used in wireless BCI systems should be flexible, or size adjusters must be added. Various designs of wireless BCI systems are shown in Figure 3.

**Figure 3.** Various designs of wireless BCI systems: (a) wearable EEG acquisition headset [27], (b) 8-channel EEG moni‐ toring headset [16], (c) baseball cap-based EEG acquisition device [14], (d) wireless EEG system for SSVEP application [25], and (e) soldier's Kevlar helmet-based ambulatory wireless EEG system for real-time workload classification [13].

#### **2.3. Signal features and applications**

low noise and low power consumption features. In the microprocessor area, various ultralow power processors have been released on the market for portable devices. The most widely used microprocessors are the PIC24 microcontroller series [35], the dsPIC digital sig‐ nal controller series [36] (manufactured by Microchip Technology) and the MSP430 micro‐ controller series [37] (manufactured by Texas Instruments). In particular, the dsPIC

Regarding the design approach of system appearance, a variety of designs have been adopt‐ ed depending on the application purpose and target users. Widely used styles of the acquisi‐ tion part of wireless BCI systems include headsets [16][17][23][25][27], head bands [18][20] [22][28][30][32], baseball caps [14][19][21][29], and military helmets [13]. In designing the ap‐ pearance of wireless BCI systems, we need to consider several factors, such as wearability, stability, and convenience of installation. To provide long-term monitoring capability, wear‐ able part of wireless BCI systems has to be light with comfort fitting. Also, convenient instal‐ lation is necessary to save time in the set up process. Appropriate pressures are also needed to maintain stable electrode positions and low impedance characteristics at the sensor-skin interfaces. Additionally, to allow for the diversity of users' head sizes, the materials used in wireless BCI systems should be flexible, or size adjusters must be added. Various designs of

**Figure 3.** Various designs of wireless BCI systems: (a) wearable EEG acquisition headset [27], (b) 8-channel EEG moni‐ toring headset [16], (c) baseball cap-based EEG acquisition device [14], (d) wireless EEG system for SSVEP application [25], and (e) soldier's Kevlar helmet-based ambulatory wireless EEG system for real-time workload classification [13].

processor series is applied as the processing unit of the Emotiv EPOC system [38].

wireless BCI systems are shown in Figure 3.

222 Brain-Computer Interface Systems – Recent Progress and Future Prospects

In EEG-based BCI systems, they translate the specific signal features that reflect the user's intentions or cognitive states into commands or feedback signals for controlling of the target applications. For these operations, BCI systems analyze and capture the user's intensions based on detection of ERPs [33] or power spectra changes in specific brain rhythms.

Most research groups have focused on sensorimotor rhythms (SMRs) [39] and event related potentials (ERPs) [40], including visual evoked potentials (VEPs) such as P300 [41] and steady-state visual evoked potentials (SSVEP) [42]. The SMRs are spontaneous responses which can be actively generated by motor imageries, such as the left hand, the right hand, or the foot movements measured by electrodes placed on the scalp over the sensorimotor cor‐ tex area. These rhythms appear as suppression or enhancement of the power spectra, called event-related desynchronization (ERD) and event-related synchronization (ERS). The VEPs are behavioral responses which are passively synchronized by the frequency of flickering visual stimulus from the occipital lobe. Because the SMRs and VEPs reflect the user's inten‐ sions, we can utilize them as a means to control commands in applications. Moreover, vari‐ ous cognitive states are also studied, such as drowsiness, alertness, and mental focusing [27] [29][30][32]. These cognitive states are related to the power changes of specific rhythms, called alpha, beta, theta rhythms and so on. Several studies have shown that the power of the alpha rhythm has a negative relationship with mental concentration [27][29][30]. Also, researchers have found that when subjects feel sleepy or fall into a deep sleep, the power spectra of alpha and theta bands change depending on these drowsiness conditions [29][30] [32]. Using these relationships, the BCI systems detect the user's cognitive states and pro‐ vide feedbacks such as focusing indicator [27] and sleep warning [32].

Among the signal features mentioned above, many researchers have chosen VEPs or users' cognitive states as a means for BCI based controls. The reason is that these features are easy to generate and provide good accuracy in the application of BCI systems. Also, the features of VEPs and cognitive state make it easy to classify users' intentions with a relatively simple feature extraction method, and a small number of electrodes and training session are needed to achieve higher accuracy. Compared with ERPs like VEPs, the SMRs can be generated by voluntary imagination, such as motor imagery, generally require long-term training to ach‐ ieve higher accuracy in BCI applications. Furthermore, approximately 30% of normal people cannot generate SMRs due to the phenomenon of BCI illiteracy [43].

In the application parts of BCI systems a variety of promising rehabilitation-related applica‐ tions have already been developed with EEG-based wired BCI systems. Because BCI sys‐ tems measure and analyze neurophysiological signals, these systems provide practical assistance for patient diagnosis, treatment, and rehabilitation. For example, using BCI sys‐ tems, long-term monitoring of EEG signals assists the diagnosis of epilepsy and the predic‐ tion of epileptic seizures [44]. For people with severe motor disabilities, the P300 speller [41] and wheelchair control [45] applications provide practical assistance in everyday life by pro‐ viding non-muscular motor functions.

In spite of these useful applications, the dissemination of BCI systems is limited because of the drawbacks of wired BCI systems. Wired BCI systems are generally bulky, complicated, and expensive. Also, the users of wired BCI systems are confined to a limited space without freedom of postures and movements. To overcome these limitations, wired BCI systems are gradually being replaced with the wireless BCI systems.

**Figure 4.** Applications of wireless BCI systems: (a) workload classification application screenshot (data collection and engagement classifier running on the gaming subject) [13], (b) EEG-based BCI archery game screenshot [27], and (c) SSVEP-based dialing application using smart phone [28].

Recently, with the development of wireless BCI systems, researchers have shifted their focus from applications for disabled people to applications of interest to the general public in the entertainment, smart living environment, and cognitive neuroscience areas. In the entertain‐ ment area, Liao *et al.* developed an EEG-based gaming interface based on a real-time focus‐ ing detection algorithm with a wireless EEG acquisition device [27]. For smart living environment, Lin *et al.* developed an environmental auto-control system based on human physiological states, such as drowsiness and alertness [29][30]. Similarly, Guge *et al.* devel‐ oped a smart home control system based on P300 EEG response [31]. For mobile applica‐ tions, Wang *et al.* developed a cellular phone dialing application [28]. In cognitive state monitoring, D'Arcy *et al.* developed a diagnostic device which provides an evaluation of an individual's conscious awareness based on various ERP components [33]. In this research, they found that sensation, perception, attention, memory, and language are properly related with the P1, mismatch negativity (MMN), P300 (tones), P300 (speech), and N400 responses. Also, Matthews *et al.* developed a real-time workload classification system during subject motion with a compact ambulatory wireless EEG system [13]. Figure 4 shows application examples of wireless BCI systems.

#### **3. Review of wireless BCI systems**

In spite of these useful applications, the dissemination of BCI systems is limited because of the drawbacks of wired BCI systems. Wired BCI systems are generally bulky, complicated, and expensive. Also, the users of wired BCI systems are confined to a limited space without freedom of postures and movements. To overcome these limitations, wired BCI systems are

**Figure 4.** Applications of wireless BCI systems: (a) workload classification application screenshot (data collection and engagement classifier running on the gaming subject) [13], (b) EEG-based BCI archery game screenshot [27], and (c)

Recently, with the development of wireless BCI systems, researchers have shifted their focus from applications for disabled people to applications of interest to the general public in the entertainment, smart living environment, and cognitive neuroscience areas. In the entertain‐ ment area, Liao *et al.* developed an EEG-based gaming interface based on a real-time focus‐ ing detection algorithm with a wireless EEG acquisition device [27]. For smart living environment, Lin *et al.* developed an environmental auto-control system based on human physiological states, such as drowsiness and alertness [29][30]. Similarly, Guge *et al.* devel‐ oped a smart home control system based on P300 EEG response [31]. For mobile applica‐ tions, Wang *et al.* developed a cellular phone dialing application [28]. In cognitive state monitoring, D'Arcy *et al.* developed a diagnostic device which provides an evaluation of an individual's conscious awareness based on various ERP components [33]. In this research,

gradually being replaced with the wireless BCI systems.

224 Brain-Computer Interface Systems – Recent Progress and Future Prospects

SSVEP-based dialing application using smart phone [28].

Over the past 20 years, many research achievements associated with BCI and the neuro‐ sciences have been made and they have helped stimulate the interest of the general public. Owing to the advances in wireless BCI systems, bulky wired biopotential acquisition sys‐ tems have been replaced with portable and wearable devices. Following this trend, the num‐ ber of published papers with the topic of wireless BCI systems and (their) applications is being continuously increased. A few commercial companies have developed and released portable wireless EEG acquisition systems with interesting new entertainment applications. Now, measuring brain activity is no longer limited to hospital-based medical diagnostics, but includes more courageous applications aiming at changing the lifestyle of users. In this section, we aim to first review several wireless BCI systems which have appeared in recent research articles. Second, we will introduce several examples of wireless BCI systems which have been lately released into the market for consumer and research usages.

#### **3.1. Wireless BCI systems in scientific papers**

In the research field, many research articles have been published in the last decade with the topic of wireless BCI systems. There are some distinct features in them such as the use of dry electrodes and novel applications which we are interested in reviewing in this subsection. For example, see the wireless BCI systems listed in Table 1. This table shows system specifi‐ cations such as the number of channels and operation hours. They are all wireless and wear‐ able systems, some aiming for applications that average people can find useful, including drowsiness detection and workload monitoring. Specifications for each system are opti‐ mized for its own target application.

In what follows, we will briefly review each of the system listed in Table 1

#### *3.1.1. Wireless BCI system for archery game control*

Recently, utilizing wireless BCI systems, the number of game applications has been in‐ creased. For example, Liao *et al.* [27] have developed an EEG based BCI device for an arch‐ ery game control. This device is designed as a user friendly headset (see Figure 3 (a)) and equipped with three channels, each channel with a dry EEG sensor. For the control of arch‐ ery game, the sensors measure the power of alpha rhythm collected off of user's forehead when a user concentrates on a target. This power value is converted to a measure of focus‐ ing intensity in real time. Any user can test out the level of one's concentration effort using


**Table 1.** Comparison of wireless BCI systems in scientific articles

this game (see Figure 4 (b)). If a user maintains for a certain period a high level of concentra‐ tion state, for example, the arrow will hit the center of target.

#### *3.1.2. Mobile BCI using dry and noncontact EEG sensors*

These days, smart phones come with high performance processors. They can be useful for wireless EEG monitoring. In [5], Yu Mike Chi *et al*. have developed a wireless BCI device based on smart phones. They have developed an smartphone application with a GUI inter‐ face for signal monitoring and analysis. They have tested two types of electrodes, dry and noncontact EEG sensors. Dry sensors consist of several spring-loaded electrodes, each comes with a finger post and a unity gain buffer. Noncontact electrodes employ CMOS amplifiers to improve their impedance performance. Both electrodes offer easy installation and good signal quality.

#### *3.1.3. Wireless BCI system based on dry electrodes with a needle-shape structure*

As mentioned in Section 2, novel electrode design is important part in wireless BCI systems. In [6], Nuno Sérgio Dias *et al.* have developed a wireless EEG acquisition system with novel dry electrodes. For the electrode design, they have employed 16 micro tip structures each looks like a needle (see Figure 1 (f)). They are designed to make direct contacts with electro‐ lyte fluids of the inner skin layers of the scalp. Therefore, signal quality is satisfactory with‐ out the use of conductive gels. The proposed system consists of an EEG brain cap with five dry electrodes, an acquisition device which is attached to the brain cap, and a wireless base station connected with a computer. The acquisition device can operate for 25 hours with two AA batteries.

#### *3.1.4. Design of wireless BCI system for drowsiness detection*

Wireless BCI systems can be utilized in practical applications such as house control system [29][30][31] and drowsiness detection system [20] for drivers. In [20], Chin-Teng Lin *et al.* have proposed a real-time wireless EEG-based BCI system for drowsiness detection and shown usefulness in providing sleep alerts to a driver in car driving simulation. They have designed a wireless signal acquisition module and a signal processing module. The acquisi‐ tion module is small enough to be embedded into a wearable headband. These modules are linked with each other via a Bluetooth connection. Therefore, it provides the advantages of mobility and long-term monitoring (more than 33 hours with a 1100-mA Li-Ion battery). Al‐ so, they have developed a real-time drowsiness detection algorithm. The algorithm detects the user's drowsiness by analyzing the theta rhythm and the alpha rhythm of the EEG sig‐ nals.

#### *3.1.5. Ambulatory wireless BCI system for real time workload classification*

A compact, lightweight, and ultra-low power ambulatory wireless BCI system has been de‐ veloped in [13]. This system consists of soldier's helmet with biosensors and a data acquisi‐ tion unit (see Figure 3 (e)). They use Quasar hybrid biosensors which are equipped with dry electrodes shaped like a set of fingers (see Figure 1 (d)). This system provides high freedom of motions with data quality as good as that of wet electrodes. They have also developed a real time classifier for determination of the cognitive workload of a user.

#### *3.1.6. Design of wireless EEG monitoring headset*

this game (see Figure 4 (b)). If a user maintains for a certain period a high level of concentra‐

These days, smart phones come with high performance processors. They can be useful for wireless EEG monitoring. In [5], Yu Mike Chi *et al*. have developed a wireless BCI device based on smart phones. They have developed an smartphone application with a GUI inter‐ face for signal monitoring and analysis. They have tested two types of electrodes, dry and noncontact EEG sensors. Dry sensors consist of several spring-loaded electrodes, each comes with a finger post and a unity gain buffer. Noncontact electrodes employ CMOS amplifiers

tion state, for example, the arrow will hit the center of target.

*3.1.2. Mobile BCI using dry and noncontact EEG sensors*

**Table 1.** Comparison of wireless BCI systems in scientific articles

**Reference**

**Signal features**

**Application**

**Sensor type**

**Frontend processing unit**

**Backend processing unit**

**Transmission protocol**

**Signal resolution and sampling rate**

**Power source and operation time**

**Design**

Liao et al., 2012 [27]

226 Brain-Computer Interface Systems – Recent Progress and Future Prospects

Mental focusing feature

Archery game control

Dry foam-based EEG sensor

PC

Bluetooth v2.0 +EDR

23 hours with a 3.7v 750mAh Liion battery

Headset with elastic band

Yu Mike Chi et al., 2012 [5]

> Dry and noncontact electrodes

**# of channels** 3 channels Not mentioned 5 channels 3 channels

Nokia N97 cellular phone

10 hours with 2 AAA batteries

TI MSP430 Microchip PIC24F

Nuno Sérgio Diasa et al., 2012 [6]

> Micro tip dry electrodes

Atmel Atmega128

PC

25 hours using 2 AA batteries

Bluetooth Bluetooth

12bit, 256Hz 24 bit 16bit, 1kHz 12bit, 512Hz

SSVEP Not mentioned

Neuro-feedback Not mentioned

Chin-Teng Lin et al., 2010 [20]

Alpha and Theta rhythms

> Drowsiness detection

Not mentioned

ADSP-BF533 embedded processor

Bluetooth v. 2.0+EDR

33 hours with a 3.7v 1.1Ah Li-Ion battery

Not mentioned Brain cap Headband Soldier helmet Headset

R. Matthews et al., 2008 [13]

Cognitive state (workload)

> Workload monitoring

Finger type hybrid biosensor

7 channels EEG, ECG, EMG, EOG 2 channels

16bit, 240Hz for EEG

80 hours with 2 AAA batteries

TI MSP430 Not mentioned

Lindsay Brown et al., 2010 [16]

Not mentioned

Not mentioned

Dry electrode with contact post

8 channels

TI MSP430 and ASIC

11bit, 256 ~ 1024Hz

30 hours with a 3.7v 140mAh Li-Ion battery

Laptop Not mentioned

Bluetooth 2.4GHz

In [16], Lindsay Brown *et al*. have introduced a design of wireless EEG monitoring headset. This headset is equipped with the 8-channel dry electrodes as shown in Figure 3 (b). Each electrode is designed with contact posts which are coated with Ag/AgCl for easy penetration into the user's hairs. Each electrode is connected to its electrode housing via a magnetic ball and a socket for tilting and vertical movement of electrodes. They have employed an 8-chan‐ nel EEG acquisition application-specific integrated circuit (ASIC) into which preamplifiers are embedded for low power consumption. This system can operate for long time over 30 hours using a 140 mAh Li-Ion battery.

#### **3.2. Wireless BCI systems for consumer use**

In the market of consumer-grade wireless BCI systems, there are many commercial compa‐ nies such as Emotiv, Neurosky, MyndPlay, PLX devices, and OCZ technology. These com‐ panies have competitively released their own wireless BCI systems along with various applications related to gaming, utilities, and mental-state monitoring. In this section, we re‐ view these wireless BCI devices for consumer use shown in Figure 5.

**Figure 5.** Pictures of Wireless BCI systems for consumer use: (a) Emotive EPOC headset [38], (b) NeuroSky Mind Set [46], (c) MyndPlay Brainband [50], (d) PLX devices XWave headset [49], (e) OCZ Neural Impulse Actuator [51].

The device released by Emotiv is the EPOC headset. The EPOC headset [38] is a multi-chan‐ nel wireless BCI system. This headset is equipped with 14 saline-based wet-contact resistive electrodes for measuring EEG, Electrooculogram (EOG), and facial Electromyogram (EMG). Additionally, the EPOC headset also has a 2-axis gyroscope for measuring the head rotation. Employing 2.4GHz wireless connectivity, this system provides wide accessibility for devices such as PCs, laptops, and smart phones. The package of the EPOC headset provides a bun‐ dle software which contains a suite of built-in signal processing algorithms for interpreta‐ tion of EEG signals. The built-in algorithms discern the user's conscious intentions, emotional states, and facial expressions based on measured EEG, Electromyogram (EMG) and Electrooculogram (EOG) signals. Through this software, the users can interact with var‐ ious applications related to virtual reality, game controlling, and brain state monitoring. These applications can be downloaded by accessing their web site.

The Neurosky Mind Set [46] is a wireless headset added with an EEG signal acquisition unit. This headset is equipped with earphones and a microphone, and a single dry-contact elec‐ trode for measuring the user's EEG signals on the user's forehead. Along with the capability of raw EEG recording, the Mind Set has the patented algorithm, named as eSense [47]. This algorithm interprets the user's mental states such as attention and meditation. These transla‐ tions are estimated by monitoring the power levels in specific frequency bands such as al‐ pha, beta and theta rhythms. The monitoring values of the brain state are utilized for making a control commands in applications.

The MyndPlay and the PLX devices are released with their own model of wireless BCI sys‐ tems. MyndPlay Brainband [50] and PLX devices XWave [49] headset utilizes the ThinkGear Application-Specific Integrated Circuit (ASIC) module [48]. This module, designed by Neu‐ rosky, is a system-on-chip integrated circuit equipped with signal acquisition components. These devices come in a headset or a harness style. They have supported the control of vari‐ ous applications such as media player, cognitive state visualization, and arcade games for PC and mobile devices.

OCZ technology, a PC component manufacturer such as solid-state drives (SSD) and power supplies, has released a game controller by utilizing an wireless BCI technology. The name of this controller is Neural Impulse Actuator (NIA) [51]. Using NIA, the users can control a PC game by translating facial expressions, eye movements, and concentrated brainwave ac‐ tivity, instead of using traditional input devices such as a keyboard and a mouse. This de‐ vice now supports various PC games including shooting, role playing, virtual worlds, and racing.

#### **3.3. Wireless BCI systems for research uses**

nel EEG acquisition application-specific integrated circuit (ASIC) into which preamplifiers are embedded for low power consumption. This system can operate for long time over 30

In the market of consumer-grade wireless BCI systems, there are many commercial compa‐ nies such as Emotiv, Neurosky, MyndPlay, PLX devices, and OCZ technology. These com‐ panies have competitively released their own wireless BCI systems along with various applications related to gaming, utilities, and mental-state monitoring. In this section, we re‐

a b

c d e

**Figure 5.** Pictures of Wireless BCI systems for consumer use: (a) Emotive EPOC headset [38], (b) NeuroSky Mind Set [46], (c) MyndPlay Brainband [50], (d) PLX devices XWave headset [49], (e) OCZ Neural Impulse Actuator [51].

The device released by Emotiv is the EPOC headset. The EPOC headset [38] is a multi-chan‐ nel wireless BCI system. This headset is equipped with 14 saline-based wet-contact resistive electrodes for measuring EEG, Electrooculogram (EOG), and facial Electromyogram (EMG). Additionally, the EPOC headset also has a 2-axis gyroscope for measuring the head rotation. Employing 2.4GHz wireless connectivity, this system provides wide accessibility for devices such as PCs, laptops, and smart phones. The package of the EPOC headset provides a bun‐ dle software which contains a suite of built-in signal processing algorithms for interpreta‐ tion of EEG signals. The built-in algorithms discern the user's conscious intentions, emotional states, and facial expressions based on measured EEG, Electromyogram (EMG) and Electrooculogram (EOG) signals. Through this software, the users can interact with var‐

view these wireless BCI devices for consumer use shown in Figure 5.

hours using a 140 mAh Li-Ion battery.

**3.2. Wireless BCI systems for consumer use**

228 Brain-Computer Interface Systems – Recent Progress and Future Prospects

Wireless BCI systems have many advantages, such as freedom of user's postures and con‐ venient installation. Therefore, wireless BCI systems are useful in the research field as well. Some companies have been involved in research with universities and research institutes to develop wireless BCI systems for research uses. They include, but not limited to, Advanced Brain Monitoring, Quasar, Starlab and Guger technologies (G.tec). In this subsection, we re‐ view their wireless BCI systems, as shown in Figure 6.

Advanced Brain Monitoring has recently released the B-Alert X series wireless EEG systems [52] for mobile neurophysiological data acquisition and analysis. These systems include three models that have different numbers of channels, i.e., 4, 10, and 24. Among these mod‐ els, the B-Alert X24 system is equipped with 24 channel electrodes for biopotential measure‐ ments, such as EEG, Electrocardiogram (ECG), Electromyogram (EMG), and Electrooculogram (EOG). This system measures and delivers real-time EEG signals via a Bluetooth connection. The system provides more than 8 hours of operation time and less than 10 minutes of installation time. Also, this system supports a variety of applications such as drowsiness, cognitive workload, and neuro-dynamics monitoring.

**Figure 6.** Pictures of Wireless BCI systems for research use: (a) Advanced Brain Monitoring B-Alert X24 system [52], (b) Quasar DSI 10/20 system [53], (c) Starlab Enobio system [54].

Quasar [53] has developed and released a wireless BCI solution. This solution includes Dry Sensor Interface (DSI) 10/20, wireless Data Acquisition (DAQ), and a suite of software of its own. DSI 10/20 is a wireless BCI headset which is equipped with up to 21 EEG sensors. The EEG sensors are dry electrodes and provide high input impedance good for measuring the high fidelity EEG signals. Wireless DAQ is a peripheral device for signal transmission and onboard recording using a flash memory. QStreamer is a suite of software which contains data acquisition algorithms as well as different cognitive state classification algorithms. The classification algorithms estimate user's mental states in terms of workload, engagement, and fatigue.

Enobio [54] is a wireless EEG acquisition device developed by Starlab. This device is a cap style with light weight feature (only 65g). It has multiple channels, supporting an option of 8 or 20 channels in particular; each is equipped with a dry electrode. It can operate up to 16 hours long using a rechargeable Lithium Polymer battery. It is connected with a computer via a Bluetooth connection. A bundle software provides real time visualization of EEG sig‐ nals such as power spectrum and raw signal monitoring. This system has been applied to various applications associated with medical, neurofeedback, and cognitive state monitor‐ ing.

Guger technologies (G.tec) is a medical engineering company which provides comprehen‐ sive BCI solutions. This company has released a mobile biopotential acquisition system named as g.MOBIlab+ [55]. This system is available in two different modes: the 8 channel EEG acquisition mode and the multi-modal acquisition mode. In the multi-modal acquisi‐ tion mode, this system can measure the EEG signals with other physiological signals such as Electrocardiogram (ECG), Electromyogram (EMG), and Electrooculogram (EOG). With a Bluetooth connection, it operates up to 100 hours using four AA batteries.

#### **4. Challenges and future research directions for wireless BCI systems**

Many wireless BCI systems have already been developed for consumers or research uses, and they are attracting public attention. However, only limited areas of applications, such as medical and entertainment applications, have taken advantages of these systems so far. Wireless BCI systems have not yet fully made their way into our life. In this section, we aim to analyze the current challenges the wireless BCI systems research must face, and discuss a few possible future research directions to enable wireless BCI systems to be practically uti‐ lized in a much wider scale.

#### **4.1. Challenges in wireless BCI systems research**

a b c

**Figure 6.** Pictures of Wireless BCI systems for research use: (a) Advanced Brain Monitoring B-Alert X24 system [52], (b)

Quasar [53] has developed and released a wireless BCI solution. This solution includes Dry Sensor Interface (DSI) 10/20, wireless Data Acquisition (DAQ), and a suite of software of its own. DSI 10/20 is a wireless BCI headset which is equipped with up to 21 EEG sensors. The EEG sensors are dry electrodes and provide high input impedance good for measuring the high fidelity EEG signals. Wireless DAQ is a peripheral device for signal transmission and onboard recording using a flash memory. QStreamer is a suite of software which contains data acquisition algorithms as well as different cognitive state classification algorithms. The classification algorithms estimate user's mental states in terms of workload, engagement,

Enobio [54] is a wireless EEG acquisition device developed by Starlab. This device is a cap style with light weight feature (only 65g). It has multiple channels, supporting an option of 8 or 20 channels in particular; each is equipped with a dry electrode. It can operate up to 16 hours long using a rechargeable Lithium Polymer battery. It is connected with a computer via a Bluetooth connection. A bundle software provides real time visualization of EEG sig‐ nals such as power spectrum and raw signal monitoring. This system has been applied to various applications associated with medical, neurofeedback, and cognitive state monitor‐

Guger technologies (G.tec) is a medical engineering company which provides comprehen‐ sive BCI solutions. This company has released a mobile biopotential acquisition system named as g.MOBIlab+ [55]. This system is available in two different modes: the 8 channel EEG acquisition mode and the multi-modal acquisition mode. In the multi-modal acquisi‐ tion mode, this system can measure the EEG signals with other physiological signals such as Electrocardiogram (ECG), Electromyogram (EMG), and Electrooculogram (EOG). With a

Bluetooth connection, it operates up to 100 hours using four AA batteries.

Quasar DSI 10/20 system [53], (c) Starlab Enobio system [54].

230 Brain-Computer Interface Systems – Recent Progress and Future Prospects

and fatigue.

ing.

Wireless BCI systems currently face the following problems: 1) insufficiency in features con‐ trollable by EEG signals, 2) deficiency in accuracy in EEG signal interpretation, and 3) lack of killer applications.

First, the available features of EEG signals are limited to be utilized for increasing the speed and accuracy of brain computer communications. The EEG signals are not easy for a user to freely generate in user's own intension. It means that the controllable dimension of EEG sig‐ nals is not large. In the EEG-based BCI systems, sensorimotor rhythms (SMRs), visual evoked potentials (VEPs) and brain rhythms which generated by particular cognitive states are used as controllable features. Among these features, SMRs usually require long-term training to be adopted in control of BCI applications. Without enough training, the accuracy of SMR-based BCI is usually very low. Also, VEPs cannot be generated without visual stim‐ ulations. Therefore, among the wireless BCI systems on scientific articles and commercial products, many BCI systems have utilized the specific brain rhythms generated under the specific cognitive states of users [27][46][49][50][51], such as attention and relaxation, as con‐ trollable features of EEG signals. The users can control an application by intentionally changing their own cognitive state. For example, in [27], the users can control the direction of arrows in an archery game, using a wireless BCI system. The direction of arrows is deter‐ mined from the quantification of the focusing intensity of the users. As can be seen here, the intensity level is only single dimensional; the degree of control the users can issue cannot but be limited. This limitation in the control dimension comes with easiness in adapting to a program within a short learning time, but it plays a limiting role as well. Consequently, the applications BCI systems can be applied are limited as well. To remedy this limitation, some research groups have focused on discovering other controllable EEG features. One example is to use detection of various cognitive states, such as drowsiness and alertness [20][29][30] [32]. With all these efforts, however, the number of features that can be obtained is still con‐ sidered very limited. Namely, wireless BCI systems today can interpret only simple mas‐ sages from user's intensions.

Second, current BCI systems are not reliable enough to be used in accuracy-critical applica‐ tions, such as vehicle controls and data telecommunications. To utilize them in a wide range of applications, improving the accuracy of brain computer communication is one of most important issues. In wireless BCI systems, many features of the EEG signals, such as cogni‐ tive states, event related potentials (ERPs) in P300, and steady-state visual evoked potentials (SSVEP), are used for BCI based controls. These features are easily affected by various noise and inference sources. For example, in ambulatory applications, such as the workload moni‐ toring [13], a user may need to wear a wireless EEG acquisition device for a long period of time and may need to be able to move around. In such a situation, the accuracy of applica‐ tions can easily decline due to vibrations and noises. Furthermore, because every person has somewhat different characteristics in EEG features, training to find the best feature set should be carried out in the individual basis for achieving higher accuracy. For these rea‐ sons, most commercial wireless BCI systems are developed for less accuracy-critical applica‐ tions such as computer games and home appliances.

Third, killer applications for wireless BCI systems are needed. In wireless BCI systems, a killer application can be said to be a useful application which influenced on the life of an average person. What BCI system provides ultimately is a human computer interface. But it is not the only form that human can interact with the computer. Speech recognition and hand-motion recognition are, for example, other easier means perhaps with faster and more accurate performance than a BCI can provide. In commercial wireless BCI systems, applica‐ tions for game and utility control have been mostly developed for entertainment uses. In these applications, it is possible to choose an alternative control interface, such as speech rec‐ ognition and hand motion, instead of the EEG signals for application controls. Thus, these applications are not a killer application. In the research field, valuable applications related to smart living environment [29][30][32], drowsiness detection [15][19][20][32], and communi‐ cations [28] have been developed for wireless BCI systems, but they are still not suitable due to lack of sufficient field verifications. Therefore, identification of a killer application still re‐ mains to be an urgently needed research topic for wireless BCI systems to thrive.

#### **4.2. Future research directions for wireless BCI systems**

Future research on wireless BCI systems should take the following directions: 1) hybrid sig‐ nal acquisition and 2) development of adaptive classification algorithms.

First, hybrid signal acquisition is needed for higher accuracy and fast brain computer inter‐ action. Hybrid signal acquisition through simultaneous recording of multiple brain signals has been shown to ensure higher accuracy thanks to complementary analysis of user's moti‐ vations. There are two different ways to do a hybrid signal acquisition we wish to discuss here. The first is utilizing other biopotential measurements, such as Electrocardiogram (ECG), Electromyogram (EMG), and Electrooculogram (EOG). This approach is already em‐ ployed in commercial wireless BCI systems such as Emotiv's EPOC system, OCZ technolo‐ gy's Neural Impulse Actuator, and Advanced Brain Monitoring's B-Alert X series. With these additional physiological signal measurements, the users can use not only brain waves but also facial expressions and eye movements. The second is utilizing the multiple features among the available EEG features, such as sensorimotor rhythms (SMRs), P300, and steadystate visual evoked potentials (SSVEP) simultaneously. Most commercial wireless BCI sys‐ tems use only the cognitive states as the controllable feature of wireless BCI systems. If wireless BCI systems employ additional features of EEG signals which are not generated by the same motivation, the accuracy of the applications will be improved by adoption of the complementary classification. Recently, Brunner *et al.* [56] have published a paper on a hy‐ brid BCI. In that paper, a hybrid approach using both the event-related desynchronization (ERD) and the SSVEP was experimentally found to provide better accuracy with little or no training.

Second, adaptive algorithms are needed to reduce training time and achieve higher accura‐ cy. Because every person has a unique set of their own EEG characteristics, most applica‐ tions require training procedures for learning the user's EEG patterns. However, the EEG patterns typically change continuously affected by many factors such as the mental state of the users and the circumstance surrounding the users. Furthermore, long-term training can make the users tired and induce degradation in accuracy. For these reasons, the reduction of training in applications is an important issue in BCI researches. To reduce trainings, addi‐ tional signal processing schemes likes adaptive classification algorithms can be added to BCI systems. Because these schemes can discern changes in EEG features, the accuracy of appli‐ cations can be improved. In the BCI research field, some researchers have already studied adaptive classification. Vidaurre *et al.* have published papers about adaptive discriminant analysis [57][58]. This algorithm is based on quadratic discriminant analysis (QDA) and line‐ ar discriminant analysis (LDA). Using this algorithm, they have shown that accuracy can be improved greatly in their own BCI experiments built on imagery motor movements.

#### **5. Conclusion**

tive states, event related potentials (ERPs) in P300, and steady-state visual evoked potentials (SSVEP), are used for BCI based controls. These features are easily affected by various noise and inference sources. For example, in ambulatory applications, such as the workload moni‐ toring [13], a user may need to wear a wireless EEG acquisition device for a long period of time and may need to be able to move around. In such a situation, the accuracy of applica‐ tions can easily decline due to vibrations and noises. Furthermore, because every person has somewhat different characteristics in EEG features, training to find the best feature set should be carried out in the individual basis for achieving higher accuracy. For these rea‐ sons, most commercial wireless BCI systems are developed for less accuracy-critical applica‐

Third, killer applications for wireless BCI systems are needed. In wireless BCI systems, a killer application can be said to be a useful application which influenced on the life of an average person. What BCI system provides ultimately is a human computer interface. But it is not the only form that human can interact with the computer. Speech recognition and hand-motion recognition are, for example, other easier means perhaps with faster and more accurate performance than a BCI can provide. In commercial wireless BCI systems, applica‐ tions for game and utility control have been mostly developed for entertainment uses. In these applications, it is possible to choose an alternative control interface, such as speech rec‐ ognition and hand motion, instead of the EEG signals for application controls. Thus, these applications are not a killer application. In the research field, valuable applications related to smart living environment [29][30][32], drowsiness detection [15][19][20][32], and communi‐ cations [28] have been developed for wireless BCI systems, but they are still not suitable due to lack of sufficient field verifications. Therefore, identification of a killer application still re‐

mains to be an urgently needed research topic for wireless BCI systems to thrive.

nal acquisition and 2) development of adaptive classification algorithms.

Future research on wireless BCI systems should take the following directions: 1) hybrid sig‐

First, hybrid signal acquisition is needed for higher accuracy and fast brain computer inter‐ action. Hybrid signal acquisition through simultaneous recording of multiple brain signals has been shown to ensure higher accuracy thanks to complementary analysis of user's moti‐ vations. There are two different ways to do a hybrid signal acquisition we wish to discuss here. The first is utilizing other biopotential measurements, such as Electrocardiogram (ECG), Electromyogram (EMG), and Electrooculogram (EOG). This approach is already em‐ ployed in commercial wireless BCI systems such as Emotiv's EPOC system, OCZ technolo‐ gy's Neural Impulse Actuator, and Advanced Brain Monitoring's B-Alert X series. With these additional physiological signal measurements, the users can use not only brain waves but also facial expressions and eye movements. The second is utilizing the multiple features among the available EEG features, such as sensorimotor rhythms (SMRs), P300, and steadystate visual evoked potentials (SSVEP) simultaneously. Most commercial wireless BCI sys‐ tems use only the cognitive states as the controllable feature of wireless BCI systems. If wireless BCI systems employ additional features of EEG signals which are not generated by

tions such as computer games and home appliances.

232 Brain-Computer Interface Systems – Recent Progress and Future Prospects

**4.2. Future research directions for wireless BCI systems**

BCI is a useful technology for people with disabilities as it can offer them an additional means of communication, and reinstate a damaged motor control function. Recently, BCI has started its way to grab the attention of the general public as well because this technology has shown the possibility of a new type of user experience. For example, drowsiness detec‐ tion can be applied to car drivers for preventing traffic accidents. And, real time monitoring of bio-potential signals is useful for diagnosis of patients who have brain diseases such as epilepsy and Alzheimer's disease. To use BCI systems in real-life applications on a daily ba‐ sis, portable, wearable wireless BCI systems are critical, instead of bulky and cumbersome wired BCI systems. Recently, several wireless BCI systems have been introduced by leading research groups and commercial companies.

In this book chapter, we have reviewed the recent research trends in the development of wireless BCI systems. Various research groups have focused on biosensors, user friendly system designs, and more influential applications. Also, there are a few companies which have developed and released wireless BCI systems into that market, with some commercial successes. Nevertheless, research challenges, such as the insufficiency in controllable fea‐ tures, the deficiency in BCI's control accuracy, and the lack of killer applications are still the issues remained to be resolved. The first two challenges are technical issues which will be resolved in time with continuous research efforts. When good wireless BCI systems which provide high-fidelity data acquisition and fast onboard signal processing are available at low cost, they will surely promote the creation of very useful real-life applications.

#### **Acknowledgements**

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MEST) (Do-Yak Research Program, No. 2012-0005656)

#### **Author details**

Seungchan Lee, Younghak Shin, Soogil Woo, Kiseon Kim and Heung-No Lee\*

\*Address all correspondence to: heungno@gist.ac.kr, seungchan@gist.ac.kr

Gwangju Institute of Science and Technology(GIST), Cheomdan-gwagiro, Buk-gu, Gwangju, Republic of Korea

#### **References**


"ENOBIO dry electrophysiology electrode; first human trial plus wireless electrode system," 29th IEEE EMBS, pp. 6689 – 6693, 2007.

**Acknowledgements**

234 Brain-Computer Interface Systems – Recent Progress and Future Prospects

**Author details**

Republic of Korea

Iss. 1, 2004.

157, 2007.

No. 2, 2012.

11, Iss. 6, pp. 5819-5834, 2011.

nal of Neural Engineering, Vol. 8, Iss. 2, 2011.

**References**

This work was supported by the National Research Foundation of Korea (NRF) grant funded

Gwangju Institute of Science and Technology(GIST), Cheomdan-gwagiro, Buk-gu, Gwangju,

[1] Emil S Valchinov and Nicolas E Pallikarakis, "An active electrode for biopotential re‐ cording from small localized bio-sources," BioMedical Engineering Online, Vol. 3,

[2] Lun-De Liao, I-Jan Wang, Sheng-Fu Chen, Jyh-Yeong Chang and Chin-Teng Lin, "Design, Fabrication and Experimental Validation of a Novel Dry-Contact Sensor for Measuring Electroencephalography Signals without Skin Preparation," Sensors, Vol.

[3] Cristian Grozea1, Catalin D. Voinescu, and Siamac Fazli, "Bristle-sensors - Low-cost Flexible Passive Dry EEG Electrodes for Neurofeedback and BCI Applications," Jour‐

[4] Thomas J. Sullivan, Stephen R. Deiss, and Gert Cauwenberghs, "A Low-Noise, Non-Contact EEG/ECG Sensor," Biomedical Circuits and Systems Conference, pp. 154 –

[5] Yu Mike Chi, Yu-Te Wang, Yijun Wang, Christoph Maier, Tzyy-Ping Jung, and Gert Cauwenberghs, "Dry and Noncontact EEG Sensors for Mobile Brain–Computer In‐ terfaces," IEEE Trans. on Neural Systems and Rehabilitation Engineering, Vol. 20,

[6] Nuno Sérgio Diasa, João Paulo Carmo, Paulo Mateus Mendes, José Higino Correiac, "Wireless instrumentation system based on dry electrodes for acquiring EEG sig‐

[7] Giulio Ruffini, Stephen Dunne, Esteve Farrés, Ívan Cester, Paul C. P. Watts, S. Ravi P. Silva, Carles Grau, Lluís Fuentemilla, Josep Marco-Pallarés and Bjorn Vandecasteele,

nals," Medical Engineering & Physics, Vol. 34, pp. 972-981,

by the Korean government (MEST) (Do-Yak Research Program, No. 2012-0005656)

Seungchan Lee, Younghak Shin, Soogil Woo, Kiseon Kim and Heung-No Lee\*

\*Address all correspondence to: heungno@gist.ac.kr, seungchan@gist.ac.kr


ment," Journal of Medical and Biological Engineering, Vol. 30, Iss. 4, pp. 237-245, 2010.


[19] Chin-Teng Lin, Yu-Chieh Chen, Teng-Yi Huang, Tien-Ting Chiu, Li-Wei Ko, Sheng-Fu Liang, Hung-Yi Hsieh, Shang-Hwa Hsu, and Jeng-Ren Duann, "Development of Wireless Brain Computer Interface with Embedded Multitask Scheduling and its Ap‐ plication on Real-Time Driver's Drowsiness Detection and Warning," IEEE Trans. on

[20] Chin-Teng Lin, Che-Jui Chang, Bor-Shyh Lin, Shao-Hang Hung, Chih-Feng Chao, and I-Jan Wang, "A Real-Time Wireless Brain–Computer Interface System for Drowsiness Detection," IEEE Trans. on Biomedical Circuit and system, pp. 214 – 222,

[21] Chin-Teng Lin, Li-Wei Ko, Jin-Chern Chiou, Jeng-Ren Duann, Ruey-Song Huang, Sheng-Fu Liang, Tzai-Wen Chiu, and Tzyy-Ping Jung, "Noninvasive Neural Prosthe‐ ses Using Mobile and Wireless EEG," Proceedings of the IEEE, Vol. 96, No. 7, July

[22] Chin-Teng Lin, Li-Wei Ko, Che-Jui Chang, Yu-Te Wang, Chia-Hsin Chung, Fu-Shu Yang, Jeng-Ren Duann, Tzyy-Ping Jung, and Jin-Chern Chiou, "Wearable and Wire‐ less Brain-Computer Interface and Its Applications," HCII 2009, pp. 741–748, 2009.

[23] Thorsten Oliver Zander, Moritz Lehne, Klas Ihme, Sabine Jatzev, Joao Correia, Chris‐ tian Kothe, Bernd Picht and Femke Nijboer, "A dry EEG-system for scientific re‐ search and brain-computer interfaces," Frontiers in Neuroscience, Vol. 5, 2011. [24] Robert Lin, Ren-Guey Lee, Chwan-Lu Tseng, Yan-Fa Wu, Joe-Air Jiang, "Design and Implementation of Wireless Multi-Channel EEG Recording System and Study Of EEG Clustering Method," Biomedical Engineering applications Basis & Communica‐

[25] Luca Piccini, Sergio Parini, Luca Maggi and Giuseppe Andreoni, "A Wearable Home BCI system: preliminary results with SSVEP protocol," 27th IEEE EMBS, pp. 5384 –

[26] Alexandre Ribeiro, António Sirgado, João Aperta, Ana Lopes, Jorge Guilherme, Pe‐ dro Correia, Gabriel Pires and Urbano Nunes, "A Low-Cost EEG Stand-Alone Device

[27] Lun-De Liao, Chi-Yu Chen, I-Jan Wang, Sheng-Fu Chen, Shih-Yu Li, Bo-Wei Chen, Jyh-Yeong Chang and Chin-Teng Lin, "Gaming control using a wearable and wire‐ less EEG-based brain-computer interface device with novel dry foam-based sensors,"

[28] Yu-Te Wang, Yijun Wang and Tzyy-Ping Jung, "A cell-phone-based brain–computer interface for communication in daily life," Journal of Neural Engineering, Vol.8, 2011.

[29] Chin-Teng Lin, Fu-Chang Lin, Shi-An Chen, Shao-Wei Lu, Te-Chi Chen, Li-Wei Ko, "EEG-based Brain-computer Interface for Smart Living Environmental Auto-adjust‐

For Brain Computer Interface," BIODEVICES, pp. 430-433, 2009.

Journal of Neuro Engineering and Rehabilitation, Vol.9 No.5, 2012.

Biomedical Engineering, Vol. 55, No. 5, 2008.

236 Brain-Computer Interface Systems – Recent Progress and Future Prospects

2010.

2008.

tions, Vol. 18 No. 6, 2006.

5387, 2005.


### **Brain Computer Interface for Epilepsy Treatment**

L. Huang and G. van Luijtelaar

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/55800

### **1. Introduction**

[44] Ling Guo, Daniel Rivero, Alejandro Pazos, "Epileptic seizure detection using multi‐ wavelet transform based approximate entropy and artificial neural networks," Jour‐

[45] F. Galán, M. Nuttinc, E. Lewa, P.W. Ferreza, G. Vanackerc, J. Philipsc, J. del R. Mill‐ án, "A brain-actuated wheelchair: Asynchronous and non-invasive Brain–computer interfaces for continuous control of robots," IEEE Trans. on Rehabilitation Engineer‐

[55] http://www.gtec.at/Products/Hardware-and-Accessories/g.MOBIlab-Specs-Features [56] C Brunner, B Z Allison, C Altstätter and C Neuper, "A comparison of three brain– computer interfaces based on event-related desynchronization, steady state visual evoked potentials, or a hybrid approach using both signals," Journal of Neural Engi‐

[57] C. Vidaurre, A. Schlögl, R. Cabeza, R. Scherer, and G. Pfurtscheller, "A Fully On-Line Adaptive BCI," IEEE Transactions on Biomedical Engineering, Vol. 53, No. 6, 2006.

[58] C. Vidaurre, A. Schlögl, R. Cabeza, R. Scherer, and G. Pfurtscheller, "Study of On-Line Adaptive Discriminant Analysis for EEG-Based Brain Computer Interfaces,"

IEEE Transactions on Biomedical Engineering, Vol. 54, No. 3, 2007.

nal of Neuroscience Methods, Vol. 193, pp.156-163, 2010.

[47] http://developer.neurosky.com/docs/doku.php?id=esenses\_tm

[49] http://www.plxdevices.com/product\_info.php?id=XWAVESONIC

[52] http://advancedbrainmonitoring.com/neurotechnology/systems/

[48] http://www.neurosky.com/Products/ThinkGearAM.aspx

[51] http://www.ocztechnology.com/nia-game-controller.html

ing, Vol. 8, Iss. 4, pp. 441-446, 2000.

238 Brain-Computer Interface Systems – Recent Progress and Future Prospects

[50] http://myndplay.com/products.php?cat=1

[53] http://www.quasarusa.com/

neering, Vol. 8, 2011.

[54] http://neuroelectrics.com/enobio

[46] http://www.neurosky.com/Products/MindSet.aspx

A brain computer interface (BCI) is a communication system converting neural activities into signals that can control computer cursors or external devices (Fetz, 2007). BCI was initially and mainly employed for patients with severe motor disorders such as amyotrophic lateral sclerosis (ALS) by providing non-muscular bidirectional communication and control. How‐ ever, the application of BCI has been extended to control various EEG signals for therapeutic purposes, such as seizure control in epilepsy patients. Although such BCIs did not demonstrate rapid control as in non-muscular communication, it still assumes that EEG based bidirectional control is possible (Wolpaw et al., 2002). More specifically, a BCI in epilepsy research, as in the current chapter, refers to a communication system capable to acquire signal and to implement real-time seizure detection/prediction and contingent delivery of warning stimuli or therapies such as electrical stimulation to control seizures (see the diagram). Such systems became feasible with technological development, and have been implemented in animal and human to control seizures. In the current chapter we will first give an overview of application of BCI, especially with deep brain stimulation in epilepsy research. Then we will discuss different components of a BCI system: input (signal acquisition), algorithm (seizure detection/predic‐ tion) and output (application and users), in particular stressing some important issues on BCI performance.

### **2. BCI application in epilepsy research**

#### **2.1. Introduction of epilepsy treatment**

Epilepsy is a common chronic neurological disorder that afflicts 0.5-1% of the world's popu‐ lation (Hauser et al., 1993). More than one third patients do not respond to the antiepileptic

© 2013 Huang and van Luijtelaar; licensee InTech. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2013 Huang and van Luijtelaar; licensee InTech. This is a paper distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

**Diagram 1.** A brain-computer interface (BCI) in epilepsy research. A BCI system here has three components: input to acquire EEG signal (obtained by implanted electrodes or scalp electrodes, amplified, band-pass and notch filtered, and at the end was digitized), an algorithm to distinguish a seizure event from non-seizure state (seizure detection or pre‐ diction algorithm), and output of application device to deliver warning or therapies such as electrical stimulation to subjects in order to disrupt or modulate seizures.

drugs (AEDs) (Kwan and Brodie, 2000). Resective surgery, the main treatment for drug resistant patients, is not proper for certain patients such as those who have multiple foci or have generalized seizures without a clear local origin. In addition, it might be accompanied with some post-operation complications such as memory, language, sensory or motor deficits (Engel et al., 2003). Alternative treatments such as neurostimulation have been developed for the treatment of these patients. Deep brain stimulation (DBS), one type of neurostimulation, delivers electrical current to neural tissue to achieve therapeutic effects. Compared to surgery, DBS is less invasive, reversible and has the potential to customize treatment to individual patients. Following its success in the treatment of movement disorders (Nguyen et al., 2000; Pollak et al., 2002; Volkmann, 2004), DBS has received increasing attention as a viable treatment option for patients with refractory epilepsy.

While stimulation is conventionally delivered at the predefined protocol (scheduled stimula‐ tion), stimulation integrated in a BCI system is delivered depending on the neurophysiologic state of the brain (responsive stimulation). Such responsive stimulation as output application of BCI has the potential of advantages such as targeting seizure dynamics with higher temporal specificity, minimization of its side effects, reduction of neural tissue damage and of course a longer battery life. The following section will review research of responsive stimulation by BCI in animals and human respectively.

#### **2.2. Responsive stimulation by BCI in animals**

drugs (AEDs) (Kwan and Brodie, 2000). Resective surgery, the main treatment for drug resistant patients, is not proper for certain patients such as those who have multiple foci or have generalized seizures without a clear local origin. In addition, it might be accompanied with some post-operation complications such as memory, language, sensory or motor deficits (Engel et al., 2003). Alternative treatments such as neurostimulation have been developed for the treatment of these patients. Deep brain stimulation (DBS), one type of neurostimulation, delivers electrical current to neural tissue to achieve therapeutic effects. Compared to surgery, DBS is less invasive, reversible and has the potential to customize treatment to individual patients. Following its success in the treatment of movement disorders (Nguyen et al., 2000; Pollak et al., 2002; Volkmann, 2004), DBS has received increasing attention as a viable treatment

**Diagram 1.** A brain-computer interface (BCI) in epilepsy research. A BCI system here has three components: input to acquire EEG signal (obtained by implanted electrodes or scalp electrodes, amplified, band-pass and notch filtered, and at the end was digitized), an algorithm to distinguish a seizure event from non-seizure state (seizure detection or pre‐ diction algorithm), and output of application device to deliver warning or therapies such as electrical stimulation to

While stimulation is conventionally delivered at the predefined protocol (scheduled stimula‐ tion), stimulation integrated in a BCI system is delivered depending on the neurophysiologic state of the brain (responsive stimulation). Such responsive stimulation as output application of BCI has the potential of advantages such as targeting seizure dynamics with higher temporal specificity, minimization of its side effects, reduction of neural tissue damage and of course a longer battery life. The following section will review research of responsive stimulation by BCI

option for patients with refractory epilepsy.

subjects in order to disrupt or modulate seizures.

240 Brain-Computer Interface Systems – Recent Progress and Future Prospects

in animals and human respectively.

Within the framework of a BCI concept, early neurophysiologic studies mainly adopted detection algorithm and feedback control to examine the effect of responsive stimulation on interictal spikes or seizure-like activities in slices. Psatta and colleagues (1983) delivered low frequency stimulation (5 Hz) to the caudate nucleus by feedback control in the epileptic foci in adult cats. Durand and his group (Kayyali and Durand, 1991; Nakagawa and Durand, 1991; Warren and Durand, 1998) applied current with feedback control in the focal area in hippocampal slices of rats. Meanwhile, Gluckman and colleagues (2001) applied a direct current (DC) electric field to hippocampal slices by using a computer controlled feedback algorithm. The outcomes of all these studies suggested antiepileptic effects of responsive stimulation by BCI.

Further, BCI with stimulation output was tested in vivo. Stimulation was applied to the motor cortex, the focus in the penicillin induced seizure model in rats by using proportional feedback control (Colpan et al., 2007). The results showed a significant reduction of mean amplitudes of seizures, suggesting positive effects of stimulation. Studies in a genetic absence epilepsy model (GAERS rats) were aimed at the search for optimal stimulation parameters of stimula‐ tion of the substantia nigra reticularis. Bilateral, bipolar and monophasic stimulation at 60 Hz and with 60-μs pulse width was optimal in the interruption of typical spike-wave discharges for absence epilepsy. However, when used for repeated stimulations, long-term suppression did not occur and even the number of spike-wave discharges increased (Feddersen et al., 2007). Stimulation by BCI was also tested with an automated detection program in the same model (Nelson et al., 2011a). Three types of stimulation frequency were investigated (130, 500, 1000 Hz) and the two high frequencies were more effective to reduce the duration of spikewave discharges.

BCI with a prediction algorithm was also applied in experimental epilepsy animal models. Stimulation was delivered in the hippocampus during the preictal period in the status epilepticus (SE) model in rats (Nair et al., 2006). The preliminary results showed a reduction in seizure frequency and longer seizure free periods, indicating antiepileptic effects of responsive hippocampal stimulation. Recently, BCI with a seizure prediction algorithm was also implemented in the SE model of TLE in six rats by delivering stimulation in the centro‐ medial thalamus (Good et al., 2009). Apart from responsive stimulation, the study also tested scheduled stimulation. BCI with a seizure prediction algorithm was implemented in a penicillin induced seizure model in rats with low frequency stimulation (1 Hz) in the cortex (Wang et al., 2012), comparing with the scheduled stimulation and non-stimulation group. The outcomes of the last two studies favored responsive stimulation over scheduled stimulation, supporting BCI with stimulation output in seizure control.

Apart from conventional deep brain stimulation, application of BCI was recently extended to low frequency transcranial electrical stimulation (TES) in another rat model of generalized absence epilepsy (Berenyi et al., 2012). The results proved positive outcomes of responsive TES by BCI in terms of reducing the duration of spike-wave discharges without a subsequent rebound.

#### **2.3. Responsive stimulation by BCI in human**

In clinic, some early patient studies have used afterdischarges (ADs) - elicited during routinely functional mapping - as model of seizures to investigate the effects of responsive stimulation as ADs are epileptiformic activities and can evolve into seizures. The promising results from some early studies (Lesser et al., 1999; Motamedi et al., 2002) justify the application of BCI in patients.

Osorio and his group (1998) developed a generic seizure detection algorithm by using wavelet analysis for early detection of seizure events. Based on that, Peters and colleagues (2001) described a BCI system- the first bedside prototype - integrated with such a real-time seizure detection algorithm and contingent delivery of stimulation at or near seizure onset. The performance and safety of this BCI system was further evaluated in eight patients (Osorio et al., 2005). The study demonstrated that BCI with stimulation output near seizure onset was practicable in real time and could be applied in a reliable and safe manner.

Following the success of BCI with stimulation application in seizure control, there is recogni‐ tion of the need of implantable circuitry for clinical use. Recently, the first implantable BCI system – responsive neurostimulator (RNS) system (Fig 1) has been developed (NeuroPace, Mountain View, CA) in order to automatically detect early seizure events and deliver contin‐ gent stimulation. As proof of principle, Kossof and his group (2004) first evaluated the safety of BCI with output of an external stimulator in four epilepsy patients. While two patients had brief transient side effects, stimulation in general was well tolerated and safe. Although the efficacy of stimulation was not the aim of that study, electrographic seizures were altered and suppressed in these patients. Consistent with this study, Fountas and colleagues (2005) using an implantable BCI device, delivered responsive stimulation in eight patients. Seven out of eight patients displayed more than a 45% decrease in seizure frequency. Using the same BCI system, Smith and colleagues (2010) delivered responsive insular stimulation in one patient with refractory focal epilepsy after the resection of focal insular area. The patient showed a 50% reduction in seizure frequency after responsive stimulation. In another case report (Enatsu et al., 2012), a TLE patient received responsive stimulation (200 Hz) with RNS system in bilateral mesial temporal areas. An up to 50% decrease in seizure frequency was reported after delivery of stimulation.

Morrell and colleagues (2011) evaluated the efficacy and safety of the RNS device in a multicenter, double-blind, randomized controlled trial in 191 patients. These patients had improved quality of life and tolerated the treatment without obvious mood or cognitive adverse effects. In addition, seizures were significantly reduced for a 12-week blind period in the treatment group. Thus, the implantable BCI can be a promising therapeutic avenue for epilepsy treatment.

#### **3. The components of a BCI**

Like any other BCI systems, the BCI system discussed here consists of an input to obtain signal, algorithm to detect or predict seizures and output to apply warning or therapies to users.

**Figure 1.** schematic graph of an implanted RNS stimulator, depth lead and cortical strip lead (NeuroPace, Mountain View, CA). The RNS neurostimulator (inset) has up to two leads. Either depth lead or cortical (subdural) strip lead is used in the system. Each has four electrode contacts that can be used for sensing and delivering stimulation. To obtain early seizure detection and delivery of focal electrical stimulation, leads are placed close to the seizure focus (Morrell et al, 2011).

#### **3.1. Signal acquisition**

**2.3. Responsive stimulation by BCI in human**

242 Brain-Computer Interface Systems – Recent Progress and Future Prospects

patients.

delivery of stimulation.

for epilepsy treatment.

**3. The components of a BCI**

In clinic, some early patient studies have used afterdischarges (ADs) - elicited during routinely functional mapping - as model of seizures to investigate the effects of responsive stimulation as ADs are epileptiformic activities and can evolve into seizures. The promising results from some early studies (Lesser et al., 1999; Motamedi et al., 2002) justify the application of BCI in

Osorio and his group (1998) developed a generic seizure detection algorithm by using wavelet analysis for early detection of seizure events. Based on that, Peters and colleagues (2001) described a BCI system- the first bedside prototype - integrated with such a real-time seizure detection algorithm and contingent delivery of stimulation at or near seizure onset. The performance and safety of this BCI system was further evaluated in eight patients (Osorio et al., 2005). The study demonstrated that BCI with stimulation output near seizure onset was

Following the success of BCI with stimulation application in seizure control, there is recogni‐ tion of the need of implantable circuitry for clinical use. Recently, the first implantable BCI system – responsive neurostimulator (RNS) system (Fig 1) has been developed (NeuroPace, Mountain View, CA) in order to automatically detect early seizure events and deliver contin‐ gent stimulation. As proof of principle, Kossof and his group (2004) first evaluated the safety of BCI with output of an external stimulator in four epilepsy patients. While two patients had brief transient side effects, stimulation in general was well tolerated and safe. Although the efficacy of stimulation was not the aim of that study, electrographic seizures were altered and suppressed in these patients. Consistent with this study, Fountas and colleagues (2005) using an implantable BCI device, delivered responsive stimulation in eight patients. Seven out of eight patients displayed more than a 45% decrease in seizure frequency. Using the same BCI system, Smith and colleagues (2010) delivered responsive insular stimulation in one patient with refractory focal epilepsy after the resection of focal insular area. The patient showed a 50% reduction in seizure frequency after responsive stimulation. In another case report (Enatsu et al., 2012), a TLE patient received responsive stimulation (200 Hz) with RNS system in bilateral mesial temporal areas. An up to 50% decrease in seizure frequency was reported after

Morrell and colleagues (2011) evaluated the efficacy and safety of the RNS device in a multicenter, double-blind, randomized controlled trial in 191 patients. These patients had improved quality of life and tolerated the treatment without obvious mood or cognitive adverse effects. In addition, seizures were significantly reduced for a 12-week blind period in the treatment group. Thus, the implantable BCI can be a promising therapeutic avenue

Like any other BCI systems, the BCI system discussed here consists of an input to obtain signal, algorithm to detect or predict seizures and output to apply warning or therapies to users.

practicable in real time and could be applied in a reliable and safe manner.

The purpose of input in the BCI system discussed here is to acquire signal relevant for seizure detection and subsequent delivery of warning and therapy. The input is normally EEG recorded from the scalp or within the brain. The chosen input is acquired by electrode, then amplified and digitized in the BCI system. Recently, extracerebral signals such as cardiac (heart rate) or motor signals (speed, direction and joint movement) have emerged as a promising direction for seizure detection (Osorio and Schachter, 2011). Signals can be categorized as noninvasive (scalp EEG, extracerebral) or invasive (intracranial EEG). Scalp electrodes are the common sources of signals in clinic. In most cases, intracranial electrode allow better for early detection than scalp electrode, especially for partial seizures (Jouny et al., 2011).

One important factor that could affect BCI in signal acquisition operation is localization of ictal onset zone. Ictal onset zone is the area that generates the epileptic seizures. The ictal zone can be regional or broad, differing from one subject to another. A seizure event is more likely to be detected early after an ictal onset zone has been identified. One way to improve the localization of the ictal zone is to develop high-density scalp EEG recordings. These high density recordings provide higher spatial resolution with possibilities for dipole source localization compared to the more classical macroelectrode EEG recording systems. Intracra‐ nial recordings with microelectrode arrays on suspected candidate brain regions allow for a next improvement in localization. Information obtained with intracranial EEG recordings are considered the golden standard as far as spatial resolution of the source of epileptiformic activity is concerned. These intracranial recordings are also used for better understanding the interaction of local networks in the course of a seizure event (Jouny et al., 2011). With help of these new electrode recording systems, an ictal onset zone is much more likely to be identified correctly.

Another fundamental issue in BCI in this stage involves which signal to look at besides seizures, particularly for the purpose of the efficacy of therapy. For example, some interictal spikes are similar to seizures but last for only a few seconds without developing into a seizure (Jouny et al., 2011). Despite their short duration, they share similar characteristics components with a seizure. Highly localized activity such as microseizures might be correlated with partial seizure (Stead et al., 2010). If that holds true, these microseizures might be potential candidates for stimulation therapy.

#### **3.2. Seizure detection/prediction algorithm**

In the BCI system discussed here, algorithms should be able to detect a seizure event from non-seizure state (detection algorithm) or predict an upcoming seizure event (prediction algorithm). Quite some detection algorithms using various linear and nonlinear methods have been reported and obtained with various degrees of success. The validity and reliability of prediction algorithms remain questionable, despite some promising results (Mormann et al., 2007) as most of them were based on selective or limited EEG recordings or without statistical validation on sensitivity and specificity of each method (Carney et al., 2011).

Early detection is one issue that is less challenging but currently still far from being optimal for application in BCI research; it determines the time window in which warning or therapeutic devices are triggered. For warning purpose, the ideal condition is to detect seizure event before subjects lose awareness or consciousness. For therapy purpose, it requires to detect seizure as early as possible. As mentioned earlier, the use of high density arrays or microelectrodes are involved in signal acquisition and detection process; they favorably influence the ability to detect seizures early. The capability of an algorithm per se can directly determine how early a seizure event is detected. On the other hand, early detection always comes at the cost of specificity of an algorithm. Some improvements can be made to enhance the quality of early detection, such as using combination of linear and nonlinear features, multi-channel approach and decision making procedure (Jouny et al., 2011).

Another difficulty for BCI in algorithm operation is to include interaction of different states such as circadian fluctuations or vigilance state and seizures. Seizure probability can be affected by sleep stage, sleep debt and diurnal and other physiological rhythms (Haut et al., 2007; Malow, 2005). Circadian phases (Quigg et al., 2000; Smyk et al., 2012; Van Luijtelaar and Coenen, 1988) and sleep stage (Shouse et al., 2004; van Luijtelaar and Bikbaev, 2007) have been reported to have effects on seizures in rat models of epilepsy. Moreover, false positives by prediction algorithms were also found to favor some states of vigilance, in particular deep sleep (Navarro et al., 2005; Schelter et al., 2006). More efforts such as computational modeling need to be taken to probe states of vigilance and circadian phase to improve prediction of seizures.

#### **3.3. Application and users**

considered the golden standard as far as spatial resolution of the source of epileptiformic activity is concerned. These intracranial recordings are also used for better understanding the interaction of local networks in the course of a seizure event (Jouny et al., 2011). With help of these new electrode recording systems, an ictal onset zone is much more likely to be identified

Another fundamental issue in BCI in this stage involves which signal to look at besides seizures, particularly for the purpose of the efficacy of therapy. For example, some interictal spikes are similar to seizures but last for only a few seconds without developing into a seizure (Jouny et al., 2011). Despite their short duration, they share similar characteristics components with a seizure. Highly localized activity such as microseizures might be correlated with partial seizure (Stead et al., 2010). If that holds true, these microseizures might be potential candidates

In the BCI system discussed here, algorithms should be able to detect a seizure event from non-seizure state (detection algorithm) or predict an upcoming seizure event (prediction algorithm). Quite some detection algorithms using various linear and nonlinear methods have been reported and obtained with various degrees of success. The validity and reliability of prediction algorithms remain questionable, despite some promising results (Mormann et al., 2007) as most of them were based on selective or limited EEG recordings or without statistical

Early detection is one issue that is less challenging but currently still far from being optimal for application in BCI research; it determines the time window in which warning or therapeutic devices are triggered. For warning purpose, the ideal condition is to detect seizure event before subjects lose awareness or consciousness. For therapy purpose, it requires to detect seizure as early as possible. As mentioned earlier, the use of high density arrays or microelectrodes are involved in signal acquisition and detection process; they favorably influence the ability to detect seizures early. The capability of an algorithm per se can directly determine how early a seizure event is detected. On the other hand, early detection always comes at the cost of specificity of an algorithm. Some improvements can be made to enhance the quality of early detection, such as using combination of linear and nonlinear features, multi-channel approach

Another difficulty for BCI in algorithm operation is to include interaction of different states such as circadian fluctuations or vigilance state and seizures. Seizure probability can be affected by sleep stage, sleep debt and diurnal and other physiological rhythms (Haut et al., 2007; Malow, 2005). Circadian phases (Quigg et al., 2000; Smyk et al., 2012; Van Luijtelaar and Coenen, 1988) and sleep stage (Shouse et al., 2004; van Luijtelaar and Bikbaev, 2007) have been reported to have effects on seizures in rat models of epilepsy. Moreover, false positives by prediction algorithms were also found to favor some states of vigilance, in particular deep sleep (Navarro et al., 2005; Schelter et al., 2006). More efforts such as computational modeling need to be taken to probe states of vigilance and circadian phase to improve prediction of

validation on sensitivity and specificity of each method (Carney et al., 2011).

correctly.

seizures.

for stimulation therapy.

**3.2. Seizure detection/prediction algorithm**

244 Brain-Computer Interface Systems – Recent Progress and Future Prospects

and decision making procedure (Jouny et al., 2011).

The output of BCI here is the device that can trigger warning or therapies to users. Given its therapeutic purpose, therapy efficacy is the standard tool to evaluate BCI performance. In addition to electrical stimulation, other therapies have also been used such as anti-seizure compounds (Stein et al., 2000), thermal energy (cooling) (Hill et al., 2000), and operant conditioning (Osterhagen et al., 2010; Sterman, 2000). Compared to current, the main limitation of pharmacological and thermal energy therapy is their slow tissue diffusivity, which means that therapies will be delivered or arrive late to fully control the target area, even when delivered locally (Osorio and Frei, 2009). On the other side, electrical stimulation is the most common application of BCI and is also our focus in this section.

#### *3.3.1. Therapy parameters for users*

One challenge for BCI in application is how to stimulate, that is, selection of proper stimulation parameters. Stimulation parameters in human are often determined by trial and error with adjustment to avoid side effects while respecting safety limits for charge density (Sun et al., 2008). A number of studies especially in animals explored which stimulation parameters stimulation frequency, waveform, amplitude, duration- can result in better seizure control.

Stimulation frequency seems to be a key factor. High frequency stimulation (HFS) (> 100 Hz) is typically used in clinic for treatment of movement disorders such as Parkinson's disease (Pollak et al., 2002; Volkmann, 2004) and also used for treatment of patients with refractory epilepsy. HFS has been delivered in various brain areas such as the anterior nucleus of the thalamus, hippocampus and thalamus (Andrade et al., 2006; Fisher et al., 2010; Velasco et al., 2000a; Velasco et al., 2000b) and has achieved various degree of success in seizure control.

In contrast, low frequency stimulation has limited and mixed effects. Stimulation at low frequency (0.5-3 Hz) in kindled animal models were found to raise the threshold of ADs (Carrington et al., 2007; Gaito et al., 1980; Ghorbani et al., 2007; Goodman et al., 2005). Yamamoto and colleagues (2002) demonstrated that LFS in the cortex had antiepileptic effects in the focus in patients with TLE. However, no other studies have further report‐ ed such effects of LFS in animal models and human of TLE. Even some clinical study reported contradictory evidence – aggravation of seizures by LFS on the centromedian thalamic nucleus in 12 patients (Velasco et al., 1997).

Furthermore, a few studies compared HFS and LFS under the same experimental condition. Albensi and colleagues (2004) compared the effects of HFS (100 Hz) and LFS (1 Hz) on epilepti‐ form activities in hippocampal slices. Both types of stimulation suppressed epileptiform activities but with one difference: the onset of suppression by LFS was gradual but persistent while that of suppression by HFS was rapidbuttransient.The effects ofLFS (5 Hz) and HFS (130 Hz) were compared on ADs in kindled rats (Wyckhuys et al., 2010b). HFS was more effective withhigherADs thresholdandlongerlatency.Boex andcolleagues (2007) comparedbothtypes of stimulation on three patients and found that HFS was more effective in reducing seizure rate. Rajdev and colleagues (2011) examined stimulation frequency (high, medium and low), pulse width(highandlow)andamplitude (highandlow)inseizures inkainite treatedrats.The results showedthatlow(5Hz)andhighfrequency(130Hz)wereeffectivetosuppressepilepticactivities compared to medium frequency (60 Hz). Taken together, these studies suggested that HFS is most effective in seizure suppression while LFS has rather complicated effects depending on which animal model and epilepsy type is chosen for LFS application.

Other factors such as pulse width and waveform could also affect the effects of stimulation. In the aforementioned study, Rajdev and colleagues (2011) investigated the influences of different pulse widths (60, 120, 240 us) on ADs. With increase of pulse width (120, 240 us), less stimu‐ lation amplitude was needed to evoke ADs, indicating higher threshold of AD interruptions for shorter pulse width (60 us). The power of the pulse (intensity times duration) determines its efficacy in this model. In addition, the waveform of stimulation can also act on the effects of stimulation. The threshold for suppression of both somatic and axonal activity was lower with sinusoidal HFS than pulse train HFS (Jensen and Durand, 2007).

Recently, a new method of stimulation has been proposed: 'temporal coding'. Cota and colleagues (2009) reported that stimulation with a 'pseudo-randomized inter stimulus interval' significantly increased the threshold to tonic-clonic seizures in the pentylenetetrazole (PTZ) model. Similarly, Poisson distributed stimulation was found to reduce spontaneous seizures in the SE model of TLE (Wyckhuys et al., 2010a) and in the GEARs model of absence epilepsy (Nelson et al., 2011a). Further, cortical stimulation at multiple stimulation sites was investi‐ gated in different combinations of periodic/aperiodic (periodic – with fixed inter stimulus interval) and synchronous/ asynchronous stimulation manner (synchronous – stimulation at the same time) in rats (Nelson et al., 2011b). The results showed that asynchronous stimulation was more effective in suppression of seizure severity and duration.

Although it remains unclear which parameters are the best for which seizure model, these studies give us more insights on how to improve efficacy of therapy.

#### *3.3.2. Selection of target area*

Another basic question regarding BCI in application is where to stimulate. So far a variety of areas have been investigated for the effects of stimulation. These areas are either areas where seizures are generated or areas which are involved in progression of seizures. The intrinsic and extrinsic characteristics of these areas determine their different responses to stimulation. In each target area, for example, the constitutions of various neuron types and their biophysical properties are different, affecting responses of individual neuron and neuronal ensemble to stimulation. Besides, the anatomical connections among these target areas vary from each other, which can influence the dynamics of neuronal populations and sensitivity to stimulation (Sunderam et al., 2010).

#### *3.3.3. Patient-specific therapy*

It is also true that patient-specific therapy is the new direction for BCI in application operation. In clinic, stimulation parameters are usually adjusted in individual patients to avoid adverse effects while maintaining therapeutic effects. Osorio and his group (Osorio et al., 2010) used linear regression to examine in a retrospective study the results of a trial in which responsive stimulation was applied in eight patients with refractory epilepsy. Regression models tested the contributions of multiple potentially relevant factors such as parameter configurations (that is, different combinations of stimulation frequency, current intensity, duration, pulse width and location) to changes of seizure severity. The results showed that certain parameter configurations were more effective in reduction of seizure severity in some patients but not in others, indicating individual difference in parameters setting for stimulation. Thus adaptive, patients-specific stimulation settings might be a necessary new step for BCI in application in order to achieve optimal efficacy.

#### **4. Conclusion**

showedthatlow(5Hz)andhighfrequency(130Hz)wereeffectivetosuppressepilepticactivities compared to medium frequency (60 Hz). Taken together, these studies suggested that HFS is most effective in seizure suppression while LFS has rather complicated effects depending on

Other factors such as pulse width and waveform could also affect the effects of stimulation. In the aforementioned study, Rajdev and colleagues (2011) investigated the influences of different pulse widths (60, 120, 240 us) on ADs. With increase of pulse width (120, 240 us), less stimu‐ lation amplitude was needed to evoke ADs, indicating higher threshold of AD interruptions for shorter pulse width (60 us). The power of the pulse (intensity times duration) determines its efficacy in this model. In addition, the waveform of stimulation can also act on the effects of stimulation. The threshold for suppression of both somatic and axonal activity was lower

Recently, a new method of stimulation has been proposed: 'temporal coding'. Cota and colleagues (2009) reported that stimulation with a 'pseudo-randomized inter stimulus interval' significantly increased the threshold to tonic-clonic seizures in the pentylenetetrazole (PTZ) model. Similarly, Poisson distributed stimulation was found to reduce spontaneous seizures in the SE model of TLE (Wyckhuys et al., 2010a) and in the GEARs model of absence epilepsy (Nelson et al., 2011a). Further, cortical stimulation at multiple stimulation sites was investi‐ gated in different combinations of periodic/aperiodic (periodic – with fixed inter stimulus interval) and synchronous/ asynchronous stimulation manner (synchronous – stimulation at the same time) in rats (Nelson et al., 2011b). The results showed that asynchronous stimulation

Although it remains unclear which parameters are the best for which seizure model, these

Another basic question regarding BCI in application is where to stimulate. So far a variety of areas have been investigated for the effects of stimulation. These areas are either areas where seizures are generated or areas which are involved in progression of seizures. The intrinsic and extrinsic characteristics of these areas determine their different responses to stimulation. In each target area, for example, the constitutions of various neuron types and their biophysical properties are different, affecting responses of individual neuron and neuronal ensemble to stimulation. Besides, the anatomical connections among these target areas vary from each other, which can influence the dynamics of neuronal populations and sensitivity to stimulation

It is also true that patient-specific therapy is the new direction for BCI in application operation. In clinic, stimulation parameters are usually adjusted in individual patients to avoid adverse effects while maintaining therapeutic effects. Osorio and his group (Osorio et al., 2010) used linear regression to examine in a retrospective study the results of a trial in which responsive

which animal model and epilepsy type is chosen for LFS application.

246 Brain-Computer Interface Systems – Recent Progress and Future Prospects

with sinusoidal HFS than pulse train HFS (Jensen and Durand, 2007).

was more effective in suppression of seizure severity and duration.

studies give us more insights on how to improve efficacy of therapy.

*3.3.2. Selection of target area*

(Sunderam et al., 2010).

*3.3.3. Patient-specific therapy*

Unlike conventional BCIs, a BCI system in epilepsy emphasizes EEG-based communication for therapeutic purposes. The need of BCI rises from delivery of contingent therapies such as stimulation to control seizures. Although a large body of work has been done to explore contingent stimulation with help of BCI for epilepsy research in animal and human, it is still in an experimental stage. With development of technique in engineering and computer science, BCI is at the stage of becoming feasible for application of therapy in the treatment of patients in some forms of refractory epilepsy.

Future progress depends on the following crucial factors: development of electrode arrays or microelectrode, improvement of early detection, algorithms embedding variables such as vigilant states and circadian phases, exploration of patient-specific optimization of parameters, selection of proper target areas to maximize BCI performance. The development of BCI is an interdisciplinary challenge requiring vigorous and continuous efforts on relevant fields such as neuroscience, computer science, mathematics, and engineering. Although BCI in epilepsy research is still developing, it provides a promising therapeutic option to those who do not respond to conventional treatment.

#### **Author details**

L. Huang and G. van Luijtelaar

Dept Biological Psychology, Donders Center for Cognition, Donders Institute for Brain Cognition and Behaviour, Radboud University Nijmegen, Nijmegen, the Netherlands

#### **References**

[1] Albensi, B. C, Ata, G, Schmidt, E, Waterman, J. D, & Janigro, D. (2004). Activation of long-term synaptic plasticity causes suppression of epileptiform activity in rat hippocampal slices. Brain Res. , 998, 56-64.


[14] Fountas, K. N, Smith, J. R, Murro, A. M, Politsky, J, Park, Y. D, & Jenkins, P. D. (2005). Implantation of a closed-loop stimulation in the management of medically refractory focal epilepsy: a technical note. Stereotact Funct Neurosurg. , 83, 153-8.

[2] Andrade, D. M, Zumsteg, D, Hamani, C, Hodaie, M, Sarkissian, S, Lozano, A. M, & Wennberg, R. A. (2006). Long-term follow-up of patients with thalamic deep brain

[3] Berenyi, A, Belluscio, M, Mao, D, & Buzsaki, G. (2012). Closed-loop control of epilepsy

[4] Boex, C, Vulliemoz, S, Spinelli, L, Pollo, C, & Seeck, M. (2007). High and low frequency electrical stimulation in non-lesional temporal lobe epilepsy. Seizure. , 16, 664-9.

[5] Carney, P. R, Myers, S, & Geyer, J. D. (2011). Seizure prediction: methods. Epilepsy

[6] Carrington, C. A, Gilby, K. L, & Mcintyre, D. C. (2007). Effect of focal low-frequency stimulation on amygdala-kindled afterdischarge thresholds and seizure profiles in

[7] Colpan, M. E, Li, Y, Dwyer, J, & Mogul, D. J. (2007). Proportional feedback stimulation

[8] Cota, V. R. Medeiros Dde, C., Vilela, M.R., Doretto, M.C., Moraes, M.F., (2009). Distinct

[9] Enatsu, R, Alexopoulos, A, Bingaman, W, & Nair, D. (2012). Complementary effect of surgical resection and responsive brain stimulation in the treatment of bitemporal lobe

[10] Engel, J. Jr., Wiebe, S., French, J., Sperling, M., Williamson, P., Spencer, D., Gumnit, R., Zahn, C., Westbrook, E., Enos, B., (2003). Practice parameter: temporal lobe and

[11] Feddersen, B, Vercueil, L, Noachtar, S, David, O, Depaulis, A, & Deransart, C. (2007). Controlling seizures is not controlling epilepsy: a parametric study of deep brain

[13] Fisher, R, Salanova, V, Witt, T, Worth, R, Henry, T, Gross, R, Oommen, K, Osorio, I, Nazzaro, J, Labar, D, Kaplitt, M, Sperling, M, Sandok, E, Neal, J, Handforth, A, Stern, J, Desalles, A, Chung, S, Shetter, A, Bergen, D, Bakay, R, Henderson, J, French, J, Baltuch, G, Rosenfeld, W, Youkilis, A, Marks, W, Garcia, P, Barbaro, N, Fountain, N, Bazil, C, Goodman, R, & Mckhann, G. Babu Krishnamurthy, K., Papavassiliou, S., Epstein, C., Pollard, J., Tonder, L., Grebin, J., Coffey, R., Graves, N., (2010). Electrical stimulation of the anterior nucleus of thalamus for treatment of refractory epilepsy.

patterns of electrical stimulation of the basolateral amygdala influence pentylenetetrazole seizure outcome. Epilepsy Behav. 14 Suppl , 1, 26-31.

localized neocortical resections for epilepsy. Epilepsia. , 44, 741-51.

[12] Fetz, E. E. (2007). Volitional control of neural activity: implications for brain-

stimulation for epilepsy. Neurology. , 66, 1571-3.

248 Brain-Computer Interface Systems – Recent Progress and Future Prospects

Behav. 22 Suppl 1, S, 94-101.

by transcranial electrical stimulation. Science. , 337, 735-7.

fast- and slow-kindling rat strains. Epilepsia. , 48, 1604-13.

for seizure control in rats. Epilepsia. , 48, 1594-603.

epilepsy: a case report. Epilepsy Behav. , 24, 513-6.

stimulation for epilepsy. Neurobiol Dis. , 27, 292-300.

computer interfaces. J Physiol. , 579, 571-9.

Epilepsia. , 51, 899-908.


[44] Osterhagen, L, Breteler, M, & Van Luijtelaar, G. (2010). Does arousal interfere with operant conditioning of spike-wave discharges in genetic epileptic rats? Epilepsy Res. , 90, 75-82.

[30] Mormann, F, Andrzejak, R. G, Elger, C. E, & Lehnertz, K. (2007). Seizure prediction: the

[31] Morrell, M. J. (2011). Responsive cortical stimulation for the treatment of medically

[32] Motamedi, G. K, Lesser, R. P, Miglioretti, D. L, Mizuno-matsumoto, Y, Gordon, B, Webber, W. R, Jackson, D. C, Sepkuty, J. P, & Crone, N. E. (2002). Optimizing

[33] Nair, S. P, Sackellares, J. C, Shiau, D. S, Norman, W. M, Dance, L. K, Pardalos, P. M,

[34] Nakagawa, M, & Durand, D. (1991). Suppression of spontaneous epileptiform activity

[36] Nelson, T. S, Suhr, C. L, Freestone, D. R, Lai, A, Halliday, A. J, Mclean, K. J, Burkitt, A. N, & Cook, M. J. seizure control with very high frequency electrical stimulation at seizure onset in the GAERS model of absence epilepsy. Int J Neural Syst. , 21, 163-73.

[37] Nelson, T. S, Suhr, C. L, Lai, A, Halliday, A. J, Freestone, D. R, Mclean, K. J, Burkitt, A. N, & Cook, M. J. (2011b). Exploring the tolerability of spatiotemporally complex

Carpentier, A., Brugieres, P., Pollin, B., Rostaing, S., Keravel, Y., (2000). Motor cortex stimulation in the treatment of central and neuropathic pain. Arch Med Res. , 31, 263-5.

quantitative analysis of seizures and short-term prediction of clinical onset. Epilepsia. ,

[38] Nguyen, J. P, & Lefaucher, J. P. Le Guerinel, C., Eizenbaum, J.F., Nakano, N.,

[39] Osorio, I, Frei, M. G, & Wilkinson, S. B. (1998). Real-time automated detection and

[40] Osorio, I, Frei, M. G, Sunderam, S, Giftakis, J, Bhavaraju, N. C, Schaffner, S. F, & Wilkinson, S. B. (2005). Automated seizure abatement in humans using electrical

[41] Osorio, I, & Frei, M. G. (2009). Real-time detection, quantification, warning, and control of epileptic seizures: the foundations for a scientific epileptology. Epilepsy Behav. , 16,

[42] Osorio, I, Manly, B, & Sunderam, S. (2010). Toward a quantitative multivariate analysis

[43] Osorio, I, & Schachter, S. (2011). Extracerebral detection of seizures: a new era in

of the efficacy of antiseizure therapies. Epilepsy Behav. , 18, 335-43.

epileptology? Epilepsy Behav. 22 Suppl 1, S, 82-7.

[35] Navarro, V, & Martinerie, J. Le Van Quyen, M., Baulac, M., Dubeau, F., Gotman, J., (2005). Seizure anticipation: do mathematical measures correlate with video-EEG

parameters for terminating cortical afterdischarges with pulse stimulation. Epilepsia. ,

Principe, J. C, & Carney, P. R. (2006). Effects of acute hippocampal stimulation on EEG

long and winding road. Brain. , 130, 314-33.

250 Brain-Computer Interface Systems – Recent Progress and Future Prospects

43, 836-46.

39, 615-27.

391-6.

intractable partial epilepsy. Neurology. , 77, 1295-1304.

dynamics. Conf Proc IEEE Eng Med Biol Soc. , 1, 4382-6.

electrical stimulation paradigms. Epilepsy Res. , 96, 267-75.

with applied currents. Brain Res. , 567, 241-7.

evaluation? Epilepsia. , 46, 385-96.

stimulation. Ann Neurol. , 57, 258-68.


## **Emotion Recognition Based on Brain-Computer Interface Systems**

Taciana Saad Rached and Angelo Perkusich

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/56227

#### **1. Introduction**

[58] Sunderam, S, Gluckman, B, Reato, D, & Bikson, M. (2010). Toward rational design of electrical stimulation strategies for epilepsy control. Epilepsy Behav. , 17, 6-22.

[59] Van Luijtelaar, E. L, & Coenen, A. M. (1988). Circadian rhythmicity in absence epilepsy

[60] Van Luijtelaar, G, & Bikbaev, A. (2007). Midfrequency cortico-thalamic oscillations and the sleep cycle: genetic, time of day and age effects. Epilepsy Res. , 73, 259-65.

[61] Velasco, A. L, Velasco, M, Velasco, F, Menes, D, Gordon, F, Rocha, L, Briones, M, & Marquez, I. and chronic electrical stimulation of the hippocampus on intractable

[62] Velasco, M, Velasco, F, Velasco, A. L, Brito, F, Jimenez, F, Marquez, I, & Rojas, B. (1997). Electrocortical and behavioral responses produced by acute electrical stimulation of the human centromedian thalamic nucleus. Electroencephalogr Clin Neurophysiol. ,

[63] Velasco, M, Velasco, F, Velasco, A. L, Boleaga, B, Jimenez, F, Brito, F, & Marquez, I. (2000b). Subacute electrical stimulation of the hippocampus blocks intractable temporal lobe seizures and paroxysmal EEG activities. Epilepsia. , 41, 158-69.

[64] Volkmann, J. (2004). Deep brain stimulation for the treatment of Parkinson's disease. J

[65] Wang, L, Guo, H, Yu, X, Wang, S, Xu, C, Fu, F, Jing, X, Zhang, H, & Dong, X. (2012). Responsive electrical stimulation suppresses epileptic seizures in rats. PLoS One. 7,

[66] Warren, R. J, & Durand, D. M. (1998). Effects of applied currents on spontaneous

epileptiform activity induced by low calcium in the rat hippocampus. Brain Res. , 806,

[67] Wolpaw, J. R, Birbaumer, N, Mcfarland, D. J, Pfurtscheller, G, & Vaughan, T. M. (2002). Brain-computer interfaces for communication and control. Clin Neurophysiol. , 113,

[68] Wyckhuys, T, Boon, P, Raedt, R, Van Nieuwenhuyse, B, Vonck, K, & Wadman, W. (2010a). Suppression of hippocampal epileptic seizures in the kainate rat by Poisson

[69] Wyckhuys, T, Raedt, R, Vonck, K, Wadman, W, & Boon, P. (2010b). Comparison of

[70] Yamamoto, J, Ikeda, A, Satow, T, Takeshita, K, Takayama, M, Matsuhashi, M,

hippocampal Deep Brain Stimulation with high (130Hz) and low frequency (5Hz) on

Matsumoto, R, Ohara, S, Mikuni, N, Takahashi, J, Miyamoto, S, Taki, W, Hashimoto, N, Rothwell, J. C, & Shibasaki, H. (2002). Low-frequency electric cortical stimulation has an inhibitory effect on epileptic focus in mesial temporal lobe epilepsy. Epilepsia. ,

distributed stimulation. Epilepsia. , 51, 2297-304.

afterdischarges in kindled rats. Epilepsy Res. , 88, 239-46.

temporal lobe seizures: preliminary report. Arch Med Res. , 31, 316-28.

in rats. Epilepsy Res. , 2, 331-6.

252 Brain-Computer Interface Systems – Recent Progress and Future Prospects

Clin Neurophysiol. , 21, 6-17.

102, 461-71.

e38141.

186-95.

767-91.

43, 491-5.

Emotions are intrinsically related to the way that individuals interact with each other as well as machines [1]. A human being can understand the emotional state of another human being and behave in the best manner to improve the communication in a certain situation. This is because emotions can be recognized through words, voice intonation, facial expressions and body language. In contrast, machines cannot understand the feelings of an individual.

In this context, affective computing aims to improve the communication among individuals and machines by recognizing human emotions and thus making that interaction easier, usable and effective. There are several studies using different approaches in human emotion detection [2]. For instance, McDaniel et. al. in [3] investigated facial expressions to detect emotions in a learning activity by the interaction among university students and a computer. The students were asked to show their emotions while interacting with a software called AutoTutor. Facial expressions were recorded by video cameras to recognize six kinds of emotions, namely: confusion, surprise, boredom, frustration, pleasure and acceptance. The authors observed that all emotions were able to be detected, except the boredom which it was indistinguishable from a neutral facial expression.

Lee et. al. in [4] explored the use of information concerning the dialogues and speeches along with voice intonation to recognize emotions from speech signals. The focus of the study was to detect negative and non-negative emotions using the information obtained from the spoken language of a call center. The main problems in emotion recognition systems based on facial expressions or spoken language is that these two sources of information are susceptible to ambiguity and false simulations. Bernhardt in [5] developed a emotion recognition system based on body language in daily activities such as walking, picking up an object, among others.

© 2013 Rached and Perkusich; licensee InTech. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2013 Rached and Perkusich; licensee InTech. This is a paper distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Beyond the detection systems based on affective facial expressions, spoken language and body language, there are several applications in affective computing that focus in detecting emotions through learning techniques to identify patterns in physiological activity that match the expression of different emotions. Liu et. al. in [6] investigated the use of cardiovascular signals, electrodermal activity, electromyography and peripheral temperature for affective detection. The aim of this study was to recognize the emotions of children affected by autism to develop a system that works as a therapist. Systems based on electroencephalogram (EEG) signals have also been used to detect emotions. For instance, in Schaaff and Schultz [7] implemented a system based on brain signals to enable a robot to recognize human emotions. Emotions were elicited by images and classified in three categories, namely: pleasant, unpleasant and neutral. Brain signals are a reliable information source due to the fact that the process of emotion interpretation starts in the central nervous system. Furthermore, an individual cannot control his brain signals to simulate a fake emotional state.

This chapter presents an emotion recognition system based on brain signals. We used a brain computer interface (BCI) as a technique to acquire and classify the brain signals into emotions. This BCI system is based on EEG signals. We used the discrete wavelet transform to select EEG features and a neural network to map these features to emotions.

The rest of this chapter is divided in six sections. In Section 2 the main concepts related to BCI and the key problem are presented. Section 3 shows some application areas of emotion recognition systems based on brain computer interface. Section 4 discusses the research course. Section 5 illustrates the methods used to acquire and process the brain signals into human emotions. Section 5 presents the results achieved using our approach. Finally, in conclusion Section 6 are presented.

#### **2. Problem statement**

BCIs are systems that enable any user to exchange information with the environment and control devices by using brain activity, i.e., without using the neuromuscular output pathways of the brain [8]. Brain signals can be acquired by means of invasive or non-invasive methods. In the former, electrodes are implanted directly in the brain. In the latter, the signal is acquired from the scalp of the user. Despite the existence of several methods to acquire brain signals, the most used method is the electroencephalogram (EEG) because it is non-invasive, portable, inexpensive, and can be used in almost all environments [9]. Moreover, low cost and increas‐ ingly portable EEG equipment have been developed in the last years.

As discussed by Wang et al. In [10], BCIs systems have been used in rehabilitation, e.g., speller systems, neuroscience, e.g., monitoring attention systems and cognitive psychology, e.g., treatment of attention-deficit hyperactivity disorder. BCIs systems have been investigated recently in recognizing emotions and are seen as a promising technique in this area because the emotions are generated in the brain.

There are several challenges in using BCIs systems for detecting emotions, such as the choice of the method and the channels of acquisition of brain signals that best provide information regarding the emotional state of an individual as well as processing techniques in order to reach a good accuracy in the recognition of emotions.

### **3. Application area**

The emotion recognition systems based on BCI can be applied to many areas, such as:


Beyond the detection systems based on affective facial expressions, spoken language and body language, there are several applications in affective computing that focus in detecting emotions through learning techniques to identify patterns in physiological activity that match the expression of different emotions. Liu et. al. in [6] investigated the use of cardiovascular signals, electrodermal activity, electromyography and peripheral temperature for affective detection. The aim of this study was to recognize the emotions of children affected by autism to develop a system that works as a therapist. Systems based on electroencephalogram (EEG) signals have also been used to detect emotions. For instance, in Schaaff and Schultz [7] implemented a system based on brain signals to enable a robot to recognize human emotions. Emotions were elicited by images and classified in three categories, namely: pleasant, unpleasant and neutral. Brain signals are a reliable information source due to the fact that the process of emotion interpretation starts in the central nervous system. Furthermore, an individual cannot control

This chapter presents an emotion recognition system based on brain signals. We used a brain computer interface (BCI) as a technique to acquire and classify the brain signals into emotions. This BCI system is based on EEG signals. We used the discrete wavelet transform to select EEG

The rest of this chapter is divided in six sections. In Section 2 the main concepts related to BCI and the key problem are presented. Section 3 shows some application areas of emotion recognition systems based on brain computer interface. Section 4 discusses the research course. Section 5 illustrates the methods used to acquire and process the brain signals into human emotions. Section 5 presents the results achieved using our approach. Finally, in conclusion

BCIs are systems that enable any user to exchange information with the environment and control devices by using brain activity, i.e., without using the neuromuscular output pathways of the brain [8]. Brain signals can be acquired by means of invasive or non-invasive methods. In the former, electrodes are implanted directly in the brain. In the latter, the signal is acquired from the scalp of the user. Despite the existence of several methods to acquire brain signals, the most used method is the electroencephalogram (EEG) because it is non-invasive, portable, inexpensive, and can be used in almost all environments [9]. Moreover, low cost and increas‐

As discussed by Wang et al. In [10], BCIs systems have been used in rehabilitation, e.g., speller systems, neuroscience, e.g., monitoring attention systems and cognitive psychology, e.g., treatment of attention-deficit hyperactivity disorder. BCIs systems have been investigated recently in recognizing emotions and are seen as a promising technique in this area because

There are several challenges in using BCIs systems for detecting emotions, such as the choice of the method and the channels of acquisition of brain signals that best provide information

his brain signals to simulate a fake emotional state.

254 Brain-Computer Interface Systems – Recent Progress and Future Prospects

Section 6 are presented.

**2. Problem statement**

the emotions are generated in the brain.

features and a neural network to map these features to emotions.

ingly portable EEG equipment have been developed in the last years.


An example of application in the field of entertainment is EEG-based music player [11]. In this application, the current emotional state of the user is identified, and a music related to this state is played. The songs are classified into six emotion types: fear, sad, frustrated, happy, satisfied and pleasant.

#### **4. Research course**

The way which a person acts with other people, objects and situations in their day-to-day is fully connected with their emotions. In this context, the recognition of human emotions has been target of several studies recently.

The recognition of emotions can be performed from the facial and body expressions, voice and physiological signals, among others. Since the focus of this work is the definition of processing techniques of EEG signals to provide better results in the classification of the brain signals into emotions, this section presents works based on EEG signals for recognition of emotions.

In [12], the authors used the EEG signals and facial expressions together for the recognition of emotions. The authors aim was to investigate which emotion (positive or negative) is generated from the execution of a particular song. EEG signals were acquired by electrodes placed in the temporal region of the brain and facial expressions were acquired from video images. As a result of this work, the authors determined that it was not possible to distinguish the type of emotion from the signals used together.

Liu et al in [11] presented an algorithm for classification of brain electrical signals in human emotions. This algorithm was based on the model of fractal dimension. The authors used some songs in the first experiment and sounds of the international affective digitized sounds in a second experiment to induce certain emotions in the participants of this study, The brain signals were acquired from three channels, FC6, F4 and AF3. Through the channel FC6 was possible to classify emotions regarding the level of excitement. The channels AF3 and F4 were used in the classification of emotions with respect to valence. According to the authors, in the forebrain of an individual can be identified greater activation in one hemisphere during the feeling of positive emotion and greater activation in the other hemisphere during feeling a negative emotion. But what hemisphere corresponds to that kind of emotion depends on each individual. Therefore, for this approach to be used, a training phase was included in the work. Using the fractal dimension model for recognition of emotions, the authors identified six basic emotions in the bi-dimensional valence-arousal graph, as shown in Figure 1.

**Figure 1.** Bi-dimensional valence-arousal approach

Bos in [13] investigated the use of EEG signals for recognition of human emotions. The author used auditory and visual stimuli extracted from such international affective digitized sounds and affective figures, respectively, to induce the feeling of a certain emotion in the participants of the experiment. The brain electrical signals were acquired through three channels, F3, F4 and FPZ, according to the international 10-20 system [14]. The signal characteristics of EEG, alpha (8-12 Hz) and beta (12-30 Hz), were selected to be used in the recognition of emotions. According to the authors of this paper, the noise signals due to the electrooculogram (EOG) are dominant below the frequency of 4 Hz, the noise related to the electrocardiogram (ECG) signals is around 1.2 Hz and above 30 Hz we can find the noise related to electromyogram (EMG). Therefore extracting features only alpha and beta reduced the noises. Bos used a band pass filter to extract only the features in the frequency range 8-30 Hz.

possible to classify emotions regarding the level of excitement. The channels AF3 and F4 were used in the classification of emotions with respect to valence. According to the authors, in the forebrain of an individual can be identified greater activation in one hemisphere during the feeling of positive emotion and greater activation in the other hemisphere during feeling a negative emotion. But what hemisphere corresponds to that kind of emotion depends on each individual. Therefore, for this approach to be used, a training phase was included in the work. Using the fractal dimension model for recognition of emotions, the authors identified six basic

Bos in [13] investigated the use of EEG signals for recognition of human emotions. The author used auditory and visual stimuli extracted from such international affective digitized sounds and affective figures, respectively, to induce the feeling of a certain emotion in the participants of the experiment. The brain electrical signals were acquired through three channels, F3, F4 and FPZ, according to the international 10-20 system [14]. The signal characteristics of EEG, alpha (8-12 Hz) and beta (12-30 Hz), were selected to be used in the recognition of emotions. According to the authors of this paper, the noise signals due to the electrooculogram (EOG) are dominant below the frequency of 4 Hz, the noise related to the electrocardiogram (ECG) signals is around 1.2 Hz and above 30 Hz we can find the noise related to electromyogram

emotions in the bi-dimensional valence-arousal graph, as shown in Figure 1.

256 Brain-Computer Interface Systems – Recent Progress and Future Prospects

**Figure 1.** Bi-dimensional valence-arousal approach

The algorithm frequency Fourier analysis divided the original signal into frequency bands. The principal components analysis reduced the number of features. And finally, Bos classified the EEG signals into emotions with the binary Fisher linear classifier. The classification of emotions was based in the bi-dimensional valence-arousal approach. As a result, Bos obtained a rate of 82.1% accuracy in classification of emotions in brain signals.

In [7], the authors investigated the use of EEG for the recognition of human emotions by humanoid robots. The goal was to provide the ability for robots to detect emotion and react to it in the same way as occurs in a human-human interaction. Also, an EEG device developed by the authors was used to obtain the brain signals. The EEG device consists of only four channels for data acquisition, located in the forebrain according to 10-20 International system in the positions Fp1, Fp2, F7 and F8.

Images of the international affective pictures induced the emotional states pleasant, neutral and unpleasant in participants of the experiments. The classification of emotions was per‐ formed using the support vector machine method. It was verified an accuracy of 47.11 % in the recognition of three emotional states mentioned above. According to the authors, to improve the accuracy of the system would require the development of a more complex system, considering multiple data sources, such as cameras and microphones.

Savran et al. [15] studied the use of brain signals and facial expressions for the recognition of human emotions. According to the authors, due to the sensitivity of the signals of EEG to electrical signals generated by the facial muscles while emotions are expressed, the EEG signals and facial expressions cannot coexist. For this reason, the authors used the near infrared spectroscopy as a technique for acquiring brain signals together with facial expressions. Although the spectroscopic technique is non-invasive and low cost, its main problem is the low temporal resolution, which limits the use of this technique in real time applications. The authors used the international affective pictures to induce emotions in participants of the experiment for the recognition of emotions and built a database with the information regarding the facial expressions and brain signals acquired through video and spectroscopy, respectively. Despite the development of the database, the authors performed the data analysis separately, not reporting results on fusion of brain signals and facial expressions for recognizing emotions.

The authors also constructed a second database with the EEG, near infrared spectroscopy and physiological signals, such as skin conductance and heart rate. These signals were acquired during an experiment of induction of emotions taking pictures of the international affective pictures as stimuli. They used a EEG device with 64 channels, but 10 channels were eliminated due to obstruction of the near infrared spectroscopy equipment. The authors did not discuss the results obtained in the classification of brain signals, only presented the protocol used in the construction of databases.

Murugappan et al. [16] showed a brain-computer interface system (ICC) for the recognition of human emotions. The acquisition of brain signals was performed by an EEG device with 64 channels. A Laplace filter was applied in pre-processing of EEG signals. The authors used the wavelets transform algorithm analysis in selecting the characteristics of brain signals and two methods for features classification, the k nearest neighbors and linear discriminant. The authors of this study chose the classification using the discrete emotions approach (happiness, surprise, fear, disgust and neutral).

Twenty subjects aged between 21 and 39 years participated in this experiment. Audio-visual induced emotions in participants of the aforementioned experiment. EEG signals were acquired at a rate of 256 Hz and have been preprocessed using the Laplace filter, as previously mentioned. The wavelet decomposed the EEG signals into five frequency bands (delta, theta, alpha, beta and gamma). The authors calculated the statistical data of alpha band (entropy, energy, standard deviation and variance) and applied this information as input for the classifiers k nearest neighbors and linear discriminant analysis. Table 1 presents the results obtained by the authors of this study.


**Table 1.** Results obtained in [16]

#### **5. Methods**

This section presents the system developed for the recognition of human emotions based on a BCI system. We use only information from brain signals to detect the emotional state of an individual in this work. Furthermore, due to the lack of an apparatus for reading brain signals, we use one database of EEG signals [17]. Figure 2 shows the architecture of the emotion recognition system proposed in this chapter.

The architecture presented in Figure 2 includes a normal BCI system composed by the brain signal acquisition, signal pre-processing, features selection and classification. The output of the BCI system can be one between four different kind of emotions: positive/excited, positive/ calm, negative/excited, and negative/calm. The next subsections discuss about each step of our emotion recognition system based on BCI interface.

#### **5.1. Database**

We use an EEG database as the source of brain signals [17]. This database was recorded by using music videos to induce emotions in the participants of the experiment. Initially, the authors selected 120 stimuli. Half of these stimuli was selected by a semi-automatically method [18] and the another half was selected manually. After the stimuli selection, one minute of each video was extracted to be used in the research, as stimuli. Finally, the authors of the database chose 40 stimuli [18]. Those stimuli were selected to elicit four different emotions in the

**Figure 2.** Emotion recognition system architecture

wavelets transform algorithm analysis in selecting the characteristics of brain signals and two methods for features classification, the k nearest neighbors and linear discriminant. The authors of this study chose the classification using the discrete emotions approach (happiness,

Twenty subjects aged between 21 and 39 years participated in this experiment. Audio-visual induced emotions in participants of the aforementioned experiment. EEG signals were acquired at a rate of 256 Hz and have been preprocessed using the Laplace filter, as previously mentioned. The wavelet decomposed the EEG signals into five frequency bands (delta, theta, alpha, beta and gamma). The authors calculated the statistical data of alpha band (entropy, energy, standard deviation and variance) and applied this information as input for the classifiers k nearest neighbors and linear discriminant analysis. Table 1 presents the results

**Results 62 channels 24 channels 8 channels**

k nearest neighbors 78,04 % 77,61 % 71,3 % linear discriminant analysis 77,83 % 70,65 % 56,09 %

This section presents the system developed for the recognition of human emotions based on a BCI system. We use only information from brain signals to detect the emotional state of an individual in this work. Furthermore, due to the lack of an apparatus for reading brain signals, we use one database of EEG signals [17]. Figure 2 shows the architecture of the emotion

The architecture presented in Figure 2 includes a normal BCI system composed by the brain signal acquisition, signal pre-processing, features selection and classification. The output of the BCI system can be one between four different kind of emotions: positive/excited, positive/ calm, negative/excited, and negative/calm. The next subsections discuss about each step of our

We use an EEG database as the source of brain signals [17]. This database was recorded by using music videos to induce emotions in the participants of the experiment. Initially, the authors selected 120 stimuli. Half of these stimuli was selected by a semi-automatically method [18] and the another half was selected manually. After the stimuli selection, one minute of each video was extracted to be used in the research, as stimuli. Finally, the authors of the database chose 40 stimuli [18]. Those stimuli were selected to elicit four different emotions in the

surprise, fear, disgust and neutral).

258 Brain-Computer Interface Systems – Recent Progress and Future Prospects

obtained by the authors of this study.

recognition system proposed in this chapter.

emotion recognition system based on BCI interface.

**Table 1.** Results obtained in [16]

**5. Methods**

**5.1. Database**

**Figure 3.** Experiment protocol

individuals: calm/positive, calm/negative, excited/positive and excited/negative. Figure 3 presents the protocol used to conduct the experiment.

The experiments were conducted in two laboratory environments with controlled lighting. Thirty-two participants took part in the experiment, and their EEG signals and peripheral physiological signals (eye movements, facial muscles movements, temperature and blood pressure, among others) were acquired with the system Biosemi Active Two. The signals were recorded from 32 channels according to the 10-10 international system. Moreover, these EEG signals in the database were filtered and the electrooculogram (EOG) artifacts were removed. It was used a camera to capture the images of 22 among the 32 participants in the frontal position. Figure 4 presents the 10-10 international system.

**Figure 4.** Illustration of the 10-10 international system

Two computers were used in this experiment, one for storing data and another for presentation of stimuli. To keep two computers synchronized bookmarks were sent from one computer to another [18].

#### The EEG data was stored

The experiments were conducted in two laboratory environments with controlled lighting. Thirty-two participants took part in the experiment, and their EEG signals and peripheral physiological signals (eye movements, facial muscles movements, temperature and blood pressure, among others) were acquired with the system Biosemi Active Two. The signals were recorded from 32 channels according to the 10-10 international system. Moreover, these EEG signals in the database were filtered and the electrooculogram (EOG) artifacts were removed. It was used a camera to capture the images of 22 among the 32 participants in the frontal

Two computers were used in this experiment, one for storing data and another for presentation of stimuli. To keep two computers synchronized bookmarks were sent from one computer to

position. Figure 4 presents the 10-10 international system.

260 Brain-Computer Interface Systems – Recent Progress and Future Prospects

**Figure 4.** Illustration of the 10-10 international system

another [18].

There are some areas of the brain that are related with the human emotional behavior: brainstem, hypothalamus, thalamus, prefrontal area and limbic system. We chose to use the EEG signals acquired from the channel FP1 in this work. The channel FP1 is located on the prefrontal area of the brain. The prefrontal area is involved in the following functions:


#### **5.2. Signals pre-processing**

In the pre-processing of data from database used in this study, first the data had their sample rate reduced from 512Hz to 128 Hz. The authors of the database removed the artifacts due to eye movements from EEG signals using the technique discussed in [18]. These signals were filtered with a band pass filter with minima cutoff frequency of 4 Hz and maxima of 45 Hz. A common reference was used for all EEG channels. The data was segmented into 60 second samples being the 3 first seconds eliminated. The preprocessing of data in the database was not done in the context of this work.

The authors of the database [17] stored the EEG data pre-processed in 32.mat (matlab) files, one per participant. Each participant file contains two arrays, as illustrated in Table 2.


**Table 2.** Contents of each participant file

#### **5.3. Signals processing**

The processing of EEG signals in a BCI system is divided into two parts: the selection of the signal characteristics and classification of these characteristics. The choice of the method to be used in the first step depends if the signal characteristics are time or frequency domain. In the second stage, the choice of method is independent of the signal domain.

#### *5.3.1. Signal characteristics selection*

Wavelets [16] has been widely used to select the characteristics of the EEG signals in emotion recognition systems and are defined as small waves that have limited duration and average values as zeros. They are mathematical functions, in which a function or data set are located on both time and frequency.

Wavelet analysis consists in the decomposition of a signal into different shifted versions in different scales from the original wavelet. The wavelet analysis is divided into continuous and discrete.

We used the Daubechies 4 discrete wavelet (db4) in this work. We chose the Daubechies 4 based on [16]. The authors performed several experiments with several families of wavelets. In those experiments, the authors found that the wavelet db4 best represents the EEG signals. Figure 5 illustrates an example of the wavelet db4.

**Figure 5.** An example of the wavelet db4

The family of Daubechies wavelets was invented by Ingrid Daubechies, one of the most important people in wavelets research. These wavelets are orthonormal compact, making discrete wavelet transform practice.

We developed a routine in Matlab for reading the EEG signals from the database used in this work. Beyond that, the routine selects the features delta, theta, alpha and beta from the EEG signals. Table 3 presents the rhythmic characteristics of the EEG signals along with their frequency bands.


**Table 3.** Rhythmic characteristics of the EEG signal

values as zeros. They are mathematical functions, in which a function or data set are located

Wavelet analysis consists in the decomposition of a signal into different shifted versions in different scales from the original wavelet. The wavelet analysis is divided into continuous and

We used the Daubechies 4 discrete wavelet (db4) in this work. We chose the Daubechies 4 based on [16]. The authors performed several experiments with several families of wavelets. In those experiments, the authors found that the wavelet db4 best represents the EEG signals.

The family of Daubechies wavelets was invented by Ingrid Daubechies, one of the most important people in wavelets research. These wavelets are orthonormal compact, making

We developed a routine in Matlab for reading the EEG signals from the database used in this work. Beyond that, the routine selects the features delta, theta, alpha and beta from the EEG signals. Table 3 presents the rhythmic characteristics of the EEG signals along with their

on both time and frequency.

**Figure 5.** An example of the wavelet db4

discrete wavelet transform practice.

frequency bands.

Figure 5 illustrates an example of the wavelet db4.

262 Brain-Computer Interface Systems – Recent Progress and Future Prospects

discrete.

The EEG signals from the database presented in Section 5.1 were charged using the aforemen‐ tioned routine. We extracted the EEG data of the FP1 channel from each participant file, discussed in Section 5.2 The EEG data extracted was stored in an new file per participant. Table 4 shows the content of each participant file.


**Table 4.** Content of the new participant file

The EEG signals were sampled at a sampling rate of 512 Hz, but as discussed in Section 5.2, in the pre-processing of the signal that rate was reduced to 128 Hz. Therefore, we used a db4 wavelet of order 4 for selecting the characteristics of the signal.

First, we calculated the wavelets coefficients using two matlab functions: detcoef (it returns the details coefficients of a wavelet) and appcoef(it returns the approximation coefficients of a wavelet). Then, we calculated the details and approximations of the wavelet using the upcoef matlab function. These steps were applied for each row of the data array

The human emotions are related with the theta and alpha characteristics. For that reason, we chose to select theta and alpha features to classify them into emotions. We calculated the values of entropy and energy components of the alpha and theta to evaluate which parameter can provide better results when classified into emotions.

We calculated the entropy and energy using the wentropy and wenergy matlab functions, respectively. That functions receive as parameter an array. In this work, we used arrays with theta and alpha features to estimate the entropy and energy.

#### *5.3.2. Signal characteristics classification*

The second stage in EEG signals processing is the classification of the signals into signals of interest for a given application using translation algorithms. Examples of translation algo‐ rithms include linear discriminant analysis, k-nearest neighbor, support vector machine, and artificial neural network [19], among others.

Artificial neural network has been widely used as algorithm to classify different kind of human information into human emotions. We chose use this technique based on the literature where we can find good results with the use of the artificial neural network.

Artificial neural networks are computational learning models inspired in the biology of the human brain. These models consist of neurons interconnected by synapses. From a functional point of view, neural networks copy the ability of the brain to learn and ideally can be trained to recognize any information, given a set of input data, by adjusting the synaptic weights. A properly trained network, in principle, should be empowered to apply their knowledge and respond appropriately to completely new entries. The most common application of neural networks is the supervised classification and therefore it requires a set of training and test data. Since learning is performed by the training data, the mathematical formalization is based on these data.

The classification of the characteristic of the brain signals in emotions was performed by neural networks algorithm in this work. A neural network consists of input layer, hidden layer and output layer. The input layer is composed of neurons that receive input stimuli. The output layer is composed by neurons which have as their output the network output. The hidden layer or intermediate layer is composed of neurons which perform any data processing network. This layer may be composed of only one layer or several layers of neurons depending on the complexity of the network. In Figure 6, is shown a neural network with two layers.

**Figure 6.** Representation of a neural network

In a neural network there are several parameters to be defined as the number of hidden layers, the number of neurons in each layer of the network and training method. The choice of these parameters in this study was performed based on the literature and in some experiments with the data from the database discussed Section 5.1.

There are several methods to train a neural network. In this work, experiments were performed with three of these methods: Levenberg-Marquardt, Bayesian regularization and resilient propagation algorithms. With the Levenberg-Marquardt algorithm, despite being the fastest, it were not obtained good results, since this method is suitable for solving problems of nonlinear regression and not to problems of pattern recognition. The worst results were obtained with the Bayesian regularization algorithm. As expected, the best results were obtained with the resilient propagation technique that according to the literature is more suitable for pattern recognition.

After the experiments, it was defined that the most appropriate neural network to classify the characteristics of alpha and theta EEG signals on emotions has the following parameters:


The neural network described above was used to classify the data of the database on human emotions. The results of this experiment are discussed in Section 6.

#### **6. Results**

Artificial neural network has been widely used as algorithm to classify different kind of human information into human emotions. We chose use this technique based on the literature where

Artificial neural networks are computational learning models inspired in the biology of the human brain. These models consist of neurons interconnected by synapses. From a functional point of view, neural networks copy the ability of the brain to learn and ideally can be trained to recognize any information, given a set of input data, by adjusting the synaptic weights. A properly trained network, in principle, should be empowered to apply their knowledge and respond appropriately to completely new entries. The most common application of neural networks is the supervised classification and therefore it requires a set of training and test data. Since learning is performed by the training data, the mathematical formalization is based on

The classification of the characteristic of the brain signals in emotions was performed by neural networks algorithm in this work. A neural network consists of input layer, hidden layer and output layer. The input layer is composed of neurons that receive input stimuli. The output layer is composed by neurons which have as their output the network output. The hidden layer or intermediate layer is composed of neurons which perform any data processing network. This layer may be composed of only one layer or several layers of neurons depending on the complexity of the network. In Figure 6, is shown a neural network with two layers.

we can find good results with the use of the artificial neural network.

264 Brain-Computer Interface Systems – Recent Progress and Future Prospects

these data.

**Figure 6.** Representation of a neural network

We chose to use the EEG signals acquired by channel FP1 due to its location. As discussed previously, the prefrontal lobe of the brain is intrinsically related to human emotions.

We processed a total of 1280 trials of EEG signals among all thirty two participants of our experiment. We selected theta and alpha rhythms from the EEG signals and calculated the energy and entropy from both features. We applied energy and entropy as inputs for the neural network used to classify brain signals into four emotional human states. As discussed in Section 5.3.2, the neural network used in classifying EEG signals into emotions in this work has just one input, therefore, we applied those parameters as input for the neural network one at a time.

First, we trained, validate, and tested the neural network with the energy calculated based on the theta feature. Then, we used the entropy obtained from the theta rhythm as the input of the neural network. The same steps were done with the energy and entropy calculated based on the alpha feature. Finally, we estimate for each classified emotion the mean and standard deviation of the results achieved for the thirty two participants of the experiment.

The Table 5 illustrates the mean and standard deviation of the results of our experiments for the emotion positive/excited.


**Table 5.** Mean and standard deviation of the results related to the emotion positive/excited

One can observe in Table 5 that the best results in classifying EEG signals into the emotion positive/excited was achieved when we used the entropy calculated based on the alpha features as the input for the neural network. When we applied the entropy based on the theta features as the input for the neural network, a good result was obtained.

The Table 6 illustrates the mean and standard deviation of the results of our experiments for the emotion positive/calm.


**Table 6.** Mean and standard deviation of the results related to the emotion positive/calm

One can observe in Table 6 that the best results in classifying EEG signals into the emotion positive/calm was achieved when we used the entropy calculated based on the theta features as the input for the neural network. When we applied the entropy based on the alpha features as the input for the neural network, a good result was obtained.

The Table 7 illustrates the mean and standard deviation of the results of our experiments for the emotion negative/calm.


**Table 7.** Mean and standard deviation of the results related to the emotion negative/calm

One can observe in Table 7 that the best results in classifying EEG signals into the emotion positive/excited was achieved when we used the entropy calculated based on the theta features as the input for the neural network. When we applied the entropy based on the alpha features as the input for the neural network, a good result was obtained.

The Table 8 illustrates the mean and standard deviation of the results of our experiments for the emotion negative/excited.


**Table 8.** Mean and standard deviation of the results related to the emotion negative/excited

One can observe in Table 8 that the best results in classifying EEG signals into the emotion negative/excited was achieved when we used the entropy calculated based on the alpha features as the input for the neural network. When we applied the entropy based on the theta features as the input for the neural network, a good result was obtained.

Finally, we observed that using FP1 channel for EEG acquisition, wavelets as the algorithm to select characteristics from EEG signals, and the neural network in classifying those features into emotions, we can recognize at least four human emotions with a good accuracy. Further‐ more, we achieved the best results achieved when we used the entropy calculated based on theta or alpha features as the input for the neural network.

#### **7. Conclusions**

the neural network. The same steps were done with the energy and entropy calculated based on the alpha feature. Finally, we estimate for each classified emotion the mean and standard

The Table 5 illustrates the mean and standard deviation of the results of our experiments for

**Emotion Positive/Excited**

Theta (Energy) 78.125 % 19.572 % Theta (Entropy) 90.625 % 7.7771 % Alpha (Energy) 64.8438 % 27.7586 % Alpha (Entropy) 95.5781 % 7.6855 %

One can observe in Table 5 that the best results in classifying EEG signals into the emotion positive/excited was achieved when we used the entropy calculated based on the alpha features as the input for the neural network. When we applied the entropy based on the theta

The Table 6 illustrates the mean and standard deviation of the results of our experiments for

**Emotion Positive/Calm**

Theta (Energy) 65 % 27.0006 % Theta (Entropy) 90.625 % 11.8967 % Alpha (Energy) 71.875 % 20.8586 % Alpha (Entropy) 86.875 % 12.2967 %

One can observe in Table 6 that the best results in classifying EEG signals into the emotion positive/calm was achieved when we used the entropy calculated based on the theta features as the input for the neural network. When we applied the entropy based on the alpha features

The Table 7 illustrates the mean and standard deviation of the results of our experiments for

**Feature Mean Standard Deviation**

**Feature Mean Standard Deviation**

deviation of the results achieved for the thirty two participants of the experiment.

**Table 5.** Mean and standard deviation of the results related to the emotion positive/excited

features as the input for the neural network, a good result was obtained.

**Table 6.** Mean and standard deviation of the results related to the emotion positive/calm

as the input for the neural network, a good result was obtained.

the emotion positive/excited.

266 Brain-Computer Interface Systems – Recent Progress and Future Prospects

the emotion positive/calm.

the emotion negative/calm.

The emotional state of a person defines their interaction with other people or objects. Therefore, the recognition of human emotions is becoming a concern in the development of systems that require human-machine interaction. The goal in recognizing human emotions is easier and more enjoyable computer use, for example.

There are several sources of information to assist in the recognition of emotions, such as facial expressions, voice and physiological signals, among others. In this study we implemented an emotion recognition system based on the BCI interface. We used a database of EEG signals acquired during experiments to induce emotions in the participants.

The database includes brain signals from thirty-two subjects. Those signals were recorded from thirty-two channels according the 10-10 international system. The database signals were preprocessed and the artifacts due to eye movements were removed. We chose use just the signals from the channel FP1 because your location and to avoid wasting time processing unnecessary information.

We selected the characteristics theta and alpha with the algorithm wavelets. We used in this work a discrete wavelet transform Daubechies db4. We calculated the parameters energy and entropy based on theta and alpha rhythms. The classification of these parameters into emotional states was accomplished with the method neural networks.

We could observe that we achieve good results in recognizing emotions with our approach. When we considered our system based on theta features with the entropy as input for the neural network, we had 90.625 %, 90.625 %, 90 % and 91.4031 % of accuracy for the emotions positive/excited, positive/calm, negative/calm, and negative/excited, respectively.

When we considered our system based on alpha features with the entropy as input for the neural network, we had 95.5781 %, 86.875 %, 87.125 % and 93.7562 % of accuracy for the emotions positive/excited, positive/calm, negative/calm, and negative/excited, respectively.

We recognized four different kinds of emotions based on the bi-dimensional approach: positive/excited, positive/calm, negative/excited, and negative/calm. The best result that we achieved was 95.5781 % when we classified EEG signals into the emotion positive/excited using the entropy calculated based on the alpha characteristics.

Therefore, we could conclude that the combination of wavelets and neural network algorithms is a good choice for classifying emotions by emotion recognition systems based on BCI interface. Furthemore, the FP1 as the signal acquisiton was a good choise based on the results achieved in this work.

According to [20], some individuals have their theta features more active than alpha features during the feeling of emotions. In other cases, the opposite happens, i. e., the subjects have the alpha rhythms more active than the theta rhythms.

As future work, we plan improve our results analyzing what EEG feature is more active during the feeling of emotion by each participant of our experiment. According to this evaluation, we will insert a step to identify the feature that is more active during the experiments in each user and adapt our emotion recognition system to receive the more significant rhythm for each participant.

#### **Author details**

require human-machine interaction. The goal in recognizing human emotions is easier and

There are several sources of information to assist in the recognition of emotions, such as facial expressions, voice and physiological signals, among others. In this study we implemented an emotion recognition system based on the BCI interface. We used a database of EEG signals

The database includes brain signals from thirty-two subjects. Those signals were recorded from thirty-two channels according the 10-10 international system. The database signals were preprocessed and the artifacts due to eye movements were removed. We chose use just the signals from the channel FP1 because your location and to avoid wasting time processing unnecessary

We selected the characteristics theta and alpha with the algorithm wavelets. We used in this work a discrete wavelet transform Daubechies db4. We calculated the parameters energy and entropy based on theta and alpha rhythms. The classification of these parameters into

We could observe that we achieve good results in recognizing emotions with our approach. When we considered our system based on theta features with the entropy as input for the neural network, we had 90.625 %, 90.625 %, 90 % and 91.4031 % of accuracy for the emotions

When we considered our system based on alpha features with the entropy as input for the neural network, we had 95.5781 %, 86.875 %, 87.125 % and 93.7562 % of accuracy for the emotions positive/excited, positive/calm, negative/calm, and negative/excited, respectively.

We recognized four different kinds of emotions based on the bi-dimensional approach: positive/excited, positive/calm, negative/excited, and negative/calm. The best result that we achieved was 95.5781 % when we classified EEG signals into the emotion positive/excited using

Therefore, we could conclude that the combination of wavelets and neural network algorithms is a good choice for classifying emotions by emotion recognition systems based on BCI interface. Furthemore, the FP1 as the signal acquisiton was a good choise based on the results

According to [20], some individuals have their theta features more active than alpha features during the feeling of emotions. In other cases, the opposite happens, i. e., the subjects have the

As future work, we plan improve our results analyzing what EEG feature is more active during the feeling of emotion by each participant of our experiment. According to this evaluation, we will insert a step to identify the feature that is more active during the experiments in each user and adapt our emotion recognition system to receive the more significant rhythm for each

positive/excited, positive/calm, negative/calm, and negative/excited, respectively.

acquired during experiments to induce emotions in the participants.

emotional states was accomplished with the method neural networks.

the entropy calculated based on the alpha characteristics.

alpha rhythms more active than the theta rhythms.

more enjoyable computer use, for example.

268 Brain-Computer Interface Systems – Recent Progress and Future Prospects

information.

achieved in this work.

participant.

Taciana Saad Rached and Angelo Perkusich

Campina Grande Federal University, Brazil

#### **References**


[11] Liu, Y, Sourina, O, & Nguyen, M. K. Real-time EEG-based emotion recognition and its applications. In Transactions on computational science XII, Berlin, Heidelberg, (2011). Marina L. Gavrilova and C. J. Kenneth Tan (Eds.). Springer-Verlag., 256-277.

[12] SuprijantoSari L., Nadhira V., Merthayasa IGN., Farida I. M. Development system for emotion detection based on brain signals and facial images. World Academy of

[13] Bos, D. O. EEG-based emotion recognition the influence of visual and auditory stim‐

[14] Kubler, A, & Muller, K. R. Toward brain-computer interfacing, chapter An introduc‐

[15] Savran, A, Ciftci, K, Chanel, G, Mota, J. C, Viet, L. H, Sankur, B, Akarun, L, Caplier, A, & Rombaut, M. Emotion detection in the loop brain signals and facial images. In

[16] Murugappan, M, Nagarajan, R, & Yaacob, S. Appraising human emotions using time frequency analysis based EEG alpha band features. In Innovative Technologies in In‐ telligent Systems and Industrial Applications, (2009). CITISIA 2009, July 2009., 70-75.

[17] DEAPdataset: a dataset for emotion analysis using eegphysiological and video sig‐ nals. http://www.eecs.qmul.ac.uk/mmv/datasets/deap/accessed 23 February (2012).

[18] Koelstra, S, Muhi, C, Soleymani, M, Lee, J-S, Yazdani, A, Ebrahimi, T, Pun, T, Nijholt, A, & Patras, I. DEAP: A database for emotion analysis using physiological signals.

[19] JananKhani PKodogiannis V., Revelt K. EEG signal classification using wavelet fea‐ ture extraction and neural networks. In Procedings of the IEEE John Vincent Atanas‐ off 2006 International Symposium on Modern Computing, Washington, DC, USA,

[20] Benbadis, S, Husain, A, Kaplan, P, & Tatum, W. Handbook of EEG interpretation.

Demos Medical Publishing. Springer Demos Medic Series, (2007).

Affective Computing IEE Transactions on, Jan.-March (2012). , 3(1), 18-31.

Proceedings of the eNTERFACE 2006 Workshop, Dubrovnik, July (2006).

tion to Brain-computer interfacing, The MIT Press, (2007). , 1-25.

Science, Engineering and Technology 50, (2009).

uli. Emotion, (2006). , 57(7), 1798-806.

270 Brain-Computer Interface Systems – Recent Progress and Future Prospects

(2006). IEEE Computer Society., 120-124.

### *Edited by Reza Fazel-Rezai*

Brain-Computer Interface (BCI) systems allow communication based on a direct electronic interface which conveys messages and commands directly from the human brain to a computer. In the recent years, attention to this new area of research and the number of publications discussing different paradigms, methods, signal processing algorithms, and applications have been increased dramatically. The objective of this book is to discuss recent progress and future prospects of BCI systems. The topics discussed in this book are: important issues concerning end-users; approaches to interconnect a BCI system with one or more applications; several advanced signal processing methods (i.e., adaptive network fuzzy inference systems, Bayesian sequential learning, fractal features and neural networks, autoregressive models of wavelet bases, hidden Markov models, equivalent current dipole source localization, and independent component analysis); review of hybrid and wireless techniques used in BCI systems; and applications of BCI systems in epilepsy treatment and emotion detections.

Photo by Ralwel / iStock

Brain-Computer Interface Systems - Recent Progress and Future Prospects

Brain-Computer Interface

Systems

Recent Progress and Future Prospects

*Edited by Reza Fazel-Rezai*