**2.1 Brain computer interfaces**

684 Amyotrophic Lateral Sclerosis

paralysis. Because these research fields are new, there is a need for development in terms of efficiency; meaning accuracy, reliability and quick responses are necessary. Many research groups from all over the world are focusing on HCI applications in order to improve alternative communication channels for the disabled. An efficient alternative channel for communication and control device without overt speaking and muscular movements is important to make life easier for individuals who are suffering from ALS or other illnesses that prevent proper limb and muscular responses. Because of this, the area of study related

In this chapter, ALS related HCI in particular, is discussed. The very common field electroencephalogram (EEG) based BCI and other approaches in this field are presented. With interdisciplinary studies, developing new interfaces and interaction techniques are opening new research fields for investigation. Especially for the paralysis patients, the classical communication or control ways, such as overt speaking or hand motions cannot be used. Using bio-signals such as EEG and its various methodologies (i.e. P300, slow cortical potentials, etc), electrooculogram (EOG), hearth rate (HR) or galvanic skin response (GSR) as well as the hemoglobin level which is related to oxygenation, are the only ways to send messages or control signals to devices regarding user's demands, intentions or expectations. This does not only give the patients the potential ability to give messages on computer screens and control a powered wheelchair or robot arm without muscular movements, but

The aim of this chapter is to present the state of the art of the technology on HCIs. This chapter also addresses the use of different bio-signals individually or the integrated hybrid/integrated multi-modal system approach for communication and control with high performance. In order to increase performance, processing combined bio-signals and multi-modal integrated systems will be discussed. For this purpose, several bio-signals such as EEG, EOG, and functional near

In general, HCIs are related to the adaptation of a human and a computer. HMIs adapt human demands to the machine. Human computer interface operation requires the effective interaction of two sides; user and system. Here, a system can be an integrated system of computers and a word speller, a robot arm, a powered wheelchair, house appliances such as door locks, TVs or musical instruments, etc. In order to reflect intent/demands, instead of overt speaking or muscular movements, other bio-signals can be used. Effective BCIs may serve as assessment tools and adaptive systems for HCI for able-bodied people and be

Research in this field is typically focused on several areas of improvement for HCIs in order

with HCIs has high expectations and are important for improving quality of life.

also can be potentially useful for the elderly as well.

proven for people with severe motor neuron diseases.

ii. User friendliness (including user training)

to increase its usefulness and effectiveness. These areas are:

**2. Human computer interactions** 

i. High performance a. Accuracy b. Reliability c. Fast d. Robustness

iii. Ease of application iv. Cost effectiveness.

infrared (fNIR) spectroscopy based system research are introduced.

From a broad perspective, BMI refers to the interface between a brain and a machine (for review Lebedev & Nicolelis, 2006). In this section, the common term - brain computer interface (BCI) will be presented. BCI can be described as a translation of human intentions into a control signal without using the muscles. The aim of BCI is to provide communication and control for people with severe motor disabilities.

The BCI system translates the signals that are encoded by the user's intentions into messages and control commands. Research in this field has been rapidly growing in neuroscience and bioengineering. Specifically, this technology is promising for users with motor neuron diseases. Table 1 shows the estimated potential users of BCI. According to this table, there are more than 100 million potential users in the world.


Table 1. Potential users of BCI in the world (Guger, 2008).

Electrical brain activities (electroencephalography, EEG) related to human intentions can be monitored using electrodes attached to the scalp surface, non-invasively. EEG signals are gross potential of the thousands of neurons, roughly reflecting bodily functions. Because the skull and scalp play the role of a barrier for the electrical signals, EEG signals have low amplitudes (in micro-volts scale) and exist in the 0.5–30 Hz frequency band. Figure 1 shows the ongoing time series EEG signal and its power spectral density. As it is shown from this figure, 8-12 Hz ( band) and 26 Hz ( band) components are dominant.

In order to increase efficiency, brain electrical signals can be recorded by subdural electrodes (electrocorticogram, ECoG), invasively. ECoGs are neuronal activity that is acquired from smaller cortical areas when compared to EEGs. Epidural or subdural recording is less invasive than intra-cortical recording. While their applications are difficult, the resolution of these recordings can be significantly higher than conventional EEG. The BCI usefulness of intra-cortical signals is promising (Wolpaw, 2003).

Human Computer Interactions for Amyotrophic Lateral Sclerosis Patients 687

was quantified by the most reactive frequency bands were chosen then band-pass filtered within those bands. SCPs are slow shifts of the EEG with duration from 300 msec to several seconds. The group of SCPs includes the contingent negative variation, premovement

While these approaches are important for the detection of imagery motor activities, the approach of the P300 waveform is very common in BCI applications. As it is shown figure 2, the P300 response is a large, positive potential (conventionally opposite polarization) that has been studied extensively within the context of the oddball paradigm. Donchin and colleagues first reported the use of the P300 for BCI communication (Donchin et al., 2000). International approaches for the detection of P300 speller, a matrix of grey symbols on a

Fig. 2. Typical averaged event related potentials after stimulation. In BCI applications P300

Fig. 3. The matrix for P300 speller. The stimulus matrix monitored by the subject. Typically, one of the rows or columns of the matrix was intensified every 125 ms (The matrix is

potentials, the Bereitschaftspotential and expectancy waves (Pfurtscheller et al., 2005).

dark background (virtual keyboard) are shown in figure 3.

Amplitude (µV)

component is extensively used.

proposed first by Donchin and colleagues 1988).

Fig. 1. Ongoing EEG recording from 27-year old male subject's occipital lobe: (a) the time series signal, and (b) power spectral density.

BCI systems can provide a communication and control channel, which bypasses conventional neuromuscular pathways involved in speaking or movement activity made to manipulate objects (Wolpaw et al., 2002). The command for device control can be generated by self-regulated µ rhythms (Wolpaw et al., 2000), motor imagery (Pfurtscheller et al., 2000) or a visual evoked potential (Sutter, 1992).

An EEG based BCI system can use the signals listed below.


P300 and SSVEP methods require external stimulation that the user has to focus attention on or gaze into flickering lights, which the other methods do not need (self-paced systems). Because of this, the user is not free to decide on performing an action; the user depends on computer software for the synchronization. Each of these methods is advantageous or disadvantageous with respect to performance, information transfer rate and user training time. The BCI systems may not use just one of these approaches, but may use a combination of two or more signals as a hybrid approached system.

As an event related potential (ERP), the common method is using the P300 response (Farwell & Donchin, 1988). The P300 response is elicited between 300 msec and 400 msec after stimulation, as it is shown in figure 2. Because ERPs are so small, for clarity, signal (epoch) averaging is necessary. The latency and amplitude may change from user to user, however the shape of the signal is roughly the same. In addition to using P300 response, SSVEPs (Gao et al, 2003) and SCPs, µ (8-12 Hz) and high β (18-26 Hz) rhythms (sensorimotor applications) are used in BCI applications.

The SSVEP is a brain response evoked mainly in the visual and parietal cortices as a response to flickering visual stimuli. The SSVEP has gained popularity in the BCI research because it provides advantages in terms of speed and robustness. ERD and Event-related synchronization

(a) (b)

BCI systems can provide a communication and control channel, which bypasses conventional neuromuscular pathways involved in speaking or movement activity made to manipulate objects (Wolpaw et al., 2002). The command for device control can be generated by self-regulated µ rhythms (Wolpaw et al., 2000), motor imagery (Pfurtscheller et al., 2000)

P300 and SSVEP methods require external stimulation that the user has to focus attention on or gaze into flickering lights, which the other methods do not need (self-paced systems). Because of this, the user is not free to decide on performing an action; the user depends on computer software for the synchronization. Each of these methods is advantageous or disadvantageous with respect to performance, information transfer rate and user training time. The BCI systems may not use just one of these approaches, but may use a combination

As an event related potential (ERP), the common method is using the P300 response (Farwell & Donchin, 1988). The P300 response is elicited between 300 msec and 400 msec after stimulation, as it is shown in figure 2. Because ERPs are so small, for clarity, signal (epoch) averaging is necessary. The latency and amplitude may change from user to user, however the shape of the signal is roughly the same. In addition to using P300 response, SSVEPs (Gao et al, 2003) and SCPs, µ (8-12 Hz) and high β (18-26 Hz) rhythms (sensorimotor

The SSVEP is a brain response evoked mainly in the visual and parietal cortices as a response to flickering visual stimuli. The SSVEP has gained popularity in the BCI research because it provides advantages in terms of speed and robustness. ERD and Event-related synchronization

Fig. 1. Ongoing EEG recording from 27-year old male subject's occipital lobe: (a) the time

series signal, and (b) power spectral density.

or a visual evoked potential (Sutter, 1992).

 steady state visual evoked potential (SSVEP), event related desynchronisation (ERD), slow cortical potential (SCP) and sensorimotor rhythm (SMR).

of two or more signals as a hybrid approached system.

applications) are used in BCI applications.

P300 response,

An EEG based BCI system can use the signals listed below.

was quantified by the most reactive frequency bands were chosen then band-pass filtered within those bands. SCPs are slow shifts of the EEG with duration from 300 msec to several seconds. The group of SCPs includes the contingent negative variation, premovement potentials, the Bereitschaftspotential and expectancy waves (Pfurtscheller et al., 2005).

While these approaches are important for the detection of imagery motor activities, the approach of the P300 waveform is very common in BCI applications. As it is shown figure 2, the P300 response is a large, positive potential (conventionally opposite polarization) that has been studied extensively within the context of the oddball paradigm. Donchin and colleagues first reported the use of the P300 for BCI communication (Donchin et al., 2000). International approaches for the detection of P300 speller, a matrix of grey symbols on a dark background (virtual keyboard) are shown in figure 3.

Fig. 2. Typical averaged event related potentials after stimulation. In BCI applications P300 component is extensively used.

Fig. 3. The matrix for P300 speller. The stimulus matrix monitored by the subject. Typically, one of the rows or columns of the matrix was intensified every 125 ms (The matrix is proposed first by Donchin and colleagues 1988).

Human Computer Interactions for Amyotrophic Lateral Sclerosis Patients 689

glass", "Turn on the TV" or "Unlock the door", etc. After real time processing regarding the

The BCI systems may be self-paced. This means that the system allows communication or control whenever the user wishes. These systems' performance determined the true positive rate and the false positive rate. The true positive rate is the percentage of intentional control commands that are correctly detected by the BCI system. False positive rate are false positives generated by the system during the periods for which the user does not intend

A synchronous BCI uses external stimulation whereas in asynchronous BCI the subject makes self-paced decisions on when to switch from one mental task to the next. A virtual keyboard and a mobile robot which respond every 0.5 s were developed (Millan & Mourino, 2003). Additionally, an asynchronous BCI analyzes and classifies EEG data continuously (Townsend et al., 2004). These systems are designed to process only one brain state in the ongoing EEG. For this purpose, only one channel may be sufficient. To control a powered wheelchair using a delayed response task a binary classification of left and right movement intentions were classified with a classification rate of over 80% from single trial EEG

In order to improve the BCI system's performance, a rapid transition of various types of parameter estimation and classification algorithms to real-time implementation and testing (Guger, et al., 2001), classification accuracy and communication rate (Turnip et al., 2010) were realized. Identifying errors through the brain's reaction to mistakes is used to improve the robustness, flexibility, and reliability of BCIs (Buttfield et al., 2006). In addition to algorithmic improvement to increase performance, or in other words - high-resolution, multi-channel EEG systems have been developed to enhance the spatial information content

For standardization of BCI research, there is a documented general-purpose BCI research and development platform known as BCI2000. This platform can incorporate independent or a combination of any brain signals, signal processing methods, output devices and

As a new approach to drive BCIs, functional magnetic resonance imaging (fMRI), have been used. The temporal resolution of the hemodynamic deoxyhemoglobin changes in the range of 1-2 seconds, while its spatial resolution is generally observable with the current imaging techniques at a few millimeters scale. Local hemodynamic response can be measured by fMRI. Hence, fMRI responses and cortical sources of EEG data are spatially related. It is possible to estimate the cortical activity with a spatial resolution of few millimeters and with a temporal resolution of milliseconds from noninvasive EEG measurements (Astolfi et al.,

Although an fMRI based BCIs noninvasively records activity of the entire brain with a high spatial resolution, they are not suitable for everyday use. They have temporal delays of several seconds. However, they have good spatial resolution and they can sample the activity of deep brain structures (Lebedev & Nicolelis, 2006). An fMRI-based BCI platform which performs data processing and feedback of the hemodynamic brain activity within 1.3 s (Weiskopf et al., 2004), psychophysiological markers (Nijboer et al., 2009), and communication using real-time fMRI (Eklund et al., 2010) were developed. In the later system, the subject in the MR scanner sees a virtual keyboard and steers a cursor to select different letters that can be combined to create words. The cursor is moved to the left by

of EEG activity. All these factors help to increasing the efficiency of BCI systems.

operating protocols (Wolpaw et al., 2003; Schalk, et al., 2004).

goal, the next step is goal-directed feedback generation.

control.

2010).

(Kaneswaran et al., 2010).

The P300 speller paradigm has been common as a promising communication tool. Using the P300, a virtual keyboard without using or requiring any activation of skeletal muscles can be realized (Farwell & Donchin, 1988; Donchin et al., 2000; Sellers & Donchin, 2006). In the virtual keyboard, rows and columns of the matrix were randomly intensified typically every 125 ms. A P300 was produced after attended character is flashed. The attended symbol was selected by averaging responses for rows and columns. Accurate performance is obtained in users with and without motor impairments. Users with ALS are able to use the P300 based, single-stimulus system using either auditory or visual presentations. The Oddball paradigm to implement using the commands: Yes, No, Pass, and End with three presentation modes: auditory, visual, and both auditory and visual were used (Sellers et al., 2006). Detecting P300 signals the mind spelled characters (words, sentences, stories), enable the communication-disabled via internet (van Kokswijk & van Hulle, 2010) were realized.

People can learn to control EEG features consisting of SMR amplitudes and can use this control to move a cursor to a target on a screen. EEG recordings during right and left motor imagery allow users to establish a new communication channel. Such an EEG-based BCI can be used to develop a simple binary response for the control of a device (Guger et al., 2000). To control cursor movement 8-12 Hz (µ) or 18-26 Hz () frequency bands over sensorimotor cortex can be used (Fabiani et al., 2004). In the standard one-dimensional application, the cursor moves horizontally from left to right at a fixed rate while vertical cursor movement is continuously controlled by SMR (µ and rhythms) amplitude over left and/or right sensorimotor cortex (McFarland & Wolpaw, 2005). Intention of movement of left or right index finger, or right foot is recognized in EEG signals (Peters et al., 2001).

The model usually employed in many BCI systems is presented in Figure 4. Here, messages can be word speller output and commands can be for the powered wheelchair, robot hand or domotic (house) appliances control commands.

Fig. 4. A Typical BCI system.

To increase usefulness, BCI research groups are focusing several respects. In order to evaluate the usefulness of approaches for user intention, it is important to recognize that intention is normally formatted as a goal (Wolpaw, 2003). This goal may be "Write letter 'A' (On the screen)", "Move forward (powered wheelchair)" "Hit target (cursor)", "Grasp the

The P300 speller paradigm has been common as a promising communication tool. Using the P300, a virtual keyboard without using or requiring any activation of skeletal muscles can be realized (Farwell & Donchin, 1988; Donchin et al., 2000; Sellers & Donchin, 2006). In the virtual keyboard, rows and columns of the matrix were randomly intensified typically every 125 ms. A P300 was produced after attended character is flashed. The attended symbol was selected by averaging responses for rows and columns. Accurate performance is obtained in users with and without motor impairments. Users with ALS are able to use the P300 based, single-stimulus system using either auditory or visual presentations. The Oddball paradigm to implement using the commands: Yes, No, Pass, and End with three presentation modes: auditory, visual, and both auditory and visual were used (Sellers et al., 2006). Detecting P300 signals the mind spelled characters (words, sentences, stories), enable the communication-disabled via internet (van Kokswijk & van Hulle, 2010) were realized. People can learn to control EEG features consisting of SMR amplitudes and can use this control to move a cursor to a target on a screen. EEG recordings during right and left motor imagery allow users to establish a new communication channel. Such an EEG-based BCI can be used to develop a simple binary response for the control of a device (Guger et al., 2000). To control cursor movement 8-12 Hz (µ) or 18-26 Hz () frequency bands over sensorimotor cortex can be used (Fabiani et al., 2004). In the standard one-dimensional application, the cursor moves horizontally from left to right at a fixed rate while vertical cursor movement is continuously controlled by SMR (µ and rhythms) amplitude over left and/or right sensorimotor cortex (McFarland & Wolpaw, 2005). Intention of movement of left or right

index finger, or right foot is recognized in EEG signals (Peters et al., 2001).

or domotic (house) appliances control commands.

Fig. 4. A Typical BCI system.

The model usually employed in many BCI systems is presented in Figure 4. Here, messages can be word speller output and commands can be for the powered wheelchair, robot hand

To increase usefulness, BCI research groups are focusing several respects. In order to evaluate the usefulness of approaches for user intention, it is important to recognize that intention is normally formatted as a goal (Wolpaw, 2003). This goal may be "Write letter 'A' (On the screen)", "Move forward (powered wheelchair)" "Hit target (cursor)", "Grasp the glass", "Turn on the TV" or "Unlock the door", etc. After real time processing regarding the goal, the next step is goal-directed feedback generation.

The BCI systems may be self-paced. This means that the system allows communication or control whenever the user wishes. These systems' performance determined the true positive rate and the false positive rate. The true positive rate is the percentage of intentional control commands that are correctly detected by the BCI system. False positive rate are false positives generated by the system during the periods for which the user does not intend control.

A synchronous BCI uses external stimulation whereas in asynchronous BCI the subject makes self-paced decisions on when to switch from one mental task to the next. A virtual keyboard and a mobile robot which respond every 0.5 s were developed (Millan & Mourino, 2003). Additionally, an asynchronous BCI analyzes and classifies EEG data continuously (Townsend et al., 2004). These systems are designed to process only one brain state in the ongoing EEG. For this purpose, only one channel may be sufficient. To control a powered wheelchair using a delayed response task a binary classification of left and right movement intentions were classified with a classification rate of over 80% from single trial EEG (Kaneswaran et al., 2010).

In order to improve the BCI system's performance, a rapid transition of various types of parameter estimation and classification algorithms to real-time implementation and testing (Guger, et al., 2001), classification accuracy and communication rate (Turnip et al., 2010) were realized. Identifying errors through the brain's reaction to mistakes is used to improve the robustness, flexibility, and reliability of BCIs (Buttfield et al., 2006). In addition to algorithmic improvement to increase performance, or in other words - high-resolution, multi-channel EEG systems have been developed to enhance the spatial information content of EEG activity. All these factors help to increasing the efficiency of BCI systems.

For standardization of BCI research, there is a documented general-purpose BCI research and development platform known as BCI2000. This platform can incorporate independent or a combination of any brain signals, signal processing methods, output devices and operating protocols (Wolpaw et al., 2003; Schalk, et al., 2004).

As a new approach to drive BCIs, functional magnetic resonance imaging (fMRI), have been used. The temporal resolution of the hemodynamic deoxyhemoglobin changes in the range of 1-2 seconds, while its spatial resolution is generally observable with the current imaging techniques at a few millimeters scale. Local hemodynamic response can be measured by fMRI. Hence, fMRI responses and cortical sources of EEG data are spatially related. It is possible to estimate the cortical activity with a spatial resolution of few millimeters and with a temporal resolution of milliseconds from noninvasive EEG measurements (Astolfi et al., 2010).

Although an fMRI based BCIs noninvasively records activity of the entire brain with a high spatial resolution, they are not suitable for everyday use. They have temporal delays of several seconds. However, they have good spatial resolution and they can sample the activity of deep brain structures (Lebedev & Nicolelis, 2006). An fMRI-based BCI platform which performs data processing and feedback of the hemodynamic brain activity within 1.3 s (Weiskopf et al., 2004), psychophysiological markers (Nijboer et al., 2009), and communication using real-time fMRI (Eklund et al., 2010) were developed. In the later system, the subject in the MR scanner sees a virtual keyboard and steers a cursor to select different letters that can be combined to create words. The cursor is moved to the left by

Human Computer Interactions for Amyotrophic Lateral Sclerosis Patients 691

(a) (b)

(c) (d) Fig. 6. Voluntary EOG signals: a) Horizontal, b) Vertical, c) Eye blink and d) double blink

Many studies exist in the literature concerning the application of eye movements to the HCI (Bahill, 1982; Yoshiaki, 1997; Juhola, 1985; Allison et al., 1996; Hutchinson et al., 1989; Norris & Wilson, 1997; Kuno, et al., 1998; Chen et al., 1999; Ihara et al., 2009). The research focused mainly on the word speller. Recognizing eye movements; such as left, right, up and down, and eye blinks to select characters from the virtual keyboard on the screen can use giving a message (Kherlopian et al., 2006; Akan & Argunsah, 2007; Usakli et al, 2009). With the EOG based word speller, it is reported that 5 letter-word such as "Water" can be written in 25 seconds (Usakli & Gurkan, 2010). The menus of this system are shown in figure 7. EOGcontrolled cursor interfaces, where the cursor can be controlled by eye movement (Tomita et

As a communication device, a prototype of a head-mounted display with the eye-gaze detection function was developed which a user can operate by eye movements (Handa & Ebisawa, 2008). With this device, the eye-gaze point was determined from the relative position between the pupil center and the corneal reflection of the light source which were detected by the camera. An eye-gaze controlled navigation and electromyography (EMG) enter (confirm) the selection of letter (Dhillon et al. 2009), and an eye-movement tracking system (Krueger & Stieglitz, 2007; Deng et al., 2009; Septanto et al., 2009) were developed. Another promising EOG based method is a Morse code generator (Wu et al., 2007). Additionally, an EOG-based a powered wheelchair (Barea et al., 2002; Chung-Hsien et al., 2009) and a portable wireless device (Zheng et al., 2009) were realized. The EOG based device allows the patients to generate decisions on a screen by means of simple eye movement signals. These signals can be measured with EMG/EEG electrodes, without the need of complex systems or infra-red cameras. Then, patients are able to select letters on the

(Usakli & Gurkan, 2010).

al., 2006; Tamura et al., 2010) are realized, successfully.

activating the left hand, to the right by activating the right hand, down by activating the left toes and up by activating the right toes (Eklund et al., 2010).

It can be concluded that as an alternative method for communication through speaking and muscular movements, BCIs are allowing communication and control for individuals with motor neuron disease such as ALS. A BCI system consists of recognition by a computation of the patterns of brain electrical activity on the scalp acquired from an array of electrodes. Although this technology is quite useful, it still needs to be developed in terms of efficiency.

### **2.2 Electrooculography based systems**

Paralyzed patients are unable to communicate normally with their environment. For these patients, the only part of their body that is under their control, in terms of muscular movement, is their eyes. Some research in this area has been focused on investigating new efficient communication tools for paralyzed patients to translate their eye movements into appropriate communication messages or control signals.

With eye movements, a potential across the cornea and retina exists. This potential is called the cornea-retinal potential and it is the source of the electrooculogram (EOG). Communicating and controlling with EOG can be used for the disabled. An EOG based HCI device is able to recognize the subject's eye movements by using the electrical activity generated by the eye movements. EOG signals have certain patterns for each kind of eye movement (left, right, up, down, blink or wink). These signal patterns can be acquired and then recognized as signals which can be used for controlling external devices like a virtual keyboard, a powered wheelchair, a robot arm or a movable robot. As a very common application, the EOG-based virtual keyboard provides a means for paralyzed patients to write letters on a screen with eye movements without using a conventional keyboard. An EOG-based system for HCI application is presented in figure 5 and EOG signal samples are shown in figure 6.

Fig. 5. Typical EOG-based interface.

activating the left hand, to the right by activating the right hand, down by activating the left

It can be concluded that as an alternative method for communication through speaking and muscular movements, BCIs are allowing communication and control for individuals with motor neuron disease such as ALS. A BCI system consists of recognition by a computation of the patterns of brain electrical activity on the scalp acquired from an array of electrodes. Although this technology is quite useful, it still needs to be developed in terms of efficiency.

Paralyzed patients are unable to communicate normally with their environment. For these patients, the only part of their body that is under their control, in terms of muscular movement, is their eyes. Some research in this area has been focused on investigating new efficient communication tools for paralyzed patients to translate their eye movements into

With eye movements, a potential across the cornea and retina exists. This potential is called the cornea-retinal potential and it is the source of the electrooculogram (EOG). Communicating and controlling with EOG can be used for the disabled. An EOG based HCI device is able to recognize the subject's eye movements by using the electrical activity generated by the eye movements. EOG signals have certain patterns for each kind of eye movement (left, right, up, down, blink or wink). These signal patterns can be acquired and then recognized as signals which can be used for controlling external devices like a virtual keyboard, a powered wheelchair, a robot arm or a movable robot. As a very common application, the EOG-based virtual keyboard provides a means for paralyzed patients to write letters on a screen with eye movements without using a conventional keyboard. An EOG-based system for HCI

application is presented in figure 5 and EOG signal samples are shown in figure 6.

toes and up by activating the right toes (Eklund et al., 2010).

appropriate communication messages or control signals.

**2.2 Electrooculography based systems** 

Fig. 5. Typical EOG-based interface.

Fig. 6. Voluntary EOG signals: a) Horizontal, b) Vertical, c) Eye blink and d) double blink (Usakli & Gurkan, 2010).

Many studies exist in the literature concerning the application of eye movements to the HCI (Bahill, 1982; Yoshiaki, 1997; Juhola, 1985; Allison et al., 1996; Hutchinson et al., 1989; Norris & Wilson, 1997; Kuno, et al., 1998; Chen et al., 1999; Ihara et al., 2009). The research focused mainly on the word speller. Recognizing eye movements; such as left, right, up and down, and eye blinks to select characters from the virtual keyboard on the screen can use giving a message (Kherlopian et al., 2006; Akan & Argunsah, 2007; Usakli et al, 2009). With the EOG based word speller, it is reported that 5 letter-word such as "Water" can be written in 25 seconds (Usakli & Gurkan, 2010). The menus of this system are shown in figure 7. EOGcontrolled cursor interfaces, where the cursor can be controlled by eye movement (Tomita et al., 2006; Tamura et al., 2010) are realized, successfully.

As a communication device, a prototype of a head-mounted display with the eye-gaze detection function was developed which a user can operate by eye movements (Handa & Ebisawa, 2008). With this device, the eye-gaze point was determined from the relative position between the pupil center and the corneal reflection of the light source which were detected by the camera. An eye-gaze controlled navigation and electromyography (EMG) enter (confirm) the selection of letter (Dhillon et al. 2009), and an eye-movement tracking system (Krueger & Stieglitz, 2007; Deng et al., 2009; Septanto et al., 2009) were developed. Another promising EOG based method is a Morse code generator (Wu et al., 2007). Additionally, an EOG-based a powered wheelchair (Barea et al., 2002; Chung-Hsien et al., 2009) and a portable wireless device (Zheng et al., 2009) were realized. The EOG based device allows the patients to generate decisions on a screen by means of simple eye movement signals. These signals can be measured with EMG/EEG electrodes, without the need of complex systems or infra-red cameras. Then, patients are able to select letters on the

Human Computer Interactions for Amyotrophic Lateral Sclerosis Patients 693

Fig. 8. Absorbtion spectrum in near-infrared window (Izzetoglu et al., 2007).

elicited voluntarily (Ayaz et al., 2007).

Figure 9 and 10 show measurement principles of oxygen level in a general NIR system. Photons transmitted with capillary vessels detected by detectors and measured photon intensity related to the oxygen level. fNIR spectroscopy can be used for BCI for the patients with ALS diseases (Bunce et al., 2006); to detect cognitive activity from prefrontal cortex

Fig. 9. Measurement of oxygen level in a NIR system. More light absorption means more

hemoglobin, and consequently, more oxygenation.

screen or even communicate basic needs (food drinks, etc.) to the caregiver with a simple movement of their eyes (Usakli et al, 2009). All these studies show that EOG signals can be used as an input for efficient HCI applications. Since the EOG signals are larger, measurement of these signals are easy compared to EEG. This property makes EOG applications much more efficient.

Fig. 7. The EOG-based user interface (Usakli & Gurkan, 2010): a) Main menu, on the top, vertical and horizontal EOG signals are shown. b) Virtual keyboard: A 5-letter word can be written in 25 seconds.

### **2.3 Near-infrared spectroscopy based systems**

Another approach for noninvasive BCI is based on optical means that measure brain activity by monitoring the hemodynamic response. Near Infrared Spectroscopy (NIRS) optical recording technology measures changes in the brain's oxygen absorption based on the optical properties of hemoglobin. Optical imaging spectroscopy can provide high spatial temporal resolution information about fractional changes in the hemodynamic response to increased neural activity (Mayhev et al, 2000). Human performance and cognitive activities such as attention, working memory, problem solving, etc., can be assessed by fNIR technology (Izzetoglu et al, 2007). NIRS technology usage is growing throughout the world for better understanding cortical activity during cognitive tasks.

NIRS is a spectroscopic method that uses about 800 nm wavelengths in the electromagnetic spectrum. The absorption spectrum in the near-infrared window is presented in figure 8. The level of light absorption related to the amount of oxyhemoglobin can be measured with detectors. While concentrating, brain uses much more oxygen than normal state. This demand meets with clean blood, then the number of oxyhemoglobin increases. Therefore, it causes more absorption of the light (Chance et al., 1998). This method gives an idea of oxygenation (changes of (de)oxy-hemoglobin) of blood in cortical capillary vessels.

The primary application of NIRS in the human body is seen through the measurement of the transmission and absorption of NIR light in human body tissues containing information about the changes in hemoglobin concentration. When a specific area of the brain is activated, the localized blood volume in that area changes quickly. Maximum response is observed between the 5th – 9th seconds (Malonek et al. 1997). Optical imaging can measure the location and activity of specific regions of the brain by continuously monitoring blood hemoglobin levels through the determination of optical absorption coefficients.

screen or even communicate basic needs (food drinks, etc.) to the caregiver with a simple movement of their eyes (Usakli et al, 2009). All these studies show that EOG signals can be used as an input for efficient HCI applications. Since the EOG signals are larger, measurement of these signals are easy compared to EEG. This property makes EOG

(a) (b) Fig. 7. The EOG-based user interface (Usakli & Gurkan, 2010): a) Main menu, on the top, vertical and horizontal EOG signals are shown. b) Virtual keyboard: A 5-letter word can be

Another approach for noninvasive BCI is based on optical means that measure brain activity by monitoring the hemodynamic response. Near Infrared Spectroscopy (NIRS) optical recording technology measures changes in the brain's oxygen absorption based on the optical properties of hemoglobin. Optical imaging spectroscopy can provide high spatial temporal resolution information about fractional changes in the hemodynamic response to increased neural activity (Mayhev et al, 2000). Human performance and cognitive activities such as attention, working memory, problem solving, etc., can be assessed by fNIR technology (Izzetoglu et al, 2007). NIRS technology usage is growing throughout the world

NIRS is a spectroscopic method that uses about 800 nm wavelengths in the electromagnetic spectrum. The absorption spectrum in the near-infrared window is presented in figure 8. The level of light absorption related to the amount of oxyhemoglobin can be measured with detectors. While concentrating, brain uses much more oxygen than normal state. This demand meets with clean blood, then the number of oxyhemoglobin increases. Therefore, it causes more absorption of the light (Chance et al., 1998). This method gives an idea of oxygenation (changes of (de)oxy-hemoglobin) of

The primary application of NIRS in the human body is seen through the measurement of the transmission and absorption of NIR light in human body tissues containing information about the changes in hemoglobin concentration. When a specific area of the brain is activated, the localized blood volume in that area changes quickly. Maximum response is observed between the 5th – 9th seconds (Malonek et al. 1997). Optical imaging can measure the location and activity of specific regions of the brain by continuously monitoring blood

hemoglobin levels through the determination of optical absorption coefficients.

applications much more efficient.

written in 25 seconds.

blood in cortical capillary vessels.

**2.3 Near-infrared spectroscopy based systems** 

for better understanding cortical activity during cognitive tasks.

Fig. 8. Absorbtion spectrum in near-infrared window (Izzetoglu et al., 2007).

Figure 9 and 10 show measurement principles of oxygen level in a general NIR system. Photons transmitted with capillary vessels detected by detectors and measured photon intensity related to the oxygen level. fNIR spectroscopy can be used for BCI for the patients with ALS diseases (Bunce et al., 2006); to detect cognitive activity from prefrontal cortex elicited voluntarily (Ayaz et al., 2007).

Fig. 9. Measurement of oxygen level in a NIR system. More light absorption means more hemoglobin, and consequently, more oxygenation.

Human Computer Interactions for Amyotrophic Lateral Sclerosis Patients 695

recordings are transformed into a Haar-wavelet series (Kawakami et al., 1996) and variational Kalman filtering (Sykacek et al., 2004) for adaptive classification in the BCI system was used. The later algorithm translates EEG segments adaptively into probabilities of cognitive states. It allows for nonstationarities in the joint process over cognitive states and generated EEG which may occur during a consecutive number of trials. The wavelet features are used to determine the characteristic of eye movement waveform (Daud &

A new two-stage approach to extract the µ rhythm component was developed. The first stage uses second-order blind identification with stationary wavelet transform to automatically remove the artifacts. In the second stage, second-order blind identification is applied again to find the µ rhythm component. In this method artifact removal enhances the extraction of the µ rhythm component (Ng & Raveendran, 2009). For classification of motor execution signals, fractal approach provides promising results (Usakli, 2010). An EEG based BCI for users to control a cursor on a computer display is one of the common study area. The developed system uses an adaptive algorithm, based on kernel partial least squares classification, to associate patterns in multichannel EEG frequency spectra with cursor controls (Trejo et al., 2006). For the BCI related classification review can be found in (Lotte et

To increase HCI efficiency, hybrid approaches offer good results. Three physiological signals: blood volume pulse, galvanic skin response and pupil diameter, to automatically monitor stress were used, successfully (Zhai et al., 2005). A vision-based multimodal human computer interface system using eye and hand motion tracking was developed. This vision-based virtual interface integrates the function of the motion tracking of eye blinking and hand gestures with the function of their recognition as a virtual interface (Shin & Chun, 2007). EEG error-related signals present a hybrid approach for HCI. This approach uses human gestures to send commands to a computer and exploits brain activity to provide implicit feedback about the recognition of such commands

**4.1 Progressing study for hybrid multi-modal human computer interaction** 

In order to contribute to the HCI field, a novel multi-modal integrated design is completed and establishing an efficient communication and control channel for individuals with motor neuron diseases such as ALS has been continuing. The preliminary results are promising. Using experience in relevant fields such as EEG (Usakli & Gencer, 2007a; Usakli & Gencer, 2007b), EOG (Usakli & Gurkan, 2010; Usakli et al., 2009), BCI prototyping (Erdogan et al, 2009), feature extraction (Usakli, 2010), this novel system attempts to increase usefulness and performance. HR and GSR signals are to be processed with the bio-signals mentioned above, simultaneously. Evaluation of these signals is whether individually or combined, depends on the user. The situation of the severity of the disease also determines the mode of the hybrid multi-mode operation. Through focusing on one of the tasks and sending correct or wrong messages, the bio signals of the subjects will be acquired and processed. Detecting these changes can be

**4. Hybrid approach of human computer interaction** 

used to send data without speaking or control the device.

Sudirman, 2010).

al., 2007).

(Chavarriaga et al., 2010).

Fig. 10. The block diagram of a NIR system.

### **2.4 Other approaches**

The behavior of active motor units identified via analysis of EMG signals recorded from the first dorsal interosseous muscle using a quadrifilar needle electrode is investigated. According to this study, the motor unit action potential waveforms recorded from patients were more complex than those recorded from control subjects as often observed in motor neuron diseases (Kasi et al., 2009). An eating assistant robot used to assist in eating independence was developed. This assistant robot is useful for people with severe disabilities. A spoon and a camera are attached on the tip of the robotic arm (Takahashi et al., 2001). Additionally, detecting the stress level of the computer user could possibly develop the computers' ability to respond intelligently and help calm negative emotional states of the user during HCI.
