**4. Signal processing method for NIRS-BCI**

#### **4.1 Detection method of brain activity**

NIRS signals also include signals that are not related to brain activity (e.g., measuring device noise, influences of respiration, and fluctuations of blood pressure). Therefore, to evaluate brain activity in detail, it is necessary to remove these extraneous signals. For this purpose, we selected the channels that have a good S/N ratio and exhibit remarkable task-related fluctuations (44 channels (contralateral motor cortex) in this case), and subjected the NIRS signals obtained from them to magnetic resonance angiogram analysis (MRA) (Kojima et. al, 2006, Yanagisawa et. al, 2009, Tsunashima et. al, 2009) through discrete wavelet transformation to decompose and reconfigure the signals.

Brain-Computer Interface Using Near-Infrared Spectroscopy for Rehabilitation 345

Reconstructed signals from channel 44 are illustrated in Figure 7. Results revealed that oxygenated hemoglobin increased and the brain was activated during grasping tasks.

oxy-Hb

100 125 150 175 200 225 250 Time (s)

a. Original signal


Hb conc. (mM

Hb conc. (mM -0.04 -0.03 -0.02 -0.01 0 0.01 0.02

 cm)

 cm)

b. Reconstructed NIRS signal

al, 2009 ).

occurring.

Fig. 7. Comparison of original signal and reconstructed signal (channel44)

We propose the detection method of brain activity. NIRS can measure changes in concentrations of oxy-Hb and deoxy-Hb, as well as the total-Hb signal (the sum of these two values). In this study, we focus on the oxy-Hb signal because changes in concentration of oxy-Hb are highly correlated with the regional cerebral blood flow (rCBF) (Sakatani et, al, 2007), and an increase in rCBF reflects an increase in neural activity (Jueptner, et. al, 1195) Furthermore, we evaluate brain activity using two indexes, oxy-Hb and its differential, because the differential of oxy-Hb is correlated with the workload of the task (Shimizu T. et.

Time (s)

100 125 150 175 200 225 250

Common fluctuations of oxy-Hb and its differential are depicted in Fig. 8 (a). As the detection method, we propose that oxy-Hb be taken on the horizontal axis and the differential of oxy-Hb on the vertical axis, and detection be made from the area of its trajectory. The trajectory of the activity in Fig. 8(a) is depicted in Fig. 8(b). When the trajectory passes through the red zone in Fig. 8(b), it can be assumed that brain activity is

Figure 6 presents the MRA results for oxy-Hb in the 44 channels, where task-related changes were remarkable. The components of the very short frequency bands of d1 and d2 correspond to the measurement noises. The components of d3 and d4 which contains the signal period from 3 to 4 seconds correspond to the breathing. The components of d5 and d6 include changes of blood pressure (Elwell C.E et al. ,1999). The trend of the whole experiment was extracted on the approximated component (a10). Because the interval of repetition of tasks and rests was 50sec, the d8 component was the central component of taskrelated changes. Therefore, signals were reconstructed by adding the d7,d8,d9 and d10 components.

Fig. 6. Decomposition of NIRS signal (channel 44)

Reconstructed signals from channel 44 are illustrated in Figure 7. Results revealed that oxygenated hemoglobin increased and the brain was activated during grasping tasks.

b. Reconstructed NIRS signal

344 Infrared Spectroscopy – Life and Biomedical Sciences

Figure 6 presents the MRA results for oxy-Hb in the 44 channels, where task-related changes were remarkable. The components of the very short frequency bands of d1 and d2 correspond to the measurement noises. The components of d3 and d4 which contains the signal period from 3 to 4 seconds correspond to the breathing. The components of d5 and d6 include changes of blood pressure (Elwell C.E et al. ,1999). The trend of the whole experiment was extracted on the approximated component (a10). Because the interval of repetition of tasks and rests was 50sec, the d8 component was the central component of taskrelated changes. Therefore, signals were reconstructed by adding the d7,d8,d9 and d10

0 50 100 150 200 250

0 50 100 150 200 250

0 50 100 150 200 250

0 50 100 150 200 250

0 50 100 150 200 250

0 50 100 150 200 250

0 50 100 150 200 250

0 50 100 150 200 250

0 50 100 150 200 250

0 50 100 150 200 250

0 50 100 150 200 250

components.

Fig. 6. Decomposition of NIRS signal (channel 44)

d1(0.3~0.5s)

d2(0.5~1.0s)

d3(1.0~2.1s)

d4(2.1~4.2s)

d5(4.2~8.3s)

d6(8.3~16.6s)

d7(16.6~33.3s)

d8(33.3~66.6s)

d9(66.6~133s)

d10(133~266s)












a 10(266s~ )

Time (s)

We propose the detection method of brain activity. NIRS can measure changes in concentrations of oxy-Hb and deoxy-Hb, as well as the total-Hb signal (the sum of these two values). In this study, we focus on the oxy-Hb signal because changes in concentration of oxy-Hb are highly correlated with the regional cerebral blood flow (rCBF) (Sakatani et, al, 2007), and an increase in rCBF reflects an increase in neural activity (Jueptner, et. al, 1195) Furthermore, we evaluate brain activity using two indexes, oxy-Hb and its differential, because the differential of oxy-Hb is correlated with the workload of the task (Shimizu T. et. al, 2009 ).

Common fluctuations of oxy-Hb and its differential are depicted in Fig. 8 (a). As the detection method, we propose that oxy-Hb be taken on the horizontal axis and the differential of oxy-Hb on the vertical axis, and detection be made from the area of its trajectory. The trajectory of the activity in Fig. 8(a) is depicted in Fig. 8(b). When the trajectory passes through the red zone in Fig. 8(b), it can be assumed that brain activity is occurring.

Fig. 7. Comparison of original signal and reconstructed signal (channel44)

Brain-Computer Interface Using Near-Infrared Spectroscopy for Rehabilitation 347

d(oxy-Hb)/dt

0.004

oxy-Hb

trajectory depicted in Fig. 8 (b), the trajectory passes through the red zone, where brain


0


Task Rest Activation area

The area where both oxy-Hb and its differential increase (first quadrant) is assumed to be the area where brain activity is occurring. When the differential of oxy-Hb is large, it can be assumed that brain activity is occurring even if the value of oxy-Hb is small. When the activity level is low, such as during rest time, the trajectory tends to draw a circle of the area of activity around the datum point. To avoid judging the trajectory of the signals that pass through the area around the datum point as a sign of activity, a threshold will be set on the differential of oxy-Hb. However, the differential of oxy-Hb does not always continue to increase when activity is occurring, so we assume that activity is also occurring when oxy-Hb is above a threshold level. Furthermore, when the trajectory does not stay in the area for more than a specified period (2.0s), we assume that it is an influence of artifact and that no

Figure 10 presents the detection results with the proposed detection method applied. Comparison with the results of the conventional detection method presented in Fig. 35confirmed the ON detection during all tasks. Furthermore, the threshold was set on the value of oxy-Hb alone in the conventional detection method, so it was impossible to make the ON detection during the first half of a task with a low level of oxy-Hb, and the detection result was delayed even if relevant brain activity could be confirmed. The proposed detection method, however, reduced the delay due to using the differential of oxy-Hb.The proposed detection method did enable highly accurate detection of many subjects,

Fig. 9. Trajectory of oxy-Hb and differential value (grasping task)

activity is assumed to be occurring during the task.

compared with the conventional detection method.

activity is occurring.

**4.2 Detection result** 

a. Change of oxy-Hb and differential value Time (s)

b. Trajectory of oxy-Hb and d(oxy-Hb)/dt

Fig. 8. Relations between oxy-Hb and differential

About grasping tasks, we selected the channels that have a good S/N ratio and exhibit remarkable task-related fluctuations (44 channels (contralateral motor cortex) in this case), and subjected the NIRS signals obtained from them to MRA through discrete wavelet transformation to decompose and reconfigure the signals.

Figure 9 depicts the trajectory with oxy-Hb on the horizontal axis and the differential of oxy-Hb on the vertical axis using the signals from grasping tasks as reconfigured. As with the

Fig. 9. Trajectory of oxy-Hb and differential value (grasping task)

trajectory depicted in Fig. 8 (b), the trajectory passes through the red zone, where brain activity is assumed to be occurring during the task.

The area where both oxy-Hb and its differential increase (first quadrant) is assumed to be the area where brain activity is occurring. When the differential of oxy-Hb is large, it can be assumed that brain activity is occurring even if the value of oxy-Hb is small. When the activity level is low, such as during rest time, the trajectory tends to draw a circle of the area of activity around the datum point. To avoid judging the trajectory of the signals that pass through the area around the datum point as a sign of activity, a threshold will be set on the differential of oxy-Hb. However, the differential of oxy-Hb does not always continue to increase when activity is occurring, so we assume that activity is also occurring when oxy-Hb is above a threshold level. Furthermore, when the trajectory does not stay in the area for more than a specified period (2.0s), we assume that it is an influence of artifact and that no activity is occurring.

#### **4.2 Detection result**

346 Infrared Spectroscopy – Life and Biomedical Sciences

oxy-Hb d(oxy-Hb)/dt

(ii)Task (iii)Rest

Time (s)

d(oxy-Hb)/dt

**+**

**- +**

**-**

(ii)

Differential value (mM

oxy-Hb

 cm/s)

(i)Rest

About grasping tasks, we selected the channels that have a good S/N ratio and exhibit remarkable task-related fluctuations (44 channels (contralateral motor cortex) in this case), and subjected the NIRS signals obtained from them to MRA through discrete wavelet

Task Rest Activation area

(iii)

Figure 9 depicts the trajectory with oxy-Hb on the horizontal axis and the differential of oxy-Hb on the vertical axis using the signals from grasping tasks as reconfigured. As with the

a. Change of oxy-Hb and differential value

Hb. conc

 (mM cm)

b. Trajectory of oxy-Hb and d(oxy-Hb)/dt Fig. 8. Relations between oxy-Hb and differential

transformation to decompose and reconfigure the signals.

(i)

Figure 10 presents the detection results with the proposed detection method applied. Comparison with the results of the conventional detection method presented in Fig. 35confirmed the ON detection during all tasks. Furthermore, the threshold was set on the value of oxy-Hb alone in the conventional detection method, so it was impossible to make the ON detection during the first half of a task with a low level of oxy-Hb, and the detection result was delayed even if relevant brain activity could be confirmed. The proposed detection method, however, reduced the delay due to using the differential of oxy-Hb.The proposed detection method did enable highly accurate detection of many subjects, compared with the conventional detection method.

Brain-Computer Interface Using Near-Infrared Spectroscopy for Rehabilitation 349

signals that are transferred; when the signals exceed the threshold, the percutaneous electrical stimulator gives percutaneous electrical stimulation to the biceps brachii muscle to induce an elbow joint refraction movement. Therefore, even those who cannot move their

We conducted experiments using the NIRS-BCI rehabilitation system depicted in Fig. 12. In the experiment, five cycles are performed: each cycle consists of 10sec of pre-task rest, 30sec of task, and 10sec of post-task rest. The threshold is set during the first two cycles, and muscle stimulation is applied when oxy-Hb has exceeded the threshold during the third and

Two kinds of tasks were set: one to perform actual grasping and the other to imagine grasping. The subject was instructed to perform grasping with right hand during actual grasping tasks, and to imagine grasping without moving the hand during the imagined grasping tasks. The biceps brachii muscles of the left arm were stimulated. In either task, the

To consider application of this system for rehabilitating patients with hemiplegia, muscles must be stimulated by detecting brain activity during imagined grasping tasks instead of during actual grasping tasks. However, large differences exist between individuals' brain activity during imagined grasping tasks. Therefore, in this study, we confirm the validity of the proposed detection method by conducting grasping tasks associated with significant brain activity appears remarkably. Furthermore, we will apply the proposed detection

Fig. 12. NIRS-BCI rehabilitation system

**5.2 Detection of brain activity** 

succeeding cycles.

arms can actually participate in rehabilitation.

subject was instructed to rest during rest time.

method to brain activity during imagined grasping tasks.

Fig. 10. Result of ON/OFF decision using proposed method (grasping task)

On the same condition, the subject was instructed to imagine grasping without moving the hand during the tasks. The detection of brain activity during imagined grasping tasks using the proposed detection method is presented in Fig. 11. Also during imagined grasping tasks, as in actual grasping tasks in Fig. 10, the ON detection was confirmed during all tasks. Furthermore, a correct detection could be made with little time delay, due to using the differential of oxy-Hb. These results confirm that the proposed detection method is valid for imagined grasping tasks as well.

Fig. 11. Result of ON/OFF decision using proposed method (imagined grasping tasks)

#### **5. IRS-BCI system for rehabilitation**

#### **5.1 BCI-rehabilitation system**

In the NIRS-BCI rehabilitation system (Fig.12), the signals measured with NIRS are transferred to the analysis workstation in real time. A threshold is set based on oxy-Hb

Fig. 12. NIRS-BCI rehabilitation system

oxy-Hb d(oxy-Hb)/dt ON/OFF

Differential value. (mM

Differential value. (mM

OFF

ON



0.0025

0.005

0

 cm/s) OFF

ON



0.0025

0.005

0

 cm/s)

R T R RR T T

Fig. 10. Result of ON/OFF decision using proposed method (grasping task)

Time (s)

R T R RR T T

100 150 200 250

imagined grasping tasks as well.

Hb conc. (mM

Hb conc. (mM



0

0.02

0.04

 cm) -0.04


0

0.02

0.04

 cm)

**5. IRS-BCI system for rehabilitation** 

**5.1 BCI-rehabilitation system** 

On the same condition, the subject was instructed to imagine grasping without moving the hand during the tasks. The detection of brain activity during imagined grasping tasks using the proposed detection method is presented in Fig. 11. Also during imagined grasping tasks, as in actual grasping tasks in Fig. 10, the ON detection was confirmed during all tasks. Furthermore, a correct detection could be made with little time delay, due to using the differential of oxy-Hb. These results confirm that the proposed detection method is valid for

oxy-Hb d(oxy-Hb)/dt ON/OFF

Fig. 11. Result of ON/OFF decision using proposed method (imagined grasping tasks)

Time (s)

100 150 200 250

In the NIRS-BCI rehabilitation system (Fig.12), the signals measured with NIRS are transferred to the analysis workstation in real time. A threshold is set based on oxy-Hb signals that are transferred; when the signals exceed the threshold, the percutaneous electrical stimulator gives percutaneous electrical stimulation to the biceps brachii muscle to induce an elbow joint refraction movement. Therefore, even those who cannot move their arms can actually participate in rehabilitation.

### **5.2 Detection of brain activity**

We conducted experiments using the NIRS-BCI rehabilitation system depicted in Fig. 12. In the experiment, five cycles are performed: each cycle consists of 10sec of pre-task rest, 30sec of task, and 10sec of post-task rest. The threshold is set during the first two cycles, and muscle stimulation is applied when oxy-Hb has exceeded the threshold during the third and succeeding cycles.

Two kinds of tasks were set: one to perform actual grasping and the other to imagine grasping. The subject was instructed to perform grasping with right hand during actual grasping tasks, and to imagine grasping without moving the hand during the imagined grasping tasks. The biceps brachii muscles of the left arm were stimulated. In either task, the subject was instructed to rest during rest time.

To consider application of this system for rehabilitating patients with hemiplegia, muscles must be stimulated by detecting brain activity during imagined grasping tasks instead of during actual grasping tasks. However, large differences exist between individuals' brain activity during imagined grasping tasks. Therefore, in this study, we confirm the validity of the proposed detection method by conducting grasping tasks associated with significant brain activity appears remarkably. Furthermore, we will apply the proposed detection method to brain activity during imagined grasping tasks.

Brain-Computer Interface Using Near-Infrared Spectroscopy for Rehabilitation 351

R T T R R T R

R:Rest T:Task oxy-Hb signal

Time (s)

oxy-Hb d(oxy-Hb)/dt signal

OFF


Differential

value (mM

 cm/s)

OFF

ON

ON

100 125 150 175 200 225 250

Fig. 14. Result of ON/OFF decision using conventional method

Fig. 15. Result of ON/OFF decision using proposed method (grasping task)

detection method.


Hb. Conc (mM cm) -0.4

R:Rest T:Task


0

Hb conc.

0.2

0.4

0.6

well.

Some problems exist in the proposed detection method. For example, the ON detection was made during the rest time before starting the task because the differential had increased, and the OFF detection was made during the second task because the differential of oxy-Hb decreased during the rest time between tasks. However, the proposed detection method did enable highly accurate detection of many subjects, compared with the conventional

Time (s)

100 125 150 175 200 225 250

R T T R R T R

The detection of brain activity during imagined grasping tasks using the proposed detection method is presented in Fig. 16. Also during imagined grasping tasks, as in actual grasping tasks in Fig. 15, the ON detection was confirmed during all tasks. Furthermore, a correct detection could be made with little time delay, due to using the differential of oxy-Hb. These results confirm that the proposed detection method is valid for imagined grasping tasks as

Brain activity in the motor area was measured using NIRS. The measurement device was a near-infrared device made by Shimadzu Corporation (multichannel NIRS instrument, OMM-2001, Shimadzu Corporation, Japan). Figure 13 depicts a scene of the experiment. With 4×4 probes arranged on the left and right sides, measurement was performed with a total of 48 channels (Fig. 4). Seven healthy males in their twenties were selected as subjects, and informed consent was obtained from them before conducting the experiment after explanation of the experiment purpose.

Fig. 13. Scene of the experiment

In the conventional detection method, the z-score was established from the mean value of the first two cycles of the signals measured by NIRS, and the standard deviation and the moving average were obtained. The threshold was set to 20% of the maximum oxy-Hb value obtained during the first two cycles; detection was made on the third and succeeding cycles, where the values of oxy-Hb that exceeded the threshold were judged as ON.

Changes in concentration of oxy-Hb during grasping tasks and the detection results are presented in Fig. 14 ON detection was observed during the first task; however, it was confirmed that no ON detection was made during the succeeding tasks, even though some changes in oxy-Hb concentration of were detected.

Figure 15 presents the detection results with the proposed detection method applied. Comparison with the results of the conventional detection method presented in Fig. 14 confirmed the ON detection not only during the first task but during all tasks. Furthermore, the threshold was set on the value of oxy-Hb alone in the conventional detection method, so it was impossible to make the ON detection during the first half of a task with a low level of oxy-Hb, and the detection result was delayed even if relevant brain activity could be confirmed. The proposed detection method, however, reduced the delay due to using the differential of oxy-Hb.

Brain activity in the motor area was measured using NIRS. The measurement device was a near-infrared device made by Shimadzu Corporation (multichannel NIRS instrument, OMM-2001, Shimadzu Corporation, Japan). Figure 13 depicts a scene of the experiment. With 4×4 probes arranged on the left and right sides, measurement was performed with a total of 48 channels (Fig. 4). Seven healthy males in their twenties were selected as subjects, and informed consent was obtained from them before conducting the experiment after

In the conventional detection method, the z-score was established from the mean value of the first two cycles of the signals measured by NIRS, and the standard deviation and the moving average were obtained. The threshold was set to 20% of the maximum oxy-Hb value obtained during the first two cycles; detection was made on the third and succeeding cycles,

Changes in concentration of oxy-Hb during grasping tasks and the detection results are presented in Fig. 14 ON detection was observed during the first task; however, it was confirmed that no ON detection was made during the succeeding tasks, even though some

Figure 15 presents the detection results with the proposed detection method applied. Comparison with the results of the conventional detection method presented in Fig. 14 confirmed the ON detection not only during the first task but during all tasks. Furthermore, the threshold was set on the value of oxy-Hb alone in the conventional detection method, so it was impossible to make the ON detection during the first half of a task with a low level of oxy-Hb, and the detection result was delayed even if relevant brain activity could be confirmed. The proposed detection method, however, reduced the delay due to using the

where the values of oxy-Hb that exceeded the threshold were judged as ON.

changes in oxy-Hb concentration of were detected.

explanation of the experiment purpose.

Fig. 13. Scene of the experiment

differential of oxy-Hb.

Fig. 14. Result of ON/OFF decision using conventional method

Fig. 15. Result of ON/OFF decision using proposed method (grasping task)

Some problems exist in the proposed detection method. For example, the ON detection was made during the rest time before starting the task because the differential had increased, and the OFF detection was made during the second task because the differential of oxy-Hb decreased during the rest time between tasks. However, the proposed detection method did enable highly accurate detection of many subjects, compared with the conventional detection method.

The detection of brain activity during imagined grasping tasks using the proposed detection method is presented in Fig. 16. Also during imagined grasping tasks, as in actual grasping tasks in Fig. 15, the ON detection was confirmed during all tasks. Furthermore, a correct detection could be made with little time delay, due to using the differential of oxy-Hb. These results confirm that the proposed detection method is valid for imagined grasping tasks as well.

Brain-Computer Interface Using Near-Infrared Spectroscopy for Rehabilitation 353

Hochberg LR., Serruya MD., Friehs GM., Mukand JA., Seleh M., Caplan AH, Branner A.,

devices by a human with tetraplegia", Nature Vol.442 (7099), pp.164 -171 Pfurtscheller G., Leeb, R., Keinrath C., Friedman D., Neuper, C. (2006)"Walking from

Pfurtscheller G., Guger C., Müller G., Krausz G., Neuper C. (2000), "Brain oscillations control hand orthosis in a tetraplegic", Neurosci Lett Vol.292 (3), pp.211-214 Pfurtscheller G., Müller GR., Pfurtscheller J., Gerner HJ., Rupp R. (2003), "'Thought' -

Vaughan TM, McFarland DJ, Schalk G, Sarnacki WA, Krusienski DJ, Sellers EW, Wolpaw JR.

Nagaoka T., Sakatani K, Awano T., Yokose N., Hoshino T., Murata Y., Katayama Y.,

Jöbsis, F.F. (1977), "Non-invasive infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters", Science, Vol. 198, pp. 1264-1267 Huettel, S. A. (2004). *Functional Magnetic Resonance Imaging,* Sinauer Associate, Inc., ISBN

Kojima T., Tsunashima H., Shiozawa T., Takada H., Sakai T. (2006), "Measurement of Train

Yanagisawa K., Tsunashima H., Marumo Y., Hirose S., Shimizu T., Taira M., Haji T. (2009),

Tsunashima H., Yanagisawa K. (2009), "Measurement of brain function of car driver using

Elwell C.E., Springett R., Hillman E., Delpy D.T. (1999), "Oscillations in Cerebral

Sakatani K., Murata Y., Fujiwara N., Hoshino T., Nakamura S., Kano T., Katayama Y. (2007),

Jueptner, M., Willer,C. (1995), "Dose measurement of regogional cerebral blood flow reflects

Shimizu T., Hirose S., Obara H., Yanagisawa K., Tsunashima H., Marumo Y., Haji T., Taira

Experimental Medicine and Biology, Vol. 471, pp. 57-65

thought", Brain Res Vol.1071 (1), pp.145-152

tetraplegia", Neurosci Lett 351, pp.33-36

and Other Transit Systems, pp. 245-254

978-0878932887, USA

183-191 (in Japanese)

Neuroscience, Vol.2009

Vol. 12 (6), p. 062110

BCI", IEEE Trans Neural Syst Eng Vol.14 (2), pp.229 -233

*Experimental Medicine and Biology*, Vol. 662, pp. 497-503

Chen D., Penn RD, Donoghue JP. (2006), "Neuronal ensemble control of prosthetic

control of functional electrical stimulation to restore hand grasp in a patient with

(2006), "The Wadsworth BCI Research and Development Program: at home with

Ishikawa A., Eda H. (2010), "Development of a new rehabilitation system based on brain-computer interface using near infrared spectroscopy," *Advances in* 

Driver's Brain Activity by Functional Near-Infrared Spectroscopy (fNIRS)," Computers in Railways X: Computer System Design and Operation in the Railway

"Measurement and evaluation of higher brain function by using functional nearinfrared spectroscopy (fNIRS)", Journal of Human Interface Society, Vol. 11(2) , pp.

functional near-infrared spectroscopy (fNIRS)," Computational Intelligence and

Haemodynamics, Implications for Functional Activation Studies", Advances in

"Comparison of blood-oxygen-level-dependent functional magnetic resonance imaging and near-infrared spectroscopy recording during functional brain activation in patients with stroke and brain tumors," Journal of Biomedical Optics,

synaptic activity?- implications for PET and fMRI," Neuroimage, Vol. 2, pp.148-156

M. (2009), "Measurement of frontal cortex brain activity attributable to the driving

**8. References** 

Fig. 16. Result of ON/OFF decision using proposed method (imagined grasping tasks)
