**Measurement of Brain Function Using Near-Infrared Spectroscopy (NIRS)**

Hitoshi Tsunashima, Kazuki Yanagisawa and Masako Iwadate *Nihon University Japan* 

### **1. Introduction**

74 Neuroimaging – Methods

Villringer, A., Planck, J., Hock, C., Schleinkofer, L. & Dirnagl, U. (1993). Near infrared

Villringer, A. & Chance, B. (1997). Non-invasive optical spectroscopy and imaging of human

Ward, K.R., Ivatury, R.R., Barbee, R.W., Terner, J., Pittman, R., Filho, I.P. & Spiess, B. (2006).

Wharton, D.C. & Tzagoloff, A. (1964). Studies on the electron transfer system. LVII. The near

Williams L.M., Phillips M.L., Brammer M.J., Skerrett D., Lagopoulos J., Rennie C., Bahramali

Wobst, P., Wenzel, R., Kohl, M., Obrig, H. & Villringer, A. (2001). Linear aspects of changes

Wolf, M., Wolf, U., Choi, J.H., Gupta, R., Safonova, L.P., Paunescu, L.A., Michalos, A., &

Wolf, M., Ferrari, M. & Quaresima, V. (2007). Progress of near-infrared spectroscopy and

Zabel, T.A. & Chute, D.L. (2002). Educational neuroimaging: a proposed neuropsychological

Zygun, D.A., Laupland, K.B., Hader, W.J., Kortbeek, J.B., Findlay, C., Doig, C.J. & Hameed,

*rehabilitation*, Vol. 17, No, 5, (October 2002), pp. 477-88, ISSN 1550-509X Zaramella, P., Freato, F., Amigoni, A., Salvadori, S., Marangoni, P., Suppjei, A., Schiavo, B. &

Vol. 12, No. 6, (November-December 2007), pp. 062104, ISSN 1560-2281 Wong Riley, M.T. (1989). Cytochrome oxidase: an endogenous metabolic marker for

Vol. 22, No. 1, (May 2004), pp.109-19, ISSN 1095-9572

No. 6592, (August 1996), pp. 629-32, ISSN 1476-4687

Vol. 239, (June 1964), pp. 2036-41, ISSN 1083-351X

pp. 101-4, ISSN 1872-7972

ISSN 1878-108X

9, ISSN 1095-9572

101, ISSN 1878-108X

pp. 213-9, ISSN 1530-0447

ISSN 0317-1671

2002), pp. 1868-75, ISSN 1095-9572

1095-9572

infrared spectroscopy cannot be explained by a mere cross talk artefact. *Neuroimage*,

spectroscopy (NIRS): a new tool to study hemodynamic changes during activation of brain function in human adults. *Neuroscience letters*, Vol. 154, No. 1-2, (May 1993),

brain function. *Trends in Neuroscience*, Vol. 20, No. 10, (October 1997), pp. 435–442,

Near infrared spectroscopy for evaluation of the trauma patient: a technology review. *Resuscitation*, Vol. 68, No. 1, (January 2006), pp. 27-44, ISSN 1873-1570 Watanabe, M. (1996). Reward expectancy in primate prefrontal neurons. *Nature*, Vol. 382,

infrared absorption band of cytochrome oxidase. *The Journal of biological chemistry*,

H., Olivieri G., David, A.S. Peduto A. & Gordon, E. (2001). Arousal dissociates amygdala and hippocampal fear responses: evidence from simultaneous fMRI and skin conductance recording, *Neuroimage* Vol. 14, No. 5, (November 2001), pp. 1070-

in deoxygenated hemoglobin concentration and cytochrome oxidase oxidation during brain activation. *Neuroimage*, Vol. 13, No. 3, (March 2001), pp. 520-30, ISSN

Gratton, E. (2002). Functional frequency-domain near-infrared spectroscopy detects fast neuronal signal in the motor cortex. *Neuroimage*, Vol. 17, No. 4, (December

topography for brain and muscle clinical applications. *Journal of biomedical optics*.

neuronal activity. *Trends in neurosciences*, Vol. 12, No. 3, pp. (March 1989), pp. 94-

application of near-infrared spectroscopy (nIRS). *The Journal of head trauma* 

Chiandetti, L. (2001). Brain auditory activation measured by near-infrared spectroscopy (NIRS) in neonates. *Pediatric research*, Vol.49, No.2, (February 2001),

S.M. (2005). Severe traumatic brain injury in a large Canadian health region. *Canadian Journal of Neurological Sciences*, Vol. 32, No. 1, (February 2005), pp. 87-92, Near-infrared spectroscopy (NIRS) has gained attention in recent years (Hoshi et al., 2001; Tamura, 2003). This non-invasive technique uses near-infrared light to evaluate increases or decreases in oxygenated hemoglobin or deoxygenated hemoglobin in tissues below the body surface.

NIRS can detect the hemodynamics of the brain in real time while the subject is moving. Brain activity can therefore be measured in various environments. Recent research has used NIRS to measure brain activity in a train driver (Kojima et al., 2005, 2006). NIRS has also been used to evaluate the mental activity of an individual driving a car in a driving simulator (Shimizu et al., 2009).

Various arguments have focused on interpretation of signals obtained from NIRS, and no uniform signal-processing method has yet been established. Averaging and baseline correction are conventional signal-processing methods used for the NIRS signal. These methods require block design, an experimental technique that involves repeating the same stimuli (tasks) and resting multiple times in order to detect brain activation during a task. However, brain activation has been noted to gradually decline when a subject repeats the same task multiple times (Takahashi et al., 2006).

Fourier analysis, which is frequently used for signal analysis, transforms information in the time domain into the frequency domain through the Fourier transform. However, time information is lost in the course of the transform. As the NIRS signal fluctuates, timefrequency analysis is suitable for the NIRS signal.

The wavelet transform is an efficient method for time-frequency analysis (Mallat, 1998). This approach adapts the window width in time and frequency so that the window width in frequency becomes smaller when the window width in time is large, or the window width in frequency becomes larger when the window width in time is small. Multi-resolution analysis (MRA) (Mallat, 1989) decomposes the signal into different scales of resolution. MRA with an orthonormal wavelet base effectively facilitates complete decomposition and reconstruction of the signal without losing original information from the signal.

Oxygenated hemoglobin and deoxygenated hemoglobin as measured in NIRS are relative values from the beginning of measurement and vary between subjects and parts of the brain. Simple averaging of the NIRS signal thus should not be applied for statistical analysis. To solve this problem, we propose the Z-scored NIRS signal.

Measurement of Brain Function Using Near-Infrared Spectroscopy (NIRS) 77

The absorption coefficients of oxygenated hemoglobin and deoxygenated hemoglobin at

be obtained by performing measurements with near-infrared rays of two different wavelengths and solving Equation 3. However, the quantity obtained here is the product of the change in concentration and the averaged path length. In general, this averaged path length *l* varies greatly from one individual to another, and from one part of the brain to

As mental calculation tasks, a low-level task comprising simple one-digit addition (e.g., 3 + 5) and a high-level task comprising subtraction and division with a decimal fraction (e.g., 234/(0.61 − 0.35)), were set to obtain NIRS signals. Brain activity in the prefrontal lobe was measured using NIRS. The measurement instrument was the OMM-3000 multichannel NIRS

Figure 1 illustrates the arrangement of optical-fiber units and the location of each channel (37 matrix, 32 channels). Figure 2 shows the recorded temporal histories of oxygenated hemoglobin (red line, indicated as oxy-Hb) and deoxygenated hemoglobin (blue line,

0 90 180 270 360 450 540

Time [s]

L R HRL RH R L RHR

R oxy-Hb deoxy-Hb :Rest L:Low difficulty task H:High difficulty task

, are known. As a result, *oxy l X* and *deoxy l X* can

each wavelength, ( ) *oxy <sup>i</sup>*

**3.1 Recording of NIRS signal** 

indicated as deoxy-Hb) in channel 20.

 **3. Signal processing methods for NIRS** 

system (Shimadzu Corporation, Japan) (Kohno, 2006).

Illuminator Detectot channel

Fig. 1. Position of optical fibers and channels for recording NIRS signals

Fig. 2. Temporal history of NIRS signals in mental calculation (channel 20)

1 2 3 4 7 8 9 10 11 14 15 16 17 20 21 22 23 24 27 28 29 30

(mental calculation task: matrix, 32 channels)


Hb conc. [mM cm]  and ( ) *deoxy <sup>i</sup>* 

another. Caution must therefore be exercised in evaluating the results.

The aim of this chapter is to propose a signal processing method suitable for the NIRS signal and applicable for neuroimaging studies. We first describe the principle underlying measurement of brain activity with NIRS in Section 2. We then propose discrete waveletbased MRA to extract the task-related signal from original NIRS recordings in Section 3.

In Section 4, we describe simultaneous measurement experiments with NIRS and functional magnetic resonance imaging (fMRI) using mental calculation tasks to confirm the validity of the proposed signal processing method. To investigate relationships between brain blood flow and skin blood flow, measurement is carried out using NIRS and the laser Doppler skin blood flow probe. The Z-scored NIRS signal is proposed for statistical analysis.

Two application examples with the proposed method for NIRS signals are shown in Section 5. We demonstrate the possibility of using the proposed method for evaluating brain activity associated with driving a car in a realistic driving environment. Another application is to brain-computer interfaces (BCIs), which are used in actively conducted studies. A BCI is a system that controls machines and devices by extracting neural information from human brain activity. BCIs are expected to become prominent as nursing robotics, such as artificial hands. The proposed method is applied for BCI systems that can control a robot arm using NIRS. We measured brain activity during actual grasping tasks and imagined grasping tasks using the BCI system to demonstrate the validity of the proposed BCI system.

Finally, some conclusions are given in Section 6.

### **2. Near-infrared Spectroscopy (NIRS)**

Using near-infrared rays, NIRS non-invasively measures changes in cerebral blood flow. The principle of measurement was developed by Jöbsis (1977), based on the measurement of hemoglobin oxygenation in the cerebral blood flow.

In uniformly distributed tissue, incident light is attenuated by absorption and scattering. The following expression, a modified Lambert-Beer law, was therefore used:

$$Abs = -\log(I\_{out} \mid I\_{in}) = \varepsilon \overline{I} \text{C} + \text{S} \text{ .} \tag{1}$$

Here, *Iin* is the irradiated quantity of light; *Iout* is the detected quantity of light; is the absorption coefficient; *C* is the concentration; *l* is the averaged path length; and *S* is the scattering term.

If it is assumed that no scattering changes in brain tissue occur during activation of the brain, the change in absorption across the activation can be expressed by the following expression:

$$
\Delta A bs = -\log(\Delta I\_{out} \, \prime \, \Delta I\_{in}) = \varepsilon \, \bar{l} \, \Delta C (\Delta X\_{oxy} \, \Delta X\_{deoxy}) \, . \tag{2}
$$

Furthermore, if the change in concentration ( *C* ) is assumed to be proportional to the changes in oxygenated hemoglobin ( *Xoxy* ) and deoxygenated hemoglobin ( *Xdeoxy* ), the following relational expression can be obtained:

$$
\Delta A \text{bs} (\mathcal{X}\_i) = I \left[ \varepsilon\_{\alpha xy} (\mathcal{X}\_i) \Delta X\_{\alpha xy} + \varepsilon\_{decay} (\mathcal{X}\_i) \Delta X\_{devxy} \right]. \tag{3}
$$

The absorption coefficients of oxygenated hemoglobin and deoxygenated hemoglobin at each wavelength, ( ) *oxy <sup>i</sup>* and ( ) *deoxy <sup>i</sup>* , are known. As a result, *oxy l X* and *deoxy l X* can be obtained by performing measurements with near-infrared rays of two different wavelengths and solving Equation 3. However, the quantity obtained here is the product of the change in concentration and the averaged path length. In general, this averaged path length *l* varies greatly from one individual to another, and from one part of the brain to another. Caution must therefore be exercised in evaluating the results.

### **3. Signal processing methods for NIRS**

### **3.1 Recording of NIRS signal**

76 Neuroimaging – Methods

The aim of this chapter is to propose a signal processing method suitable for the NIRS signal and applicable for neuroimaging studies. We first describe the principle underlying measurement of brain activity with NIRS in Section 2. We then propose discrete waveletbased MRA to extract the task-related signal from original NIRS recordings in Section 3. In Section 4, we describe simultaneous measurement experiments with NIRS and functional magnetic resonance imaging (fMRI) using mental calculation tasks to confirm the validity of the proposed signal processing method. To investigate relationships between brain blood flow and skin blood flow, measurement is carried out using NIRS and the laser Doppler skin blood flow probe. The Z-scored NIRS signal is proposed for

Two application examples with the proposed method for NIRS signals are shown in Section 5. We demonstrate the possibility of using the proposed method for evaluating brain activity associated with driving a car in a realistic driving environment. Another application is to brain-computer interfaces (BCIs), which are used in actively conducted studies. A BCI is a system that controls machines and devices by extracting neural information from human brain activity. BCIs are expected to become prominent as nursing robotics, such as artificial hands. The proposed method is applied for BCI systems that can control a robot arm using NIRS. We measured brain activity during actual grasping tasks and imagined grasping tasks using the BCI system to demonstrate the

Using near-infrared rays, NIRS non-invasively measures changes in cerebral blood flow. The principle of measurement was developed by Jöbsis (1977), based on the measurement of

In uniformly distributed tissue, incident light is attenuated by absorption and scattering.

log( / ) *A out in bs I I lC S*

absorption coefficient; *C* is the concentration; *l* is the averaged path length; and *S* is the

If it is assumed that no scattering changes in brain tissue occur during activation of the brain, the change in absorption across the activation can be expressed by the following

> log( /) ( , ) *Abs I I l C X X out in oxy deoxy*

Furthermore, if the change in concentration ( *C* ) is assumed to be proportional to the changes in oxygenated hemoglobin ( *Xoxy* ) and deoxygenated hemoglobin ( *Xdeoxy* ), the

() [ () () ] *Abs l X*

 

 *i oxy i oxy deoxy i deox X <sup>y</sup>* . (3)

. (1)

. (2)

is the

The following expression, a modified Lambert-Beer law, was therefore used:

Here, *Iin* is the irradiated quantity of light; *Iout* is the detected quantity of light;

statistical analysis.

scattering term.

expression:

validity of the proposed BCI system.

Finally, some conclusions are given in Section 6.

hemoglobin oxygenation in the cerebral blood flow.

following relational expression can be obtained:

 

**2. Near-infrared Spectroscopy (NIRS)** 

As mental calculation tasks, a low-level task comprising simple one-digit addition (e.g., 3 + 5) and a high-level task comprising subtraction and division with a decimal fraction (e.g., 234/(0.61 − 0.35)), were set to obtain NIRS signals. Brain activity in the prefrontal lobe was measured using NIRS. The measurement instrument was the OMM-3000 multichannel NIRS system (Shimadzu Corporation, Japan) (Kohno, 2006).

Figure 1 illustrates the arrangement of optical-fiber units and the location of each channel (37 matrix, 32 channels). Figure 2 shows the recorded temporal histories of oxygenated hemoglobin (red line, indicated as oxy-Hb) and deoxygenated hemoglobin (blue line, indicated as deoxy-Hb) in channel 20.

Fig. 1. Position of optical fibers and channels for recording NIRS signals (mental calculation task: matrix, 32 channels)

Fig. 2. Temporal history of NIRS signals in mental calculation (channel 20)

Measurement of Brain Function Using Near-Infrared Spectroscopy (NIRS) 79

Figure 4 shows the result of baseline correction applied for NIRS signals (Fig. 2) after removing high-frequency noise using a moving average of 25 data. Figure 5 indicates the functional brain imaging of the frontal lobe. It should be noted that brain activation

0 90 180 270 360 450 540

① ② ③


0.08

Hb conc. (mM cm)

0

( )*t* , and then analyzes the

Time [s]

①② ③

Fig. 4. Results of signal processing with baseline correction and de-noising

However, time information is lost in the course of the transform.

in frequency becomes larger when the window width in time is small.

shifting and dilating the waveform called the mother wavelet,

which is dilated with a scale parameter *a* and translated by *b* as

is a function of zero average

**3.3 Decomposition and reconstruction of NIRS signals using wavelet transform** 

Fourier analysis, which is frequently used for frequency analysis, transforms information in the time domain into information in the frequency domain through the Fourier transform.

A short-time Fourier transform, or windowed Fourier transform, can be used for timefrequency analysis of signals. However, the detection capacity varies widely, depending on

In contrast, wavelet transform is an efficient method of time-frequency analysis. This approach adapts the window width in time and frequency so that the window width in frequency becomes smaller when the window width in time is large, or the window width

Wavelet transform expresses the local shape of the waveform to be analyzed, *S t*( ) , by

() 0 *t dt* (4)

L R HRL RHR L RHR

R oxy-Hb deoxy-Hb :Rest L:Low difficulty task H:High difficulty task

gradually declines when a subject repeats the same task multiple times.


**3.3.1 Wavelet transform** 

the setting of the window.

waveform. A wavelet

Hb conc. [mM cm]

### **3.2 Analysis of NIRS signals**

In NIRS signal analysis, brain activity related to a task must be separated from that which is not, since NIRS measures not only signals of brain activity during a task, but also other signals, including measurement noise.

In general, changes in oxygenated hemoglobin and deoxygenated hemoglobin when the brain is activated and restored to the original state exhibit the trend illustrated in Figure 3 (Huettel, 2004). Therefore, if these signals can be extracted from the measured signals, the brain has obviously been activated.

Averaging and baseline correction are conventional signal-processing methods. These methods require block design, an experimental technique that involves repeating the same stimuli (tasks) and resting multiple times in order to detect brain activation during a task.

Averaging is the method by which data are averaged for each task. Randomly generated noise approaches zero with averaging, and only periodic data are left. Averaging is effective when similar reactions are generated repeatedly. However, for cerebral blood flow that shows large variations in reactions to the same stimuli, the reliability of averaged signals is low, and false signals may be created. Furthermore, even significant signals may become undetectable after averaging.

Baseline correction corrects the start and end points of a block to zero to remove gradual trends, based on the assumption that blood flow is restored to its original state during a task block. However, because blood flow involves irregular fluctuations, the reference points are unstable. Therefore, if the whole block is corrected based on those two points alone, signals may be distorted.

Fig. 3. Schematic of changes in hemoglobin concentration due to neural activity

In NIRS signal analysis, brain activity related to a task must be separated from that which is not, since NIRS measures not only signals of brain activity during a task, but also other

In general, changes in oxygenated hemoglobin and deoxygenated hemoglobin when the brain is activated and restored to the original state exhibit the trend illustrated in Figure 3 (Huettel, 2004). Therefore, if these signals can be extracted from the measured signals, the

Averaging and baseline correction are conventional signal-processing methods. These methods require block design, an experimental technique that involves repeating the same stimuli (tasks) and resting multiple times in order to detect brain activation during a task. Averaging is the method by which data are averaged for each task. Randomly generated noise approaches zero with averaging, and only periodic data are left. Averaging is effective when similar reactions are generated repeatedly. However, for cerebral blood flow that shows large variations in reactions to the same stimuli, the reliability of averaged signals is low, and false signals may be created. Furthermore, even significant signals may become

Baseline correction corrects the start and end points of a block to zero to remove gradual trends, based on the assumption that blood flow is restored to its original state during a task block. However, because blood flow involves irregular fluctuations, the reference points are unstable. Therefore, if the whole block is corrected based on those two points alone, signals

0 2 4 6 8 10 12

Oxygenated hemoglobin

> Deoxygenated hemoglobin

Time [s]

Fig. 3. Schematic of changes in hemoglobin concentration due to neural activity

**3.2 Analysis of NIRS signals** 

signals, including measurement noise.

brain has obviously been activated.

undetectable after averaging.


0

Relativconerati

Relative concentration

1

2

3

may be distorted.

Figure 4 shows the result of baseline correction applied for NIRS signals (Fig. 2) after removing high-frequency noise using a moving average of 25 data. Figure 5 indicates the functional brain imaging of the frontal lobe. It should be noted that brain activation gradually declines when a subject repeats the same task multiple times.

Fig. 4. Results of signal processing with baseline correction and de-noising

### **3.3 Decomposition and reconstruction of NIRS signals using wavelet transform 3.3.1 Wavelet transform**

Fourier analysis, which is frequently used for frequency analysis, transforms information in the time domain into information in the frequency domain through the Fourier transform. However, time information is lost in the course of the transform.

A short-time Fourier transform, or windowed Fourier transform, can be used for timefrequency analysis of signals. However, the detection capacity varies widely, depending on the setting of the window.

In contrast, wavelet transform is an efficient method of time-frequency analysis. This approach adapts the window width in time and frequency so that the window width in frequency becomes smaller when the window width in time is large, or the window width in frequency becomes larger when the window width in time is small.

Wavelet transform expresses the local shape of the waveform to be analyzed, *S t*( ) , by shifting and dilating the waveform called the mother wavelet, ( )*t* , and then analyzes the waveform.

A wavelet is a function of zero average

$$\int\_{-\infty}^{+\infty} \psi'(t)dt = 0\tag{4}$$

which is dilated with a scale parameter *a* and translated by *b* as

Measurement of Brain Function Using Near-Infrared Spectroscopy (NIRS) 81

, , *m mn mn*( )

*Daubechies* wavelet (Daubechies, 1992), which is orthonormal base and is a compactly supported wavelet. The vanishing moments of the *Daubechies* wavelet can be changed by an

> L H 60 30 3060

R R R R R R

Time [s]

Fig. 5. Decomposition of NIRS signal using a wavelet base (*Daubechie* 7)

0

*m m m m St a d*

. (12)

. (13)

L H

R:Rest

60 30 60 30 [s]

L: Low difficulty task H: High difficulty task

*m n*, is important. We employed a

*dDt* 

*n*

<sup>0</sup> ( )

Task-related components can thus be reconstructed from detailed components *dm* .

index *N*. We decided to use a relatively high-order generating index, *N* = 7.

Thus, signal *S t*( ) can be expressed as

*d*1(0.3 - 0.6s)

*d*2(0.6 - 1.3s)

*d*3(1.3 - 2.6s)

*d*4(2.6 - 5.1s)

*d*5(5.1 - 10.2s)

*d*6(10.2 - 20.5s)

*d*7(20.5 - 41.0s)

*d*8(41.0 - 82.0s)

*d*9(82.0 - 164s)

*d*10(164 - 328s)

*a*10(328s - )

*S*(*t*)

In the wavelet transform, the choice of a mother wavelet

L H 5 60 30 60 30 
$$
\psi\_{a,b}(t) = \frac{1}{\sqrt{a}} \nu(\frac{t-b}{a})\,. \tag{5}
$$

The continuous wavelet transform of signal *S t*( ) is computed with , ( ) *a b t* as

$$
\tilde{S}(a,b) = \int\_{-\infty}^{\infty} S(t) \boldsymbol{\nu}\_{a,b}^\*(t) dt \,. \tag{6}
$$

Here, \* denotes the complex conjugate of .

One can construct wavelets such that the dilated and translated function

$$
\hat{\Psi}\_{m,n}(t) = 2^{-m/2} \hat{\Psi}(2^{-m}t - n) \tag{7}
$$

is an orthonormal base. Discrete wavelet transform can be computed by

$$D\_m = \int\_{-\infty}^{\infty} S(t) \varphi\_{m,n}(t) dt \,. \tag{8}$$

In the continuous wavelet transform, information is duplicated, requiring many calculations. Discrete wavelet transform handles a smaller volume of information than continuous wavelet transform, but is able to transform signals more efficiently. Furthermore, use of an orthonormal base facilitates complete reconstruction of original signals without redundancy. The following section describes decomposition and reconstruction of signals using MRA.

### **3.3.2 Multi-Resolution Analysis (MRA)**

MRA decomposes signals into a tree structure using the discrete wavelet transform. In the case of the object time-series signals, *S t*( ) , it decomposes the signals into an approximated component (low-frequency component) and multiple detailed components (high-frequency components).

A signal *S t*( ) can be expressed as follows by discrete wavelet transform using an orthonormal base *m n*, as

$$S(\mathbf{t}) = \sum\_{n=-\infty}^{\infty} A\_{m\_0,n} \phi\_{m\_0,n}(\mathbf{t}) + \sum\_{m=-\infty}^{m\_0} \sum\_{n=-\infty}^{\infty} D\_{m,n} \varphi\_{m,n}(\mathbf{t}) \,. \tag{9}$$

Here, , *m n*( )*t* is the scaling function as defined by the following equation:

$$
\phi\_{m,n}(t) = 2^{-m/2} \phi(2^{-m}t - n) \,. \tag{10}
$$

The coefficient of the approximated component is calculated by

$$A\_{m,n} = \int\_{-\infty}^{\infty} S(t)\phi\_{m,n}(t) \,. \tag{11}$$

The detailed components of the signals on level *m* can be expressed by

$$d\_m = \sum\_{n=-\infty}^{\infty} D\_{m,n} \varphi\_{m,n}(t) \,. \tag{12}$$

Thus, signal *S t*( ) can be expressed as

80 Neuroimaging – Methods

<sup>1</sup> () ( ) *a b t b <sup>t</sup> a a*

, ( , ) () () *S a b S t t dt a b*

.

/2 , ( ) 2 (2 ) *m m*

, *m m* () () *D S t t dt n*

In the continuous wavelet transform, information is duplicated, requiring many calculations. Discrete wavelet transform handles a smaller volume of information than continuous wavelet transform, but is able to transform signals more efficiently. Furthermore, use of an orthonormal base facilitates complete reconstruction of original signals without redundancy. The following section describes decomposition and

MRA decomposes signals into a tree structure using the discrete wavelet transform. In the case of the object time-series signals, *S t*( ) , it decomposes the signals into an approximated component (low-frequency component) and multiple detailed components (high-frequency

A signal *S t*( ) can be expressed as follows by discrete wavelet transform using an

*n m n St A t D t*

*m n*( )*t* is the scaling function as defined by the following equation:

*m n* 

The detailed components of the signals on level *m* can be expressed by

The coefficient of the approximated component is calculated by

/2 , ( ) 2 (2 ) *m m*

, , *A St t m n* () () *m n*

 

0 0 0 , , , , ( ) ( ) ( ) *m mnmn mn mn*

 

*t t n* . (10)

. (11)

. (9)

 

 

\*

such that the dilated and translated function

. (5)

*t* as

. (6)

*t t n* (7)

. (8)

,

The continuous wavelet transform of signal *S t*( ) is computed with , ( ) *a b*

*m n* 

is an orthonormal base. Discrete wavelet transform can be computed by

denotes the complex conjugate of

Here, \* 

components).

Here, , 

orthonormal base

One can construct wavelets

reconstruction of signals using MRA.

**3.3.2 Multi-Resolution Analysis (MRA)** 

*m n*, as

$$S(t) = a\_{m\_0} + \sum\_{m=-\infty}^{m\_0} d\_m \,. \tag{13}$$

Task-related components can thus be reconstructed from detailed components *dm* .

In the wavelet transform, the choice of a mother wavelet *m n*, is important. We employed a *Daubechies* wavelet (Daubechies, 1992), which is orthonormal base and is a compactly supported wavelet. The vanishing moments of the *Daubechies* wavelet can be changed by an index *N*. We decided to use a relatively high-order generating index, *N* = 7.

Fig. 5. Decomposition of NIRS signal using a wavelet base (*Daubechie* 7)

Measurement of Brain Function Using Near-Infrared Spectroscopy (NIRS) 83

<sup>L</sup> Rest Task

16 28 36 28 36 28 36 28 36 28 36 28 36 28 36 28 36 28 36 [s]

A 28s task consisted of 14 questions at 2s intervals for the low level, 10 questions at 2.8s intervals for the intermediate level, or two questions at 14s intervals for the high level. The subject answered the questions displayed on the computer monitor without speaking. During the 36s rest time, the subject rested while steadily gazing at a cross mark displayed

Brain activity in the prefrontal lobe was measured using NIRS and fMRI simultaneously. NIRS data were collected using the OMM-3000 system (Shimadzu Corporation, Japan) in

Data from fMRI (3 mm thickness, 40 slices) were collected using a Siemens Symphony 1.5T (T2\*-weighted gradient-echo sequence, TR = 4000 ms, TE = 50 ms, FA = 90 degrees, 6464 pixels, FOV = 192 mm). Whole-brain images were obtained as T1-weighted images (TR =

Data from fMRI were pre-processed using Statistical Parametric Mapping (SPM99; Welcome Department of Imaging Neuroscience, UK) Normalized contrast images were smoothed with an isotropic Gaussian kernel (FWHM = 12 mm). Regions of interest (ROIs) were defined as clusters of 10 or more voxels in which estimated parameter values differed

Subjects were nine healthy volunteers (nine men, no women). The arrangement of optical

Figure 8 presents measurement results through all channels for Subject A during the first three tasks. During the mental calculation task at the high level (i.e., the third task), oxygenated hemoglobin increased and deoxygenated hemoglobin decreased on both outer

Figure 9 presents MRA results for oxygenated hemoglobin in channel 26, where task-related changes were marked. The trend of the whole experiment was extracted on the approximated component (*a*10). As the interval of repetition of tasks and rests was 64 s, the *d*8 component was the central component of task-related changes. Signals were thus

Reconstructed signals from channel 26 are illustrated in Figure 10. Results revealed that oxygenated hemoglobin increased and the brain was activated during mental calculation tasks. Furthermore, such changes became larger as the level of mental calculation task

Figure 11 shows a comparison of functional brain imaging by fMRI and NIRS with the proposed method. The rectangle in the fMRI image indicates the region of measurement with NIRS. NIRS images agree with those for fMRI at a different workload level. This result

supports the effectiveness of MRA with the discrete wavelet transform.

2200 ms, TE = 3.93 ms, FA = 15 degrees, TI = 1100 ms, 1 mm3 voxel, FOV = 256 mm).

<sup>H</sup> Rest Task

<sup>L</sup> Rest Task

<sup>R</sup> Task

<sup>L</sup> Rest Task

Fig. 7. Experimental design

on the computer monitor.

the MRI scanner.

**4.2 NIRS and fMRI signal recording** 

significantly from zero (p<0.01).

**4.3 Signal processing** 

sides of the frontal lobe.

increased.

<sup>M</sup> Rest Task

<sup>H</sup> Rest Task

fiber units and measurement positions is shown in Figure 1.

reconstructed by adding the *d*7, *d*8, and *d*9 components.

<sup>M</sup> Rest Task

L: Low difficulty task M: Middle difficulty task H: High difficulty task

<sup>H</sup> Rest Task

<sup>M</sup> Rest

Figure 5 presents MRA results for oxygenated hemoglobin in channel 20, where task-related changes were marked. Here, the measured signal was decomposed into ten levels. The trend of the whole experiment was extracted on the approximated component (*a*10), in the lowestfrequency range.

Here, *d*1 and *d*2, the highest frequency ranges, showed a relatively large amplitude. It is possible that these ranges represented measurement noise. As the interval for repetition of tasks and rests was 64 s, the *d*8 component was the central component of task-related changes. Signals were therefore reconstructed by adding the *d*7, *d*8, and *d*9 components.

Reconstructed signals are illustrated in Figure 6. Of note, the activation pattern of oxygenated hemoglobin and deoxygenated hemoglobin, shown in Figure 3, can be observed very clearly. Comparison between Figure 4 (signal processing with baseline correction and de-noising) and Figure 6 (wavelet-based method) shows the improved performance of the proposed method. Results revealed that oxygenated hemoglobin increased and the brain was activated during mental calculation tasks.

Fig. 6. Results of brain imaging using reconstructed signals

### **4. Measurement of brain functions under mental workload**

### **4.1 Setting of workload**

To confirm the validity of the signal-processing method explained in the previous section, we measured brain functions through simultaneous use of NIRS and fMRI.

To measure brain activity under workload, we used the workload of mental calculation. Mental calculation tasks were set to low, medium, and high levels as follows:


The design of the experiment is presented in Figure 7. Each set was composed of 28 s of task and 36 s of rest, in that order. By arranging three sets for each level in random order, a total of nine sets of experiment were conducted over 592 s.

Fig. 7. Experimental design

Figure 5 presents MRA results for oxygenated hemoglobin in channel 20, where task-related changes were marked. Here, the measured signal was decomposed into ten levels. The trend of the whole experiment was extracted on the approximated component (*a*10), in the lowest-

Here, *d*1 and *d*2, the highest frequency ranges, showed a relatively large amplitude. It is possible that these ranges represented measurement noise. As the interval for repetition of tasks and rests was 64 s, the *d*8 component was the central component of task-related changes. Signals were therefore reconstructed by adding the *d*7, *d*8, and *d*9 components. Reconstructed signals are illustrated in Figure 6. Of note, the activation pattern of oxygenated hemoglobin and deoxygenated hemoglobin, shown in Figure 3, can be observed very clearly. Comparison between Figure 4 (signal processing with baseline correction and de-noising) and Figure 6 (wavelet-based method) shows the improved performance of the proposed method. Results revealed that oxygenated hemoglobin increased and the brain

①② ③

To confirm the validity of the signal-processing method explained in the previous section,

To measure brain activity under workload, we used the workload of mental calculation.

 High-level task: Subtraction and division with decimal fraction (e.g., 234/(0.61 − 0.35)) The design of the experiment is presented in Figure 7. Each set was composed of 28 s of task and 36 s of rest, in that order. By arranging three sets for each level in random order, a total

Time [s]

0 90 180 270 360 450 540

① ② ③

L R HRL RHR L RHR

R oxy-Hb deoxy-Hb :Rest L:Low difficulty task H:High difficulty task


0.08

Hb conc. (mM cm)

0

frequency range.

was activated during mental calculation tasks.

Fig. 6. Results of brain imaging using reconstructed signals

Low-level task: Simple one-digit addition (e.g., 3 + 5)

of nine sets of experiment were conducted over 592 s.

**4. Measurement of brain functions under mental workload** 

we measured brain functions through simultaneous use of NIRS and fMRI.

Mental calculation tasks were set to low, medium, and high levels as follows:

Medium-level task: One-digit addition of three numbers (e.g., 6 + 5 + 9)


Hb conc. [mM cm]

**4.1 Setting of workload** 

A 28s task consisted of 14 questions at 2s intervals for the low level, 10 questions at 2.8s intervals for the intermediate level, or two questions at 14s intervals for the high level. The subject answered the questions displayed on the computer monitor without speaking. During the 36s rest time, the subject rested while steadily gazing at a cross mark displayed on the computer monitor.

### **4.2 NIRS and fMRI signal recording**

Brain activity in the prefrontal lobe was measured using NIRS and fMRI simultaneously. NIRS data were collected using the OMM-3000 system (Shimadzu Corporation, Japan) in the MRI scanner.

Data from fMRI (3 mm thickness, 40 slices) were collected using a Siemens Symphony 1.5T (T2\*-weighted gradient-echo sequence, TR = 4000 ms, TE = 50 ms, FA = 90 degrees, 6464 pixels, FOV = 192 mm). Whole-brain images were obtained as T1-weighted images (TR = 2200 ms, TE = 3.93 ms, FA = 15 degrees, TI = 1100 ms, 1 mm3 voxel, FOV = 256 mm).

Data from fMRI were pre-processed using Statistical Parametric Mapping (SPM99; Welcome Department of Imaging Neuroscience, UK) Normalized contrast images were smoothed with an isotropic Gaussian kernel (FWHM = 12 mm). Regions of interest (ROIs) were defined as clusters of 10 or more voxels in which estimated parameter values differed significantly from zero (p<0.01).

Subjects were nine healthy volunteers (nine men, no women). The arrangement of optical fiber units and measurement positions is shown in Figure 1.

### **4.3 Signal processing**

Figure 8 presents measurement results through all channels for Subject A during the first three tasks. During the mental calculation task at the high level (i.e., the third task), oxygenated hemoglobin increased and deoxygenated hemoglobin decreased on both outer sides of the frontal lobe.

Figure 9 presents MRA results for oxygenated hemoglobin in channel 26, where task-related changes were marked. The trend of the whole experiment was extracted on the approximated component (*a*10). As the interval of repetition of tasks and rests was 64 s, the *d*8 component was the central component of task-related changes. Signals were thus reconstructed by adding the *d*7, *d*8, and *d*9 components.

Reconstructed signals from channel 26 are illustrated in Figure 10. Results revealed that oxygenated hemoglobin increased and the brain was activated during mental calculation tasks. Furthermore, such changes became larger as the level of mental calculation task increased.

Figure 11 shows a comparison of functional brain imaging by fMRI and NIRS with the proposed method. The rectangle in the fMRI image indicates the region of measurement with NIRS. NIRS images agree with those for fMRI at a different workload level. This result supports the effectiveness of MRA with the discrete wavelet transform.

Measurement of Brain Function Using Near-Infrared Spectroscopy (NIRS) 85

R oxy-Hb deoxy-Hb :Rest L:Low difficulty task M: Middle difficulty task H:High difficulty task

L L R RR R R RRR R M M H H H L L

0 40 80 120 160 200 240 280 320 360 400 440 480 520 560

(a) Original signal

R:Rest L:Low difficulty task M: Middle difficulty task H:High difficulty task

Fig. 10. Comparison of original signal and reconstructed signal (channel 26)

Low difficulty task

Middle difficulty task

High difficulty task

Fig. 11. Functional brain imaging by fMRI and NIRS

But, SkBF did not increase in relation to task level.

**4.4 Influence of skin blood flow (SkBF)** 

Time (s)

L R R R RR R R R R M M H H L L H M

0 40 80 120 160 200 240 280 320 360 400 440 480 520 560 Time (s) (b) Reconstructed NIRS signal

fNIRS fMRI

0.02

Hb conc. (mM cm)

> -0.02 0

To investigate relationships between brain blood flow and SkBF, we took measurements using NIRS and the laser Doppler SkBF probe. The laser Doppler SkBF probe (Advance Corporation, Japan) was placed on the right side of the forehead, 5 cm from the nasal root. The semiconductor laser Doppler velocimeter used a monochromatic light source at 780 nm. Figure 12 shows the comparison of NIRS signal (cerebral blood flow) and SkBF. Oxygenated hemoglobin increased and the brain was activated during mental calculation tasks. Furthermore, such changes became larger as the level of mental calculation task increased.

oxy-Hb deoxy-Hb


Hb conc.(mM cm)

> Hb conc.(mM cm)


Fig. 8. Hemoglobin concentration changes in frontal lobe

Fig. 9. Decomposition of NIRS signal (channel 26)

Task M 28 36

H: High difficulty task oxy-Hb deoxy-Hb <sup>R</sup> <sup>R</sup> <sup>R</sup>


0 64 128 192 ch-10

> -0.1 0 -0.0 5 0.0 0 0.0 5 0.1 0

0 64 128 192 ch-23

> -0.1 0 -0.0 5 0.0 0 0.0 5 0.1 0

0 64 128 192 ch-4

R: Rest <sup>16</sup>

Task H 28 36


0 64 128 192 ch-17

> -0.1 0 -0.0 5 0.0 0 0.0 5 0.1 0

0 64 128 192 ch-30

<sup>0</sup> <sup>64</sup> <sup>128</sup> <sup>192</sup> <sup>256</sup> <sup>320</sup> <sup>384</sup> <sup>448</sup> <sup>512</sup> <sup>576</sup> <sup>0</sup>

<sup>0</sup> <sup>64</sup> <sup>128</sup> <sup>192</sup> <sup>256</sup> <sup>320</sup> <sup>384</sup> <sup>448</sup> <sup>512</sup> <sup>576</sup> -0.05

H 28

R:Rest L: Low difficulty task M: Middle difficulty taskH: High difficulty task

28 [s]

R 36

<sup>0</sup> <sup>64</sup> <sup>128</sup> <sup>192</sup> <sup>256</sup> <sup>320</sup> <sup>384</sup> <sup>448</sup> <sup>512</sup> <sup>576</sup> -0.01

<sup>0</sup> <sup>64</sup> <sup>128</sup> <sup>192</sup> <sup>256</sup> <sup>320</sup> <sup>384</sup> <sup>448</sup> <sup>512</sup> <sup>576</sup> -0.05

<sup>0</sup> <sup>64</sup> <sup>128</sup> <sup>192</sup> <sup>256</sup> <sup>320</sup> <sup>384</sup> <sup>448</sup> <sup>512</sup> <sup>576</sup> -0.01

<sup>0</sup> <sup>64</sup> <sup>128</sup> <sup>192</sup> <sup>256</sup> <sup>320</sup> <sup>384</sup> <sup>448</sup> <sup>512</sup> <sup>576</sup> -0.005

<sup>0</sup> <sup>64</sup> <sup>128</sup> <sup>192</sup> <sup>256</sup> <sup>320</sup> <sup>384</sup> <sup>448</sup> <sup>512</sup> <sup>576</sup> -0.01

<sup>0</sup> <sup>64</sup> <sup>128</sup> <sup>192</sup> <sup>256</sup> <sup>320</sup> <sup>384</sup> <sup>448</sup> <sup>512</sup> <sup>576</sup> -0.005

<sup>0</sup> <sup>64</sup> <sup>128</sup> <sup>192</sup> <sup>256</sup> <sup>320</sup> <sup>384</sup> <sup>448</sup> <sup>512</sup> <sup>576</sup> -0.01

<sup>0</sup> <sup>64</sup> <sup>128</sup> <sup>192</sup> <sup>256</sup> <sup>320</sup> <sup>384</sup> <sup>448</sup> <sup>512</sup> <sup>576</sup> -0.02

<sup>0</sup> <sup>64</sup> <sup>128</sup> <sup>192</sup> <sup>256</sup> <sup>320</sup> <sup>384</sup> <sup>448</sup> <sup>512</sup> <sup>576</sup> -0.01

<sup>0</sup> <sup>64</sup> <sup>128</sup> <sup>192</sup> <sup>256</sup> <sup>320</sup> <sup>384</sup> <sup>448</sup> <sup>512</sup> <sup>576</sup> -0.01

Time [s]


0 64 128 192 ch-11

> -0.1 0 -0.0 5 0.0 0 0.0 5 0.1 0

0 64 128 192 ch-24

> -0.1 0 -0.0 5 0.0 0 0.0 5 0.1 0

> > L 28

R 36

0 64 128 192 ch-5

> -0.10 -0.05 0.00 0.05 0.10

0 64 128 192 ch-18


0 64 128 192 ch-31


0 64 128 192 ch-12

> -0.1 0 -0.0 5 0.0 0 0.0 5 0.1 0

0 64 128 192 ch-25

> -0.1 0 -0.0 5 0.0 0 0.0 5 0.1 0

H 28

R 36

0 64 128 192 ch-6

L: Low difficulty task M: Middle difficulty task

> -0.10 -0.05 0.00 0.05 0.10

0 64 128 192 ch-19


0 64 128 192 ch-32

> M 28

R 36

0 64 128 192 ch-13

0 64 128 192 ch-26


0 64 128 192 ch-7

> -0.1 0 -0.0 5 0.0 0 0.0 5 0.1 0

0 64 128 192 ch-20

> -0.1 0 -0.0 5 0.0 0 0.0 5 0.1 0

R 16 L 28 R 36 M 28 R 36 H



> -0.10 -0.05 0.00 0.05 0.10

0 64 128 192 ch-14


0 64 128 192 ch-27

*d*1(0.3 - 0.6s)

Fig. 9. Decomposition of NIRS signal (channel 26)

*d*2(0.6 - 1.3s)

*d*3(1.3 - 2.6s)

*d*4(2.6 - 5.1s)

*d*5(5.1 - 10.2s)

*d*6(10.2 - 20.5s)

*d*7(20.5 - 41.0s)

*d*8(41.0 - 82.0s)

*d*9(82.0 - 164s)

*d*10(164 -328s)

*a*10(328s - )

*S*(*t*)


0 64 128 192 ch-8

> -0.1 0 -0.0 5 0.00 0.05 0.10

0 64 128 192


Fig. 8. Hemoglobin concentration changes in frontal lobe

0.005 0.01

0 0.05

0 0.01

0 0.05

0 0.01

0 0.005

0 0.01


<sup>0</sup> 0.005

0 0.01

0 0.02

0 0.01

0 0.01

0 64 128 192 ch-2

> -0.10 -0.05 0.00 0.05 0.10

0 64 128 192 ch-15

ch-21 ch-22


0 64 128 192 ch-28


Task L 28 R 36

0 64 128 192 ch-9

> -0.1 0 -0.0 5 0.00 0.05 0.10


> M 28

R 36

R 36

0 64 128 192

0 64 128 192 ch-3

> -0.10 -0.05 0.00 0.05 0.10

0 64 128 192 ch-16


0 64 128 192 ch-29

> L 28

R 36

(b) Reconstructed NIRS signal

Fig. 10. Comparison of original signal and reconstructed signal (channel 26)

Fig. 11. Functional brain imaging by fMRI and NIRS

### **4.4 Influence of skin blood flow (SkBF)**

To investigate relationships between brain blood flow and SkBF, we took measurements using NIRS and the laser Doppler SkBF probe. The laser Doppler SkBF probe (Advance Corporation, Japan) was placed on the right side of the forehead, 5 cm from the nasal root. The semiconductor laser Doppler velocimeter used a monochromatic light source at 780 nm. Figure 12 shows the comparison of NIRS signal (cerebral blood flow) and SkBF. Oxygenated hemoglobin increased and the brain was activated during mental calculation tasks. Furthermore, such changes became larger as the level of mental calculation task increased. But, SkBF did not increase in relation to task level.

Measurement of Brain Function Using Near-Infrared Spectroscopy (NIRS) 87

is the standard deviation.

(14)

oxy-Hb deoxy-Hb

1.5

Zscore


0

L L R M M R H R R R H R L R H R M R

0 40 80 120 160 200 240 280 320 360 400 440 480 520 560

(a) Averaged reconstructed fNIRS signal (channel 26)

(b) Brain activity imaged by averaged NIRS signals

fNIRS fMRI

1.0

Zscore


Fig. 15. Functional brain imaging by fMRI and NIRS (group analysis for nine subjects)

Figure 14 shows averaged NIRS signals using Z-score for nine subjects. Of note, the difference in workload level is reflected in the gradient of the oxygenated hemoglobin

0

Time (s)

Here, *X* is the signal of oxygenated hemoglobin or deoxygenated hemoglobin reconstructed

*<sup>X</sup> <sup>Z</sup>*

Low difficulty task Middle difficulty task High difficulty task

Fig. 14. Results of group analysis of NIRS signals for nine subjects

Low difficulty task

Middle difficulty task

High difficulty task

R:Rest L:Low difficulty task M: Middle difficulty task H:High difficulty task

using MRA; is the mean value; and


Zscore

Fig. 12. Comparison of NIRS signal (channel 26) and SkBF

Mean values of oxygenated hemoglobin and SkBF at each task level are shown in Figure 13. The mean value of NIRS (Fig. 13(a)) became higher with higher task level. However, the mean value of SkBF (Fig. 13(b)) did not change in relation to task level. This result confirmed that SkBF did not influence NIRS signals (cerebral blood flow) in the recognition task.

Fig. 13. Averaged result for oxy-Hb and SkBF at each task level

### **4.5 Statistical analysis**

The NIRS signal expresses the quantity of relative changes using the start point as the reference. However, comparisons of measurements between subjects or statistical processing of measurements of all subjects cannot be implemented using this signal as is. Therefore, we propose a method for converting data of oxygenated hemoglobin and deoxygenated hemoglobin reconstructed by MRA into Z-scores using the following expression, so that the mean value is 0 and standard deviation is 1.

Low difficulty task Middle difficulty task oxy-Hb deoxy-Hb SkBF

Time (s)

Blood Flow (%)

Low Middle High

0 10 20 30 40 50 60

Blood Flow (%)

Blood Flow (%)

Hb conc. (mM cm)

High difficulty task

Time (s)




Blood Flow (%)

(a) Result of oxy-Hb mean value (b) Result of SkBF mean value

The NIRS signal expresses the quantity of relative changes using the start point as the reference. However, comparisons of measurements between subjects or statistical processing of measurements of all subjects cannot be implemented using this signal as is. Therefore, we propose a method for converting data of oxygenated hemoglobin and deoxygenated hemoglobin reconstructed by MRA into Z-scores using the following


0

0 10 20 30 40 50 60

Mean values of oxygenated hemoglobin and SkBF at each task level are shown in Figure 13. The mean value of NIRS (Fig. 13(a)) became higher with higher task level. However, the mean value of SkBF (Fig. 13(b)) did not change in relation to task level. This result confirmed that SkBF did not influence NIRS signals (cerebral blood flow) in the recognition


Time (s)

Hb conc. (mM cm)

Fig. 12. Comparison of NIRS signal (channel 26) and SkBF

Low Middle High

Fig. 13. Averaged result for oxy-Hb and SkBF at each task level

expression, so that the mean value is 0 and standard deviation is 1.


0 10 20 30 40 50 60



0

0.01

Hb conc. (mM cm)

**4.5 Statistical analysis** 

0.02

0.03

Hb conc. (mM cm)

task.

$$Z = \frac{X - \mu}{\sigma} \tag{14}$$

Here, *X* is the signal of oxygenated hemoglobin or deoxygenated hemoglobin reconstructed using MRA; is the mean value; and is the standard deviation.

Fig. 14. Results of group analysis of NIRS signals for nine subjects

Fig. 15. Functional brain imaging by fMRI and NIRS (group analysis for nine subjects)

Figure 14 shows averaged NIRS signals using Z-score for nine subjects. Of note, the difference in workload level is reflected in the gradient of the oxygenated hemoglobin

Measurement of Brain Function Using Near-Infrared Spectroscopy (NIRS) 89

Car drivers obtain visual information on the surrounding environment, recognize and judge that information suitably and then control their vehicle using the steering wheel, accelerator and brake pedal operations. Human brain activity functions to control all of these processes. In situations where it is necessary to predict unexpected danger, brain activity of the driver is thought to strengthen the cognitive function by spontaneously raising the level of attention. In the course of developing driver support systems, having a clear understanding

In recent years, various driving-assistance systems have been developed to improve safety by reducing driver workloads. Examples include the adaptive cruise control (ACC) system, which maintains a safe distance between the driver's vehicle and the vehicle ahead, and the lane-keeping assistance system, which keeps the car in a lane through steering support. However, while driver workload is reduced, the attention of the driver may also be reduced, resulting in unexpected accidents. Driver workload thus needs to be examined from the perspectives of cognitive engineering and human physiology. It is necessary to clarify relationships between driver workload and brain activity, which includes recognition and judgment. Driver attention then needs to be evaluated and the relationship between brain

A small number of neuroimaging studies have used driving simulators to examine brain activity during driving. In these studies (Uchiyama, 2003; Spiers & Maguire, 2007), fMRI has been used. However, fMRI has many shortcomings in evaluating driving performance, requiring the subject to lie in a narrow cylinder during evaluation and not permitting movement of the body, particularly the head. This situation makes the driving task

To verify that the reduction in driving workload with ACC could be evaluated from brain activity, we conducted an experiment that involved the use of a driving simulator

> 40s 45s 40s 45s 40s Time (s)

The main specifications of the driving simulator were as follows. Dimensions: 2,440 mm (W) 2,280 mm (H) 1,850 mm (D), front view: wide-field (138 degrees) screen projection, DLP projector with a total pixel count of 780,000 (XGA), rear view: 3 mirror independent LCD

keeping speed 40km/h

**5. Application examples** 

**5.1.1 Background** 

**5.1 Measurement of brain functions while driving a car** 

of human brain activity is important in such driving situations.

activity and driving performance clarified.

simulating following a vehicle (Fig. 17).

Fig. 17. Speed pattern of leading car

Speed (km/h)

unrealistic and unnatural.

**5.1.2 Driving task** 

concentration. Figure 15 shows the results of group analysis for nine subjects. The rectangle in the fMRI image indicates the region of measurement with NIRS. NIRS images agree with results of fMRI regarding different workload levels. This result supports the effectiveness of the proposed method.

### **4.6 Subjective and objective evaluation of workload**

In this experiment, the workload of each subject was measured using the Japanese version of NASA-TLX to evaluate correlations between the workload of mental calculation tasks and the objective evaluation using NIRS. The NASA-TLX is composed of six measures: mental requirements; physical requirements; temporal demand; work performance; effort; and frustration. Before workload evaluation, the subject performed one-to-one comparisons of the importance of elements of the workload involved in task performance.

The weight of each measure was based on the number of times an element was selected as more important during 15 one-to-one comparisons. When evaluating the workload of each task, the subject placed a mark at the appropriate position on the segment drawn between both extremes for each of the six measures.

A weighted workload (WWL) score was obtained by reading the position of each evaluation mark on a scale of 0 to 100 and multiplying by the weight for each measure determined by one-to-one comparison, then averaging all the products. Figure 16 (left) presents the WWL score of Subject A as determined by NASA-TLX. Workload became higher with higher task level.

Figure 16 (right) shows the results for 9 subjects evaluated using the maximum gradient of oxygenated hemoglobin in the task with different workload levels. Multivariate test using the Ryan method was used. Significant differences between high- and low-level tasks, or between high- and medium-level tasks are apparent (p<0.05). The results exhibited good correlation with subjective evaluation using NASA-TLX. This result confirms the feasibility of evaluating workload using the signal of cerebral blood flow obtained from NIRS.

Fig. 16. Subjective and objective evaluation of workload

### **5. Application examples**

### **5.1 Measurement of brain functions while driving a car**

### **5.1.1 Background**

88 Neuroimaging – Methods

concentration. Figure 15 shows the results of group analysis for nine subjects. The rectangle in the fMRI image indicates the region of measurement with NIRS. NIRS images agree with results of fMRI regarding different workload levels. This result supports the effectiveness of

In this experiment, the workload of each subject was measured using the Japanese version of NASA-TLX to evaluate correlations between the workload of mental calculation tasks and the objective evaluation using NIRS. The NASA-TLX is composed of six measures: mental requirements; physical requirements; temporal demand; work performance; effort; and frustration. Before workload evaluation, the subject performed one-to-one comparisons of

The weight of each measure was based on the number of times an element was selected as more important during 15 one-to-one comparisons. When evaluating the workload of each task, the subject placed a mark at the appropriate position on the segment drawn between

A weighted workload (WWL) score was obtained by reading the position of each evaluation mark on a scale of 0 to 100 and multiplying by the weight for each measure determined by one-to-one comparison, then averaging all the products. Figure 16 (left) presents the WWL score of Subject A as determined by NASA-TLX. Workload became higher with higher task

Figure 16 (right) shows the results for 9 subjects evaluated using the maximum gradient of oxygenated hemoglobin in the task with different workload levels. Multivariate test using the Ryan method was used. Significant differences between high- and low-level tasks, or between high- and medium-level tasks are apparent (p<0.05). The results exhibited good correlation with subjective evaluation using NASA-TLX. This result confirms the feasibility

> 0 0.0005 0.001 0.0015 0.002 0.0025 0.003

Differential value

(a) Workload evaluation by NASA-TLX. (b) Maximum gradient of NIRS signal during the task

Low Middle High

\*

\*


\* p< 0.05

of evaluating workload using the signal of cerebral blood flow obtained from NIRS.

the importance of elements of the workload involved in task performance.

the proposed method.

level.

80 100

Workload (WWL)

**4.6 Subjective and objective evaluation of workload** 

both extremes for each of the six measures.

Low Middle High

Fig. 16. Subjective and objective evaluation of workload

Car drivers obtain visual information on the surrounding environment, recognize and judge that information suitably and then control their vehicle using the steering wheel, accelerator and brake pedal operations. Human brain activity functions to control all of these processes. In situations where it is necessary to predict unexpected danger, brain activity of the driver is thought to strengthen the cognitive function by spontaneously raising the level of attention. In the course of developing driver support systems, having a clear understanding of human brain activity is important in such driving situations.

In recent years, various driving-assistance systems have been developed to improve safety by reducing driver workloads. Examples include the adaptive cruise control (ACC) system, which maintains a safe distance between the driver's vehicle and the vehicle ahead, and the lane-keeping assistance system, which keeps the car in a lane through steering support.

However, while driver workload is reduced, the attention of the driver may also be reduced, resulting in unexpected accidents. Driver workload thus needs to be examined from the perspectives of cognitive engineering and human physiology. It is necessary to clarify relationships between driver workload and brain activity, which includes recognition and judgment. Driver attention then needs to be evaluated and the relationship between brain activity and driving performance clarified.

A small number of neuroimaging studies have used driving simulators to examine brain activity during driving. In these studies (Uchiyama, 2003; Spiers & Maguire, 2007), fMRI has been used. However, fMRI has many shortcomings in evaluating driving performance, requiring the subject to lie in a narrow cylinder during evaluation and not permitting movement of the body, particularly the head. This situation makes the driving task unrealistic and unnatural.

### **5.1.2 Driving task**

To verify that the reduction in driving workload with ACC could be evaluated from brain activity, we conducted an experiment that involved the use of a driving simulator simulating following a vehicle (Fig. 17).

Fig. 17. Speed pattern of leading car

The main specifications of the driving simulator were as follows. Dimensions: 2,440 mm (W) 2,280 mm (H) 1,850 mm (D), front view: wide-field (138 degrees) screen projection, DLP projector with a total pixel count of 780,000 (XGA), rear view: 3 mirror independent LCD

Measurement of Brain Function Using Near-Infrared Spectroscopy (NIRS) 91

NIRS signals include signals that are unrelated to brain activity (e.g., noise of the measurement instrument, influences of breathing, and changes in blood pressure). These unrelated signals need to be removed to evaluate brain activity in detail. The measured NIRS signals were thus decomposed through MRA using discrete wavelet transform, and components related to the driving task were reconstructed. Group analysis using Z-score

Figure 20 presents the results of group analysis for four drivers without ACC, while Figure 21 presents the results with ACC. Figure 20(a) confirms that oxygenated hemoglobin increased when the subject drove without ACC and exhibited a high value in the latter half of the task. The brain function imaged in Figure 20(b) confirms that, as common brain activity, both outer portions of the frontal lobe became active during the

Figure 21(a) indicates that oxygenated hemoglobin did not increase while driving with the use of ACC. Also, the brain function image in Figure 21(b) reveals that the frontal lobe was less active than when the subject drove without ACC. This result may reflect the reduction

<sup>0</sup> 30 60 90 120 150 180 210 Time (s)

(a) Averaged reconstructed NIRS signal (channel 26)

① ②③

(b) Functional brain imaging

Fig. 20. Results of group analysis for four drivers without ACC system

Stopping Driving Stopping Driving Stopping

① ②③ oxyHb deoxyHb

Z-score

0

1.0

‐1.0

**5.1.4 Decomposition and reconstruction of NIRS signals** 

was then conducted for all subjects.

of driving workload by ACC.


Zscore

**5.1.5 Results** 

driving task.

display 640480 pixel (VGA), computer graphics: redraw speed, 30-60 frames/s, simulation system: 6-axis motion base system using six electric screw cylinders.

Driving tests were conducted under two conditions: one involved following a vehicle while utilizing the ACC; and the other involved following a vehicle while driving without ACC. The subject performed practice runs to become somewhat skilful in handling the driving simulator and then drove twice under each condition. Brain activity during one condition was compared with that during the other condition.

### **5.1.3 Measurement method**

Brain activity in the frontal lobe was measured using NIRS. Figure 18 depicts the scene of the experiment. The measuring instrument was the OMM-300 near-infrared imaging device (Shimadzu Corporation, Japan). Figure 19 illustrates the arrangement of optical-fiber units (3 9 matrix, 42 channels).

Numbers between the light-emitting fiber unit and the light-receiving fiber unit denote the measurement channels; measurements were performed through a total of 42 channels. Furthermore, driving performance was also recorded on the driving simulator while measuring brain activity. Four male subjects in their 20s participated; all were in healthy condition and had ordinary driving licenses.

Fig. 18. Experiment with driving simulator (driver follows the proceeding vehicle with and without ACC)

Fig. 19. Position of optical fibers and channels (driving task: 39 matrix, 42 channels)

### **5.1.4 Decomposition and reconstruction of NIRS signals**

NIRS signals include signals that are unrelated to brain activity (e.g., noise of the measurement instrument, influences of breathing, and changes in blood pressure). These unrelated signals need to be removed to evaluate brain activity in detail. The measured NIRS signals were thus decomposed through MRA using discrete wavelet transform, and components related to the driving task were reconstructed. Group analysis using Z-score was then conducted for all subjects.

### **5.1.5 Results**

90 Neuroimaging – Methods

display 640480 pixel (VGA), computer graphics: redraw speed, 30-60 frames/s, simulation

Driving tests were conducted under two conditions: one involved following a vehicle while utilizing the ACC; and the other involved following a vehicle while driving without ACC. The subject performed practice runs to become somewhat skilful in handling the driving simulator and then drove twice under each condition. Brain activity during one condition

Brain activity in the frontal lobe was measured using NIRS. Figure 18 depicts the scene of the experiment. The measuring instrument was the OMM-300 near-infrared imaging device (Shimadzu Corporation, Japan). Figure 19 illustrates the arrangement of optical-fiber units

Numbers between the light-emitting fiber unit and the light-receiving fiber unit denote the measurement channels; measurements were performed through a total of 42 channels. Furthermore, driving performance was also recorded on the driving simulator while measuring brain activity. Four male subjects in their 20s participated; all were in healthy

 Fig. 18. Experiment with driving simulator (driver follows the proceeding vehicle with and

> Illuminator Detectot channel

17 34

7 24 41

1 2 3 4 9 10 11 12 13 18 19 20 21 26 27 28 29 30 35 36 37 38

Fig. 19. Position of optical fibers and channels (driving task: 39 matrix, 42 channels)

system: 6-axis motion base system using six electric screw cylinders.

was compared with that during the other condition.

condition and had ordinary driving licenses.

**5.1.3 Measurement method** 

(3 9 matrix, 42 channels).

without ACC)

Figure 20 presents the results of group analysis for four drivers without ACC, while Figure 21 presents the results with ACC. Figure 20(a) confirms that oxygenated hemoglobin increased when the subject drove without ACC and exhibited a high value in the latter half of the task. The brain function imaged in Figure 20(b) confirms that, as common brain activity, both outer portions of the frontal lobe became active during the driving task.

Figure 21(a) indicates that oxygenated hemoglobin did not increase while driving with the use of ACC. Also, the brain function image in Figure 21(b) reveals that the frontal lobe was less active than when the subject drove without ACC. This result may reflect the reduction of driving workload by ACC.

Fig. 20. Results of group analysis for four drivers without ACC system

Measurement of Brain Function Using Near-Infrared Spectroscopy (NIRS) 93

from NIRS, in order to cause the elbow joint to move. Signals measured by NIRS are unstable, since signals components that are irrelevant to the subject (e.g., noise of measurement instruments, heartbeat, and respiration) are included. Moreover, no processing method for NIRS signals has yet been established. For these reasons, obtaining a highly accurate operation by merely setting a simple threshold on NIRS signals is difficult. In this study, we assumed artificial arms for physically handicapped individuals and proposed a BCI system that would enable operation of the robot arms using NIRS to measure cranial nerve information in a non-invasive manner. Furthermore, we applied new signal-processing methods, such as multiresolution analysis with discrete wavelet transform and a brain activity-judging method that uses derivatives of NIRS signals, to achieve highly

In imaging tasks where the subject performs imaging in the brain without moving the body, signal intensity is weak and the presence of brain activity is difficult to judge. We therefore first verified the effectiveness of the system and judging method in grasping tasks, which have high signal intensity, and then verified the identification accuracy of the system from

Figure 22 depicts a robot arm control system that uses a NIRS-BCI system. This system is composed of a cerebral function measurement section, a feature extraction and recognition section, and a device control section. In the cerebral function measurement section, the oxygenated hemoglobin level of the user is measured using the OMM-3000 multichannel NIRS instrument made by Shimadzu Corporation, Japan. In the feature extraction and recognition section, the threshold is obtained by analyzing the original signal of oxygenated hemoglobin that was measured. When oxygenated hemoglobin after analysis exceeds the threshold obtained in the feature extraction and recognition section, the on signal is sent to the device control section to enable rotation of the joint of the robot arm (MR-999; Elekit,

user

Extraction/recognition

NIRS signal

Data transfer

Measurement

tests that involve imaged grasping tasks, with consideration of applying the system.

**5.2.2 A robot arm control using a NIRS-BCI system** 

control

on/off transmission of a signal

Fig. 22. Robot arm control using the NIRS-BCI system

accurate on/off operations.

Japan).

### **5.2 Brain-Computer Interface (BCI) using NIRS 5.2.1 Background**

Currently, the concept of BCIs is under intense study. BCIs extract activity from the human brain as cranial nerve information and use the information as inputs to control machinery and equipment. If this could enable the operation of machinery and equipment directly from cranial nerve information without the subject moving the hands and feet, such systems could be applied to care-taking robots for physically handicapped individuals.

BCI systems can be divided into two types. The invasive type reads cranial nerve information using electrodes embedded directly into the brain. The non-invasive type reads cranial nerve activity from the surface of the head using NIRS or electroencephalography (EEG). The invasive type offers high signal accuracy, but imposes heavy burdens on the user, such as the need for invasive procedures and the risk of infections after such procedures. The non-invasive form thus seems to have wider applicability.

In a study on the non-invasive type, Matsumura et al. extracted cranial nerve information using EEG and controlled a humanoid robot (Matsumura & Nakagawa, 2006). Many other studies of BCIs have used EEG. However, EEG may be greatly affected by movement of the user's body, and is vulnerable to electronic noise. In contrast, NIRS imposes fewer restrictions on body movement than EEG and is more resistant to electronic noise, so the load imposed on the user is less and electronic devices have no influence. The present study therefore focused on BCIs using NIRS.

In an earlier study, Nagaoka et al. developed an NIRS-BCI rehabilitation system (Nagaoka et al., 2010). In that study, electric stimuli corresponding to cranial nerve information were applied to the biceps brachii muscle of the user by setting a threshold on signals measured

① ②③

Fig. 21. Results of group analysis for four drivers with ACC system

**5.2 Brain-Computer Interface (BCI) using NIRS** 

therefore focused on BCIs using NIRS.

oxyHb deoxyHb

Z-score

0 ‐1.0

1.0

<sup>0</sup> 30 60 90 120 150 180 210 Time (s)

(a) Averaged reconstructed NIRS signal (Channel no. 26)

(b) Functional brain imaging

Currently, the concept of BCIs is under intense study. BCIs extract activity from the human brain as cranial nerve information and use the information as inputs to control machinery and equipment. If this could enable the operation of machinery and equipment directly from cranial nerve information without the subject moving the hands and feet, such systems

BCI systems can be divided into two types. The invasive type reads cranial nerve information using electrodes embedded directly into the brain. The non-invasive type reads cranial nerve activity from the surface of the head using NIRS or electroencephalography (EEG). The invasive type offers high signal accuracy, but imposes heavy burdens on the user, such as the need for invasive procedures and the risk of infections after such

In a study on the non-invasive type, Matsumura et al. extracted cranial nerve information using EEG and controlled a humanoid robot (Matsumura & Nakagawa, 2006). Many other studies of BCIs have used EEG. However, EEG may be greatly affected by movement of the user's body, and is vulnerable to electronic noise. In contrast, NIRS imposes fewer restrictions on body movement than EEG and is more resistant to electronic noise, so the load imposed on the user is less and electronic devices have no influence. The present study

In an earlier study, Nagaoka et al. developed an NIRS-BCI rehabilitation system (Nagaoka et al., 2010). In that study, electric stimuli corresponding to cranial nerve information were applied to the biceps brachii muscle of the user by setting a threshold on signals measured

could be applied to care-taking robots for physically handicapped individuals.

procedures. The non-invasive form thus seems to have wider applicability.

① ②③

Stopping Driving Stopping Driving Stopping


Zscore

**5.2.1 Background** 

from NIRS, in order to cause the elbow joint to move. Signals measured by NIRS are unstable, since signals components that are irrelevant to the subject (e.g., noise of measurement instruments, heartbeat, and respiration) are included. Moreover, no processing method for NIRS signals has yet been established. For these reasons, obtaining a highly accurate operation by merely setting a simple threshold on NIRS signals is difficult.

In this study, we assumed artificial arms for physically handicapped individuals and proposed a BCI system that would enable operation of the robot arms using NIRS to measure cranial nerve information in a non-invasive manner. Furthermore, we applied new signal-processing methods, such as multiresolution analysis with discrete wavelet transform and a brain activity-judging method that uses derivatives of NIRS signals, to achieve highly accurate on/off operations.

In imaging tasks where the subject performs imaging in the brain without moving the body, signal intensity is weak and the presence of brain activity is difficult to judge. We therefore first verified the effectiveness of the system and judging method in grasping tasks, which have high signal intensity, and then verified the identification accuracy of the system from tests that involve imaged grasping tasks, with consideration of applying the system.

### **5.2.2 A robot arm control using a NIRS-BCI system**

Figure 22 depicts a robot arm control system that uses a NIRS-BCI system. This system is composed of a cerebral function measurement section, a feature extraction and recognition section, and a device control section. In the cerebral function measurement section, the oxygenated hemoglobin level of the user is measured using the OMM-3000 multichannel NIRS instrument made by Shimadzu Corporation, Japan. In the feature extraction and recognition section, the threshold is obtained by analyzing the original signal of oxygenated hemoglobin that was measured. When oxygenated hemoglobin after analysis exceeds the threshold obtained in the feature extraction and recognition section, the on signal is sent to the device control section to enable rotation of the joint of the robot arm (MR-999; Elekit, Japan).

Fig. 22. Robot arm control using the NIRS-BCI system

Measurement of Brain Function Using Near-Infrared Spectroscopy (NIRS) 95

an increase in rCBF reflects an increase in neural activity. We therefore focused on the

Furthermore, the derivative of oxygenated hemoglobin is highly correlated with the workload of the task as shown in Section 4.6, so we judged brain activity using two indices: oxygenated hemoglobin; and its derivative. First, NIRS signals include signal components that are irrelevant to brain activity (e.g., noise of measurement instruments, influences of respiration, and fluctuations of blood pressure). Signals were thus decomposed and

The original oxygenated hemoglobin signal and reconstructed oxygenated hemoglobin signal are depicted in Figure 24(a). In Figure 24(a), oxygenated hemoglobin includes not only brain activity, but also the effects of blood pressure, breathing and measurement noises. However, in Figure 24(b), the reconstructed oxygenated hemoglobin signal can be

T:Task R:Rest t oxy-Hb T:Task R:Rest t oxy-Hb

100 150 200 250

*t*

*Xoxy oxy dtdX* Time (s)

R T R T R T R


Hb conc. (mMcm)

*oxy t*

(a) Original oxy-Hb signal (b) Reconstructed oxy-Hb signal

task

Changes in oxygenated hemoglobin and its derivative when the brain is activated are depicted in Figure 25. Oxygenated hemoglobin was defined as *Xoxy* and its derivative was defined as *dXoxy/dt*. The threshold of oxygenated hemoglobin (denoted as *y*1) and that of the derivative (denoted as *y*2) were computed by *y*1 = *K*1 max(*Xoxy*) and *y*2 = *K*2 max(*dXoxy/dt*). *K*<sup>1</sup> and *K*2 were weight factors that depended on the point where *Xoxy* reached a maximum.

Fig. 24. Changes in original and reconstructed oxygenated hemoglobin signals

2 *ta sk t*

Fig. 25. Oxygenated hemoglobin concentration change due to neural activity

oxygenated hemoglobin signal.

reconstructed by MRA.

seen to be smoothed.


Hb conc. (mMcm)

100 150 200 250

max( ) *Xoxy* max *oxy dX dt* 

Time (s)

R T R T R T R

Experiments were conducted using the robot arm control system depicted in Figure 22. Optical fibers were arranged in 44 matrices on the right and left sides to perform measurement, with a total of 48 channels.

Five trials were carried out, with each trial consisting of 10 s of pre-task rest, 30 s of task, and 10 s of post-task rest. The first two trials were defined as the learning stage, where the feature extraction and recognition section learned the fluctuation pattern of the user's oxygenated hemoglobin levels without moving the robot.

In the third and subsequent trials, the robot arm was rotated according to the learned pattern corresponding with oxygenated hemoglobin. Both grasping tasks and imaged grasping tasks were performed. The subject was instructed to rest during the rest time. The motor area was selected as the measurement site.

### **5.2.3 Judgment of brain activity**

### *Judgment with a simple threshold*

In the judgment method that used a simple threshold, the average oxygenated hemoglobin level in the first two trials and the standard deviation were converted to Z-scores, and the moving average was obtained. The threshold was set to 20% of the maximum oxygenated hemoglobin level during the first two trials; judgment was made in the third and subsequent trials, and an "on" state was judged when oxygenated hemoglobin level exceeded this threshold.

Changes in density of oxygenated hemoglobin level and the judgment results for imaged grasping tasks and actual grasping tasks are depicted in Figure 23. In Figure 23(a), "on" judgments were observed in the first and second trials, but not in the third trial. Next, in Figure 23(b), "on" judgments were made in all tasks from the first to third trials, but "on" and "off" judgments alternated within a short time, indicating instability. Furthermore, an "on" judgment was made during the rest period in the second trial, indicating erroneous judgments.

Fig. 23. Results for on/off decisions using a simple threshold

### *Judgment using oxygenated hemoglobin level and its derivative*

NIRS can measure both change in density of oxygenated hemoglobin and that of deoxygenated hemoglobin; however, the change in density of oxygenated hemoglobin is highly correlated with the change in regional cerebral blood flow (rCBF) (Hoshi, 2001), and

Experiments were conducted using the robot arm control system depicted in Figure 22. Optical fibers were arranged in 44 matrices on the right and left sides to perform

Five trials were carried out, with each trial consisting of 10 s of pre-task rest, 30 s of task, and 10 s of post-task rest. The first two trials were defined as the learning stage, where the feature extraction and recognition section learned the fluctuation pattern of the user's

In the third and subsequent trials, the robot arm was rotated according to the learned pattern corresponding with oxygenated hemoglobin. Both grasping tasks and imaged grasping tasks were performed. The subject was instructed to rest during the rest time. The

In the judgment method that used a simple threshold, the average oxygenated hemoglobin level in the first two trials and the standard deviation were converted to Z-scores, and the moving average was obtained. The threshold was set to 20% of the maximum oxygenated hemoglobin level during the first two trials; judgment was made in the third and subsequent trials, and an "on" state was judged when oxygenated hemoglobin level

Changes in density of oxygenated hemoglobin level and the judgment results for imaged grasping tasks and actual grasping tasks are depicted in Figure 23. In Figure 23(a), "on" judgments were observed in the first and second trials, but not in the third trial. Next, in Figure 23(b), "on" judgments were made in all tasks from the first to third trials, but "on" and "off" judgments alternated within a short time, indicating instability. Furthermore, an "on" judgment was made during the rest period in the second trial, indicating erroneous

> -0.02 0.01 0.04 0.07 0.1 0.13 0.16

Hb conc. (mMcm)

on

off

NIRS can measure both change in density of oxygenated hemoglobin and that of deoxygenated hemoglobin; however, the change in density of oxygenated hemoglobin is highly correlated with the change in regional cerebral blood flow (rCBF) (Hoshi, 2001), and

(a) Actual grasping tasks (b) Imagined grasping tasks

100 150 200 250

Time (s)

R T R T R T R

T:Task R:Rest t oxy-Hb on/off

on

off

measurement, with a total of 48 channels.

oxygenated hemoglobin levels without moving the robot.

motor area was selected as the measurement site.

100 150 200 250

Time (s)

Fig. 23. Results for on/off decisions using a simple threshold *Judgment using oxygenated hemoglobin level and its derivative* 

R T R T R T R

T:Task R:Rest t oxy-Hb on/off

**5.2.3 Judgment of brain activity**  *Judgment with a simple threshold* 

exceeded this threshold.

judgments.


Hb conc. (mMcm) an increase in rCBF reflects an increase in neural activity. We therefore focused on the oxygenated hemoglobin signal.

Furthermore, the derivative of oxygenated hemoglobin is highly correlated with the workload of the task as shown in Section 4.6, so we judged brain activity using two indices: oxygenated hemoglobin; and its derivative. First, NIRS signals include signal components that are irrelevant to brain activity (e.g., noise of measurement instruments, influences of respiration, and fluctuations of blood pressure). Signals were thus decomposed and reconstructed by MRA.

The original oxygenated hemoglobin signal and reconstructed oxygenated hemoglobin signal are depicted in Figure 24(a). In Figure 24(a), oxygenated hemoglobin includes not only brain activity, but also the effects of blood pressure, breathing and measurement noises. However, in Figure 24(b), the reconstructed oxygenated hemoglobin signal can be seen to be smoothed.

Fig. 24. Changes in original and reconstructed oxygenated hemoglobin signals

Fig. 25. Oxygenated hemoglobin concentration change due to neural activity

Changes in oxygenated hemoglobin and its derivative when the brain is activated are depicted in Figure 25. Oxygenated hemoglobin was defined as *Xoxy* and its derivative was defined as *dXoxy/dt*. The threshold of oxygenated hemoglobin (denoted as *y*1) and that of the derivative (denoted as *y*2) were computed by *y*1 = *K*1 max(*Xoxy*) and *y*2 = *K*2 max(*dXoxy/dt*). *K*<sup>1</sup> and *K*2 were weight factors that depended on the point where *Xoxy* reached a maximum.

Measurement of Brain Function Using Near-Infrared Spectroscopy (NIRS) 97

activity detected by NIRS and workload scores obtained from NASA-TLX. This result indicates the possibility of evaluating workload from cerebral blood flow signals obtained

Whether the reduction of driving workload by ACC can be evaluated from brain activity was evaluated through experiments using a driving simulator. The results revealed that while the outer portions of the frontal lobe were active in connection with driving performance when the subject drove without ACC, no activity related to driving performance was seen with the use of ACC. These results suggest the possibility of evaluating driving-assistance systems through the evaluation of driving workload from

We developed MRA using discrete wavelet transform as a real-time post-signal processing for NIRS-BCI system and applied this approach to control of a robot arm. With a judgment method that used a simple threshold, no stable operation could be obtained for grasping

We proposed a judgment method that uses two indices (oxygenated hemoglobin and the derivative of oxygenated hemoglobin) for judging brain activity, and adjusts the threshold of oxygenated hemoglobin and its derivative depending on the point of maximum oxygenated hemoglobin during the task. Results indicate that our proposed method enables more accurate judgments than use of a simple threshold. We thus confirmed the feasibility

This work was supported by the Nihon University Multidisciplinary Research Grant in

Hoshi, Y., Kobayashi, N., Tamura, M. (2001). Interpretation of nearinfrared spectroscopy

Tamura, M. (2003). Functional near-infrared spectoroscopy, *Advances in Neurological Sciences,* 

Kojima, T., Tsunashima, H., Shiozawa, T., Takada, H. and Sakai, T. (2005). Measurement of

Kojima, T., Tsunashima, H., Shiozawa, T. (2006). Measurement of train driver's brain activity

Shimizu, T., Hirose, S., Obara, H., Yanagisawa, K., Tsunashima, H., Marumo, Y., Haji. T. and

*of Japan Annual Conference*, Vol. 11 pp. 354-355, ISSN 1349-5062, Japan, 2006 Mallat, S. (1998). *A Wavelet Tour of Signal Processing,* Academic Press, ISBN 978-0124666061,

Driving Workload and Increased Attention, *SAE paper* NO. 2009-01-054 Takahashi, K., Watanabe, N. and Harada, T. (2006). Preliminary experiment of the

*Optical and Quantum Electronics*, Vol.37, No.13-15, pp. 1319-1338

signals, A study with a newly developed perfused rat brain model, *Journal of* 

Train Driver's Brain Activity by Functional Near-Infrared Spectroscopy (fNIRS),

by functional near-infrared spectroscopy (fNIRS), *COMPUTERS IN RAILWAYS X*,

Taira, M. (2009). Measurement of Frontal Cortex Brain Activity Attribute to the

evaluation of a VR based training system using brain activity, *Virtual Reality Society* 

using NIRS.

measurement of brain activity using NIRS.

of our proposed method for a NIRS-BCI system.

*Applied Physiology*, Vol.90, No.5, pp.1657-1662

Series C, Vol.47, No.6, pp.891-901

tasks or imaged grasping tasks.

**7. Acknowledgment** 

WIT Press

USA

2006, 2007 and 2010.

**8. References** 

The point where *X*oxy reached a maximum was defined as *e* = (*toxy* − *ttask*/2)/(*ttask*/2). Here, *ttask* was a time at which tasks were performed and *toxy* was a time from start of the task to when *Xoxy* reached a maximum. In this study, *K*1 and *K*2 were set to *K*1 = 0.8*e* + 0.2 and *K*1 = −0.8*e* + 1. If *Xoxy* reached maximum in the middle of the task, *y*1 was 20% of the maximum value of *X*oxy. If *Xoxy* reached maximum at the end of the task, *y*2 was 20% of the maximum value of *dXoxy*/*dt*. The on/off decision (on = 1, off = 0) was made by *Y* (= 1: *Xoxy*>*y*1 or *dXoxy*/*dt*>*y*2, = 0: otherwise).

The judgment result for grasping tasks when the proposed judgment method was applied is depicted in Figure 26(a), and that for imaged grasping tasks is depicted in Figure 26(b).

Fig. 26. Result of on/off decision using oxygenated hemoglobin and its derivative

In contrast to the judgment method where a simple threshold was imposed on oxygenated hemoglobin in grasping tasks (Fig. 23(a)), "on" judgments were confirmed during the first and all subsequent tasks with the method depicted in Figure 26(a). However, "on" judgments were made continually for 5 s after completion of the task in each trial, indicating that erroneous judgments may be made. The judgment method thus needs further refinement.

In contrast to the judgment method where a simple threshold was imposed on oxygenated hemoglobin in imaged grasping tasks (Fig. 23(b)), more stable judgment was made in Figure 26(b), and the problem of alternating "on" and "off" judgments within a short time was reduced.

### **6. Conclusions**

A signal-processing method to extract task-related components with MRA based on discrete wavelet transform is proposed for NIRS. Integration of data from multiple subjects using Zscores was then developed for statistical group analysis.

Brain activity of subjects with workloads comprising different levels of mental calculation tasks were measured using NIRS and fMRI. NIRS images constructed using the proposed method agreed with fMRI images at different workload levels. Those results show that the proposed method is effective for evaluating brain activity measured by NIRS.

Comparison between NIRS signal (cerebral blood flow) and skin blood flow showed that skin blood flow does not affect NIRS signal in the recognition task.

Changes in brain activity in connection with workload were compared with the subjective evaluation of workload by NASA-TLX. Good correlation was observed between brain activity detected by NIRS and workload scores obtained from NASA-TLX. This result indicates the possibility of evaluating workload from cerebral blood flow signals obtained using NIRS.

Whether the reduction of driving workload by ACC can be evaluated from brain activity was evaluated through experiments using a driving simulator. The results revealed that while the outer portions of the frontal lobe were active in connection with driving performance when the subject drove without ACC, no activity related to driving performance was seen with the use of ACC. These results suggest the possibility of evaluating driving-assistance systems through the evaluation of driving workload from measurement of brain activity using NIRS.

We developed MRA using discrete wavelet transform as a real-time post-signal processing for NIRS-BCI system and applied this approach to control of a robot arm. With a judgment method that used a simple threshold, no stable operation could be obtained for grasping tasks or imaged grasping tasks.

We proposed a judgment method that uses two indices (oxygenated hemoglobin and the derivative of oxygenated hemoglobin) for judging brain activity, and adjusts the threshold of oxygenated hemoglobin and its derivative depending on the point of maximum oxygenated hemoglobin during the task. Results indicate that our proposed method enables more accurate judgments than use of a simple threshold. We thus confirmed the feasibility of our proposed method for a NIRS-BCI system.

### **7. Acknowledgment**

This work was supported by the Nihon University Multidisciplinary Research Grant in 2006, 2007 and 2010.

### **8. References**

96 Neuroimaging – Methods

The point where *X*oxy reached a maximum was defined as *e* = (*toxy* − *ttask*/2)/(*ttask*/2). Here, *ttask* was a time at which tasks were performed and *toxy* was a time from start of the task to when *Xoxy* reached a maximum. In this study, *K*1 and *K*2 were set to *K*1 = 0.8*e* + 0.2 and *K*1 = −0.8*e* + 1. If *Xoxy* reached maximum in the middle of the task, *y*1 was 20% of the maximum value of *X*oxy. If *Xoxy* reached maximum at the end of the task, *y*2 was 20% of the maximum value of *dXoxy*/*dt*. The on/off decision (on = 1, off = 0) was made by *Y* (= 1: *Xoxy*>*y*1 or

The judgment result for grasping tasks when the proposed judgment method was applied is depicted in Figure 26(a), and that for imaged grasping tasks is depicted in Figure 26(b).

> -0.05 -0.03 -0.01 0.01 0.03 0.05

Hb conc. (mMcm)

(a) Actual grasping tasks (b) Imagined grasping tasks

In contrast to the judgment method where a simple threshold was imposed on oxygenated hemoglobin in grasping tasks (Fig. 23(a)), "on" judgments were confirmed during the first and all subsequent tasks with the method depicted in Figure 26(a). However, "on" judgments were made continually for 5 s after completion of the task in each trial, indicating that erroneous judgments may be made. The judgment method thus needs further

In contrast to the judgment method where a simple threshold was imposed on oxygenated hemoglobin in imaged grasping tasks (Fig. 23(b)), more stable judgment was made in Figure 26(b), and the problem of alternating "on" and "off" judgments within a short time was

A signal-processing method to extract task-related components with MRA based on discrete wavelet transform is proposed for NIRS. Integration of data from multiple subjects using Z-

Brain activity of subjects with workloads comprising different levels of mental calculation tasks were measured using NIRS and fMRI. NIRS images constructed using the proposed method agreed with fMRI images at different workload levels. Those results show that the

Comparison between NIRS signal (cerebral blood flow) and skin blood flow showed that

Changes in brain activity in connection with workload were compared with the subjective evaluation of workload by NASA-TLX. Good correlation was observed between brain

proposed method is effective for evaluating brain activity measured by NIRS.

skin blood flow does not affect NIRS signal in the recognition task.

off -0.01

100 150 200 250

Time (s)

R T R T R T R

T:Task R:Rest t oxy-Hb d(oxy-Hb)/dt on/off


d(oxy-Hb)/dt (mM cm/s) on

off


d(oxy-Hb)/dt (mM cm/s)

Fig. 26. Result of on/off decision using oxygenated hemoglobin and its derivative

on

*dXoxy*/*dt*>*y*2, = 0: otherwise).

100 150 200 250

Time (s)

scores was then developed for statistical group analysis.

R T R T R T R

T:Task R:Rest t oxy-Hb d(oxy-Hb)/dt on/off


refinement.

reduced.

**6. Conclusions** 

Hb conc. (mMcm)


**5** 

*Germany* 

**Towards Model-Based Brain Imaging** 

Brain imaging has been a key technology in advancing our understanding of the neuronal basis of cognition. However, in order to fully unleash the power of brain imaging it needs to be combined with other sources of information such as gene expression, behavioural

In so-called "model-based brain imaging" computational models of how the brain processes information are employed in order to interpret the data. A prominent example is the use of models from reinforcement learning in order to interpret responses in basal ganglia, or frontal cortex. However, while such a model-based brain imaging certainly adds a new level of explanation beyond the mere description of the evoked neuronal activations, it will always be

Here, we argue that the application of multi-scale modeling, which bridges the gap between the various spatial and temporal scales, is a necessary next step in the analysis of brain imaging data. For example, this way it will be possible to simulate the effects of altered membrane currents under pharmacological manipulation on brain-wide network dynamics

We emphasize that one important benefit of models is to operationalize as many assumptions as possible (Section 2). Then, we summarize the state-of-the-art in Neuroinformatics tool support, which is necessary for multi-scale models in model-based brain imaging. We also argue that even without complete physical models, which bridge the various scales, a data-driven approach using partly phenomenological model can be pursued, but this calls for new Neuroinformatics tools (Section 3). We summarize some of our recent work in network modeling of the visual system, where we performed systematic model comparisons (Section 4), and we close with identifying a challenging test case for multi-scale modeling in model-based brain imaging, namely the investigation of the

**2. Two dimensions of theory-dependence of observations in neuroscience**  Brain imaging studies are already heavily dependent on various kinds of models. We argue that a key advantage of models in brain imaging (and in the interpretation of neuronal data

limited by the spatial and temporal resolution of the employed imaging technologies.

and compare simulation results with recorded brain imaging data.

**1. Introduction** 

performance, as well as with computational models.

neuronal basis of the self (Section 5).

**with Multi-Scale Modeling** 

*Dept of Computer Science and Electrical Engineering* 

Lars Schwabe and Youwei Zheng *Adaptive and Regenerative Software Systems* 

*Universität Rostock, Rostock,* 

