Abstract

In this paper, we present a cooperative game, Brainio Bros 300, using a brain-computer interface (BCI). The game is cooperatively controlled by two people using P300-generating color discrimination. The two users advance through the game together, one as the "player" and the other as the "supporter" providing assistance. We assumed that players would be able-bodied, while supporters would include people with severe disabilities. Through experiments using human subjects, we evaluated the subjects' impressions of the game and its usefulness. The results of the impression evaluation showed that the subjects generally had good impressions, and there were many opinions that such cooperative games are interesting. We also discuss the possibilities of using the P300 BCI.

Keywords: P300, brain computer interface, EEG, cooperative game, game for people with major disabilities, game design

## 1. Introduction

A brain-computer interface (BCI) offers a noninvasive means of enabling a human to send messages and commands directly from his or her brain to a computer without moving, by wearing a simple scalp probe (a set of electrodes or sensors) [1–4]. This noninvasive technology differs from invasive, surgical approaches that can cause irreversible damage to brain tissue.

In this paper, we present a BCI-based cooperative game, Brainio Bros 300. The BCI uses the P300 brain wave [5–9], a typical electrophysiological response to internal and external event-related potential (ERP) stimuli, measured using an EEG. The P300 wave has proven relatively easy to use for a variety of controlsignaling purposes in much recent practical research.

When a human experiences interest in any kind of target, there is a measurable brain activity response. It is known that when a subject recognizes a specified photograph among a series of randomly presented photographs (i.e., the "ah!" response), P300 can be measured around the top of the head. Regardless of the type of stimulus (visual, tactile, auditory, olfactory, gustatory, etc.), P300 appears around 300 ms after the stimulus, which makes it a very useful brain feature that

can be used by healthy or handicapped people for controlling an external device or communicating with the environment in real time. Notable applications of a P300 BCI include previous engineering studies in which a locomotive robot [10] and wheelchair [11] were controlled. Some neuroscientific experimental studies have also been conducted using "brain painting" [12, 13] for patient rehabilitation. A P300 BCI has also been used with virtual reality spaces [14] and as an interface for Twitter and Second Life [15].

Simple BCI games can also be useful for helping a user to control his or her brain activity. Games controlled using EEG signals have been designed to improve the power and duration of concentration, increase the speed and accuracy of brain waves, and improve cognitive function [16]. BCI-based games appear in both medical- and entertainment-focused varieties. Medical applications include games that use a steady-state visual evoked potential (SSVEP) to improve the concentration power necessary to control a BCI [17]. Here, SSVEPs refer to the brain signals induced by using a pattern reversal stimulus (i.e., the use of a checkerboard pattern to stimulate vision) or a flash stimulus (i.e., the use of a flickering light source such as an LED to stimulate vision). Medical games to promote the speedy generation of BCI commands, thus improving the user experience, have also been developed [18]. In the entertainment realm, games have been developed using a number of signals, including P300, to play popular games such as Pong and Tetris and to control a dancing robot [19–22]. Most such games depend on the player's degree of concentration [22].

In this research, we developed Brainio Bros 300 as a game controlled cooperatively by two people using P300-generating color discrimination. The two users advance together through the game, one as the "player" and the other as the "supporter" providing assistance. The player controls the game character by using a keyboard's arrow keys to navigate through a series of colored blocks, while the supporter removes blocks obstructing the player's path by thinking of the appropriate color via the P300 BCI. By cooperating, the participants can reach the end of the game. We assumed that players would be able-bodied, while supporters could include people with severe disabilities.

#### 2. Brainio Bros 300 cooperative game

This section describes the Brainio Bros 300 game, with the discussion divided into two parts: the P300 BCI used and the components and design of the cooperative game.

#### 2.1 P300-based brain-computer interface

We used a new, portable wireless EEG cap called g.Unicorn EEG (g.tec medical engineering, Austria), as shown in Figure 1, for recording brain activity in real time. For our experimental paradigm, we adapted the P300 speller idea to our problem by using MATLAB/Simulink to develop the game-based BCI. Ten dry electrodes were used to record EEG signals at a sampling frequency of 256Hz. The electrodes were placed according to the international 10–20 system, using the Fz (forehead), Cz (crown), P3, Pz, P4, PO7, Oz, and PO8 (all at the back of the head) locations, with references placed at A2 (earlobe) and Fz (forehead), as shown in Figure 2.

blue, green, yellow, purple, and white. The six colors were randomly arranged into two rows of three, with each box flickering 30 times between a photograph of a human face and a color. The subject choses a color and counted in their head how many times that color was displayed. Because research results have indicated that flickering photographs of famous people increases calibration accuracy by increasing the amplitude of the P300 wave, we used images of widely recognizable people, such as Albert Einstein and President Donald Trump. The procedures of the

Calibration of the EEG experiment using the g.Unicorn EEG cap for a P300-based BCI.

A Cooperative Game Using the P300 EEG-Based Brain-Computer Interface

DOI: http://dx.doi.org/10.5772/intechopen.84621

calibration and game phases are detailed below:

Figure 1.

Figure 2.

177

Electrode positions of the eight channels.

Before the cooperative game begins, the first user (designated as the "supporter") had to put on the EEG cap and perform calibration for sending color commands. We developed a program enabling the user to calibrate six colors: red, A Cooperative Game Using the P300 EEG-Based Brain-Computer Interface DOI: http://dx.doi.org/10.5772/intechopen.84621

Figure 1. Calibration of the EEG experiment using the g.Unicorn EEG cap for a P300-based BCI.

Figure 2. Electrode positions of the eight channels.

blue, green, yellow, purple, and white. The six colors were randomly arranged into two rows of three, with each box flickering 30 times between a photograph of a human face and a color. The subject choses a color and counted in their head how many times that color was displayed. Because research results have indicated that flickering photographs of famous people increases calibration accuracy by increasing the amplitude of the P300 wave, we used images of widely recognizable people, such as Albert Einstein and President Donald Trump. The procedures of the calibration and game phases are detailed below:

can be used by healthy or handicapped people for controlling an external device or communicating with the environment in real time. Notable applications of a P300 BCI include previous engineering studies in which a locomotive robot [10] and wheelchair [11] were controlled. Some neuroscientific experimental studies have also been conducted using "brain painting" [12, 13] for patient rehabilitation. A P300 BCI has also been used with virtual reality spaces [14] and as an interface for

Simple BCI games can also be useful for helping a user to control his or her brain activity. Games controlled using EEG signals have been designed to improve the power and duration of concentration, increase the speed and accuracy of brain waves, and improve cognitive function [16]. BCI-based games appear in both medical- and entertainment-focused varieties. Medical applications include games that use a steady-state visual evoked potential (SSVEP) to improve the concentration power necessary to control a BCI [17]. Here, SSVEPs refer to the brain signals induced by using a pattern reversal stimulus (i.e., the use of a checkerboard pattern to stimulate vision) or a flash stimulus (i.e., the use of a flickering light source such as an LED to stimulate vision). Medical games to promote the speedy generation of BCI commands, thus improving the user experience, have also been developed [18]. In the entertainment realm, games have been developed using a number of signals, including P300, to play popular games such as Pong and Tetris and to control a dancing robot [19–22]. Most such games depend on the player's

In this research, we developed Brainio Bros 300 as a game controlled cooperatively by two people using P300-generating color discrimination. The two users advance together through the game, one as the "player" and the other as the "supporter" providing assistance. The player controls the game character by using a keyboard's arrow keys to navigate through a series of colored blocks, while the supporter removes blocks obstructing the player's path by thinking of the appropriate color via the P300 BCI. By cooperating, the participants can reach the end of the game. We assumed that players would be able-bodied, while supporters could

This section describes the Brainio Bros 300 game, with the discussion divided

We used a new, portable wireless EEG cap called g.Unicorn EEG (g.tec medical engineering, Austria), as shown in Figure 1, for recording brain activity in real time. For our experimental paradigm, we adapted the P300 speller idea to our problem by using MATLAB/Simulink to develop the game-based BCI. Ten dry electrodes were used to record EEG signals at a sampling frequency of 256Hz. The electrodes were placed according to the international 10–20 system, using the Fz (forehead), Cz (crown), P3, Pz, P4, PO7, Oz, and PO8 (all at the back of the head) locations, with references placed at A2 (earlobe) and Fz (forehead), as shown in

into two parts: the P300 BCI used and the components and design of the

Before the cooperative game begins, the first user (designated as the "supporter") had to put on the EEG cap and perform calibration for sending color commands. We developed a program enabling the user to calibrate six colors: red,

Twitter and Second Life [15].

Assistive and Rehabilitation Engineering

degree of concentration [22].

include people with severe disabilities.

cooperative game.

Figure 2.

176

2. Brainio Bros 300 cooperative game

2.1 P300-based brain-computer interface

	- 1. Select the colors to be used in calibration, as illustrated in Figure 3. For example, red, blue, green, and yellow can be displayed in order, as shown in the figure under "Item selections."

The information on the selected color was sent by a UDP broadcast on port 1000 and received by a Unity application. As seen in Figure 3, the selected colors were also shown under "Written text," but this was not actually used in the game.

The Brainio Bros 300 cooperative game was developed in Unity 2018 and runs on the Windows 10 operating system. It is a game in the style of Nintendo's Super Mario Bros., in which the player maneuvers a character through various obstacles to reach a goal flag and complete the stage. Along the way, the player encounters blocks that are too high to clear and points where it is impossible to pass, but the

The game contains blocks of five colors: red, blue, green, yellow, and purple. Figure 4 illustrates a green block obstructing the player, preventing him or her from advancing. In such situation, the player communicates an instruction to the supporter (e.g., "green's in the way" or "destroy green"), and the supporter then thinks of the color green. By doing so, the green block in the game can be destroyed, as shown in Figure 5. The game was designed so that it cannot be completed

Because the game would not be enjoyable with only one obstruction, multiple blocks of different colors are placed in one spot, as shown in Figure 4, requiring the player to consider which color to destroy. The game was also designed to offer multiple courses, allowing the player to choose his or her own course. For example, Figure 6 shows a case of two paths. If the red block is destroyed, the player can

player and supporter work together to complete the game.

A Cooperative Game Using the P300 EEG-Based Brain-Computer Interface

without the player and supporter cooperating.

2.2 Brainio Bros 300 design

DOI: http://dx.doi.org/10.5772/intechopen.84621

Figure 4.

Figure 5.

179

Obstructed by the green block.

After destroying the green block.

	- 1. As colors are randomly flashed on the screen, the user looks at the color he or she wants to select and mentally counts its flashes. We used 20 flashes for a balance between selection accuracy and enjoyability of the game.
	- 2. The system guesses which color the user selected according to how the EEG changes. The EEG readings at these moments are measured, and the instants at which the low-frequency stimuli are displayed are averaged as the trigger.

Figure 3. Calibration screen for our P300 experimental paradigm.

The information on the selected color was sent by a UDP broadcast on port 1000 and received by a Unity application. As seen in Figure 3, the selected colors were also shown under "Written text," but this was not actually used in the game.

## 2.2 Brainio Bros 300 design

• Calibration phase

• Game phase

Figure 3.

178

of the game.

averaged as the trigger.

Calibration screen for our P300 experimental paradigm.

2. Begin calibration.

Assistive and Rehabilitation Engineering

1. Select the colors to be used in calibration, as illustrated in Figure 3. For example, red, blue, green, and yellow can be displayed in order, as shown

4.The colors randomly flicker. The user mentally adds to the count every time he or she sees the color red. After red has flickered 30 times, the

5. The system records the speed of the response every time the color red was

6.This process continues three–five more times for all the other colors. Each

1. As colors are randomly flashed on the screen, the user looks at the color he or she wants to select and mentally counts its flashes. We used 20 flashes for a balance between selection accuracy and enjoyability

2. The system guesses which color the user selected according to how the EEG changes. The EEG readings at these moments are measured, and the instants at which the low-frequency stimuli are displayed are

in the figure under "Item selections."

system is configured to stop.

flashed and the EEG information.

3. The system directs the user to look at "red."

color takes 20 s, for a total of around 2 min.

The Brainio Bros 300 cooperative game was developed in Unity 2018 and runs on the Windows 10 operating system. It is a game in the style of Nintendo's Super Mario Bros., in which the player maneuvers a character through various obstacles to reach a goal flag and complete the stage. Along the way, the player encounters blocks that are too high to clear and points where it is impossible to pass, but the player and supporter work together to complete the game.

The game contains blocks of five colors: red, blue, green, yellow, and purple. Figure 4 illustrates a green block obstructing the player, preventing him or her from advancing. In such situation, the player communicates an instruction to the supporter (e.g., "green's in the way" or "destroy green"), and the supporter then thinks of the color green. By doing so, the green block in the game can be destroyed, as shown in Figure 5. The game was designed so that it cannot be completed without the player and supporter cooperating.

Because the game would not be enjoyable with only one obstruction, multiple blocks of different colors are placed in one spot, as shown in Figure 4, requiring the player to consider which color to destroy. The game was also designed to offer multiple courses, allowing the player to choose his or her own course. For example, Figure 6 shows a case of two paths. If the red block is destroyed, the player can

Figure 4. Obstructed by the green block.

Figure 5. After destroying the green block.

advance via the blue block, whereas if the blue block is destroyed, the player can jump onto the red block and continue onward.

possible to destroy the player character's foothold. When the character

A Cooperative Game Using the P300 EEG-Based Brain-Computer Interface

such a BCI by focusing on evaluating the Brainio Bros 300 game.

ter touches the flag, the game is completed.

DOI: http://dx.doi.org/10.5772/intechopen.84621

ment with the supporter as a reference.

each color to its corresponding key.

the game.

3. Game experiment

system [23].

groups, respectively.

table.

181

falls, however, it returns to the start point, from where the player can continue

Such blocks were programmed in advance to behave as such in the game in advance. We included such features to prevent the game from becoming a monotonous experience for the users. A flag is beyond all the obstacles. When the charac-

The game also features obstacles in which blocks move horizontally or vertically.

For experiments using an EEG capable of measuring the P300 brain wave, it is extremely difficult at present to prepare an environment supporting easy calibration for many subjects. For example, even preparing only one EEG device capable of measuring P300 requires a considerable cost, and it takes time to perform the calibration. Therefore, this paper considers the possibilities of games for use with

Specifically, we evaluated Brainio Bros 300 in terms of the user's impression of

For Brainio Bros 300, although the supporter wore the EEG device, the game was actually controlled with a keyboard. We thus conducted an evaluation experi-

The experiment was conducted with 25 students at Future University Hakodate: 19 men and 6 women. Of these students, 12 were designated as supporters and 13 as

20.76 years, with average ages of 20.75 and 20.77 years for the supporter and player

To use the WOZ approach, we created an experimental system using keyboard input to destroy blocks as a substitute for the P300 BCI component of Super Brainio Bros 300. In this system, pressing the "R," "G," "B," "Y," or "C" keys caused the red, green, blue, yellow, or purple blocks, respectively, to be destroyed. For the ease of distinguishing which key corresponded to which color, we applied a sticker of

We also created a post-experiment questionnaire to evaluate the users' impressions and opinions of the usefulness of Super Brainio Bros 300. Table 1 lists the details of the questionnaire, which included questions using the semantic differential (SD) method, a five-point scale method, and free responses. For the five-point scale, the responses consisted of "strongly agree," "agree," "neither," "disagree," and "strongly disagree." The questions are listed with abbreviated forms in the

Table 2 lists the details of each condition for the SD method and the reasons for its selection. Each condition showing a positive impression is filled in gray. On the questionnaire form, the positive and negative responses were distributed between the left and right sides as a counterbalancing measure. The SD method used a sevenstage evaluation, with responses consisting of "extremely" (positive or negative), "very" (positive or negative), "a little" (positive or negative), and "neither."

We also used a keyboard, display, notebook PC, desk, chair, and EEG headset as

experimental materials, arranged as shown in Figure 9.

the game and its usefulness. For our methodology, we used the Wizard of Oz (WOZ) approach. A WOZ system involves a user interacting with a person acting as a computer system (i.e., the "wizard"), allowing for effective simulation of a real

players. The supporters consisted of 8 men and 4 women, while the players consisted of 11 men and 2 women. The average age of the whole group was

The game was designed to destroy (i.e., hide) only the one color currently recognized by the BCI, so that only one color could be destroyed at a time. For example, if a red block is first destroyed and then a green block, the game displays the destroyed red block again. Figure 7 shows a case of the player unable to advance with only a blue block displayed, while Figure 8 shows the result with the blue block destroyed, causing the red blocks to reappear. The game was designed as indicated by these images, so that, even though there are no blocks beyond the blue block, destroying the blue block causes the red blocks to reappear, allowing the player to advance.

This system, with the combined limitations of using only five colors and allowing only one color to be destroyed at a time, gives the player manual control over which blocks are destroyed. Because an unintended color can be destroyed, it is

Figure 6. Start screen with two ways of advancing.

Figure 7. Unable to reach the blue block.

Figure 8. Blue block destroyed, causing red blocks to reappear.

A Cooperative Game Using the P300 EEG-Based Brain-Computer Interface DOI: http://dx.doi.org/10.5772/intechopen.84621

possible to destroy the player character's foothold. When the character falls, however, it returns to the start point, from where the player can continue the game.

The game also features obstacles in which blocks move horizontally or vertically. Such blocks were programmed in advance to behave as such in the game in advance. We included such features to prevent the game from becoming a monotonous experience for the users. A flag is beyond all the obstacles. When the character touches the flag, the game is completed.

### 3. Game experiment

advance via the blue block, whereas if the blue block is destroyed, the player can

The game was designed to destroy (i.e., hide) only the one color currently recognized by the BCI, so that only one color could be destroyed at a time. For example, if a red block is first destroyed and then a green block, the game displays the destroyed red block again. Figure 7 shows a case of the player unable to advance with only a blue block displayed, while Figure 8 shows the result with the blue block destroyed, causing the red blocks to reappear. The game was designed as indicated by these images, so that, even though there are no blocks beyond the blue block, destroying the blue block causes the red blocks to reappear, allowing the

This system, with the combined limitations of using only five colors and allowing only one color to be destroyed at a time, gives the player manual control over which blocks are destroyed. Because an unintended color can be destroyed, it is

jump onto the red block and continue onward.

Assistive and Rehabilitation Engineering

player to advance.

Figure 6.

Figure 7.

Figure 8.

180

Unable to reach the blue block.

Blue block destroyed, causing red blocks to reappear.

Start screen with two ways of advancing.

For experiments using an EEG capable of measuring the P300 brain wave, it is extremely difficult at present to prepare an environment supporting easy calibration for many subjects. For example, even preparing only one EEG device capable of measuring P300 requires a considerable cost, and it takes time to perform the calibration. Therefore, this paper considers the possibilities of games for use with such a BCI by focusing on evaluating the Brainio Bros 300 game.

Specifically, we evaluated Brainio Bros 300 in terms of the user's impression of the game and its usefulness. For our methodology, we used the Wizard of Oz (WOZ) approach. A WOZ system involves a user interacting with a person acting as a computer system (i.e., the "wizard"), allowing for effective simulation of a real system [23].

For Brainio Bros 300, although the supporter wore the EEG device, the game was actually controlled with a keyboard. We thus conducted an evaluation experiment with the supporter as a reference.

The experiment was conducted with 25 students at Future University Hakodate: 19 men and 6 women. Of these students, 12 were designated as supporters and 13 as players. The supporters consisted of 8 men and 4 women, while the players consisted of 11 men and 2 women. The average age of the whole group was 20.76 years, with average ages of 20.75 and 20.77 years for the supporter and player groups, respectively.

To use the WOZ approach, we created an experimental system using keyboard input to destroy blocks as a substitute for the P300 BCI component of Super Brainio Bros 300. In this system, pressing the "R," "G," "B," "Y," or "C" keys caused the red, green, blue, yellow, or purple blocks, respectively, to be destroyed. For the ease of distinguishing which key corresponded to which color, we applied a sticker of each color to its corresponding key.

We also created a post-experiment questionnaire to evaluate the users' impressions and opinions of the usefulness of Super Brainio Bros 300. Table 1 lists the details of the questionnaire, which included questions using the semantic differential (SD) method, a five-point scale method, and free responses. For the five-point scale, the responses consisted of "strongly agree," "agree," "neither," "disagree," and "strongly disagree." The questions are listed with abbreviated forms in the table.

Table 2 lists the details of each condition for the SD method and the reasons for its selection. Each condition showing a positive impression is filled in gray. On the questionnaire form, the positive and negative responses were distributed between the left and right sides as a counterbalancing measure. The SD method used a sevenstage evaluation, with responses consisting of "extremely" (positive or negative), "very" (positive or negative), "a little" (positive or negative), and "neither."

We also used a keyboard, display, notebook PC, desk, chair, and EEG headset as experimental materials, arranged as shown in Figure 9.


#### Table 1. Questionnaire details.

We next describe the experimental procedure. The total experiment time was around 30 minutes. The first step was to gather all the subjects in a room and form them into pairs. The participants were instructed to pair with someone they did not know, where possible. As there was an odd number of participants, resulting in one excess player, a researcher acted as the supporter for that player and did not fill out a questionnaire. After pairing the participants, we explained the experiment to the whole group. This included explanations of the following: (1) an overview of the game (a game controlled by brain waves, requiring the player and supporter to cooperate to reach the goal), (2) the role of the player (to control the character by using the keyboard), (3) the role of the supporter (to assist the player by using brain waves), (4) the control method, and (5) an image of the game screen. The players were then taken into separate rooms containing the experimental materials. Each player was seated in the chair indicated by "Player's chair" in Figure 9 and asked to wait until the supporter arrived.

After the players had been moved, we explained to the supporters that this experiment used the WOZ approach, that brain waves were not actually used, and that they would advance through the game by using the keyboard. The supporters

Details of the conditions used for the SD method, together with the reasons for their use.

A Cooperative Game Using the P300 EEG-Based Brain-Computer Interface

DOI: http://dx.doi.org/10.5772/intechopen.84621

Table 2.

Figure 9.

183

Arrangement of the experimental materials.

A Cooperative Game Using the P300 EEG-Based Brain-Computer Interface DOI: http://dx.doi.org/10.5772/intechopen.84621


#### Table 2.

Details of the conditions used for the SD method, together with the reasons for their use.

#### Figure 9.

We next describe the experimental procedure. The total experiment time was around 30 minutes. The first step was to gather all the subjects in a room and form them into pairs. The participants were instructed to pair with someone they did not know, where possible. As there was an odd number of participants, resulting in one excess player, a researcher acted as the supporter for that player and did not fill out a questionnaire. After pairing the participants, we explained the experiment to the whole group. This included explanations of the following: (1) an overview of the game (a game controlled by brain waves, requiring the player and supporter to cooperate to reach the goal), (2) the role of the player (to control the character by using the keyboard), (3) the role of the supporter (to assist the player by using brain waves), (4) the control method, and (5) an image of the game screen. The players were then taken into separate rooms containing the experimental materials. Each player was seated in the chair indicated by "Player's chair" in

Figure 9 and asked to wait until the supporter arrived.

Table 1.

182

Questionnaire details.

Assistive and Rehabilitation Engineering

Arrangement of the experimental materials.

After the players had been moved, we explained to the supporters that this experiment used the WOZ approach, that brain waves were not actually used, and that they would advance through the game by using the keyboard. The supporters

were asked to wear the EEG headset and appear to the player as if they were controlling the game through brain waves. The supporters were then taken to the separate rooms in which the players were waiting, and each was seated in the "Supporter's chair" shown in Figure 9. As soon as we had confirmed that the player and supporter were both seated, a brief explanation was once again provided to both of them. At this time, the participants were instructed to communicate regarding the color of the block when the player wanted a certain block cleared, when the supporter was trying to clear a block, and so on.

After this instruction, the participants began playing the game. After they had completed it, they returned to the room where the initial explanation had been provided, and they filled out the questionnaires. Once they had completed the questionnaires, the experiment was over. At this point, they were instructed not to reveal the details of the experiment.

Finally, we explain our analysis methods for the experimental data. For the SD method in which pairs of words were displayed side by side (e.g., easy vs. difficult) to assess the impression of the participants, they were asked to assign a score in the range of 3 and 3, with 0 representing "neither", 3 representing "extremely leaning to the left word", and 3 representing "extremely leaning to the right word". We then compiled descriptive statistics and plotted a semantic profile.

Next, the five-point scale was tallied with "strongly agree" as 5 points, "agree" as 4 points, "neither" as 3 points, "disagree" as 2 points, and "strongly disagree" as 1 point. After compiling descriptive statistics for this data, we performed a chi-square test.

Finally, for the free responses, we counted experiences and keywords shared among participants and collected them into overall viewpoints. We also collected strongly held minority opinions as necessary.
