**5. Application programs for eye-gaze input**

Our research group has developed application programs for eye-gaze input to assist ALS patients. The interface of the application programs employs indicators displayed on the PC monitor, as shown in Section 4-2. We present our application programs below.

#### **5.1. Text input system**

Text input is the most important function to aid communication by ALS patients. Inputting text by eye gaze increases the convenience of their communication. We designed indicators for text input by eye gaze, considering the success rate of gaze selection with our proposed system [9]. There are 12 indicators (2 rows and 6 columns). With this system, users can input Japanese or English text at a faster pace. However, around 60 indicators are required to input Japanese text. Similarly, 12 indicators are insufficient for English text input, because the English language contains uppercase letters, lowercase letters, and punctuation marks. To resolve this problem, we designed a new interface through which users can select any character (English or Japanese) by first choosing the indicator group. An overview of the interface is shown in Figure 8.

This interface requires two selections: one for character group selection (e.g., Group "A–E") and another for character input. Letters require two selections, and punctuation marks ("etc." in Figure 8) also require two. However, commonly used characters such as the space ("SPC" in Figure 8) require just one selection. To input the character "C," the user first selects the indicator for Group "A–E" and then the indicator for the uppercase letter "C." Japanese characters can be input in the same way.

This interface requires two selections: one for character group selection (e.g., Group "A–E") and another for character input. Letters require two selections, and punctuation marks ("etc." in Figure 8) also require two. However, commonly used characters such as Eye-Gaze Input System Suitable for Use under Natural Light and Its Applications Toward a Support for ALS Patients http://dx.doi.org/10.5772/56560 251

E" and then the indicator for the uppercase letter "C." Japanese characters can be input in the same way.

first choosing the indicator group. An overview of the interface is shown in Figure 8.

general Windows software cannot recognize the icon or menu item that is gazed at by the user.

Generally, the eye-gaze input decision with such an interface requires the detection of not only the user's gaze point but also the user's command for an indicator (assigned character) selection. An input decision can be made by using eye fixation, measuring the time for which the eye fixates on a target such as one of the indicators. The abovementioned interface using indicator selection requires special application programs. However, the operability of the system can be increased by using suitably designed indicators. The users need to sufficiently practice operating this system to operate the application programs at a faster pace.

When users operate a PC with the mouse, they gaze at the mouse cursor routinely. In other words, if the mouse cursor can be moved to the user's gaze point, the eye gaze of the user can be utilized for an input interface. Our eye-gaze input system can obtain the coordinates of the user's gaze point. In other words, when a user gazes at a point on the PC screen, the mouse cursor moves to

If the mouse cursor is controlled by eye gaze, the user gazes over the entire area of screen. Hence, the eye gaze of the user must be detected with a high degree of accuracy. Therefore, a system with this interface is calibrated using the indicators shown in Figure 6(b). By using an interface for mouse operation, the general Windows software can be operated without any special application programs. In addition, Windows operations such as copying a file can be performed by eye gaze. The method for operating this

We developed special application programs for eye-gaze input. However, users may want to run other Windows software. By selecting the abovementioned interface, users can fulfill this desire. However, the icons and menu items of the general Windows software are small for eye-gaze input. In other words, it is difficult to select the icons and menu items with mouse operations by eye gaze. When users gaze at one point on the object viewed, their eye fixation has micromotions (called involuntary eye movements). Therefore, it is difficult to keep the mouse cursor on the viewed object for the time required for eye-gaze input. In addition, the general Windows software and the eye-gaze detection software are executed on Windows separately. Hence, the

To resolve these issues, the interface for mouse operation requires a different method for input decisions. We think that an eye blink should provide the information used in this input decision method. The details of this method are given in Section 6.

Our research group has developed application programs for eye-gaze input to assist ALS patients. The interface of the application programs employs indicators displayed on the PC monitor, as shown in Section 4-2. We present our application programs below.

Text input is the most important function to aid communication by ALS patients. Inputting text by eye gaze increases the convenience of their communication. We designed indicators for text input by eye gaze, considering the success rate of gaze selection with our proposed system [9]. There are 12 indicators (2 rows and 6 columns). With this system, users can input Japanese or English text at a faster pace. However, around 60 indicators are required to input Japanese text. Similarly, 12 indicators are insufficient for English text input, because the English language contains uppercase letters, lowercase letters, and punctuation marks. To resolve this problem, we designed a new interface through which users can select any character (English or Japanese) by

the space ("SPC" in Figure 8) require just one selection. To input the character "C," the user first selects the indicator for Group "A–

Figure 8. Text input system **Figure 8.** Text input system

interface for mouse operation, the general Windows software can be operated without any special application programs. In addition, Windows operations such as copying a file can be performed by eye gaze. The method for operating this interface is clear and simple; therefore,

We developed special application programs for eye-gaze input. However, users may want to run other Windows software. By selecting the abovementioned interface, users can fulfill this desire. However, the icons and menu items of the general Windows software are small for eyegaze input. In other words, it is difficult to select the icons and menu items with mouse operations by eye gaze. When users gaze at one point on the object viewed, their eye fixation has micromotions (called involuntary eye movements). Therefore, it is difficult to keep the mouse cursor on the viewed object for the time required for eye-gaze input. In addition, the general Windows software and the eye-gaze detection software are executed on Windows separately. Hence, the general Windows software cannot recognize the icon or menu item that

To resolve these issues, the interface for mouse operation requires a different method for input decisions. We think that an eye blink should provide the information used in this input decision

Our research group has developed application programs for eye-gaze input to assist ALS patients. The interface of the application programs employs indicators displayed on the PC

Text input is the most important function to aid communication by ALS patients. Inputting text by eye gaze increases the convenience of their communication. We designed indicators for text input by eye gaze, considering the success rate of gaze selection with our proposed system [9]. There are 12 indicators (2 rows and 6 columns). With this system, users can input Japanese or English text at a faster pace. However, around 60 indicators are required to input Japanese text. Similarly, 12 indicators are insufficient for English text input, because the English language contains uppercase letters, lowercase letters, and punctuation marks. To resolve this problem, we designed a new interface through which users can select any character (English or Japanese) by first choosing the indicator group. An overview of the interface is shown in

This interface requires two selections: one for character group selection (e.g., Group "A–E") and another for character input. Letters require two selections, and punctuation marks ("etc." in Figure 8) also require two. However, commonly used characters such as the space ("SPC" in Figure 8) require just one selection. To input the character "C," the user first selects the indicator for Group "A–E" and then the indicator for the uppercase letter "C." Japanese

monitor, as shown in Section 4-2. We present our application programs below.

method. The details of this method are given in Section 6.

**5. Application programs for eye-gaze input**

this interface is user-friendly.

250 Current Advances in Amyotrophic Lateral Sclerosis

is gazed at by the user.

**5.1. Text input system**

characters can be input in the same way.

Figure 8.

#### **5.2. Support system for personal computer operation**

functions can be performed.

**4.3. Interface for mouse operation** 

interface is clear and simple; therefore, this interface is user-friendly.

**5. Application programs for eye-gaze input** 

**5.1. Text input system** 

that point.

If users can operate general Windows functions by eye gaze, they can operate commonly used application programs such as mailers and Web browsers. Users can input text to these applications using the abovementioned interface for text input. Such applications are normally operated by keyboard or mouse, especially the latter. When an eye-gaze input system is used, the functions of the application programs must be assigned to indicators. We have extended our system to general Windows functions. Many guidelines have been proposed for the development of application programs for the disabled. To satisfy these guidelines, we assign the following Windows functions to indicators: cursor control; execution of application programs; use of shortcut keys to copy, cut, and paste; and selection of items from a menu bar. Hence, commercial applications can be used with our system [9]. The Windows functions are organized as shown in Figure 9, and the user can switch indicator group. The "main operation screen" has indicators for cursor operation, object selection, decision input (enter), etc. The "extended operation screen" has indicators for operating the mouse, activating the desktop, switching, or closing the window, etc. Using these indicators, all general Windows functions can be performed. **5.2. Support system for personal computer operation**  If users can operate general Windows functions by eye gaze, they can operate commonly used application programs such as mailers and Web browsers. Users can input text to these applications using the abovementioned interface for text input. Such applications are normally operated by keyboard or mouse, especially the latter. When an eye-gaze input system is used, the functions of the application programs must be assigned to indicators. We have extended our system to general Windows functions. Many guidelines have been proposed for the development of application programs for the disabled. To satisfy these guidelines, we assign the following Windows functions to indicators: cursor control; execution of application programs; use of shortcut keys to copy, cut, and paste; and selection of items from a menu bar. Hence, commercial applications can be used with our system [9]. The Windows functions are organized as shown in Figure 9, and the user can switch indicator group. The "main operation screen" has indicators for cursor operation, object selection, decision input (enter), etc. The "extended operation screen" has indicators for operating the mouse, activating the desktop, switching, or closing the window, etc. Using these indicators, all general Windows

**5.3. Mailer software for sending fixed phrases** 

this mailer software is shown in Figure 10.

Fixed phrase area

Figure 10.Mailer software for sending fixed messages

Extended operation screen

Letter input area

As described in Section 4-3, general Windows functions can be extended by controlling the mouse cursor by eye gaze. However,

By using the text input system described in Section 5-1, users can input English text by eye gaze at approximately 16 characters per minute [9]. This input rate is not adequate to send an emergency message. To resolve this concern, we developed mailer software for sending fixed phrases by eye gaze. This software requires only a few steps for sending a message. In addition, combinations of the fixed phrases can be sent, and each phrase is customizable. Users can send a message to a pager or a smart phone outside the room. Therefore, users can communicate their requests (such as "I would like a drink of water") to their helpers. A screenshot of

indicators that include the commonly used functions actualize a comfortable and high-speed operation of Windows.

Figure 9. Support system for personal computer operation **Figure 9.** Support system for personal computer operation

As described in Section 4-3, general Windows functions can be extended by controlling the mouse cursor by eye gaze. However, indicators that include the commonly used functions actualize a comfortable and high-speed operation of Windows.

Web browser

"↓" is selected.

**5.4. Web browser using eye gaze** 

"→"is selected. Weather

Eye-Gaze Input System Suitable for Use under Natural Light and Its Applications Toward a Support for ALS Patients

With the popularization of the Internet, people now frequently browse Web pages to collect information. We paid great attention to this point; hence, we developed a Web browser for the eye-gaze input system. Generally, a Web page is related to others via hyperlinks. In addition, users often input text to a Web page when using a social networking service (SNS), online shopping, etc. When browsing these Web pages, the users make selections via hyperlinks, radio buttons, and text boxes that must be detected on the Web pages. Our system determines the locations of these selectable objects on a Web page. The system then stores the locations of these objects. Consequently, the mouse cursor jumps to the object of the candidate input. Therefore, our system enables Web browsing at a faster pace. An overview of the object selection method that uses information on the arrangement is shown in Figure

http://dx.doi.org/10.5772/56560

253

Studies have reported that the three principal functions of an environmental control system are for a television, reclining bed, and air conditioner. In other words, physically disabled people such as ALS patients would like to operate the functions of these devices. We focused our attention on a PC with a television tuner, and developed a television program viewing system for eyegaze input. This system displays television programs on the PC screen along with the indicators for function control. The five indicators for the television program viewing system are displayed in the upper part of the screen as shown in Figure 12. The functions of a channel selector, volume control, and power switch are assigned to the five indicators. When users view television programs, the indicators are not required. Therefore, we set up two modes designated as viewer mode and control mode. In control mode, the five indicators are displayed on the screen. In viewer mode, the five indicators are not displayed, but an indicator for mode change is displayed. If the user gazes at the indicator for mode change, the other five indicators appear instead.

**off** 

As described above, we have developed not only an eye-gaze input system for natural light but also an application system. When the application programs are used in combination, the quality of life (QoL) of ALS patients is improved. However, in order to provide additional improvements in QoL, a more versatile environment for eye-gaze input is required. For example, some users would like to explore the newer Web services, such as Facebook and Twitter. It is difficult to develop new software for these users individually. To resolve this problem, we need to improve our interface for mouse operation by eye gaze (presented in Section 4-3).

Studies have reported that the three principal functions of an environmental control system are for a television, reclining bed, and air conditioner. In other words, physically disabled people such as ALS patients would like to operate the functions of these devices. We focused our attention on a PC with a television tuner, and developed a television program viewing system for eye-gaze input. This system displays television programs on the PC screen along with the indicators for function control. The five indicators for the television program viewing system are displayed in the upper part of the screen as shown in Figure 12. The functions of a channel selector, volume control, and power switch are assigned to the five indicators. When users view television programs, the indicators are not required. Therefore, we set up two modes designated as viewer mode and control mode. In control mode, the five indicators are displayed on the screen. In viewer mode, the five indicators are not displayed, but an indicator for mode change is displayed. If the user gazes at the indicator for mode change, the other five

"→" Weather is selected.

Figure 11.Object selection method using arrangement information

**↑ CH ↓ ↑ Vol. ↓ TV**

TV screen

**5.5. Television program viewing system** 

Figure 12.Television program viewing system

**6. Next-generation eye-gaze input system** 

As described above, we have developed not only an eye-gaze input system for natural light but also an application system. When the application programs are used in combination, the

News

Sports

Web browser

"↓" is selected.

News

Sports

**Figure 11.** Object selection method using arrangement information

11.

**5.5. Television program viewing system**

indicators appear instead.

**Figure 12.** Television program viewing system

**6. Next-generation eye-gaze input system**

#### **5.3. Mailer software for sending fixed phrases**

By using the text input system described in Section 5-1, users can input English text by eye gaze at approximately 16 characters per minute [9]. This input rate is not adequate to send an emergency message. To resolve this concern, we developed mailer software for sending fixed phrases by eye gaze. This software requires only a few steps for sending a message. In addition, combinations of the fixed phrases can be sent, and each phrase is customizable. Users can send a message to a pager or a smart phone outside the room. Therefore, users can communicate their requests (such as "I would like a drink of water") to their helpers. A screenshot of this mailer software is shown in Figure 10.

**Figure 10.** Mailer software for sending fixed messages

#### **5.4. Web browser using eye gaze**

With the popularization of the Internet, people now frequently browse Web pages to collect information. We paid great attention to this point; hence, we developed a Web browser for the eye-gaze input system. Generally, a Web page is related to others via hyperlinks. In addition, users often input text to a Web page when using a social networking service (SNS), online shopping, etc. When browsing these Web pages, the users make selections via hyperlinks, radio buttons, and text boxes that must be detected on the Web pages. Our system determines the locations of these selectable objects on a Web page. The system then stores the locations of these objects. Consequently, the mouse cursor jumps to the object of the candidate input. Therefore, our system enables Web browsing at a faster pace. An overview of the object selection method that uses information on the arrangement is shown in Figure 11.

of these objects. Consequently, the mouse cursor jumps to the object of the candidate input. Therefore, our system enables Web

In control mode, the five indicators are displayed on the screen. In viewer mode, the five indicators are not displayed, but an indicator for mode change is displayed. If the user gazes at the indicator for mode change, the other five indicators appear instead.

would like to explore the newer Web services, such as Facebook and Twitter. It is difficult to develop new software for these users

**Figure 11.** Object selection method using arrangement information browsing at a faster pace. An overview of the object selection method that uses information on the arrangement is shown in Figure 11.

#### **5.5. Television program viewing system** Web browser

As described in Section 4-3, general Windows functions can be extended by controlling the mouse cursor by eye gaze. However, indicators that include the commonly used functions

By using the text input system described in Section 5-1, users can input English text by eye gaze at approximately 16 characters per minute [9]. This input rate is not adequate to send an emergency message. To resolve this concern, we developed mailer software for sending fixed phrases by eye gaze. This software requires only a few steps for sending a message. In addition, combinations of the fixed phrases can be sent, and each phrase is customizable. Users can send a message to a pager or a smart phone outside the room. Therefore, users can communicate their requests (such as "I would like a drink of water") to their helpers. A screenshot of this

Letter input

area

With the popularization of the Internet, people now frequently browse Web pages to collect information. We paid great attention to this point; hence, we developed a Web browser for the eye-gaze input system. Generally, a Web page is related to others via hyperlinks. In addition, users often input text to a Web page when using a social networking service (SNS), online shopping, etc. When browsing these Web pages, the users make selections via hyperlinks, radio buttons, and text boxes that must be detected on the Web pages. Our system determines the locations of these selectable objects on a Web page. The system then stores the locations of these objects. Consequently, the mouse cursor jumps to the object of the candidate input. Therefore, our system enables Web browsing at a faster pace. An overview of the object

selection method that uses information on the arrangement is shown in Figure 11.

actualize a comfortable and high-speed operation of Windows.

**5.3. Mailer software for sending fixed phrases**

252 Current Advances in Amyotrophic Lateral Sclerosis

mailer software is shown in Figure 10.

Fixed phrase

area

**Figure 10.** Mailer software for sending fixed messages

**5.4. Web browser using eye gaze**

Studies have reported that the three principal functions of an environmental control system are for a television, reclining bed, and air conditioner. In other words, physically disabled people such as ALS patients would like to operate the functions of these devices. We focused our attention on a PC with a television tuner, and developed a television program viewing system for eye-gaze input. This system displays television programs on the PC screen along with the indicators for function control. The five indicators for the television program viewing system are displayed in the upper part of the screen as shown in Figure 12. The functions of a channel selector, volume control, and power switch are assigned to the five indicators. When users view television programs, the indicators are not required. Therefore, we set up two modes designated as viewer mode and control mode. In control mode, the five indicators are displayed on the screen. In viewer mode, the five indicators are not displayed, but an indicator for mode change is displayed. If the user gazes at the indicator for mode change, the other five indicators appear instead. Figure 11.Object selection method using arrangement information **5.5. Television program viewing system**  Studies have reported that the three principal functions of an environmental control system are for a television, reclining bed, and air conditioner. In other words, physically disabled people such as ALS patients would like to operate the functions of these devices. We focused our attention on a PC with a television tuner, and developed a television program viewing system for eyegaze input. This system displays television programs on the PC screen along with the indicators for function control. The five indicators for the television program viewing system are displayed in the upper part of the screen as shown in Figure 12. The functions of a channel selector, volume control, and power switch are assigned to the five indicators. When users view television programs, the indicators are not required. Therefore, we set up two modes designated as viewer mode and control mode. News Sports "→" Weather is selected. "↓" is selected.


Figure 12.Television program viewing system **Figure 12.** Television program viewing system

#### As described above, we have developed not only an eye-gaze input system for natural light but also an application system. When the application programs are used in combination, the quality of life (QoL) of ALS patients is improved. However, in order to provide additional improvements in QoL, a more versatile environment for eye-gaze input is required. For example, some users **6. Next-generation eye-gaze input system**

individually. To resolve this problem, we need to improve our interface for mouse operation by eye gaze (presented in Section 4-3). As described above, we have developed not only an eye-gaze input system for natural light but also an application system. When the application programs are used in combination, the

**6. Next-generation eye-gaze input system** 

quality of life (QoL) of ALS patients is improved. However, in order to provide additional improvements in QoL, a more versatile environment for eye-gaze input is required. For example, some users would like to explore the newer Web services, such as Facebook and Twitter. It is difficult to develop new software for these users individually. To resolve this problem, we need to improve our interface for mouse operation by eye gaze (presented in Section 4-3).

phrase mailer, Web browser, and television program viewing system. When these programs

Eye-Gaze Input System Suitable for Use under Natural Light and Its Applications Toward a Support for ALS Patients

http://dx.doi.org/10.5772/56560

255

Our eye-gaze input system can obtain the coordinates of the user's gaze point. Accordingly, when a user gazes at a point on the PC screen, the mouse cursor moves to that point. By using this input interface, users can operate the general application software of Windows. In addition, our system is expected to contribute to the development of a next-generation eyegaze input system. This new eye-gaze input system will be developed using our new method for eye-gaze and eye-blink detection. We believe that our new eye-gaze input system can

and Ohyama Minoru2

1 College of Engineering, Kanto Gakuin University, Kanazawa-ku, Yokohama-shi, Kanaga‐

[1] Huchinson T.E., White K.P., Martin W.N. Jr., Reichert K.C., Frey, L.A. Human-com‐ puter Interaction using Eye-gaze Input. IEEE Transactions on Systems, Man, and Cy‐

[2] Ward D.J., MacKay D.J.C. Fast Hands-free Writing by Gaze Direction. Nature, 2002

[3] Hansen J.P., Torning K., Johansen A.S., Itoh K., Aoki H. Gaze Typing Compared with Input by Head and Hand. Proceedings of Eye Tracking Research and Applications

[4] Corno F., Farinetti L., Signorile I. A Cost-effective Solution for Eye-gaze Assistive Technology. Proceedings of IEEE International Conference on Multimedia and Expo,

[5] Kim K.N., Ramakrishna R.S. Vision-based Eye-gaze Tracking for Human Computer Interface. Proceedings of IEEE International Conference on Systems, Man and Cyber‐

Symposium on Eye Tracking Research and Applications, 2004, 131-138

2 School of Information Environment, Tokyo Denki University, Inzai-shi, Chiba, Japan

are used in combination, the QoL of ALS patients is improved.

ameliorate the QoL of ALS patients.

, Ohi Shoichi2

bernetics, 1989;19(7) 1527-1534.

\*Address all correspondence to: abe@kanto-gakuin.ac.jp

**Author details**

Abe Kiyohiko1

wa, Japan

**References**

418; 838.

2002;2.433-436.

netics, 1999;2, 324-329.

As shown in Section 4-2, if a user gazes at the indicator for a desired input, that input is easily decided upon, because the application program can recognize the indicator viewed. The interface for mouse operation can move the cursor to the gaze point of the user; however, it is difficult for this type of interface to recognize the icon viewed. To resolve this problem fundamentally, we are developing an interface that utilizes information on eye gaze and eye blinks. Many such interfaces have been proposed, but no truly practical system has been developed. When using this type of interface, unconscious eye blinks occur. In other words, the input errors are often attributable to unconscious blinks. This phenomenon is known as the "Midas touch problem."

We think that if involuntary (unconscious) blinks can be recognized, the input errors can be significantly decreased. In fact, we are presently developing an eye-gaze input system that can recognize voluntary blinks. Most conventional methods for measuring eye blinks analyze images of the eye (and its surrounding skin) captured by a video camera. Commonly used NTSC video cameras are capable of detecting eye blinks. However, it is difficult for these cameras to measure the detailed temporal changes that occur during the process of eye blinking, because an eye blink occurs relatively fast (within a few hundred milliseconds). The eye-gaze input system also uses an NTSC camera and therefore it is necessary to take account of this problem.

NTSC video cameras capture moving images at 60 fields/s, and these field images are mixed to produce field-interlaced images at a rate of 30 frames/s (fps). We have proposed a new method for using NTSC video cameras to measure eye blinks [13]. This method utilizes the non-interlaced eye images captured by an NTSC video camera. These images are odd- and even-field images in the NTSC format and are generated by splitting NTSC frames (interlaced images). The proposed method has a time resolution that is twice that of the NTSC format. Therefore, the detailed temporal changes that occur during the process of eye blinking can be measured. By using this new method for eye blink detection, we can develop a next-generation eye-gaze input system that is more user-friendly.
