**4. Input interfaces based on eye gaze**

We developed a new eye-gaze input system using the methods discussed in Section 3. This system detects the eye gaze of a user under natural light and operates the application programs for communication aids such as text input. Two interfaces to operate the application programs have been developed. One of the interfaces has indicators displayed on the PC monitor. The functions of application programs are executed by gazing at these indicators. The other interface allows eye gaze to control the mouse cursor. By means of this interface, a user can operate the general Windows software. We describe our eye-gaze input system and its input interface below.

#### **4.1. Eye-gaze input system**

Our eye-gaze input system comprises a PC, a home video camera, and an IEEE 1394 interface for image capture from the camera. For eye-gaze detection, the computer runs image analysis software on Windows (XP, Vista, or 7). This system (illustrated in Figure 5) does not require a device exclusively for image processing. The characteristics of eye gaze vary from one individual to another. Therefore, the eye-gaze input system requires calibration. The indicators for calibration are shown in Figure 6. Users must calibrate the system before using it for tasks. After the camera location is adjusted, the calibration begins. While the calibration is being performed, users gaze at each indicator when its color switches to red. Our eye-gaze input system has two types of indicators, which are specific to each application. In particular, the five calibration indicators shown in Figure 6(a) are used for the input interface with a work‐ space displayed at the center of the PC monitor. The workspace is used for displaying an application software window. In addition, the nine calibration indicators shown in Figure 6(b) are used for the interface to operate the mouse cursor, because this interface requires a higher accuracy of measurement.

eye-gaze input system needs 6 to 12 indicators. In particular, some kinds of application programs require 6 to 10 indicators. These application programs utilize the arrangement in Figure 7(b). For example, fixed-phrase mailers or Web browsers use this arrangement pattern.

Eye-Gaze Input System Suitable for Use under Natural Light and Its Applications Toward a Support for ALS Patients

http://dx.doi.org/10.5772/56560

249

(a) (b) (c)

In addition, text input is a popular application for eye-gaze input. Around 60 indicators are required to input Japanese text. In fact, English text input systems require a similar number of indicators, because the English language contains uppercase letters, lowercase letters, and punctuation marks. Moreover, control keys are required for text input. If around 60 indicators are displayed on a screen, the window for text input cannot be arranged on the same screen. In other words, its operability is greatly decreased. Therefore, we developed a text input system for Japanese and English using the indicators shown in Figure 7(c). Its interface requires two selections: one for character group selection and another for character input (the details are given in Section 5). For an eye-gaze input system with the indicators shown in Figure 7(a), (b), or (c), there is no necessity to detect eye gaze when the user gazes at the center of the PC monitor. Therefore, an eye-gaze input system using any of these arrangements is calibrated

Generally, the eye-gaze input decision with such an interface requires the detection of not only the user's gaze point but also the user's command for an indicator (assigned character) selection. An input decision can be made by using eye fixation, measuring the time for which the eye fixates on a target such as one of the indicators. The abovementioned interface using indicator selection requires special application programs. However, the operability of the system can be increased by using suitably designed indicators. The users need to sufficiently

When users operate a PC with the mouse, they gaze at the mouse cursor routinely. In other words, if the mouse cursor can be moved to the user's gaze point, the eye gaze of the user can be utilized for an input interface. Our eye-gaze input system can obtain the coordinates of the user's gaze point. In other words, when a user gazes at a point on the PC screen, the mouse

If the mouse cursor is controlled by eye gaze, the user gazes over the entire area of screen. Hence, the eye gaze of the user must be detected with a high degree of accuracy. Therefore, a system with this interface is calibrated using the indicators shown in Figure 6(b). By using an

practice operating this system to operate the application programs at a faster pace.

**Figure 7.** Arrangement patterns of indicators

with the simple indicators shown in Figure 6(a).

**4.3. Interface for mouse operation**

cursor moves to that point.

**Figure 5.** Appearance of proposed system

**Figure 6.** Indicators for calibration

#### **4.2. Interface using indicators displayed on PC monitor**

An interface suitable for eye-gaze input can be designed when developing application programs for an eye-gaze input system. For example, an interface using indicators is most commonly used. The indicators are displayed on the windows of the application program and are selected by the gaze of the user. The arrangement of indicators depends on the measure‐ ment accuracy of the eye-gaze input system. Our system treats each indicator as one of 27 indicators (3 rows and 9 columns), which permits high accuracy. However, in the interest of usefulness, our practical eye-gaze input system utilizes an interface with 5 to 12 indicators. The arrangement patterns of the indicators are shown in Figure 7.

The arrangement pattern in Figure 7(a) is used when the eye-gaze input system needs fewer than five indicators. This arrangement pattern can be used when the application program requires a small number of indicators. However, it can demand a wide display area for the application program. Therefore, this arrangement is best used by application programs such as a television program viewer. The arrangements in Figures 7(b) and (c) are used when the eye-gaze input system needs 6 to 12 indicators. In particular, some kinds of application programs require 6 to 10 indicators. These application programs utilize the arrangement in Figure 7(b). For example, fixed-phrase mailers or Web browsers use this arrangement pattern.

**Figure 7.** Arrangement patterns of indicators

6(b) are used for the interface to operate the mouse cursor, because this interface requires a

Video camera

(a) (b)

An interface suitable for eye-gaze input can be designed when developing application programs for an eye-gaze input system. For example, an interface using indicators is most commonly used. The indicators are displayed on the windows of the application program and are selected by the gaze of the user. The arrangement of indicators depends on the measure‐ ment accuracy of the eye-gaze input system. Our system treats each indicator as one of 27 indicators (3 rows and 9 columns), which permits high accuracy. However, in the interest of usefulness, our practical eye-gaze input system utilizes an interface with 5 to 12 indicators.

The arrangement pattern in Figure 7(a) is used when the eye-gaze input system needs fewer than five indicators. This arrangement pattern can be used when the application program requires a small number of indicators. However, it can demand a wide display area for the application program. Therefore, this arrangement is best used by application programs such as a television program viewer. The arrangements in Figures 7(b) and (c) are used when the

PC

**4.2. Interface using indicators displayed on PC monitor**

The arrangement patterns of the indicators are shown in Figure 7.

User

higher accuracy of measurement.

248 Current Advances in Amyotrophic Lateral Sclerosis

**Figure 5.** Appearance of proposed system

**Figure 6.** Indicators for calibration

In addition, text input is a popular application for eye-gaze input. Around 60 indicators are required to input Japanese text. In fact, English text input systems require a similar number of indicators, because the English language contains uppercase letters, lowercase letters, and punctuation marks. Moreover, control keys are required for text input. If around 60 indicators are displayed on a screen, the window for text input cannot be arranged on the same screen. In other words, its operability is greatly decreased. Therefore, we developed a text input system for Japanese and English using the indicators shown in Figure 7(c). Its interface requires two selections: one for character group selection and another for character input (the details are given in Section 5). For an eye-gaze input system with the indicators shown in Figure 7(a), (b), or (c), there is no necessity to detect eye gaze when the user gazes at the center of the PC monitor. Therefore, an eye-gaze input system using any of these arrangements is calibrated with the simple indicators shown in Figure 6(a).

Generally, the eye-gaze input decision with such an interface requires the detection of not only the user's gaze point but also the user's command for an indicator (assigned character) selection. An input decision can be made by using eye fixation, measuring the time for which the eye fixates on a target such as one of the indicators. The abovementioned interface using indicator selection requires special application programs. However, the operability of the system can be increased by using suitably designed indicators. The users need to sufficiently practice operating this system to operate the application programs at a faster pace.

#### **4.3. Interface for mouse operation**

When users operate a PC with the mouse, they gaze at the mouse cursor routinely. In other words, if the mouse cursor can be moved to the user's gaze point, the eye gaze of the user can be utilized for an input interface. Our eye-gaze input system can obtain the coordinates of the user's gaze point. In other words, when a user gazes at a point on the PC screen, the mouse cursor moves to that point.

If the mouse cursor is controlled by eye gaze, the user gazes over the entire area of screen. Hence, the eye gaze of the user must be detected with a high degree of accuracy. Therefore, a system with this interface is calibrated using the indicators shown in Figure 6(b). By using an interface for mouse operation, the general Windows software can be operated without any special application programs. In addition, Windows operations such as copying a file can be performed by eye gaze. The method for operating this interface is clear and simple; therefore, this interface is user-friendly.

Generally, the eye-gaze input decision with such an interface requires the detection of not only the user's gaze point but also the user's command for an indicator (assigned character) selection. An input decision can be made by using eye fixation, measuring the time for which the eye fixates on a target such as one of the indicators. The abovementioned interface using indicator selection requires special application programs. However, the operability of the system can be increased by using suitably designed indicators. The users need to sufficiently practice operating this system to operate the application programs at a faster pace.

When users operate a PC with the mouse, they gaze at the mouse cursor routinely. In other words, if the mouse cursor can be moved to the user's gaze point, the eye gaze of the user can be utilized for an input interface. Our eye-gaze input system can obtain the coordinates of the user's gaze point. In other words, when a user gazes at a point on the PC screen, the mouse cursor moves to

If the mouse cursor is controlled by eye gaze, the user gazes over the entire area of screen. Hence, the eye gaze of the user must be detected with a high degree of accuracy. Therefore, a system with this interface is calibrated using the indicators shown in Figure 6(b). By using an interface for mouse operation, the general Windows software can be operated without any special application programs. In addition, Windows operations such as copying a file can be performed by eye gaze. The method for operating this

We developed special application programs for eye-gaze input. However, users may want to run other Windows software. By selecting the abovementioned interface, users can fulfill this desire. However, the icons and menu items of the general Windows software are small for eye-gaze input. In other words, it is difficult to select the icons and menu items with mouse operations by eye gaze. When users gaze at one point on the object viewed, their eye fixation has micromotions (called involuntary eye movements). Therefore, it is difficult to keep the mouse cursor on the viewed object for the time required for eye-gaze input. In addition, the general Windows software and the eye-gaze detection software are executed on Windows separately. Hence, the

To resolve these issues, the interface for mouse operation requires a different method for input decisions. We think that an eye blink should provide the information used in this input decision method. The details of this method are given in Section 6.

Our research group has developed application programs for eye-gaze input to assist ALS patients. The interface of the application programs employs indicators displayed on the PC monitor, as shown in Section 4-2. We present our application programs below.

Text input is the most important function to aid communication by ALS patients. Inputting text by eye gaze increases the convenience of their communication. We designed indicators for text input by eye gaze, considering the success rate of gaze selection with our proposed system [9]. There are 12 indicators (2 rows and 6 columns). With this system, users can input Japanese or English text at a faster pace. However, around 60 indicators are required to input Japanese text. Similarly, 12 indicators are insufficient for English text input, because the English language contains uppercase letters, lowercase letters, and punctuation marks. To resolve this problem, we designed a new interface through which users can select any character (English or Japanese) by

This interface requires two selections: one for character group selection (e.g., Group "A–E") and another for character input. Letters require two selections, and punctuation marks ("etc." in Figure 8) also require two. However, commonly used characters such as the space ("SPC" in Figure 8) require just one selection. To input the character "C," the user first selects the indicator for Group "A–

http://dx.doi.org/10.5772/56560

251

A B C D E Back

Indicators 2 (Group "A –E")

If users can operate general Windows functions by eye gaze, they can operate commonly used application programs such as mailers and Web browsers. Users can input text to these applications using the abovementioned interface for text input. Such applications are normally operated by keyboard or mouse, especially the latter. When an eye-gaze input system is used, the functions of the application programs must be assigned to indicators. We have extended our system to general Windows functions. Many guidelines have been proposed for the development of application programs for the disabled. To satisfy these guidelines, we assign the following Windows functions to indicators: cursor control; execution of application programs; use of shortcut keys to copy, cut, and paste; and selection of items from a menu bar. Hence, commercial applications can be used with our system [9]. The Windows functions are organized as shown in Figure 9, and the user can switch indicator group. The "main operation screen" has indicators for cursor operation, object selection, decision input (enter), etc. The "extended operation screen" has indicators for operating the mouse, activating the desktop, switching, or closing the window, etc. Using these indicators, all general Windows

As described in Section 4-3, general Windows functions can be extended by controlling the mouse cursor by eye gaze. However,

By using the text input system described in Section 5-1, users can input English text by eye gaze at approximately 16 characters per minute [9]. This input rate is not adequate to send an emergency message. To resolve this concern, we developed mailer software for sending fixed phrases by eye gaze. This software requires only a few steps for sending a message. In addition, combinations of the fixed phrases can be sent, and each phrase is customizable. Users can send a message to a pager or a smart phone outside the room. Therefore, users can communicate their requests (such as "I would like a drink of water") to their helpers. A screenshot of

indicators that include the commonly used functions actualize a comfortable and high-speed operation of Windows.

Back

Mouse operation mode

Work area

Extended operation screen

(Indicators for applications)

Letter input area

Sw. Close Desk top

Edit mode Main

a b c d e

**4.3. Interface for mouse operation** 

interface is clear and simple; therefore, this interface is user-friendly.

**5. Application programs for eye-gaze input** 

A–E F-J K-O P-T U-Y Z,

BS TAB

JPN ENT SPC

Indicators 1

**5.1. Text input system** 

Figure 8. Text input system

**5.2. Support system for personal computer operation**

functions can be performed.

Work area

Main operation screen

Text Next Prev. Edit Ext.

Text input mode

**Figure 9.** Support system for personal computer operation

Figure 9. Support system for personal computer operation

Ent.

Sel.

this mailer software is shown in Figure 10.

Fixed phrase area

Figure 10.Mailer software for sending fixed messages

**5.3. Mailer software for sending fixed phrases** 

**Figure 8.** Text input system

can be performed.

To Japanese mode (Main indicators)

general Windows software cannot recognize the icon or menu item that is gazed at by the user.

first choosing the indicator group. An overview of the interface is shown in Figure 8.

Eye-Gaze Input System Suitable for Use under Natural Light and Its Applications Toward a Support for ALS Patients

etc.

If users can operate general Windows functions by eye gaze, they can operate commonly used application programs such as mailers and Web browsers. Users can input text to these applications using the abovementioned interface for text input. Such applications are normally operated by keyboard or mouse, especially the latter. When an eye-gaze input system is used, the functions of the application programs must be assigned to indicators. We have extended our system to general Windows functions. Many guidelines have been proposed for the development of application programs for the disabled. To satisfy these guidelines, we assign the following Windows functions to indicators: cursor control; execution of application programs; use of shortcut keys to copy, cut, and paste; and selection of items from a menu bar. Hence, commercial applications can be used with our system [9]. The Windows functions are organized as shown in Figure 9, and the user can switch indicator group. The "main operation screen" has indicators for cursor operation, object selection, decision input (enter), etc. The "extended operation screen" has indicators for operating the mouse, activating the desktop, switching, or closing the window, etc. Using these indicators, all general Windows functions

**5.2. Support system for personal computer operation** 

E" and then the indicator for the uppercase letter "C." Japanese characters can be input in the same way.

that point.

We developed special application programs for eye-gaze input. However, users may want to run other Windows software. By selecting the abovementioned interface, users can fulfill this desire. However, the icons and menu items of the general Windows software are small for eyegaze input. In other words, it is difficult to select the icons and menu items with mouse operations by eye gaze. When users gaze at one point on the object viewed, their eye fixation has micromotions (called involuntary eye movements). Therefore, it is difficult to keep the mouse cursor on the viewed object for the time required for eye-gaze input. In addition, the general Windows software and the eye-gaze detection software are executed on Windows separately. Hence, the general Windows software cannot recognize the icon or menu item that is gazed at by the user.

To resolve these issues, the interface for mouse operation requires a different method for input decisions. We think that an eye blink should provide the information used in this input decision method. The details of this method are given in Section 6.
