**5. An implementation**

Within the proposed human-robot framework, we identify three different groups of people that may interact with the robot. These people, who are known as interactants, are grouped according to their roles, relating to the robot actions, and on the nature of the information they may require.

Group 1 Interactants – *Operator* who is responsible for the control and supervision of the robot. The operator is equipped with an LCD display and the Multi-Modal Human Input Device (HID). A wearable see-through transparent LCD display is provided to allow the operator an unobstructed visual awareness of the environment. The LCD display functionality allows for the projecting of high-resolution dialog actions proposed by the partner-robot. In addition, relevant information required by the operator to allow for timely intervention would also be displayed. This information includes communication strength and battery health. With the augmented view, the operator is able to provide the necessary support and commands to the robot. Navigation through the menu options and selection of specific items are executed using the Multi-Modal Human Input Device.

Group 2 Interactants – *Observers* who are interested in monitoring the tasks being executed. This group of humans may be equipped with the transparent LCD displays where they can share the actions of the human in controlling the tasks. Without the wearable displays, this group of observers would only be able to share in the notifications by the robot through the 'Laser Writer'. They would, however, not be permitted or able to control the actions of the robot, as control is only permitted through the Multi-Modal Human Input Device. This restriction provides a clear differentiation between the two groups.

Group 3 Interactants – *Passerby* who are in the vicinity of interaction, but who are not directly related to, or interested in the task being executed by the robot. The interest of this group of passerby arises from the sharing of common space and the need to accommodate the robot's motion. Predominantly, the interest may be restricted to one of avoiding the robot and its workspace. Their interest is in the near-term actions of the robot as in the robot's current actions or in its next action. They need to be provided with the ability to identify the robot's actions or its intention. The robot can support this need for situation awareness by this group through the visual prompts generated by the laser writer (**Table 1**).


**Table 1.** Framework for multimodal AR GUI.

**4.3. Camera-projector calibration and software**

**Figure 8.** Projection of calibration target and corner extraction.

**Figure 9.** Laser input detection and arrow projection.

**5. An implementation**

project an arrow that follows the laser marker indicated by the user.

formation parameters.

174 Recent Advances in Robotic Systems

Our multimodal handheld device is equipped with a laser pointer, which is used to project a marker point for indicating a reference or indicating a chosen item. The marker and its projected position are identifiable by the camera, located close to the laser writer. To determine the position of the marker, a calibration process is performed to obtain the necessary trans‐

The calibration process is performed by using the laser writer to draw a square with known parameters onto the floor within a region, in the camera field of view. The camera captures the image of the square that was projected on the floor. The corners of the square in the camera frame are subsequently extracted. These values, together with the known projector frame, are

**Figure 9** shows the camera-projector system and demonstrates the use of the laser marker to indicate a position. The system confirms the position of the marker by responding with the projection of an arrow head that points to the marker location. In this scenario, the robot will

Within the proposed human-robot framework, we identify three different groups of people that may interact with the robot. These people, who are known as interactants, are grouped

used to obtain the projector-camera homography. **Figure 8** illustrates the procedures.

**Figure 10** shows the dialog generated by the wearable display. Each robot is equipped with a unique augmented reality marker for facilitating the process of overlaying the computergenerated information over the real scene. The operator, with the multimodal hand controller, will have the capability of visually selecting the options suggested by the robot to complete the task. This offers a more intuitive and convenient way to interact with the robot.

**Figure 10.** Dialog menu for Group 1 interactants.

While the GUI on the wearable display enables the operator to intuitively control and monitor the robot, other humans who share the same working environment (Passerby) might encounter difficulties in accommodating this robot's motion and inadvertently cross into its intended path. The laser writer is provided to overcome this problem by providing passersby with visual indications of the robot's intentions. **Figure 11** shows one possible implementation for this system. In this scenario, this laser system allows the robot to project the direction of its trajectory, before moving from one point to another. Particularly, the robot is able to indicate: 'stop', 'forward', 'backward', 'turn left' and 'turn right' as text or line-graphic symbols.

**Figure 11.** Laser notification to all interactants and LCD menus.
