**4. Laser projection-based AR system**

A laser projection system has been installed on the robot to provide for a spatial augmented reality. It facilitates the presentation of projected digital AR information such as the robot motion behaviours and its motion trajectories onto the floor surface of the work environment.

**Figure 5** depicts a schematic of the implemented laser projection system. To create laser graphics, two tiny computer-controlled mirrors are used to direct the laser beam onto a suitable surface. The first mirror rotates about the horizontal axis while the second mirror rotates about the vertical axis. A pair of galvanometers is used to produce the rotating motions, which subsequently aims the laser beam to any point on a square or rectangular raster. The position of the laser point is controlled by changing the electric current through its coil in a magnetic field. The shaft, of which the mirror is attached to, will rotate to an angle proportional to the coil current. In this manner, by combining the motions of the two galvanometers in orthogonal planes, the x-y position of the projected laser spot can be changed.

**Figure 5.** Schematic diagram of the laser writer display system.

The computer 'connects the dots' by rotating the mirrors at a very high speed. This causes the laser spot to move sufficiently fast from one position to another, resulting in a viewer seeing a single outline drawing. This process is called 'scanning' and computer-controlled mirrors are galvanometer 'scanners'. The scanners move from point to point at a rate of approximately 30–40 kpps. To add more detail to a scene, additional sets of scanners can be used to overcome the limitations in scanning speeds.

A multicolour laser projection system would consist of red, green and blue lasers, each with its individual driver and optics. The drivers also control the intensity of each laser source independently. Red, green and blue laser beams are mixed in the transparent mirror system and the combined beam is subsequently projected onto the mirrors of the galvanometers. Together, these three laser diodes combine their output to produce a white or an 'infinitely' varied coloured beam.

### **4.1. Image generation**

An International Laser Display Association (ILDA) interface can be used to import custom graphics, text and effects into laser animation format. Files containing scalable vector pictures or videos are loaded to the graphic controller in a special format. The control software converts these files into a list of sequential points, each of which is characterised by the angular deflections of the galvanometers in the vertical and horizontal planes. The intensity of laser radiation is also controlled via the interface.

The ILDA laser control standard produces a sequence of digital-to-analog converter (DAC) outputs on differential wire pairs with average amplitude of ± 24V for the galvanometer control and ±5V for the laser diode drivers. When the galvanometers receive a new value for mirror deflections, it drives the mirrors to the next desired angular position.

### **4.2. Transformations for inclined surface projection**

As the projector frame is not necessarily perpendicular to the projection surface, there will be visible distortions in the source image (Projector Frame) projected onto the surface (Projected Image Frame) as shown in **Figure 6**.

**Figure 6.** Projected image is distorted when projected onto an oblique surface.

This distortion needs to be corrected through a pre-warping process that is applied to the projection image, before being projected to the particular surface.

**Figure 7.** Projected image distortion correction.

**Figure 5.** Schematic diagram of the laser writer display system.

the limitations in scanning speeds.

radiation is also controlled via the interface.

varied coloured beam.

172 Recent Advances in Robotic Systems

**4.1. Image generation**

The computer 'connects the dots' by rotating the mirrors at a very high speed. This causes the laser spot to move sufficiently fast from one position to another, resulting in a viewer seeing a single outline drawing. This process is called 'scanning' and computer-controlled mirrors are galvanometer 'scanners'. The scanners move from point to point at a rate of approximately 30–40 kpps. To add more detail to a scene, additional sets of scanners can be used to overcome

A multicolour laser projection system would consist of red, green and blue lasers, each with its individual driver and optics. The drivers also control the intensity of each laser source independently. Red, green and blue laser beams are mixed in the transparent mirror system and the combined beam is subsequently projected onto the mirrors of the galvanometers. Together, these three laser diodes combine their output to produce a white or an 'infinitely'

An International Laser Display Association (ILDA) interface can be used to import custom graphics, text and effects into laser animation format. Files containing scalable vector pictures or videos are loaded to the graphic controller in a special format. The control software converts these files into a list of sequential points, each of which is characterised by the angular deflections of the galvanometers in the vertical and horizontal planes. The intensity of laser

The ILDA laser control standard produces a sequence of digital-to-analog converter (DAC) outputs on differential wire pairs with average amplitude of ± 24V for the galvanometer control and ±5V for the laser diode drivers. When the galvanometers receive a new value for mirror

deflections, it drives the mirrors to the next desired angular position.

The pre-warping process can be accomplished using the concept of homography in computer vision [26], as shown in **Figure 7**. The idea is to find the transformation matrix between the source image frame and the projected image frame, as illustrated in **Figure 7** step (1). Typically, a square outline will be projected onto the desired surface and its corner's positions will be estimated. The corner positions of this square, in the source projector frame, are also calculated. The appropriate transformation matrix will be determined using the corresponding points. The details of this method are described in the paper by Rahul, Robert and Matthew [27]. The next step, as seen in **Figure 7** step (2), is to multiply this matrix with our original image to obtain the pre-warped image frame. Finally, as depicted in **Figure 7** step (3), this image is projected onto the inclined surface to produce the undistorted square image (Projected Image Frame).

### **4.3. Camera-projector calibration and software**

Our multimodal handheld device is equipped with a laser pointer, which is used to project a marker point for indicating a reference or indicating a chosen item. The marker and its projected position are identifiable by the camera, located close to the laser writer. To determine the position of the marker, a calibration process is performed to obtain the necessary trans‐ formation parameters.

The calibration process is performed by using the laser writer to draw a square with known parameters onto the floor within a region, in the camera field of view. The camera captures the image of the square that was projected on the floor. The corners of the square in the camera frame are subsequently extracted. These values, together with the known projector frame, are used to obtain the projector-camera homography. **Figure 8** illustrates the procedures.

**Figure 8.** Projection of calibration target and corner extraction.

**Figure 9** shows the camera-projector system and demonstrates the use of the laser marker to indicate a position. The system confirms the position of the marker by responding with the projection of an arrow head that points to the marker location. In this scenario, the robot will project an arrow that follows the laser marker indicated by the user.

**Figure 9.** Laser input detection and arrow projection.
