**Augmented Reality Platform for Collaborative E-Maintenance Systems**

Samir Benbelkacem1, Nadia Zenati-Henda1, Fayçal Zerarga1, Abdelkader Bellarbi1, Mahmoud Belhocine1, Salim Malek1 and Mohamed Tadjine2 *1Development Centre of Advanced Technologies 2Polytechnic National School Algeria* 

## **1. Introduction**

One of the recent design goals in Human-Computer Interaction is the extension of the sensorymotor capabilities of computer systems enabling a combination of the real and the virtual in order to assist the user in performing his task in a physical setting. Such systems are called Augmented Reality (short: AR). The growing interest of designers for this paradigm is due to the dual need of users to benefit from computers and interact with the real world.

AR is a new interactive approach, where virtual objects (such as texts, 2D images and 3D models) are added to real scenes in real time by using sensing and visualization technology. The computer generated digital information is overlaid on the user's physical environment so that he can perceive currently important information where needed.

Augmented Reality (AR) is derived from Virtual Reality (VR) in which the user is completely immersed in an artificial world. In VR systems, there is no way for the user to interact with objects in the real world. Using AR technology, users can thus interact with mixed virtual and real worlds in a natural way (Zhong & Boulanger, 2002).

AR research is of major interest in several domains. Azuma gives a description of various applications of AR systems in (Azuma, 1997) including medical visualisation, manufacturing and repair, robot path planning, entertainment and military aircraft.

AR application is an excellent domain for maintenance tasks in industrial environment (Changzhi et al, 2006). AR allows the user to see virtual objects increased upon real world scenes through display devices such as PCs, Laptops, Pc-Pockets**,** Video-projectors or Head Mounted Displays (HMD). The technician can interact with the virtual world and may dispose of additional information in various forms; for instance additional maintenance tasks instructions may be given in the form of texts, images, video or audio augmentations.

Several maintenance platforms based on AR have been developed. ARVIKA (Marsot et al., 2009) introduces AR in the life cycle of industrial products, AMRA implements a mobile AR system in an industrial setting (Didier & Roussel, 2005), STARMATE (Schwald et al., 2001) assists the operator during maintenance tasks on complex mechanical systems, ULTRA (Riess & Stricker, 2006) has developed a software architecture that enables production of

Augmented Reality Platform for Collaborative E-Maintenance Systems 213

To address this need, we have developed concepts concerning the data transfer (augmented

Tracking is a very important research subject in the field of augmented reality (Fiala, 2004), (Comport et al., 2005), and the vision-based tracking method is usually appropriate. Tracking methods use cameras that capture real scene as sensing devices. So, to apply AR to maintenance support, the positions and orientations of users in real time must be measured with high accuracy. Among the tracking technologies proposed in previous studies (Comport et al., 2005), marker-based tracking, which uses image processing technique to measure the relative position and orientation between a camera and markers (transformation matrix marker/camera), seems to be appropriate to industrial maintenance context. In fact, the most popular tracking method, which uses square markers, is ARToolkit

ARToolkit (Augmented Reality Toolkit) is an open-source software library used to develop augmented reality applications. It was developed by "The Human Interface Technology Laboratory" (HIT Lab) at Washington University. It uses vision techniques to calculate position and orientation of the camera relative to markers. The programmer can use this

ARToolkit guarantees the virtual object tracking when the camera (or user) changes position. The "ARToolkit" library has several types of markers (Kato & Billinghurst, 1999)

The markers are in a black square form with a code inside. This marker is a simple form that can easily be identified for the insertion of a virtual object. The position and orientation of

However, like other vision-based tracking, the tracking with ARToolkit suffers from the lack of robustness. In this case, our team proposes (Bellarbi et al., 2010(b)) a new version of ARToolkit called i-ARToolkit (improved ARToolkit) that gives solutions to ARToolkit

The first problem of ARToolkit is the use of static thresholding method that cannot adapt to changing environmental parameters (brightness level). i-ARToolkit applies dynamic thresholding approach to guarantee markers recognition even if environmental properties

The second problem concerns the virtual objects instability in the real scene. This is due to the uncertainty of the transformation matrix. i-Artoolkit proposes an approach which takes

information to draw the 3D object and to insert it correctly in the real scene.

the camera can be calculated by identifying the markers in a video stream.

**Hiro**

2D/3D objects) and the remote collaboration.

(Kato & Billinghurst, 1999).

Fig. 1. Examples of ARToolkit's markers.

(see Fig. 1).

problems.

change.

into account this uncertainty.

**2.1 Tracking system and virtual objects transfer** 

augmented reality manuals and on-site support of (mobile) maintenance workers. Also, A prototype of automobile maintenance (BMW) based on AR are presented in (Platonov et al., 2006) using see through system. More recently, the project ARMAR (Henderson et al., 2010) has been interested in exploring the extent to which AR can increase the productivity, the precision and safety of maintenance personnel.

These platforms provide many advantages for repair task improvement, but not sufficiently. In some cases, technicians cannot repair equipments even with AR means. This occurs when a new failure appears, in which case, the corresponding scenario does not exist in the database. The main solution is to contact remote experts (with required qualification level) to provide maintenance procedures through collaboration with the field technician.

In recent years, systems supporting remote collaborative work for industrial maintenance have appeared. But, increasing importance is given more to collaborative principles (Bangemann et al., 2006) rather than maintenance and AR aspects (Bottecchia et al., 2010).

In (Zhong & Boulanger, 2002) a prototype is presented in which operators equipped with display devices are supervised by an expert. This latter can only provide audio indications. Sakata and Kurata (Sakata et al., 2003), (Kurata et al., 2004) developed the Wearable Active Camera/Laser (WACL) which allows the remote instructor not only to independently look into the worker's task space, but also to point to real objects in the task space with the laser spot (Alem et al., 2011). In (Bottecchia et al., 2010) a maintenance collaborative system entitled CAMEKA is described, which enables the expert to give visual indications to an operator with an AR display device fitted with a camera. What the camera sees is sent to the expert who can "capture" an image from the video flow, add notes, then send back the enriched image to the operator's display device.

Existing AR systems are based on a single technician who repairs a single machine. Others are based on a local collaboration through available on-site experts. This situation does not guarantee direct task assistance. In some cases, the experts are not available while in others, the existing local experts lack sufficient competence. It, then, is necessary to call onto a remote expertise for the technician's performances improvement. Also, the capitalization of expert's know-how is guaranteed.

Our area of interest is the establishment of a distributed platform allowing collaboration between technicians and remote experts based on AR benefits. Two main aspects are studied and developed in our case. The first aspect addresses a new collaboration strategy based on Service Oriented Architecture (SOA) which offers efficient solutions in terms of information transfer and exchange, data heterogeneous management, etc. The second aspect ensures the virtual objects (maintenance procedures) transfer from the remote expert in real time. The result is a visual space shared by the technician and the remote expert.

The content of this chapter is organised as follows. In section 2, a collaborative platform supporting AR interaction is presented. Section 3 describes a global industrial maintenance scenario. Implementation and results are presented in section 4. Finally, in the last section, a conclusion and perspectives are given.

#### **2. E-maintenance platform based on augmented reality concept**

In this section, the collaborative platform supporting AR interaction is built. The expert should be able to insert augmentation into video stream to increase the operator's view in real time, so that he understands the tasks to be achieved.

To address this need, we have developed concepts concerning the data transfer (augmented 2D/3D objects) and the remote collaboration.

## **2.1 Tracking system and virtual objects transfer**

212 Augmented Reality – Some Emerging Application Areas

augmented reality manuals and on-site support of (mobile) maintenance workers. Also, A prototype of automobile maintenance (BMW) based on AR are presented in (Platonov et al., 2006) using see through system. More recently, the project ARMAR (Henderson et al., 2010) has been interested in exploring the extent to which AR can increase the productivity, the

These platforms provide many advantages for repair task improvement, but not sufficiently. In some cases, technicians cannot repair equipments even with AR means. This occurs when a new failure appears, in which case, the corresponding scenario does not exist in the database. The main solution is to contact remote experts (with required qualification level)

In recent years, systems supporting remote collaborative work for industrial maintenance have appeared. But, increasing importance is given more to collaborative principles (Bangemann et al., 2006) rather than maintenance and AR aspects (Bottecchia et al., 2010). In (Zhong & Boulanger, 2002) a prototype is presented in which operators equipped with display devices are supervised by an expert. This latter can only provide audio indications. Sakata and Kurata (Sakata et al., 2003), (Kurata et al., 2004) developed the Wearable Active Camera/Laser (WACL) which allows the remote instructor not only to independently look into the worker's task space, but also to point to real objects in the task space with the laser spot (Alem et al., 2011). In (Bottecchia et al., 2010) a maintenance collaborative system entitled CAMEKA is described, which enables the expert to give visual indications to an operator with an AR display device fitted with a camera. What the camera sees is sent to the expert who can "capture" an image from the video flow, add notes, then send back the

Existing AR systems are based on a single technician who repairs a single machine. Others are based on a local collaboration through available on-site experts. This situation does not guarantee direct task assistance. In some cases, the experts are not available while in others, the existing local experts lack sufficient competence. It, then, is necessary to call onto a remote expertise for the technician's performances improvement. Also, the capitalization of

Our area of interest is the establishment of a distributed platform allowing collaboration between technicians and remote experts based on AR benefits. Two main aspects are studied and developed in our case. The first aspect addresses a new collaboration strategy based on Service Oriented Architecture (SOA) which offers efficient solutions in terms of information transfer and exchange, data heterogeneous management, etc. The second aspect ensures the virtual objects (maintenance procedures) transfer from the remote expert in real

The content of this chapter is organised as follows. In section 2, a collaborative platform supporting AR interaction is presented. Section 3 describes a global industrial maintenance scenario. Implementation and results are presented in section 4. Finally, in the last section, a

In this section, the collaborative platform supporting AR interaction is built. The expert should be able to insert augmentation into video stream to increase the operator's view in

time. The result is a visual space shared by the technician and the remote expert.

**2. E-maintenance platform based on augmented reality concept** 

real time, so that he understands the tasks to be achieved.

to provide maintenance procedures through collaboration with the field technician.

precision and safety of maintenance personnel.

enriched image to the operator's display device.

expert's know-how is guaranteed.

conclusion and perspectives are given.

Tracking is a very important research subject in the field of augmented reality (Fiala, 2004), (Comport et al., 2005), and the vision-based tracking method is usually appropriate. Tracking methods use cameras that capture real scene as sensing devices. So, to apply AR to maintenance support, the positions and orientations of users in real time must be measured with high accuracy. Among the tracking technologies proposed in previous studies (Comport et al., 2005), marker-based tracking, which uses image processing technique to measure the relative position and orientation between a camera and markers (transformation matrix marker/camera), seems to be appropriate to industrial maintenance context. In fact, the most popular tracking method, which uses square markers, is ARToolkit (Kato & Billinghurst, 1999).

ARToolkit (Augmented Reality Toolkit) is an open-source software library used to develop augmented reality applications. It was developed by "The Human Interface Technology Laboratory" (HIT Lab) at Washington University. It uses vision techniques to calculate position and orientation of the camera relative to markers. The programmer can use this information to draw the 3D object and to insert it correctly in the real scene.

ARToolkit guarantees the virtual object tracking when the camera (or user) changes position. The "ARToolkit" library has several types of markers (Kato & Billinghurst, 1999) (see Fig. 1).

Fig. 1. Examples of ARToolkit's markers.

The markers are in a black square form with a code inside. This marker is a simple form that can easily be identified for the insertion of a virtual object. The position and orientation of the camera can be calculated by identifying the markers in a video stream.

However, like other vision-based tracking, the tracking with ARToolkit suffers from the lack of robustness. In this case, our team proposes (Bellarbi et al., 2010(b)) a new version of ARToolkit called i-ARToolkit (improved ARToolkit) that gives solutions to ARToolkit problems.

The first problem of ARToolkit is the use of static thresholding method that cannot adapt to changing environmental parameters (brightness level). i-ARToolkit applies dynamic thresholding approach to guarantee markers recognition even if environmental properties change.

The second problem concerns the virtual objects instability in the real scene. This is due to the uncertainty of the transformation matrix. i-Artoolkit proposes an approach which takes into account this uncertainty.

Augmented Reality Platform for Collaborative E-Maintenance Systems 215

Once the transformation matrix between the marker and the camera is calculated,

augmented objects which represent maintenance procedures are easily inserted.

temporary markers to predetermined mount points prior to a repair sequence.

The configuration given in Fig.3 allows the technician to view maintenance scenarios even if he changes position. A number of markers attached to preregistered locations in the area of a repair could serve as anchor points for labels, instructions, and other virtual content. In most cases, maintenance units could install these markers permanently without interfering with system components. For cases when this was not possible, technician could affix

 b. Image captured by a camera at a short distance.

In most cases, the technician has difficulties to place the markers in adequate and precise position. For that, it is recommended to choose particular preregistered locations (example:

To resolve the placement's marker problem, the best solution is to replace printed markers with natural features (such as visually unique parts of an engine), making possible markerless tracking (Bleser et al., 2005), (Comport et al., 2005). In our future work we plan to

When a technician performs a difficult task, he can be assisted by a remote expert. The difficulty lays in how to send the augmented scene from remote the expert to the technician's visual scene. For this reason, two approaches are proposed. The first one consists in computing the technician's position (transformation matrix marker/camera) from the captured images. The captured images and the transformation matrix are both

As observed in the above figure (Fig. 5), the technician just sends the captured images. The expert should have a tracking system to detect the makers and to calculate the transformation matrix. The remote expert's tracking system takes an additional time to perform the different operations. In addition, to calculate the transformation matrix, the images must be sent to the remote expert without being compressed. The markers detection

Fig. 3. Example of car engine with markers

a. Image captured by a camera at a long distance.

corners) to perform easily markers placement.

transmitted to the remote expert (see Fig. 4).

in the compressed images gives poor results.

**2.1.1 Virtual objects transfer** 

develop a markerless based method for maintenance field.

In summary, i-ARToolkit tracking system performs the following steps (Bellarbi et al., 2010(b)):

Step1: Open a video stream

Step2: For each captured image:


Another problem usually appears when applying AR in maintenance; it is the non-detection of markers when the technician works in a large space. This occurs when the camera is far from the marker. Fig. 2 shows an estimation of the maximum detection distance of i-ARToolkit markers.

Fig. 2. Detection distance vs marker width

The proposed solution, inspired from (Hirotake et al., 2010), is to establish a relationship between a number of detected markers in the captured image and the distance between the camera and the markers.

Fig. 3.a shows that the detected markers become very numerous and the size of each marker on the image becomes very small when the distance between the camera and the markers is long. It therefore becomes difficult to calculate the transformation matrix for each marker. The markers size on the image is too small. In this case, a global transformation matrix is calculated to encompass the transformation matrices of the detected markers. This principle enhances the stability of inserted object.

On the other hand, as shown in Fig. 3.b, when the distance between the camera and the markers is short, the number of the detected markers becomes small and the markers sizes on the image become very large. In this case, it becomes possible to obtain more stable transformation matrix since the marker size on the captured image is sufficiently large.

Once the transformation matrix between the marker and the camera is calculated, augmented objects which represent maintenance procedures are easily inserted.

214 Augmented Reality – Some Emerging Application Areas

In summary, i-ARToolkit tracking system performs the following steps (Bellarbi et al.,

2.1 Calculate the optimal threshold value from the captured image using Otsu method

2.2 Transform the captured image to a binary image using the calculated threshold value.

2.4 Calculate the camera's position and orientation (calculate the transformation matrix).

2.6 Superimpose the virtual objects upon the captured image (using the calculated

Another problem usually appears when applying AR in maintenance; it is the non-detection of markers when the technician works in a large space. This occurs when the camera is far from the marker. Fig. 2 shows an estimation of the maximum detection distance of

The proposed solution, inspired from (Hirotake et al., 2010), is to establish a relationship between a number of detected markers in the captured image and the distance between the

Fig. 3.a shows that the detected markers become very numerous and the size of each marker on the image becomes very small when the distance between the camera and the markers is long. It therefore becomes difficult to calculate the transformation matrix for each marker. The markers size on the image is too small. In this case, a global transformation matrix is calculated to encompass the transformation matrices of the detected markers. This principle

On the other hand, as shown in Fig. 3.b, when the distance between the camera and the markers is short, the number of the detected markers becomes small and the markers sizes on the image become very large. In this case, it becomes possible to obtain more stable transformation matrix since the marker size on the captured image is sufficiently large.

2010(b)):

Step1: Open a video stream

(Otsu, 1979).

i-ARToolkit markers.

Step2: For each captured image:

transformation matrix).

Fig. 2. Detection distance vs marker width

enhances the stability of inserted object.

camera and the markers.

2.3 Detect black squares markers in the binary image.

2.5 Apply the stabilization algorithm (Bellarbi et al., 2010(a)).

a. Image captured by a camera at a long distance.

 b. Image captured by a camera at a short distance.

Fig. 3. Example of car engine with markers

The configuration given in Fig.3 allows the technician to view maintenance scenarios even if he changes position. A number of markers attached to preregistered locations in the area of a repair could serve as anchor points for labels, instructions, and other virtual content. In most cases, maintenance units could install these markers permanently without interfering with system components. For cases when this was not possible, technician could affix temporary markers to predetermined mount points prior to a repair sequence.

In most cases, the technician has difficulties to place the markers in adequate and precise position. For that, it is recommended to choose particular preregistered locations (example: corners) to perform easily markers placement.

To resolve the placement's marker problem, the best solution is to replace printed markers with natural features (such as visually unique parts of an engine), making possible markerless tracking (Bleser et al., 2005), (Comport et al., 2005). In our future work we plan to develop a markerless based method for maintenance field.

#### **2.1.1 Virtual objects transfer**

When a technician performs a difficult task, he can be assisted by a remote expert. The difficulty lays in how to send the augmented scene from remote the expert to the technician's visual scene. For this reason, two approaches are proposed. The first one consists in computing the technician's position (transformation matrix marker/camera) from the captured images. The captured images and the transformation matrix are both transmitted to the remote expert (see Fig. 4).

As observed in the above figure (Fig. 5), the technician just sends the captured images. The expert should have a tracking system to detect the makers and to calculate the transformation matrix. The remote expert's tracking system takes an additional time to perform the different operations. In addition, to calculate the transformation matrix, the images must be sent to the remote expert without being compressed. The markers detection in the compressed images gives poor results.

Augmented Reality Platform for Collaborative E-Maintenance Systems 217

expert's computer. With this approach the expert does not need a tracking system to calculate the required parameters. Also, the images can be sent to the remote expert in

According to this comparison, the first method is adopted to guarantee the best data exchange between the actors. The next section gives more details about the data exchange

In this section, our aim is to develop approaches for remote collaboration based essentially on the Web Services concept. In this case, we are interested in developing a communication module (for chat, file, image and augmented video transfer) to take into account the heterogeneity problem between different hardware (Pc-Pocket and PC-Tablet) and software platforms (operating systems, programming languages...). The use of Web Services ensures

When the technician lacks competence to repair broken down equipment he contacts a remote expert to obtain appropriate solutions. A distributed platform is then required to

A distributed system is a set of interconnected devices which collaborate to perform a set of tasks. The tasks are called by a remote services exposed in a web server. Based on Web Services, this system can manage heterogeneous applications geographically dispersed.

Expertise centre

Satellite

GSM

server

Technician

Send **/** Receive


Repair order

Update

Supervisor

the scene Web

Access point

Maintenance scenarios

compressed form. So, the computing time is reduced.

the interoperability between the applications of different actors.

**2.2.1 Distributed and mobile e-maintenance platform** 

principle in a collaborative context.

**2.2 Remote collaboration** 

facilitate this collaboration.

Fig. 6. ARIMA platform configuration.

SOAP Messages

Augmenting

AR Device -PDA -Eyes glass -Pc tablet

Expert

Fig. 4. Essential steps of the first method.

The second approach consists in sending the captured images alone, from the technician to the remote expert. Besides the technician, the transformation matrix is also computed at the remote expert, (see Fig. 5).

Fig. 5. Essential steps of the second method.

The first method gives better results since the markers detection is performed at the technician's device. The transformation matrix and the images are transferred to the remote expert's computer. With this approach the expert does not need a tracking system to calculate the required parameters. Also, the images can be sent to the remote expert in compressed form. So, the computing time is reduced.

According to this comparison, the first method is adopted to guarantee the best data exchange between the actors. The next section gives more details about the data exchange principle in a collaborative context.

### **2.2 Remote collaboration**

216 Augmented Reality – Some Emerging Application Areas

9. Display captured image and virtual objects

> 8. Send virtual objects and their position

> > 5. Receive Image and transformation matrix

7. Display Augmented

5. Receive Image

6. Detect Markers 7. Calculate transformation matrix 8. Insert virtual objects

9. Display Augmented

image

6. Insert virtual objects based on transformation

image.

matrix

Remote Expert

The second approach consists in sending the captured images alone, from the technician to the remote expert. Besides the technician, the transformation matrix is also computed at the

> 9. Display captured image and virtual objects

The first method gives better results since the markers detection is performed at the technician's device. The transformation matrix and the images are transferred to the remote

Remote Expert

8. Send virtual objects and their position

Fig. 4. Essential steps of the first method.

Technician

4. Send Image and transformation matrix

Technician

4. Send Image

Fig. 5. Essential steps of the second method.

remote expert, (see Fig. 5).

2. Detect Markers 1. Capture Image

3. Calculate transformation matrix

2. Detect Markers 1. Capture Image

3. Calculate transformation matrix

> In this section, our aim is to develop approaches for remote collaboration based essentially on the Web Services concept. In this case, we are interested in developing a communication module (for chat, file, image and augmented video transfer) to take into account the heterogeneity problem between different hardware (Pc-Pocket and PC-Tablet) and software platforms (operating systems, programming languages...). The use of Web Services ensures the interoperability between the applications of different actors.

#### **2.2.1 Distributed and mobile e-maintenance platform**

When the technician lacks competence to repair broken down equipment he contacts a remote expert to obtain appropriate solutions. A distributed platform is then required to facilitate this collaboration.

A distributed system is a set of interconnected devices which collaborate to perform a set of tasks. The tasks are called by a remote services exposed in a web server. Based on Web Services, this system can manage heterogeneous applications geographically dispersed.

Fig. 6. ARIMA platform configuration.

Augmented Reality Platform for Collaborative E-Maintenance Systems 219

Service provider

Web Server

WSDL

WSDL SOAP

Expert 1 Expert 2 Technician

Send SOAP files and run web methods in WSDL documents

Initially, the technician receives an alarm about failed equipment. He proceeds to a first analysis and tries to formulate a diagnosis. When he is unable to repair, he contacts a remote expert using internet network. The essential aspect is to guarantee the reliable data exchange

The data transfer process is described as follows (see Fig 8): the technician captures a video of failure's location. For every captured frame, a tracking system detects the markers and calculates a transformation matrix. Both captured video and transformation matrix are transmitted to the remote expert. The expert adds virtual objects (scenarios) according to the scene. The inserted virtual objects and their positions are sent to the technician. The virtual objects are received and displayed according to their positions

The data transfer is based on a client/server architecture. Each user must authenticate to the system. For that, the user runs a login method "login (username, password)". The data is encapsulated in a SOAP messages. The server checks the username and the password in the database and sends the answer to the user. For video transfer, the technician runs the web method "send(image, transformation matrix)" to send the captured frame and the calculated transformation matrix to the web server. The expert performs the web method "receive(image, markers)" to retrieve the captured frame and the transformation matrix. Then, he adds augmentations. The expert returns the result to the technician by running the method "sendObject(objectPostion, Object)". The technician runs the web method receiveObject "(objectPostion, Object)" to receive and displays the virtual objects (see

Fig. 7. Collaboration principle between actors.

Service broker

**2.3 Internal operation of ARIMA platform** 

affected by the remote expert.

Fig. 8).

using SOAP file through an access point in Internet network.

In order to build infrastructure for data exchange via services, we adopt "SOA" (Service Oriented Architecture) concept which is a paradigm that allows organizing and exploiting distributed capabilities that may be under the control of different ownership domains (Nickul, 2007). Based on this architecture, we have developed our platform named ARIMA (Augmented Reality and Image processing in Maintenance Application).

Fig.6 shows a distributed and mobile platform allowing a dialogue between technicians using mobile devices (such as PDA, Pc-tablet, eye glass...) and remote experts.

## **2.2.2 Web services for e-maintenance**

The interaction between actors (experts and technicians) requires communication platforms. Despite the advantages of the existing technologies, it cannot effectively manage the heterogeneous running environment especially when using various communication tools such as Pc-Pocket, Pc-Tablet, HMD and others. The Web Service presents efficient solution to resolve this problem. The main role is to facilitate data transfer between actors using the Internet network.

A Web Service is a software module which performs a set of discrete tasks. It is a set of services which can be invoked through a network, especially the World Wide Web, accessible via standard Internet protocols (Booth et al., 2004). A Web Service has an interface described in a WSDL (Web Service Description Language) format which exposes the method to be invoked by a web server (binding, port, services). Other systems interact with the Web Service by sending SOAP (Simple Object Access Protocol) messages to the web methods. These messages use HTTP with an XML serialization in conjunction with other Web-related standards. In other words, a Web Service is a set of related application methods that can be remotely accessed through a network (such as a corporate Intranet or Internet itself). WDSL documents are indexed in searchable Universal Description Discovery and Integration (UDDI) Business Registries, so that developers and applications can locate the Web Services.

Web Services are both characterized by the reuse facilitation. They are independent from used platforms (Windows, UNIX...) and programming languages (C #, JAVA, VB...). This interoperability form makes Web Services one of the most used technologies to design distributed applications (Leymann, 2003). In our case, Web Services principle is adopted to establish a distributed mobile maintenance platform.

As a most important concept of Web Services, we are interested to use WCF (Windows Communication Foundation). It is a new feature of Dot NET Framework version. It supports the sending of HTTP data. HTTP protocol facilitates the communication in the Internet network (Scott, 2007).

The actors consume Web Services by sending SOAP file. The services are described in a WSDL file and distributed through a UDDI registry. This allows easy collaboration between actors to resolve several technicians' task. The actors communicate by exchanging SOAP files through a service provider in order to run web methods. These web methods are described in WSDL file which is hosted in the service broker**.** Fig. 7 shows an example of a Web Service requested by two experts and one technician in a collaboration context.

218 Augmented Reality – Some Emerging Application Areas

In order to build infrastructure for data exchange via services, we adopt "SOA" (Service Oriented Architecture) concept which is a paradigm that allows organizing and exploiting distributed capabilities that may be under the control of different ownership domains (Nickul, 2007). Based on this architecture, we have developed our platform named ARIMA

Fig.6 shows a distributed and mobile platform allowing a dialogue between technicians

The interaction between actors (experts and technicians) requires communication platforms. Despite the advantages of the existing technologies, it cannot effectively manage the heterogeneous running environment especially when using various communication tools such as Pc-Pocket, Pc-Tablet, HMD and others. The Web Service presents efficient solution to resolve this problem. The main role is to facilitate data transfer between actors using the

A Web Service is a software module which performs a set of discrete tasks. It is a set of services which can be invoked through a network, especially the World Wide Web, accessible via standard Internet protocols (Booth et al., 2004). A Web Service has an interface described in a WSDL (Web Service Description Language) format which exposes the method to be invoked by a web server (binding, port, services). Other systems interact with the Web Service by sending SOAP (Simple Object Access Protocol) messages to the web methods. These messages use HTTP with an XML serialization in conjunction with other Web-related standards. In other words, a Web Service is a set of related application methods that can be remotely accessed through a network (such as a corporate Intranet or Internet itself). WDSL documents are indexed in searchable Universal Description Discovery and Integration (UDDI) Business Registries, so that developers and applications can locate the

Web Services are both characterized by the reuse facilitation. They are independent from used platforms (Windows, UNIX...) and programming languages (C #, JAVA, VB...). This interoperability form makes Web Services one of the most used technologies to design distributed applications (Leymann, 2003). In our case, Web Services principle is adopted to

As a most important concept of Web Services, we are interested to use WCF (Windows Communication Foundation). It is a new feature of Dot NET Framework version. It supports the sending of HTTP data. HTTP protocol facilitates the communication in the Internet

The actors consume Web Services by sending SOAP file. The services are described in a WSDL file and distributed through a UDDI registry. This allows easy collaboration between actors to resolve several technicians' task. The actors communicate by exchanging SOAP files through a service provider in order to run web methods. These web methods are described in WSDL file which is hosted in the service broker**.** Fig. 7 shows an example of a Web Service requested by two experts and one technician in a

(Augmented Reality and Image processing in Maintenance Application).

**2.2.2 Web services for e-maintenance** 

establish a distributed mobile maintenance platform.

Internet network.

Web Services.

network (Scott, 2007).

collaboration context.

using mobile devices (such as PDA, Pc-tablet, eye glass...) and remote experts.

Fig. 7. Collaboration principle between actors.

#### **2.3 Internal operation of ARIMA platform**

Initially, the technician receives an alarm about failed equipment. He proceeds to a first analysis and tries to formulate a diagnosis. When he is unable to repair, he contacts a remote expert using internet network. The essential aspect is to guarantee the reliable data exchange using SOAP file through an access point in Internet network.

The data transfer process is described as follows (see Fig 8): the technician captures a video of failure's location. For every captured frame, a tracking system detects the markers and calculates a transformation matrix. Both captured video and transformation matrix are transmitted to the remote expert. The expert adds virtual objects (scenarios) according to the scene. The inserted virtual objects and their positions are sent to the technician. The virtual objects are received and displayed according to their positions affected by the remote expert.

The data transfer is based on a client/server architecture. Each user must authenticate to the system. For that, the user runs a login method "login (username, password)". The data is encapsulated in a SOAP messages. The server checks the username and the password in the database and sends the answer to the user. For video transfer, the technician runs the web method "send(image, transformation matrix)" to send the captured frame and the calculated transformation matrix to the web server. The expert performs the web method "receive(image, markers)" to retrieve the captured frame and the transformation matrix. Then, he adds augmentations. The expert returns the result to the technician by running the method "sendObject(objectPostion, Object)". The technician runs the web method receiveObject "(objectPostion, Object)" to receive and displays the virtual objects (see Fig. 8).

Augmented Reality Platform for Collaborative E-Maintenance Systems 221

When the intervention is achieved, the technician fills out a maintenance report which contains all the information pertaining to the failure and the intervention procedure. This report is transmitted to the supervisor (for validation) and then stored in the interventions

Texts Indicates the components name or the maintenance scenarios Pointing arrows Indicates the components or how to execute the maintenance scenario

In this section, a maintenance application using ARIMA platform is implemented. Our tests

The technician observes that the engine has failed, but he cannot identify the problem. So, he contacts a remote expert in order to perform a detailed diagnosis. The two actors collaborate

In our case, two failure types are treated. So, two maintenance scenarios are proposed. For the first failure, the collaboration is essentially based on a video exchange data. This requires the use of i-Artoolkit tracking system. For the second failure, we use images for exchange

data (no video transfer). In this case, the tracking system is not necessary.

Tools Shows the tool type to perform the operation Sound Guides the technician through voice indications

**Augmentation Function** 

history database.

Table 1. Types of augmentations.

**4. Implementation and results** 

are performed on a car engine (see Fig. 9).

Fig. 9. Example of equipment (car engine).

by exchanging data: chat, files, images and videos.

Fig. 8. Collaboration between expert and technician.

## **3. Global industrial scenario**

ARIMA platform operation is based on a set of scenarios which are given as follows:

First, the supervisor (or team leader) sends the repair order to the technician who checks his schedule.

When the failure is identified, a maintenance scenario is then displayed in "augmented" form (see table 1) through the technician's output device.

As for the technician, due to the lack of competence, he collaborates with a remote expert. The server establishes a link between the expert and the technician. A dialogue is then performed between the two actors in order to identify the failure. To obtain more information, the expert can request documentation and/or the previous maintenance reports. Once the failure is analysed, the expert sends the augmented maintenance procedure to the technician's display device.

When the intervention is achieved, the technician fills out a maintenance report which contains all the information pertaining to the failure and the intervention procedure. This report is transmitted to the supervisor (for validation) and then stored in the interventions history database.


Table 1. Types of augmentations.

220 Augmented Reality – Some Emerging Application Areas

Fig. 8. Collaboration between expert and technician.

form (see table 1) through the technician's output device.

procedure to the technician's display device.

ARIMA platform operation is based on a set of scenarios which are given as follows:

First, the supervisor (or team leader) sends the repair order to the technician who checks his

When the failure is identified, a maintenance scenario is then displayed in "augmented"

As for the technician, due to the lack of competence, he collaborates with a remote expert. The server establishes a link between the expert and the technician. A dialogue is then performed between the two actors in order to identify the failure. To obtain more information, the expert can request documentation and/or the previous maintenance reports. Once the failure is analysed, the expert sends the augmented maintenance

**3. Global industrial scenario** 

schedule.

## **4. Implementation and results**

In this section, a maintenance application using ARIMA platform is implemented. Our tests are performed on a car engine (see Fig. 9).

Fig. 9. Example of equipment (car engine).

The technician observes that the engine has failed, but he cannot identify the problem. So, he contacts a remote expert in order to perform a detailed diagnosis. The two actors collaborate by exchanging data: chat, files, images and videos.

In our case, two failure types are treated. So, two maintenance scenarios are proposed. For the first failure, the collaboration is essentially based on a video exchange data. This requires the use of i-Artoolkit tracking system. For the second failure, we use images for exchange data (no video transfer). In this case, the tracking system is not necessary.

Augmented Reality Platform for Collaborative E-Maintenance Systems 223

Fig. 11. Example of maintenance collaboration (second case study).

giving augmented information on the user's field of view.

wireless 802.11.

**5. Conclusion** 

time even if he moves.

For hardware environment, technicians use a Pc-Pocket (HP, Windows mobile 5 with a video camera and internet connection) or a Pc-Tablet (Sony VAIO UX 280P, Windows XP, video camera and internet connection). Remote experts use a Laptop (Dell, Windows XP, Wi-Fi and camera). The collaboration is performed through access points which support

An AR e-maintenance platform design is presented in this paper. The aim is to help a technician during his intervention. As a result, we are focused on synchronous and remote collaboration between technicians and experts to complete maintenance and repair tasks by

Two principal concepts are treated and developed: remote collaboration based on Web Services and virtual objects transfer. The adopted strategy allows the technician to collaborate easily with a remote expert. Also, he can receive the augmented 3D scene in real

This platform is applied in the case of a car engine repair. The maintenance operation is performed by a technician who collaborates with a distant expert to obtain maintenance scenarios displayed on the user's viewed scene. The results show the benefits of remote

collaboration and AR for maintenance assistance (security, flexibility, saving time ...).

#### *First case study: the engine makes much noise*

The technician uses his Pc-Tablet, captures the video scene and sends it to the expert. He indicates, by chat, that the engine makes much noise. The expert views the video scene and proposes a maintenance procedure (check the oil level). The expert uses the augmentation editor to insert corresponding 3D augmentations (text and arrows) in the viewed scene. The augmented video is then transmitted to the technician (see Fig. 10). The collaboration continues until the engine is repaired.

Fig. 10. Example of maintenance collaboration (first case study).

#### *Second case study: the engine overheats*

Using a Pc-Pocket, the technician captures an image (photo) of the scene (failure's location), and sends it to the expert. The failure concerns the engine temperature. Besides the chat, the expert uses a simple editor to insert 2D augmentations (screwdriver and arrows) that show the maintenance procedure (check the radiator) (see Fig. 11). The enhanced image is transmitted to the technician's Pc-Pocket.

Fig. 11. Example of maintenance collaboration (second case study).

For hardware environment, technicians use a Pc-Pocket (HP, Windows mobile 5 with a video camera and internet connection) or a Pc-Tablet (Sony VAIO UX 280P, Windows XP, video camera and internet connection). Remote experts use a Laptop (Dell, Windows XP, Wi-Fi and camera). The collaboration is performed through access points which support wireless 802.11.

## **5. Conclusion**

222 Augmented Reality – Some Emerging Application Areas

The technician uses his Pc-Tablet, captures the video scene and sends it to the expert. He indicates, by chat, that the engine makes much noise. The expert views the video scene and proposes a maintenance procedure (check the oil level). The expert uses the augmentation editor to insert corresponding 3D augmentations (text and arrows) in the viewed scene. The augmented video is then transmitted to the technician (see Fig. 10). The collaboration

*First case study: the engine makes much noise* 

continues until the engine is repaired.

Fig. 10. Example of maintenance collaboration (first case study).

Using a Pc-Pocket, the technician captures an image (photo) of the scene (failure's location), and sends it to the expert. The failure concerns the engine temperature. Besides the chat, the expert uses a simple editor to insert 2D augmentations (screwdriver and arrows) that show the maintenance procedure (check the radiator) (see Fig. 11). The enhanced image is

*Second case study: the engine overheats* 

transmitted to the technician's Pc-Pocket.

An AR e-maintenance platform design is presented in this paper. The aim is to help a technician during his intervention. As a result, we are focused on synchronous and remote collaboration between technicians and experts to complete maintenance and repair tasks by giving augmented information on the user's field of view.

Two principal concepts are treated and developed: remote collaboration based on Web Services and virtual objects transfer. The adopted strategy allows the technician to collaborate easily with a remote expert. Also, he can receive the augmented 3D scene in real time even if he moves.

This platform is applied in the case of a car engine repair. The maintenance operation is performed by a technician who collaborates with a distant expert to obtain maintenance scenarios displayed on the user's viewed scene. The results show the benefits of remote collaboration and AR for maintenance assistance (security, flexibility, saving time ...).

Augmented Reality Platform for Collaborative E-Maintenance Systems 225

Didier, J., Roussel, D. (2005). Amra: Augmented reality assistance in train maintenance tasks. *Workshop on Industrial Augmented Reality (ISMAR'05)*, Vienna (Austria). Fiala, M. (2004). Artag, An improved marker system based on ARToolkit. *National Research* 

Henderson,S., Feiner, Steven. (2010). Exploring the Benefits of Augmented Reality

Hirotake, I., Zhiqiang, B., Hidenori, F., Tomoki, S., Toshinori, N., Akihisa, O., Hiroshi, S.,

Kato, H., Billinghurst, M. (1999). Marker tracking and hmd calibration for a video-based

Kurata, T., Sakata, N., Kourogi, M., Kuzuoka, H., Billinghurst, M. (2004). Remote

Leymann, F. (2003). Web Services: Distributed Applications without Limits. *Proceedings of* 

Nickul, D. (2007). Service Oriented Architecture (SOA) and Specialized Messaging Patterns.

Marsot, J., Gardeux, F., Govaere, V. (2009). Réalité augmentée et prévention des risques : apports et limites. *Revue of Hygiène et sécurité du travail,* INRS, Paris pp. 15-23. Otsu, N. (1979). A Threshold Selection Method from Gray-Level Histograms. *IEEE* 

Platonov, J., Heibel, H., Meyer, P. and Grollmann, B. (2006). A mobile markless AR

Riess, P., Stricker, D. (2006). AR on-demand: a practicable solution for augmented reality on

Sakata, N., Kurata, T., Kato, T., Kourogi, M., Kuzuoka, H. (2003). WACL: supporting

Schwald, B. and all. (2001). STARMATE: Using Augmented Reality technology for computer

Scott, K. (2007). Professional WCF Programming: .NET development with the Windows

system for maintenance and repair. *Mixed and Augmented Reality (ISMAR'06)*, pp.

low-end handheld devices. *AR/VR Workshop of the Germany Computer Science Society*.

elecommunications using - wearable active camera with laser pointer Wearable Computers. *Proceedings of Seventh IEEE International Symposium on Wearable* 

guided maintenance of complex mechanical elements. *E-work and E-Commerce, IOS* 

*Transactions on Systems, Man, and Cybernetics*, Vol.9, pp. 62–66.

Barcelona, Spain.

*Council*, Canada.

China.

105–108.

Coblence, Germany.

*Computer Graphics*, 16(1), pp.4-16.

*Augmented Reality (IWAR 1999)*, pp.85–92.

*10th Conference on Database Systems for Business*.

*Adobe Systems Incorporated*, San-Jose, USA.

*Computers*, New-York, USA, pp 53-56.

*Press*, Vol. 1, pp. 17-19, Venice, Italy.

communication foundation. *Amazon Edition, 430 pages*.

Arlington, Virginia, USA, pp. 62-69.

*International Conference on Robotics and Automation (ICRA'05)*, pp.2841-2846,

Documentation for Maintenance and Repair. *IEEE Transactions on Visualization and* 

Masanori, I., Yoshinori, K., Yoshitsugu, M. (2010). Augmented Reality Applications for Nuclear Power Plant Maintenance Work. *The 3rd International Symposium on Symbiotic Nuclear Power Systems for 21st Century (ISSNP2010)*, Harbin, Heilongjiang,

augmented reality conferencing system. *Proceedings of ACM/IEEE Workshop on* 

collaboration using a shoulder-worn active camera/laser Wearable Computers. ISWC 2004. *Proceedings of Eighth International Symposium on Wearable Computers*,

For future work, we aim to improve the proposed prototype in various ways. The user and interaction method need enhancement to make visualisation and manipulation of graphical objects easier. Also, the user's equipment weight can be reduced by using HMD. Moreover, sensors network installation is necessary to provide the equipment's state.

#### **6. References**


224 Augmented Reality – Some Emerging Application Areas

For future work, we aim to improve the proposed prototype in various ways. The user and interaction method need enhancement to make visualisation and manipulation of graphical objects easier. Also, the user's equipment weight can be reduced by using HMD. Moreover,

Alem, L., Tecchia, F. and Huang, W. (2011). ReMoTe: A tele-assistance system for

Azuma, R. T. (1997). A Survey of Augmented Reality. *Presence: Teleoperators and Virtual* 

Badard, T. (2006). Geospatial Service Oriented Architectures for Mobile Augmented Reality.

Bangemann, T., Rebeuf, X., Reboul, D., Schulze, A., Szymanski, J., Thomesse, J-P., Thron, M.,

Bellarbi, A., Benbelkacem, S., Malek, M., Zenati-Henda, N., Belhocine, M. (2010 (a)).

Bellarbi, A., Benbelkacem, S., Malek, M., Zenati-Henda, N., Belhocine, M. (2010 (b)).

Bleser, G., Pastarmov, Y., Stricker, D. (2005). Real-time 3D Camera Tracking for Industrial

Booth, D., Haas,H., McCabe, F., Newcomer, E., Champion, M., Ferris, C., Orchard, D. (2004).

Bottecchia, S., Cieutat, J-M., Merlo, C., Jessel, J-P. (2009). A new AR interaction paradigm for

Bottecchia, S., Cieutat, J.-M., Jessel J.-P. (2010) T.A.C: Augmented Reality System for

Changzhi, K., Bo, K., Dongyi C., Xinyu, L. (2006) .An Augmented reality based Application

Comport, A. I., Kragic, D., Marchand, E., Chaumette, F. (2005). Robust real-time visual

maintenance operators in mines. *Proceedings of 11th Underground Coal Operators* 

*Proceedings of the 1st International Workshop on Mobile Geospatial Augmented Reality*.

Zerhouni, N. (2006). PROTEUS—Creating distributed maintenance systems through an integration platform. *Computers in Industry Journal, Elsevier*, Vol 57,

Amélioration des performances d'ARToolKit pour la réalisation d'applications de réalité augmentée. *International Conference on Image and Signal Processing and their* 

Dynamic thresholding technique for ARToolKit recognition markers. *International Conference on Electrical Engineering, Electronics and Automatics, ICEEEA'10,* Bejaia,

Augmented Reality Applications. *Proceedings of International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision*, Plzen - Bory, Czech

Web Services Architecture. In: *W3C Working Group*. Available from:

collaborative teleassistance system: the POA. *International Journal on Interactive* 

Collaborative Tele-Assistance in the Field of Maintenance through Internet.

for Equipment Maintenance. *Book Chapter, Advances in Artificial Reality and Tele-*

tracking: comparison, theoretical analysis and performance evaluation. *In IEEE* 

sensors network installation is necessary to provide the equipment's state.

*Conference (COAL2011)*. Wollongong University.

Environments, Vol 6, pp.355-385.

*Applications, ISPA'2010*, Biskra, Algeria.

<http://www.w3.org/TR/ws-arch>.

*Existence*, Springer, Vol. 4282.

*Design and Manufacturing, Springer*, Vol. 3 N. 1.

*Augmented Human (AH'2010)*, Megève, France.

Banff, Canada, pp. 73-77.

pp.539-551.

Algeria.

Republic, pp. 47-54.

**6. References** 

*International Conference on Robotics and Automation (ICRA'05)*, pp.2841-2846, Barcelona, Spain.


**The Design and Implementation of** 

*National Taichung University of Education/ Hyweb Technology Co., Ltd.* 

Today's technological advancements, many innovative applications continue to emerge, and in supporting education and learning has brought many changes. With these changes, the application of using virtual reality technology has greatly different on the educational learning way compared to the traditional computer-assisted instruction. Such as abstract concepts simulation, virtual object manipulation, and interactive 3D gaming system, etc.

Through innovative technology-based learning, many learners do produce effective learning, and based on this learning effectiveness, more and more different kinds of technology-transfer medium system were requested to support learners more realistic environment in their computer-based learning system. Therefore, Augmented Reality (AR) technology gains attention in educational use because of its feature of combing real-life

AR technology, through the heavy helmet display, until using the webcam and markers directly to display the result, the combing of real-life scenes and the virtual objects, on the screen, is a mature technology can be applied to assist students with learning. However, as the fixed correspondence of each maker and virtual object in AR learning system, the development of each learning courseware based on AR technology is time-consuming. Thus, how to apply software engineering methods, combined with the AR technology in education to promote more effective use of AR-based learning programs, is a subject can't

Cooperative learning is an important way of learning in a modern educational environment. Remote cooperative learning is also an inevitable way of learning in today's internet era. Collaborative learning, from the past face to face discussions to the current Internet remote distance learning, is constantly changing, but the key is how learners can really communicate with each other between the meaning of sharing and the achievement of interaction. Based on this critical factor, many Internet-based collaborative learning systems were proposed. These systems are common, mainly through the use of the real environment among learners or the share of the virtual world generated by the system, to allow learners to engage in a dialogue between the real scenes and common virtual objects for discussion.

**1. Introduction** 

be ignored.

situation and the characteristics of virtual objects.

**On-Line Multi-User Augmented** 

**Reality Integrated System** 

Hsiao-shen Wang and Chih-Wei Chiu

*Taiwan* 

Zhong, X., Boulanger, P. (2002). Collaborative augmented reality: A prototype for industrial training. *21th Biennial Symposium on Communication*, Canada. **12** 
