**6.2 Reconstruction**

*Advanced Methods and New Materials for Cultural Heritage Preservation*

tion in most of the existing AR systems.

following section.

real environments [23]. Any AR system without an accurate registration leads to unsuccessful mixed environments because of the results of the defect wrong. The registration process is the overlay of virtual objects onto a real scene by using information that have extracted from the scene. Especially, this information is the feature points that extracted from the real scene using some tracking techniques. There are two categories of registration techniques, sensor-based and computer vision-based techniques. In sensor-based technique, there is a need to calibrate the external sensors, but the available sensors equipment's are either huge or expensive, or lack satisfactory levels of accuracy. Computer vision-based methods techniques work to avert calibration of external sensors, as well as offer the possibility for accurate tracking without huge and costly equipment. It can be categorised as two main types depending on camera calibration requirements [33]. The first kind does not require any calibration of camera parameters in advance, which includes the use of a known 3D calibration object. However, the second type is assuming that the intrinsic camera parameters are pre-calibrated. This is a common assump-

There are several researches that work to develop the registration of the virtual elements in the real-world environments. These researches will be explained in the

Gao et al. [6] introduced a new technique to improve the stabilisation and the accuracy of marker-less registration in AR. Based on three-dimensional map information generated by visual simultaneous localisation-SLAM. The proposed technique allows tracking and registration of virtual objects in order to ensure a stable in addition of real-time performance of marker-less AR applications. The stability of the system can be performed by integrating the Hough voting algorithm with the repeated Closest Points (ICP) technique. The proposed technique is faster than the standard methods. In addition, it is able to achieve more accurate registration results when compared with the previous techniques. The experimental results showed that the proposed technique can efficiently repress the virtual object jittering, as well as a higher tracking accuracy with good performance [6]. This technique can identify only one object for each recognition. Kanade-Lucas-Tomasi (KLT) natural feature tracker and the reconstruction technology is presented by Pang et al. [33]. KLT tracker technique is used to track the identical feature points in two control images. The authors presented three key stages in the proposed technique. The first stage is the affine reconstruction. In this stage, two control images from the video sequence are chosen and the KLT tracker is used for the extraction of the natural feature points. After that, the Affine Coordinate System (ACS) is defined by using these natural feature points. The user is responsible to select four planar points for setting the Euclidean WCS in two control images, respectively, and then the affine coordinates of the specific points are reconstructed by using the affine reconstruction method. While, the second stage is re-projection. Compute the corresponding affine reprojection matrix in the live video frame by using the natural feature points that have been tracked by the KLT technique. The image projections of the selected points are predestined in the live video sequence by using the affine re-projection matrix. However, the third stage is the camera extrinsic parameters such as camera pose, which are predestined in terms of the four selected points achieved in the second stage. Eventually, the virtual objects can be rendered on the real scene by using the graphics pipeline techniques such as OpenGL. The experiment results showed some improvement compared to the previous work [33]. The main limitation of this research is that the user has to manually determine the four points in the initialization stage, as well as, the authors do not consider

**64**

tracking the feature points.

Reconstruction is one of the basic processes in the AR. It refers to the construction of virtual objects in a similar way to replicate the original building [23]. Many cultural heritage applications require to reconstruct of real-world objects and scenes. Reconstruction process becomes increasingly common to use for modelling purpose of cultural heritage. This is fundamentally because of rapid development in laser-scanning techniques, 3D modelling, image-based modelling techniques, the power of the computer and virtual reality. The default objects appear on an appropriate model that covers the details of accurate enough is essential [23]. Objects must be exactly identical to the original ones which visitors can see clearly at the background of live videos. In addition, interest in objects' shadows is an essential part of the reconstruction process. Real-time shadows are created in relation to the sun position in a specified location, date and time. Eventually, the influence of the sky lighting on the virtual building during the daytime is the last part of creating the realistic virtual heritage in AR systems. Most virtual reconstruction techniques are based mainly on 3D scanning techniques, in order to get the objects faithfully [34]. **Figure 5** shows the reconstruction of the building and place it in the real environment.

## **6.3 Tracking**

Tracking is a substantial subject in a real-time augmented reality context. The key requirements for tracking are the high level of accuracy and low level of latency at a sensible cost. Objects' tracking in the scene is defined as the amount of the pose between the camera and the objects. Virtual objects can be displayed into the scene using the pose. A local moving edges tracker have been used to provide real-time tracking of points normal to the object contours [27].

A new method for conception of vision-based augmented reality systems is presented by considering either 3D model-based tracking techniques or 3D modelfree tracking approaches [1]. The method depends on decreasing the cost function expressed in the image and this decreasing is achieved via a visual serving control law. The main feature of a model-based method is that the information about the scene allows improvement of robustness and system' performance by the ability for predicting hidden movement of the object and acts in order to reduce the effects of outlier data introduced in the tracking process [35]. It is occasionally necessary to achieve the pose computation with minimal constraining information on the viewed scene because 3D information is not readily available in certain circumstances. The algorithm has been tested on different image sequences and for diverse applications, which illustrate a real usability of this approach [1]. This research has

**Figure 5.** *Realistic reconstruction of cultural heritage.*

several limitations. The first limitation is the system that relies on a course manual initialization on the very first image. The second limitation is the system that does not take spatiotemporal aspect of the tracking process in depth consideration. Robustness can also be treated from one time-step to another. A novel marker-less camera tracking system and user interaction methodology for augmented reality (AR) on unprepared table-top environments is presented by Lee et al. [36]. A real-time system architecture is presented to merge two kinds of feature tracking. Marker-less tracking method is initialised by a simple hand gesture using the Handy AR system that used to estimate a camera pose from a user's outstretched hand. Detecting distinctive image features of the scene and tracking frame-to-frame by computing optical flow. The proposed system used distinctive image features for recognising the scene and to correct for accumulated tracking errors. For achieving real-time performance, multiple operations are processed in a synchronised multithreaded method: capturing a video frame, tracking features using optical flow, detecting distinctive invariant features and rendering an output frame. The speed and accuracy of hybrid feature tracking system have been evaluate and demonstrate a proof-of-concept application to enable AR in unprepared table-top environments, by using bare hands for interaction [36]. One of the significant limitation of this research is the system applied on 2D scene.

Novel interactive techniques for outdoor augmentation have presented to use a mobile device [37]. The system can be executed and perform real time on simple mini PC equipment. Feature tracking have been used for estimating camera motion when user turns the mobile device and examines the augmented scene. The authors have considered two scenarios. The first scenario is constantly applicable with any 3D model for ad hoc use without prior information or calibration process. The second scenario uses GPS for realising the viewing location and Google Earth KML files for locating the augmented object and its placement. This method, 3D object placed on Google Earth can be viewed on site without any addition data transformation steps. The systems have been tested with potential end users. The authors believe that the system is useful in diverse current real-life applications [35].

A model-based hybrid tracking system is proposed for outdoor AR applied for urban environments that allows accurate, real-time overlays for a handheld device [38]. The system merges different well-known techniques in order to provide a powerful experience that surpasses each of the individual components alone: an edge-based tracker that used for accurate localisation, gyroscope measurements to cope with fast motions, gravity measurements and magnetic field to avert drift and a rear store of reference frames with online frame chosen used to re-initialise automatically after dynamic occlusions or failures. A novel edge-based tracker distributes with the traditional edge model, and uses instead of a coarse, but textured, 3D model. This technique has several features [39]. The first feature is automatically disposing from scale-based detail, appearancebased edge signatures can be used to improve conformity and the models required are more usually available. The second feature is the system's accuracy and robustness is pretending with comparisons to map-based ground truth data. The tracking system have the possibility to apply to other types of display such as head mounted displays using video see-through overlays, while optical seethrough displays would demand further calibration of the HMD's virtual camera with take in account of the video camera [38]. This system has the resulting asymmetry in the information display capabilities of the two environments (virtual and real-time environments). An integral natural feature-based tracking system is proposed to support the creation of AR applications that concentrated

**67**

*Cultural Heritage in Marker-Less Augmented Reality: A Survey*

**7. 3D reconstruction techniques for cultural heritage**

AR technologies have become increasingly popular. These techniques are not just practical for developers of AR system, but also to the scientific community. The standard approach to create a 3D model is to build it from scratch using tools such as the unity 3D programme, which provides building blocks in the form of primitive 3D shapes. Many new technologies aim to increase the level of automation and realism by beginning with the real images of the object or converting it to direct

This technology includes vastly available devices, so the same system can handle a wide range of objects and scenes. These systems have the ability to create a realistic model, and those rely on photogrammetry have high geometric accuracy. This technique is usually used for geometric surfaces of architecture objects or for modelling precise terrain. It uses a mathematical model to capture 3D object information from 2D image dimensions or obtain 3D data using methods such as shading, texture, theory, contour and 2D edge gradient [41]. Deriving 3D measurements from images naturally requires that interest points be appearance in the image. Often, this is not potential, either because the area is hidden or covered behind an object or surface or because there is no mark, edge or visible feature to extract [28]. The main goal of image-based reconstruction is the ability to represent arbitrary geometry. For modelling complete geometric structures, it is usually necessary to remove the labour-intensive task through this approach [41]. The mechanism can also deal with the real-world effects that images take, but difficult to reproduce with

3D geometry information for an object can be captured directly by this technique [28]. The 3D measurement of images requires that interest points or edges

on the automotive sector [40]. An AR application was construct on top of the system to refer to the location of 3D coordinates in a specific environment. It can be applied to many various applications in cars, for example, a maintenance assistant, an intelligent manual and many more. The system is evaluated during the Volkswagen/ISMAR Tracking Challenge 2014, which designed to test stateof-the-art tracking technique based on requirements encountered in automotive industrial settings. Evaluation results illustrate that the system allowed users to correctly determine tasks points that involved tracking a revolving vehicle, tracking data on an integral vehicle and tracking with high accuracy. The evaluation of the system is allowed to understand the applicability boundaries of texture-based technique in the texture less automotive environment, a problem not addressed considerably in the literature [40]. This research has several limitations. The first limitation is low frame rate when the number of 3D key-points in the model is large. The second limitation is error accumulation when the entire vehicle is reconstructed in a single take. The third limitation is lack of temporal continuity, which may result for shivering; sensibility to extreme illumination conditions. The fourth limitation is accidental failures when cope with scenes that have mini-

*DOI: http://dx.doi.org/10.5772/intechopen.80975*

mum of texture information.

digitisation using a laser scanner [28].

the customary graphics techniques.

**7.2 Range-based modelling**

**7.1 Image-based modelling**

#### *Cultural Heritage in Marker-Less Augmented Reality: A Survey DOI: http://dx.doi.org/10.5772/intechopen.80975*

*Advanced Methods and New Materials for Cultural Heritage Preservation*

research is the system applied on 2D scene.

applications [35].

several limitations. The first limitation is the system that relies on a course manual initialization on the very first image. The second limitation is the system that does not take spatiotemporal aspect of the tracking process in depth consideration. Robustness can also be treated from one time-step to another. A novel marker-less camera tracking system and user interaction methodology for augmented reality (AR) on unprepared table-top environments is presented by Lee et al. [36]. A real-time system architecture is presented to merge two kinds of feature tracking. Marker-less tracking method is initialised by a simple hand gesture using the Handy AR system that used to estimate a camera pose from a user's outstretched hand. Detecting distinctive image features of the scene and tracking frame-to-frame by computing optical flow. The proposed system used distinctive image features for recognising the scene and to correct for accumulated tracking errors. For achieving real-time performance, multiple operations are processed in a synchronised multithreaded method: capturing a video frame, tracking features using optical flow, detecting distinctive invariant features and rendering an output frame. The speed and accuracy of hybrid feature tracking system have been evaluate and demonstrate a proof-of-concept application to enable AR in unprepared table-top environments, by using bare hands for interaction [36]. One of the significant limitation of this

Novel interactive techniques for outdoor augmentation have presented to use a mobile device [37]. The system can be executed and perform real time on simple mini PC equipment. Feature tracking have been used for estimating camera motion when user turns the mobile device and examines the augmented scene. The authors have considered two scenarios. The first scenario is constantly applicable with any 3D model for ad hoc use without prior information or calibration process. The second scenario uses GPS for realising the viewing location and Google Earth KML files for locating the augmented object and its placement. This method, 3D object placed on Google Earth can be viewed on site without any addition data transformation steps. The systems have been tested with potential end users. The authors believe that the system is useful in diverse current real-life

A model-based hybrid tracking system is proposed for outdoor AR applied for urban environments that allows accurate, real-time overlays for a handheld device [38]. The system merges different well-known techniques in order to provide a powerful experience that surpasses each of the individual components alone: an edge-based tracker that used for accurate localisation, gyroscope measurements to cope with fast motions, gravity measurements and magnetic field to avert drift and a rear store of reference frames with online frame chosen used to re-initialise automatically after dynamic occlusions or failures. A novel edge-based tracker distributes with the traditional edge model, and uses instead of a coarse, but textured, 3D model. This technique has several features [39]. The first feature is automatically disposing from scale-based detail, appearancebased edge signatures can be used to improve conformity and the models required are more usually available. The second feature is the system's accuracy and robustness is pretending with comparisons to map-based ground truth data. The tracking system have the possibility to apply to other types of display such as head mounted displays using video see-through overlays, while optical seethrough displays would demand further calibration of the HMD's virtual camera with take in account of the video camera [38]. This system has the resulting asymmetry in the information display capabilities of the two environments (virtual and real-time environments). An integral natural feature-based tracking system is proposed to support the creation of AR applications that concentrated

**66**

on the automotive sector [40]. An AR application was construct on top of the system to refer to the location of 3D coordinates in a specific environment. It can be applied to many various applications in cars, for example, a maintenance assistant, an intelligent manual and many more. The system is evaluated during the Volkswagen/ISMAR Tracking Challenge 2014, which designed to test stateof-the-art tracking technique based on requirements encountered in automotive industrial settings. Evaluation results illustrate that the system allowed users to correctly determine tasks points that involved tracking a revolving vehicle, tracking data on an integral vehicle and tracking with high accuracy. The evaluation of the system is allowed to understand the applicability boundaries of texture-based technique in the texture less automotive environment, a problem not addressed considerably in the literature [40]. This research has several limitations. The first limitation is low frame rate when the number of 3D key-points in the model is large. The second limitation is error accumulation when the entire vehicle is reconstructed in a single take. The third limitation is lack of temporal continuity, which may result for shivering; sensibility to extreme illumination conditions. The fourth limitation is accidental failures when cope with scenes that have minimum of texture information.
