Preface

Chapter 8 **How to Create Suitable Augmented Reality Application to Teach Social Skills for Children with ASD 119**

**VI** Contents

I-Jui Lee, Ling-Yi Lin, Chien-Hsu Chen and Chi-Hsuan Chung

Virtual reality (VR) and augmented reality (AR) have long been attributed to science fiction movies and gaming industry. Although VR was popular for decades, AR has been democra‐ tised through famous Pokemon Go. Combination of VR/AR and the physical environment commonly called mixed reality on smartphones has prompted developers towards new heights of creativity and innovation. No need for high-tech wearables to experience VR/AR environments the cloud is quite helpful in processing large multimedia and code processing. One may think that VR/AR is more about multimedia development and visualisation techni‐ ques; nevertheless, there is a huge algorithmic component behind VR/AR systems. Major ICT players have come up with libraries for digital objects that have further accelerated the development life cycle of VR/AR systems.

Regarding challenges, there are technical limitations but also reluctance from users to adopt VR/AR technology. Nonetheless, few mobile manufacturers are not willing to implement such technologies. The main challenge remains dynamic content that would keep the inter‐ est of the users and with exciting user experience so that applications do not fall obsolete within a short time span. Like Multimedia Message Service (MMS), VR/AR has had poor adoption due to lack of content and appropriate business models between service providers and content developers.

On the other hand, VR/AR has proven to be very much useful in healthcare and engineering fields, without mentioning educational technologies where it is already a legacy. Medical scientists, educators and engineers require deeper understanding of how VR/AR can assist them in their daily activities as well as pertinent issues. So far from user perspective, this book showcases research and development to enlighten readers with background in VR/AR.

Moreover, it compiles latest advancement in the fields of VR/AR from a broad perspective. Chapters cover core areas where presently VR/AR technologies are applied, namely, health‐ care, education, building industry and visualisation. The chapters' content provides experi‐ mental validations and high scientific value.

We would like to express our whole-hearted appreciations to all the authors and their re‐ spective institutions who have contributed for the accomplishment of this book. Everything has been possible thanks to the InTech publishing process managers Mr. Julian Virag and Mrs. Danijela Vladika, the technical board, and the commissioning editor for their endeav‐ our in disseminating knowledge globally.

> **Associate Professor (Dr) Nawaz Mohamudally** University of Technology, Mauritius

**Chapter 1**

**Provisional chapter**

**Introductory Chapter: Enhancing Augmented Reality**

**Introductory Chapter: Enhancing Augmented Reality** 

One of most amazing aspects of VR and AR is user experience that can be defined as personal or group feelings about VR and AR applications and products. VR/AR user experience is more than usability, perception or customer adoption patterns, it deals with the emotional interaction with technology as well as physical and virtual environments. Although VR/AR is mostly a visualisation technique, other senses like touch and hearing are increasingly present. One example of how user experience can drive a business model is actually the success of social networks on the Internet. Smartphones too have provided rich user experience with multimedia and VoIP applications. AR has brought richer user experience on smartphones that are accessible through the camera or a VR glass. However the resources limitations on handheld devices have prompted novel methods of computation. Extensive work exists on mobile code offloading towards a server or a cloud to minimise battery depletion. Another method is VM cloning, but despite techniques to overcome hardware and software challenges little is done in view of understanding and improving user experience. This chapter gives a brief apercu of design thinking, a technique that can enhance VR/AR applications user experi-

> © 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution,

© 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

and reproduction in any medium, provided the original work is properly cited.

*"In 2021, the augmented and virtual reality market is expected to reach a market size of 215 billion U.S. dollars."*<sup>1</sup> AR applications are today multidisciplinary. Below are listed some of the

1 https://www.statista.com/statistics/591181/global-augmented-virtual-reality-market-size/

DOI: 10.5772/intechopen.76850

**User Experience (AR-UX) with Design Thinking**

**User Experience (AR-UX) with Design Thinking**

Nawaz Mohamudally

**1. Introduction**

**2. Use cases**

Additional information is available at the end of the chapter

ence. Focus is on AR applications for smartphones.

Nawaz MohamudallyAdditional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.76850

#### **Introductory Chapter: Enhancing Augmented Reality User Experience (AR-UX) with Design Thinking Introductory Chapter: Enhancing Augmented Reality User Experience (AR-UX) with Design Thinking**

DOI: 10.5772/intechopen.76850

Nawaz Mohamudally

Additional information is available at the end of the chapter Nawaz MohamudallyAdditional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.76850

#### **1. Introduction**

 One of most amazing aspects of VR and AR is user experience that can be defined as personal or group feelings about VR and AR applications and products. VR/AR user experience is more than usability, perception or customer adoption patterns, it deals with the emotional interaction with technology as well as physical and virtual environments. Although VR/AR is mostly a visualisation technique, other senses like touch and hearing are increasingly present. One example of how user experience can drive a business model is actually the success of social networks on the Internet. Smartphones too have provided rich user experience with multimedia and VoIP applications. AR has brought richer user experience on smartphones that are accessible through the camera or a VR glass. However the resources limitations on handheld devices have prompted novel methods of computation. Extensive work exists on mobile code offloading towards a server or a cloud to minimise battery depletion. Another method is VM cloning, but despite techniques to overcome hardware and software challenges little is done in view of understanding and improving user experience. This chapter gives a brief apercu of design thinking, a technique that can enhance VR/AR applications user experience. Focus is on AR applications for smartphones.

#### **2. Use cases**

*"In 2021, the augmented and virtual reality market is expected to reach a market size of 215 billion U.S. dollars."*<sup>1</sup> AR applications are today multidisciplinary. Below are listed some of the

<sup>1</sup> https://www.statista.com/statistics/591181/global-augmented-virtual-reality-market-size/

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

areas and AR use cases with specific user experiences. AR can transform one's home interior into an archaeological site with artefacts 15,000 years back, it is indeed another level of user experience.

**4. Design thinking for AR**

come with experience or by sharing.

information about user experience.

**Figure 1.** Design thinking framework.

design thinking. The process is iterative and non-linear.

box thinking and observing nature are ways to get inspired.

psychological aspect of the users towards the AR app should be captured.

Design a way of thinking was initially meant for creation and problem solving. It is actually a combination of analytical thinking and intuitive thinking that dates back to the 1950s and 1960s. On its way, design thinking to improve UX has gained momentum. Today's smartphone resulted from this approach. **Figure 1** depicts global view of the main components of

Introductory Chapter: Enhancing Augmented Reality User Experience (AR-UX) with Design…

http://dx.doi.org/10.5772/intechopen.76850

3

*Inspiration:* involves an accurate understanding of a problem or an eventual product where AR can add value. It is like envisioning the application even if it appears to be beyond the capacity of current development tools. Inspiration does not come out of the blue, out of the

*Empathy:* is to look at the needs of the users, not only gathering user requirements but what users would like to experience. It could be higher image resolution, larger display, and quicker response and so on. Ease of use and fulfilling needs are parts of user experience. The

*Ideation*: this is perhaps the most important milestone in enhancing user experience as it deals with challenges and ability to find innovative answers to the findings during the empathy phase but also the opportunities and limitations provide during the prototyping phase. Ideas

*Prototyp*ing: involves implementation and testing of the AR app in an agile manner. Testing is crucial in producing an application that is reliable and answers to the user expectations. The element of surprise or extra excitement is the plus in user experience, a differentiator. A survey conducted on testing of AR apps has revealed that most of the testing fields are technology oriented that are camera settings, native or hybrid app, and so on. There is little

Coming back to the AR development for smartphones, marker-less applications are often more challenging but appreciated compared to marker based applications. Users love the element of surprise and astonishment. But the smartphone is not suitable to plug hardware devices that can add extra sensation apart from video, audio, 3D interaction. Nonetheless, the presence of sensors and access to data from cloud puts forward mobile cloud computing as a serious option for future AR applications with new user experience. Sophisticated scripts and AI code can run on the cloud and render user information and experience in real time on the smartphone. Currently effects are on the graphical objects more than the business rules. Thus an AR developer is first and foremost a graphic designer and a multimedia developer. To

*Gaming:* Gaming experience with AR picked up with the famous PokemonGO. Candy Camera is a simple app that allows to add features to selfies. 3D animation and mixed realities combining physical environments and digital objects provide a special user experience in AR games. Players look for entertainment and sensations that are different from video games with shootings.

*Education:* AR applications bring new learning and teaching experiences aiming at developing critical thinking and self-learning. In engineering visualisation of artefacts with interaction is another form of instructions. Likewise the study of bones in medicine with an AR app enhances the student perspective.

*Healthcare*: AR is more and more adopted for patient rehabilitation and for diagnosis. Recent advances advocate that AR will be able to provide surgeons with an environment to prepare themselves prior to a surgery operation. This is an example of kind of user experience in healthcare that is unique to this situation.

*Building industry:* AR is widely used in building information systems, however AR is used also to safeguard patrimonies in digital format or holograms. History discovery and tourism are typical experiences from such applications.

*Navigation:* Popular examples are Flight Simulator and Immersion experience. In such applications there is contrast between the physical environment and virtual environment. Immersion generally is fully virtual, whereas navigating in a building or a site is about mixed realities.

*VuMark:* This is a product code just like the QR code that uses Vuforia technology to display additional information about a product for example when the camera of a smartphone points at the VuMark. It could provide another type of sales experience for consumers. It has been shown that pricing is not always the major determining factor, sales experience makes a difference.

#### **3. AR apps development life cycle**

Developing AR apps requires 3D Graphics and Object Design software (Example 3D MAX, a game engine (Unity3D), Qualcomm Vuforia (an AR SDK to track Image Targets), Head Mounted Display or a Smartphone with a high resolution camera). To enhance user experience, there are existing software engineering techniques like co-construction and user involvement. The time consuming activity is about design and production of the graphics objects that amount to more than 80% of a project. Google Libraries for AR objects and AR Kit have facilitated this phase.

## **4. Design thinking for AR**

areas and AR use cases with specific user experiences. AR can transform one's home interior into an archaeological site with artefacts 15,000 years back, it is indeed another level of user

*Gaming:* Gaming experience with AR picked up with the famous PokemonGO. Candy Camera is a simple app that allows to add features to selfies. 3D animation and mixed realities combining physical environments and digital objects provide a special user experience in AR games. Players look for entertainment and sensations that are different from video games

*Education:* AR applications bring new learning and teaching experiences aiming at developing critical thinking and self-learning. In engineering visualisation of artefacts with interaction is another form of instructions. Likewise the study of bones in medicine with an AR app

*Healthcare*: AR is more and more adopted for patient rehabilitation and for diagnosis. Recent advances advocate that AR will be able to provide surgeons with an environment to prepare themselves prior to a surgery operation. This is an example of kind of user experience in

*Building industry:* AR is widely used in building information systems, however AR is used also to safeguard patrimonies in digital format or holograms. History discovery and tourism

*Navigation:* Popular examples are Flight Simulator and Immersion experience. In such applications there is contrast between the physical environment and virtual environment. Immersion generally is fully virtual, whereas navigating in a building or a site is about mixed realities.

*VuMark:* This is a product code just like the QR code that uses Vuforia technology to display additional information about a product for example when the camera of a smartphone points at the VuMark. It could provide another type of sales experience for consumers. It has been shown that pricing is not always the major determining factor, sales experience makes

Developing AR apps requires 3D Graphics and Object Design software (Example 3D MAX, a game engine (Unity3D), Qualcomm Vuforia (an AR SDK to track Image Targets), Head Mounted Display or a Smartphone with a high resolution camera). To enhance user experience, there are existing software engineering techniques like co-construction and user involvement. The time consuming activity is about design and production of the graphics objects that amount to more than 80% of a project. Google Libraries for AR objects and AR Kit

experience.

with shootings.

a difference.

enhances the student perspective.

healthcare that is unique to this situation.

are typical experiences from such applications.

2 State of the Art Virtual Reality and Augmented Reality Knowhow

**3. AR apps development life cycle**

have facilitated this phase.

Design a way of thinking was initially meant for creation and problem solving. It is actually a combination of analytical thinking and intuitive thinking that dates back to the 1950s and 1960s. On its way, design thinking to improve UX has gained momentum. Today's smartphone resulted from this approach. **Figure 1** depicts global view of the main components of design thinking. The process is iterative and non-linear.

*Inspiration:* involves an accurate understanding of a problem or an eventual product where AR can add value. It is like envisioning the application even if it appears to be beyond the capacity of current development tools. Inspiration does not come out of the blue, out of the box thinking and observing nature are ways to get inspired.

*Empathy:* is to look at the needs of the users, not only gathering user requirements but what users would like to experience. It could be higher image resolution, larger display, and quicker response and so on. Ease of use and fulfilling needs are parts of user experience. The psychological aspect of the users towards the AR app should be captured.

*Ideation*: this is perhaps the most important milestone in enhancing user experience as it deals with challenges and ability to find innovative answers to the findings during the empathy phase but also the opportunities and limitations provide during the prototyping phase. Ideas come with experience or by sharing.

*Prototyp*ing: involves implementation and testing of the AR app in an agile manner. Testing is crucial in producing an application that is reliable and answers to the user expectations. The element of surprise or extra excitement is the plus in user experience, a differentiator. A survey conducted on testing of AR apps has revealed that most of the testing fields are technology oriented that are camera settings, native or hybrid app, and so on. There is little information about user experience.

Coming back to the AR development for smartphones, marker-less applications are often more challenging but appreciated compared to marker based applications. Users love the element of surprise and astonishment. But the smartphone is not suitable to plug hardware devices that can add extra sensation apart from video, audio, 3D interaction. Nonetheless, the presence of sensors and access to data from cloud puts forward mobile cloud computing as a serious option for future AR applications with new user experience. Sophisticated scripts and AI code can run on the cloud and render user information and experience in real time on the smartphone. Currently effects are on the graphical objects more than the business rules. Thus an AR developer is first and foremost a graphic designer and a multimedia developer. To

**Figure 1.** Design thinking framework.

further enhance UX, more data analytics is required. On the network side higher bandwidth coupled with advancement in cellular networks, for instance 5G has not yet disclosed the potential for richer AR apps.

**Chapter 2**

**Provisional chapter**

**Augmented Reality and Virtual Reality: Initial**

**Augmented Reality and Virtual Reality: Initial** 

DOI: 10.5772/intechopen.74317

In this chapter, we will review the applications of augmented reality (AR) and virtual reality (VR) in diagnostic radiology. We begin the chapter by discussing state of the art medical imaging techniques to include scanner hardware, spatial resolution, and presentation methods to the radiologist or other medical professionals. We continue by discussing the methodology of a technique called Depth-3-Dimensional (D3D) imaging, which transforms cross-sectional medical imaging datasets into a left and right eye images that that can be viewed with state of the art AR or VR headsets. We include results of the D3D

The purpose of this chapter is to review the applications of augmented reality (AR) and virtual reality (VR) in diagnostic radiology. This introduction section will provide a brief general summary of AR technologies, VR technologies, AR/VR applications in medicine and AR/VR applications diagnostic radiology. The remainder of the paper will discuss state-of-the-art medical imaging systems, current methods of viewing medical images, the imaging processing techniques for generating an AR/VR image, AR/VR imaging results and path forward.

processed AR/VR imaging and conclude with a discussion of the path forward. **Keywords:** augmented reality, virtual reality, diagnostic radiology, D3D, depth

> © 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

© 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

**Successes in Diagnostic Radiology**

**Successes in Diagnostic Radiology**

David B. Douglas, Demetri Venets, Cliff Wilke, David Gibson, Lance Liotta, Emanuel Petricoin,

David B. Douglas, Demetri Venets, Cliff Wilke, David Gibson, Lance Liotta, Emanuel Petricoin,

Additional information is available at the end of the chapter

Additional information is available at the end of the chapter

Buddy Beck and Robert Douglas

Buddy Beck and Robert Douglas

http://dx.doi.org/10.5772/intechopen.74317

**Abstract**

perception

**1. Introduction**

#### **5. Conclusion**

This chapter highlights user experience component in the development of AR applications. Design thinking is proposed to augment user experience in AR applications for smartphones. It is an iterative method that can add value to the already the hardware and software challenges in AR applications. AR applications may not always be customer or user centric, so it is on a case to case basis that design thinking must be applied. Survey on the prototyping milestone has shown that in the testing phase UX is often negligible although user testing is present compared to technical issues and Empathy is not straightforward. The future of very rich AR-UX lies in mobile cloud computing. The coming chapters in this book deal with more pertinent and complex scientific issues in VR/AR, nevertheless readers will appreciate how far user experience is pervasive.

#### **Author details**

Nawaz Mohamudally

Address all correspondence to: alimohamudally@umail.utm.ac.mu

University of Technology, Mauritius

#### **Chapter 2 Provisional chapter**

#### **Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology**

DOI: 10.5772/intechopen.74317

David B. Douglas, Demetri Venets, Cliff Wilke, David Gibson, Lance Liotta, Emanuel Petricoin, Buddy Beck and Robert Douglas David B. Douglas, Demetri Venets, Cliff Wilke, David Gibson, Lance Liotta, Emanuel Petricoin, Buddy Beck and Robert Douglas

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.74317

#### **Abstract**

further enhance UX, more data analytics is required. On the network side higher bandwidth coupled with advancement in cellular networks, for instance 5G has not yet disclosed the

This chapter highlights user experience component in the development of AR applications. Design thinking is proposed to augment user experience in AR applications for smartphones. It is an iterative method that can add value to the already the hardware and software challenges in AR applications. AR applications may not always be customer or user centric, so it is on a case to case basis that design thinking must be applied. Survey on the prototyping milestone has shown that in the testing phase UX is often negligible although user testing is present compared to technical issues and Empathy is not straightforward. The future of very rich AR-UX lies in mobile cloud computing. The coming chapters in this book deal with more pertinent and complex scientific issues in VR/AR, nevertheless readers will appreciate how

potential for richer AR apps.

4 State of the Art Virtual Reality and Augmented Reality Knowhow

far user experience is pervasive.

University of Technology, Mauritius

Address all correspondence to: alimohamudally@umail.utm.ac.mu

**Author details**

Nawaz Mohamudally

**5. Conclusion**

In this chapter, we will review the applications of augmented reality (AR) and virtual reality (VR) in diagnostic radiology. We begin the chapter by discussing state of the art medical imaging techniques to include scanner hardware, spatial resolution, and presentation methods to the radiologist or other medical professionals. We continue by discussing the methodology of a technique called Depth-3-Dimensional (D3D) imaging, which transforms cross-sectional medical imaging datasets into a left and right eye images that that can be viewed with state of the art AR or VR headsets. We include results of the D3D processed AR/VR imaging and conclude with a discussion of the path forward.

**Keywords:** augmented reality, virtual reality, diagnostic radiology, D3D, depth perception

#### **1. Introduction**

The purpose of this chapter is to review the applications of augmented reality (AR) and virtual reality (VR) in diagnostic radiology. This introduction section will provide a brief general summary of AR technologies, VR technologies, AR/VR applications in medicine and AR/VR applications diagnostic radiology. The remainder of the paper will discuss state-of-the-art medical imaging systems, current methods of viewing medical images, the imaging processing techniques for generating an AR/VR image, AR/VR imaging results and path forward.

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### **1.1. Introduction to AR/VR**

AR technologies can be classified into AR systems or mixed reality (MR) systems. In both the AR and MR systems, the user wears a head mounted display (HMD), which provides the simultaneous display of a virtual image and the scene of the real-world surroundings [1]. The differences are that in AR, the virtual image is transparent like a hologram and in MR the virtual image appears solid. Examples of AR systems include the Meta and DAQRI systems. An example of an MR system is the Microsoft HoloLens. In AR systems, the user can view the virtual image and interact with the real-world scene.

**2. Current state-of-the-art diagnostic medical imaging**

methods.

focus on the latter.

images.

**2.1. Medical imaging equipment**

*2.1.1. Computed tomography (CT)*

Diagnostic radiology plays a major role in medicine as it provides precise anatomic and physiologic information to physicians enabling diagnosis of complex disease and monitoring response to treatment. This section will be organized into three subsections including medical imaging equipment, conventional imaging techniques and the advanced 3D rendering

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

7

The field of diagnostic radiology includes a wide variety of imaging equipment including systems that generate inherently 2D images (e.g., chest radiograph) as well as systems that generate volumetric medical imaging datasets (e.g., computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET)). In this section, we will

In order to perform a CT scan (also known as a CAT scan), the patient is placed in the horizontal position on the CT scanner table. See **Figure 1**. The table is then translated through a donut shaped device containing both an X-ray tube and X-ray detector. Multiple projection images

Image reconstruction algorithms, such as filtered back projection, are performed in order to generate cross-sectional images in the axial plane (x-y plane). Since sequential, contiguous axial images can be obtained, coronal plane (x-z) and sagittal plane (y-z) plane images can be reconstructed. Data is stored in digital imaging and communication in medicine

**Figure 1.** Photograph of a computed tomography (CT) scanner. The patient is being transferred from onto the mobile CT scanner table that will translate through the donut-shaped device during the CT scan and take multiple projection

are acquired as the X-ray tube and detector assembly rotate around the patient.

VR technologies may be classified as fully immersive, semi-immersive or non-immersive [2]. In fully immersive VR systems such as the Oculus Rift and the HTC Vive, the HMD displays a virtual image and the real-world surroundings are completely occluded from the user's field of view [3]. In semi-immersive VR systems such as the Samsung Gear VR, the HMD displays the virtual image, but the real-world scene is only partially occluded from the user's field of view [3]. In VR systems, the user can maneuver through the virtual world by head movements via HMD tracking or walking via external camera tracking systems. Additional ways that a user can interact with the virtual environment include voice gestures or through handheld devices with haptic feedback. The HMDs for the AR and VR systems display a unique image to each eye; therefore, stereoscopic imaging and depth perceptions is achieved. Both AR and VR are rapidly growing fields. Worldwide revenues for AR/VR were \$5 billion in 2016, but are expected to increase to \$162 billion by 2020 [3].

#### **1.2. Applications of AR/VR in medicine**

In 2015, the United States spent 17% of its gross domestic product on healthcare. 32% of the healthcare spending was during hospital stays [4]. Nearly half of hospital costs are for surgical care [5]. Therefore, approximately 3% of the gross domestic product includes costs related to surgery. There is a drive to improve efficiency in the operation room to both improve patient care and drive down costs. AR/VR holds promise in accomplishing these goals through improving pre-operative planning and enhancing intra-operative surgical procedures [6].

#### **1.3. Applications of AR/VR in diagnostic radiology**

In the United States, the total cost of diagnostic imaging has been estimated to be \$100 billion in 2006 alone [7]. Utilization rates of diagnostic imaging are on the rise [8–10]. Currently, there is no United States Food and Drug Administration (FDA) approved AR/VR system used in diagnostic radiology [11]. AR/VR provides enhanced viewing including depth perception and improved human machine interface (HMI) [12, 13]. AR/VR HMDs provide unique images to each eye yielding depth perception. AR/VR systems leverage advanced gaming controllers and joysticks to improve HMI. Because of these features of AR/VR and others discussed later in this chapter, we believe there will be increasing applications of AR/VR in diagnostic radiology in the future.

### **2. Current state-of-the-art diagnostic medical imaging**

Diagnostic radiology plays a major role in medicine as it provides precise anatomic and physiologic information to physicians enabling diagnosis of complex disease and monitoring response to treatment. This section will be organized into three subsections including medical imaging equipment, conventional imaging techniques and the advanced 3D rendering methods.

#### **2.1. Medical imaging equipment**

**1.1. Introduction to AR/VR**

by 2020 [3].

procedures [6].

virtual image and interact with the real-world scene.

6 State of the Art Virtual Reality and Augmented Reality Knowhow

**1.2. Applications of AR/VR in medicine**

**1.3. Applications of AR/VR in diagnostic radiology**

diagnostic radiology in the future.

AR technologies can be classified into AR systems or mixed reality (MR) systems. In both the AR and MR systems, the user wears a head mounted display (HMD), which provides the simultaneous display of a virtual image and the scene of the real-world surroundings [1]. The differences are that in AR, the virtual image is transparent like a hologram and in MR the virtual image appears solid. Examples of AR systems include the Meta and DAQRI systems. An example of an MR system is the Microsoft HoloLens. In AR systems, the user can view the

VR technologies may be classified as fully immersive, semi-immersive or non-immersive [2]. In fully immersive VR systems such as the Oculus Rift and the HTC Vive, the HMD displays a virtual image and the real-world surroundings are completely occluded from the user's field of view [3]. In semi-immersive VR systems such as the Samsung Gear VR, the HMD displays the virtual image, but the real-world scene is only partially occluded from the user's field of view [3]. In VR systems, the user can maneuver through the virtual world by head movements via HMD tracking or walking via external camera tracking systems. Additional ways that a user can interact with the virtual environment include voice gestures or through handheld devices with haptic feedback. The HMDs for the AR and VR systems display a unique image to each eye; therefore, stereoscopic imaging and depth perceptions is achieved. Both AR and VR are rapidly growing fields. Worldwide revenues for AR/VR were \$5 billion in 2016, but are expected to increase to \$162 billion

In 2015, the United States spent 17% of its gross domestic product on healthcare. 32% of the healthcare spending was during hospital stays [4]. Nearly half of hospital costs are for surgical care [5]. Therefore, approximately 3% of the gross domestic product includes costs related to surgery. There is a drive to improve efficiency in the operation room to both improve patient care and drive down costs. AR/VR holds promise in accomplishing these goals through improving pre-operative planning and enhancing intra-operative surgical

In the United States, the total cost of diagnostic imaging has been estimated to be \$100 billion in 2006 alone [7]. Utilization rates of diagnostic imaging are on the rise [8–10]. Currently, there is no United States Food and Drug Administration (FDA) approved AR/VR system used in diagnostic radiology [11]. AR/VR provides enhanced viewing including depth perception and improved human machine interface (HMI) [12, 13]. AR/VR HMDs provide unique images to each eye yielding depth perception. AR/VR systems leverage advanced gaming controllers and joysticks to improve HMI. Because of these features of AR/VR and others discussed later in this chapter, we believe there will be increasing applications of AR/VR in The field of diagnostic radiology includes a wide variety of imaging equipment including systems that generate inherently 2D images (e.g., chest radiograph) as well as systems that generate volumetric medical imaging datasets (e.g., computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET)). In this section, we will focus on the latter.

#### *2.1.1. Computed tomography (CT)*

In order to perform a CT scan (also known as a CAT scan), the patient is placed in the horizontal position on the CT scanner table. See **Figure 1**. The table is then translated through a donut shaped device containing both an X-ray tube and X-ray detector. Multiple projection images are acquired as the X-ray tube and detector assembly rotate around the patient.

Image reconstruction algorithms, such as filtered back projection, are performed in order to generate cross-sectional images in the axial plane (x-y plane). Since sequential, contiguous axial images can be obtained, coronal plane (x-z) and sagittal plane (y-z) plane images can be reconstructed. Data is stored in digital imaging and communication in medicine

**Figure 1.** Photograph of a computed tomography (CT) scanner. The patient is being transferred from onto the mobile CT scanner table that will translate through the donut-shaped device during the CT scan and take multiple projection images.

(DICOM) files with a typical matrix of a CT scan is 512 × 512 pixels. A pixel in the axial plane is a 2D object with a discrete length in the x-direction and discrete length in the y-direction. A voxel is a 3D object created with a pixel by adding a third dimension to create a volume.

*2.1.2. Magnetic resonance imaging (MRI)*

In order to perform an MRI scan, the patient is placed in the horizontal position on the MRI scanner table within a large cylindrical shaped device. A large magnetic field is directed through the long axis of the cylinder. Transmit coils direct a radiofrequency (RF) pulse into the patient's body and receive coils process the returning electromagnetic signal from the body to create an image. Similar to CT, contiguous planar images can be stacked and axial, sagittal and coronal reformats can be reconstructed. The imaging data for a MRI scan is similar to that of a CT scan in matrix size and the fact that each pixel has a comparable gray scale. Unlike CT scanners, MRI scanners do not employ ionizing radiation and are therefore ideal for young children or pregnant women who are more vulnerable to radiation. Furthermore, MRI scans have the ability to perform exceptional contrast resolution between tissues of similar density and can diagnose certain types of traumatic brain injury that cannot be seen on CT scans [15]. Modern MRI scans require significantly more time to perform than CT scans,

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

9

In order to perform a PET scan, a radiopharmaceutical (e.g., fludeoxyglucose F-18) is administered to the patient. Then, the patient is placed in the horizontal position on the PET scanner table within a donut shaped device. As the radiopharmaceutical decays, photons are emitted from within the patient and are received by the PET detector crystals. As with CT and MRI,

A typical matrix of a PET scan is 128 × 128, which is smaller than that of CT or MRI. However, it is similar in that each pixel has an associated numerical value associated with it indicating gray scale. In the case of PET, the gray scale corresponds to the amount of radioactivity emitted from that location. Since the radiopharmaceuticals can target certain structures in the body, it is possible for improved diagnosis of conditions like Alzheimer's disease compared with MRI [16].

In the previous section, we discussed three types of medical imaging scanners that generate volumetric data. In this section, we will review the current methods of viewing the volumetric

The conventional viewing method for reviewing volumetric datasets is a slice-by-slice viewing method for axial, sagittal and coronal imaging planes or on occasion oblique reformats. It is estimated that most radiologists spend more than 95% of their total time on cross-sectional imaging datasets using this conventional slice-by-slice approach [17]. Certain anatomical structures are easier to visualize on particular imaging planes. As an example, some radiologists have a preference for viewing the midline structures in the brain on the sagittal images,

axial images can be stacked and sagittal and coronal images can be reconstructed.

nearly one-hour of image acquisition time for a MRI scan of the brain.

*2.1.3. Positron emission tomography (PET) scanner*

**2.2. Current methods of viewing medical imaging**

*2.2.1. Conventional viewing of the volumetric data*

data.

as shown in **Figure 3**.

Each pixel has an associated gray-scale value called a Hounsfield Unit (HU), which is a function of the density and composition of the tissue. As a reference, water has a HU of 0. Soft tissues (e.g., brain, kidney, muscle, etc.) are slightly denser than water and have a HU of approximately 30–40. Compact bone can have a density of 400. Fat is slightly less dense than water and has a HU of approximately −100. Air is significantly less dense than water and has a HU of −1000.

A process called "windowing and leveling" is performed by the radiologist to set the window "level" and window "width." The window "level" refers to the HU number where mid-gray can represent. The window "width" is the range of shades of gray such that any value more extreme than the range is white (if more dense) or black (if less dense). See **Figure 2**.

Modern CT scans can be perform a total body head-to-toe scan in less than 15 seconds with spatial resolution of less than 1 mm. Thus, CT scans can be invaluable in the setting of trauma since it enables the radiologist to diagnose critical injuries anywhere in the body, such as traumatic brain injury [14]. This provides critical information to the neurosurgeon who can then perform life-saving interventions.

**Figure 2.** Axial CT scan of the head in "bone window" (A), "brain window" (B) and "scalp window" (C) at the same level within the brain. In bone window (A), the window level is set to 400 (to optimize visualization of bone which has a density of approximately 400 HU) and the width is set to 3000. In the brain window (B), the window level is set to 30 (to optimize visualization of brain which as a density of approximately 30 HU) and the width is set to 80. Any pixels that are below −10 (30 (window level) minus 40 (1/2 window width)) will be black. Air with a HU of −1000 is black because it is less than −10. Fat (yellow star) with a HU of −100 is black because it is less than −10. Any pixels that are higher than 70 (30 (window level) plus 40 (1/2 window width)) will be white. In this image, the bone is white. In the scalp window (C), the window level is set to 30 and the window width to 800. Note that fat (blue star) with a HU of −100 is now gray since it falls within the range of the window level of 30 and width of 800, which would be 30±400 or −370 to 430.

#### *2.1.2. Magnetic resonance imaging (MRI)*

(DICOM) files with a typical matrix of a CT scan is 512 × 512 pixels. A pixel in the axial plane is a 2D object with a discrete length in the x-direction and discrete length in the y-direction. A voxel is a 3D object created with a pixel by adding a third dimension to cre-

Each pixel has an associated gray-scale value called a Hounsfield Unit (HU), which is a function of the density and composition of the tissue. As a reference, water has a HU of 0. Soft tissues (e.g., brain, kidney, muscle, etc.) are slightly denser than water and have a HU of approximately 30–40. Compact bone can have a density of 400. Fat is slightly less dense than water and has a HU of approximately −100. Air is significantly less dense than water and has

A process called "windowing and leveling" is performed by the radiologist to set the window "level" and window "width." The window "level" refers to the HU number where mid-gray can represent. The window "width" is the range of shades of gray such that any value more extreme than the range is white (if more dense) or black (if less dense). See

Modern CT scans can be perform a total body head-to-toe scan in less than 15 seconds with spatial resolution of less than 1 mm. Thus, CT scans can be invaluable in the setting of trauma since it enables the radiologist to diagnose critical injuries anywhere in the body, such as traumatic brain injury [14]. This provides critical information to the neurosurgeon who can then

**Figure 2.** Axial CT scan of the head in "bone window" (A), "brain window" (B) and "scalp window" (C) at the same level within the brain. In bone window (A), the window level is set to 400 (to optimize visualization of bone which has a density of approximately 400 HU) and the width is set to 3000. In the brain window (B), the window level is set to 30 (to optimize visualization of brain which as a density of approximately 30 HU) and the width is set to 80. Any pixels that are below −10 (30 (window level) minus 40 (1/2 window width)) will be black. Air with a HU of −1000 is black because it is less than −10. Fat (yellow star) with a HU of −100 is black because it is less than −10. Any pixels that are higher than 70 (30 (window level) plus 40 (1/2 window width)) will be white. In this image, the bone is white. In the scalp window (C), the window level is set to 30 and the window width to 800. Note that fat (blue star) with a HU of −100 is now gray since

it falls within the range of the window level of 30 and width of 800, which would be 30±400 or −370 to 430.

ate a volume.

8 State of the Art Virtual Reality and Augmented Reality Knowhow

a HU of −1000.

**Figure 2**.

perform life-saving interventions.

In order to perform an MRI scan, the patient is placed in the horizontal position on the MRI scanner table within a large cylindrical shaped device. A large magnetic field is directed through the long axis of the cylinder. Transmit coils direct a radiofrequency (RF) pulse into the patient's body and receive coils process the returning electromagnetic signal from the body to create an image. Similar to CT, contiguous planar images can be stacked and axial, sagittal and coronal reformats can be reconstructed. The imaging data for a MRI scan is similar to that of a CT scan in matrix size and the fact that each pixel has a comparable gray scale.

Unlike CT scanners, MRI scanners do not employ ionizing radiation and are therefore ideal for young children or pregnant women who are more vulnerable to radiation. Furthermore, MRI scans have the ability to perform exceptional contrast resolution between tissues of similar density and can diagnose certain types of traumatic brain injury that cannot be seen on CT scans [15]. Modern MRI scans require significantly more time to perform than CT scans, nearly one-hour of image acquisition time for a MRI scan of the brain.

#### *2.1.3. Positron emission tomography (PET) scanner*

In order to perform a PET scan, a radiopharmaceutical (e.g., fludeoxyglucose F-18) is administered to the patient. Then, the patient is placed in the horizontal position on the PET scanner table within a donut shaped device. As the radiopharmaceutical decays, photons are emitted from within the patient and are received by the PET detector crystals. As with CT and MRI, axial images can be stacked and sagittal and coronal images can be reconstructed.

A typical matrix of a PET scan is 128 × 128, which is smaller than that of CT or MRI. However, it is similar in that each pixel has an associated numerical value associated with it indicating gray scale. In the case of PET, the gray scale corresponds to the amount of radioactivity emitted from that location. Since the radiopharmaceuticals can target certain structures in the body, it is possible for improved diagnosis of conditions like Alzheimer's disease compared with MRI [16].

#### **2.2. Current methods of viewing medical imaging**

In the previous section, we discussed three types of medical imaging scanners that generate volumetric data. In this section, we will review the current methods of viewing the volumetric data.

#### *2.2.1. Conventional viewing of the volumetric data*

The conventional viewing method for reviewing volumetric datasets is a slice-by-slice viewing method for axial, sagittal and coronal imaging planes or on occasion oblique reformats. It is estimated that most radiologists spend more than 95% of their total time on cross-sectional imaging datasets using this conventional slice-by-slice approach [17]. Certain anatomical structures are easier to visualize on particular imaging planes. As an example, some radiologists have a preference for viewing the midline structures in the brain on the sagittal images, as shown in **Figure 3**.

**Figure 3.** Image of the brain from an MRI scanner. This is of an axial T1-weighted sequence (A) with coronal (B) and sagittal (C) images reconstructed.

Occasionally, the radiologist needs to view an abnormality in another plane, other than the axial, sagittal, coronal imaging planes. In these instances, oblique plane reformats can be used with the images still viewed in a conventional slice-by-slice approach. See **Figure 4**. Curved planar reformats can also be performed and have been shown to be beneficial [18].

*2.2.2. Challenges in conventional viewing of the volumetric data*

deliberate slice-by-slice method takes considerable time.

rendering and volume rendering.

*2.2.3.1. Surface rendering*

and keyboard.

*2.2.3. Advanced (non-AR/VR) viewing methods of the volumetric data*

First, is the challenge of information overload. The dramatic improvements in spatial resolution (commonly smaller than 1 mm) of CT and MRI coupled with large portions of the body imaged generate immense datasets and the radiologists face the challenge of information overload. As an example, a CT scan of the chest with an axial matrix of 512 pixels (x-direction) by 512 pixels (y-direction) would have 262,144 pixels on a single slice. Thin-cut imaging of the chest provides 500 axial slices, each containing the 262,144 pixels, or roughly 131 million pixels in the data set. Second, is the challenge of detecting small lesions. One example of this is the challenge is the identification a small pulmonary nodule, which is a topic of great concern for radiologists and a top cause of litigation [19, 20]. Identifying a tumor at an small size and corresponding early stage is important in order to improve patient survival and reduce cost of treatment. A very

**Figure 5.** Image of the typical diagnostic radiology workstation displaying images of a breast MRI. Note that four flatscreen monitors are present. Methods for human machine interaction (HMI) to manipulate the images include a mouse

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

11

Third, is the challenge of mentally building a 3D image from reviewing slices [21]. Depending on the clinical scenario and body part imaged, the radiologist can be tasked with following certain twisting and turning structures through the body such as following a blood vessel or loops of intestines.

Most radiologists spend a small fraction (<5%) of their total time in interpreting their imaging scans with advanced viewing methods [17]. Such non-AR/VR techniques include surface

The first 3D rendering technique to display the human body's anatomy was surface rendering (also known as shaded surface display). Through segmentation techniques such as thresholding to display only a prescribed set of pixels, apparent surfaces are displayed within the body. A virtual light source is used to provide surface shading. In surface rendering, only

A standard viewing method includes a flat screen, high-resolution diagnostic imaging monitor with keyboard and mouse, as shown in **Figure 5**. Typically, the radiologist will use the wheel on the back of the mouse to scroll through the stacks of images. Note that the radiologist also uses a microphone to dictate the radiology reports.

**Figure 4.** CT scan of the chest in lung windows showing oblique reformats (A, B) and coronal reformat (C). Note that the image (A) is the oblique plane image corresponding to the yellow line in (C). Note that the image (B) is the oblique plane image corresponding to the green line in (C). Note that image (C) is the coronal plane image corresponding to the red line in (A) and (B).

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology http://dx.doi.org/10.5772/intechopen.74317 11

**Figure 5.** Image of the typical diagnostic radiology workstation displaying images of a breast MRI. Note that four flatscreen monitors are present. Methods for human machine interaction (HMI) to manipulate the images include a mouse and keyboard.

#### *2.2.2. Challenges in conventional viewing of the volumetric data*

First, is the challenge of information overload. The dramatic improvements in spatial resolution (commonly smaller than 1 mm) of CT and MRI coupled with large portions of the body imaged generate immense datasets and the radiologists face the challenge of information overload. As an example, a CT scan of the chest with an axial matrix of 512 pixels (x-direction) by 512 pixels (y-direction) would have 262,144 pixels on a single slice. Thin-cut imaging of the chest provides 500 axial slices, each containing the 262,144 pixels, or roughly 131 million pixels in the data set.

Second, is the challenge of detecting small lesions. One example of this is the challenge is the identification a small pulmonary nodule, which is a topic of great concern for radiologists and a top cause of litigation [19, 20]. Identifying a tumor at an small size and corresponding early stage is important in order to improve patient survival and reduce cost of treatment. A very deliberate slice-by-slice method takes considerable time.

Third, is the challenge of mentally building a 3D image from reviewing slices [21]. Depending on the clinical scenario and body part imaged, the radiologist can be tasked with following certain twisting and turning structures through the body such as following a blood vessel or loops of intestines.

#### *2.2.3. Advanced (non-AR/VR) viewing methods of the volumetric data*

Most radiologists spend a small fraction (<5%) of their total time in interpreting their imaging scans with advanced viewing methods [17]. Such non-AR/VR techniques include surface rendering and volume rendering.

#### *2.2.3.1. Surface rendering*

**Figure 4.** CT scan of the chest in lung windows showing oblique reformats (A, B) and coronal reformat (C). Note that the image (A) is the oblique plane image corresponding to the yellow line in (C). Note that the image (B) is the oblique plane image corresponding to the green line in (C). Note that image (C) is the coronal plane image corresponding to the

Occasionally, the radiologist needs to view an abnormality in another plane, other than the axial, sagittal, coronal imaging planes. In these instances, oblique plane reformats can be used with the images still viewed in a conventional slice-by-slice approach. See **Figure 4**. Curved

**Figure 3.** Image of the brain from an MRI scanner. This is of an axial T1-weighted sequence (A) with coronal (B) and

A standard viewing method includes a flat screen, high-resolution diagnostic imaging monitor with keyboard and mouse, as shown in **Figure 5**. Typically, the radiologist will use the wheel on the back of the mouse to scroll through the stacks of images. Note that the radiolo-

planar reformats can also be performed and have been shown to be beneficial [18].

gist also uses a microphone to dictate the radiology reports.

sagittal (C) images reconstructed.

10 State of the Art Virtual Reality and Augmented Reality Knowhow

red line in (A) and (B).

The first 3D rendering technique to display the human body's anatomy was surface rendering (also known as shaded surface display). Through segmentation techniques such as thresholding to display only a prescribed set of pixels, apparent surfaces are displayed within the body. A virtual light source is used to provide surface shading. In surface rendering, only

**Figure 6.** Shaded surface display (SSD) of the bones of the head and neck in three different projections (A, B, C). Note that while the density of the skull is intrinsically the same HU, the gray scale displayed changed due to the apparent light source. Note that true depth perception is not achieved because this figure is displayed on a flat computer screen or paper.

*2.2.3.2. Volume rendering*

See **Figure 8**.

AR/VR in diagnostic radiology.

The technique of volume rendering has been researched for many years by the computer graphics industry and has recently been applied to diagnostic radiology [22]. In volume rendering, a transfer function is applied to assign a color and opacity to each intensity value. As an example, voxels that correspond to the density of blood vessel are colored red and voxels that correspond to the density of bone are colored white. This has significantly helped radi-

**Figure 8.** Volume rendering image of an magnetic resonance angiogram (MRA), which is a type of magnetic resonance

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

13

One of the key limitations of volume rendering is the overlapping structures [24, 25]. This limitation is significantly worse in settings such as viewing of the vasculature of the brain.

In the first sections of this chapter, we have reviewed the medical imaging equipment, conventional slice-by-slice techniques and the advanced 3D rendering methods including surface rendering and volume rendering. We will now review AR/VR in diagnostic radiology.

The previously discussed limitation of overlapping structures can be minimized through a process called depth-3-dimensional (D3D) imaging by providing unique images to each eye and display on AR/VR HDUs. D3D transforms cross-sectional imaging datasets and displays them on AR/VR head display units (HDUs). In doing so, an overall immersive viewing experience is created. In AR/VR radiology, the radiologist will wear a head display unit, which can

This section of the paper will be organized into three sections. First, we will discuss the process for generating an AR/VR image from a cross-sectional medical imaging dataset. Second, we will include results. Third, we will discuss the path forward and future opportunities of

be either a VR, AR or MR system. The basic concept is outlined in **Figure 9**.

ologists visualize complex 3D structures [23]. See **Figure 7**.

imaging (MRI) of the brain. Note the extensive areas of overlap, which limits evaluation.

**3. Review of AR/VR in diagnostic radiology**

a single surface is used. An advantage of only displaying a single surface is the fact that surface rendering techniques are typically not limited by overlapping tissues within the human body. However, there are a few limitations.

One limitation of only displaying a single surface is the fact that thresholding is used for one tissue type at a time and it can be difficult to understand the anatomic relationship of multiple different organ systems when only a single organ system is displayed. Another limitation is the fact that many organs are of similar density to their surroundings and it can be difficult to segment these structures out. Finally, since surface rendering images have been displayed on flat screen images true depth perception is not achieved. See **Figure 6**.

**Figure 7.** Volume rendering images of the bones and blood vessels of the head and neck in three different projections (A, B, C). Note that this differs from the shaded surface display since these volume rendering images showed both the bones and the blood vessels and the SSD only displayed the bones. Note that the carotid artery can be seen in (A). Note that in image (C), the shading from the apparent light source has been removed.

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology http://dx.doi.org/10.5772/intechopen.74317 13

**Figure 8.** Volume rendering image of an magnetic resonance angiogram (MRA), which is a type of magnetic resonance imaging (MRI) of the brain. Note the extensive areas of overlap, which limits evaluation.

#### *2.2.3.2. Volume rendering*

**Figure 7.** Volume rendering images of the bones and blood vessels of the head and neck in three different projections (A, B, C). Note that this differs from the shaded surface display since these volume rendering images showed both the bones and the blood vessels and the SSD only displayed the bones. Note that the carotid artery can be seen in (A). Note that in

a single surface is used. An advantage of only displaying a single surface is the fact that surface rendering techniques are typically not limited by overlapping tissues within the human

**Figure 6.** Shaded surface display (SSD) of the bones of the head and neck in three different projections (A, B, C). Note that while the density of the skull is intrinsically the same HU, the gray scale displayed changed due to the apparent light source. Note that true depth perception is not achieved because this figure is displayed on a flat computer screen

One limitation of only displaying a single surface is the fact that thresholding is used for one tissue type at a time and it can be difficult to understand the anatomic relationship of multiple different organ systems when only a single organ system is displayed. Another limitation is the fact that many organs are of similar density to their surroundings and it can be difficult to segment these structures out. Finally, since surface rendering images have been displayed on

flat screen images true depth perception is not achieved. See **Figure 6**.

image (C), the shading from the apparent light source has been removed.

body. However, there are a few limitations.

12 State of the Art Virtual Reality and Augmented Reality Knowhow

or paper.

The technique of volume rendering has been researched for many years by the computer graphics industry and has recently been applied to diagnostic radiology [22]. In volume rendering, a transfer function is applied to assign a color and opacity to each intensity value. As an example, voxels that correspond to the density of blood vessel are colored red and voxels that correspond to the density of bone are colored white. This has significantly helped radiologists visualize complex 3D structures [23]. See **Figure 7**.

One of the key limitations of volume rendering is the overlapping structures [24, 25]. This limitation is significantly worse in settings such as viewing of the vasculature of the brain. See **Figure 8**.

### **3. Review of AR/VR in diagnostic radiology**

In the first sections of this chapter, we have reviewed the medical imaging equipment, conventional slice-by-slice techniques and the advanced 3D rendering methods including surface rendering and volume rendering. We will now review AR/VR in diagnostic radiology.

The previously discussed limitation of overlapping structures can be minimized through a process called depth-3-dimensional (D3D) imaging by providing unique images to each eye and display on AR/VR HDUs. D3D transforms cross-sectional imaging datasets and displays them on AR/VR head display units (HDUs). In doing so, an overall immersive viewing experience is created. In AR/VR radiology, the radiologist will wear a head display unit, which can be either a VR, AR or MR system. The basic concept is outlined in **Figure 9**.

This section of the paper will be organized into three sections. First, we will discuss the process for generating an AR/VR image from a cross-sectional medical imaging dataset. Second, we will include results. Third, we will discuss the path forward and future opportunities of AR/VR in diagnostic radiology.

intensity or texture surrounding a voxel, which provides a classifier algorithm with measurements that are used to distinguish one class from another, where classes correspond to the

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

15

Historical data from numerous similar imaging exams will be collected and "ground truthed" so that a neural network or deep learning classifier can be used to improve classification accuracy. Techniques will be used to first classify the input imagery into categories such as skeletal, fat, normal breast tissue, breast cancer, etc. [26, 27]. The user will need to input the tissue-type selection until an intelligent system will automatically determine the anatomy of the input imagery. User input of the tissue-type selection will be used as historical data to

The goal of the 3D filtering algorithm is to make the most understandable representation of the imagery to the radiologist [26, 27]. In order to accomplish this goal, it utilizes the image tissue-type from the 3D segmentation algorithm to determine which tissue types should be

Current graphic hardware have limitations on how many voxels that can be displayed. Therefore, it is necessary to allocate percentages to the various tissue types based on priority. Those tissue voxels that must be seen from a clinically importance perspective (e.g., tumor) are given the highest percentage. Other tissues that are not as clinically important (e.g., subcutaneous fat) are sparsely sampled. There are two benefits of this process. First of all, this allows the operator to understand the high priority tissue (e.g., tumor) in proper context (e.g., tumor is touching the spine). It can be difficult or impossible to understand exact position of the tumor when all the surrounding tissue has been removed. Second of all, this lower priority tissue needs to be somewhat more transparent so the higher priority tissue can be seen through the lower priority tissue. However, transparency allocated may not necessarily correspond to tissue density. Additionally, thinning the tissue between the viewing perspectives and the area of interest makes it much easier to see the area of interest

User selected filtering is employed to enable the operator to slice away tissue sections to facilitate viewing of areas of interest. Additional implementation of various geometric volumes and surfaces to temporarily remove sections of the voxel cloud may be beneficial. The opera-

The goal of the 3D rendering is to replicate, insofar as possible, what a radiologist would see if looking with his/her own two eyes through the skin and into the body were possible. A slightly different image would be seen from each eye from its particular vantage point. This difference allows one to see depth. The basic concept is illustrated in **Figure 10**. The geometry

tor can also enable or disable display of specific tissue types.

behind the approach is illustrated in **Figures 11** and **12**.

different tissue types.

drive the classifier.

*3.1.2. 3D filtering algorithm*

given highest priorities.

accurately.

*3.1.3. 3D rendering engine*

**Figure 9.** Overview of the system. Cross-sectional imaging data is sent via the digital recording device to the general purpose processor and then sending a unique image to the left eye and a unique image to the right eye a head display unit (HDU) [26].

#### **3.1. Image processing for AR/VR**

To optimize visualization, the D3D software suite must provide the capability for visualizing medical imagery rendered in a true 3 dimensional representation [26, 27]. In order to accomplish this, input imagery is converted from 2 dimensional images into a 3 dimensional voxel space, segmented into distinct tissue types; and then filtered. Finally, rendering is then performed wherein the rendering engine computes a left and right view. This allows the operator to visualize the data using the same stereopsis that our eyes and brains have spent a lifetime interpreting and processing.

#### *3.1.1. 3D segmentation*

During the 3D segmentation process, each image pixel in the input imagery is treated as a voxel, a three dimensional entity with length, width, and height. Each voxel is read into a large 3D array. The 3D array is analyzed for similarity to its neighbors, and for context clues that imply similarity to historical training data.

Mean and variance statistics largely drive the similarity analysis. Every voxel is compared to all the neighboring voxels in the 3x3 (or 5x5 or larger) nearest neighbor region. The most similar voxels are assigned the same class designation. The 3x3 variance is used to determine the width of the local intensity distribution. The width of the local intensity distribution in turn determines the threshold applied for declaring similarity between voxels. Simple local features on the neighborhood are computed around each voxel to describe the local average intensity or texture surrounding a voxel, which provides a classifier algorithm with measurements that are used to distinguish one class from another, where classes correspond to the different tissue types.

Historical data from numerous similar imaging exams will be collected and "ground truthed" so that a neural network or deep learning classifier can be used to improve classification accuracy. Techniques will be used to first classify the input imagery into categories such as skeletal, fat, normal breast tissue, breast cancer, etc. [26, 27]. The user will need to input the tissue-type selection until an intelligent system will automatically determine the anatomy of the input imagery. User input of the tissue-type selection will be used as historical data to drive the classifier.

#### *3.1.2. 3D filtering algorithm*

**3.1. Image processing for AR/VR**

14 State of the Art Virtual Reality and Augmented Reality Knowhow

interpreting and processing.

that imply similarity to historical training data.

*3.1.1. 3D segmentation*

unit (HDU) [26].

To optimize visualization, the D3D software suite must provide the capability for visualizing medical imagery rendered in a true 3 dimensional representation [26, 27]. In order to accomplish this, input imagery is converted from 2 dimensional images into a 3 dimensional voxel space, segmented into distinct tissue types; and then filtered. Finally, rendering is then performed wherein the rendering engine computes a left and right view. This allows the operator to visualize the data using the same stereopsis that our eyes and brains have spent a lifetime

**Figure 9.** Overview of the system. Cross-sectional imaging data is sent via the digital recording device to the general purpose processor and then sending a unique image to the left eye and a unique image to the right eye a head display

During the 3D segmentation process, each image pixel in the input imagery is treated as a voxel, a three dimensional entity with length, width, and height. Each voxel is read into a large 3D array. The 3D array is analyzed for similarity to its neighbors, and for context clues

Mean and variance statistics largely drive the similarity analysis. Every voxel is compared to all the neighboring voxels in the 3x3 (or 5x5 or larger) nearest neighbor region. The most similar voxels are assigned the same class designation. The 3x3 variance is used to determine the width of the local intensity distribution. The width of the local intensity distribution in turn determines the threshold applied for declaring similarity between voxels. Simple local features on the neighborhood are computed around each voxel to describe the local average The goal of the 3D filtering algorithm is to make the most understandable representation of the imagery to the radiologist [26, 27]. In order to accomplish this goal, it utilizes the image tissue-type from the 3D segmentation algorithm to determine which tissue types should be given highest priorities.

Current graphic hardware have limitations on how many voxels that can be displayed. Therefore, it is necessary to allocate percentages to the various tissue types based on priority. Those tissue voxels that must be seen from a clinically importance perspective (e.g., tumor) are given the highest percentage. Other tissues that are not as clinically important (e.g., subcutaneous fat) are sparsely sampled. There are two benefits of this process. First of all, this allows the operator to understand the high priority tissue (e.g., tumor) in proper context (e.g., tumor is touching the spine). It can be difficult or impossible to understand exact position of the tumor when all the surrounding tissue has been removed. Second of all, this lower priority tissue needs to be somewhat more transparent so the higher priority tissue can be seen through the lower priority tissue. However, transparency allocated may not necessarily correspond to tissue density. Additionally, thinning the tissue between the viewing perspectives and the area of interest makes it much easier to see the area of interest accurately.

User selected filtering is employed to enable the operator to slice away tissue sections to facilitate viewing of areas of interest. Additional implementation of various geometric volumes and surfaces to temporarily remove sections of the voxel cloud may be beneficial. The operator can also enable or disable display of specific tissue types.

#### *3.1.3. 3D rendering engine*

The goal of the 3D rendering is to replicate, insofar as possible, what a radiologist would see if looking with his/her own two eyes through the skin and into the body were possible. A slightly different image would be seen from each eye from its particular vantage point. This difference allows one to see depth. The basic concept is illustrated in **Figure 10**. The geometry behind the approach is illustrated in **Figures 11** and **12**.

**Figure 10.** The 3D volume shows a gray voxel and a black voxel. In volume rendering (A), the gray voxel is hidden behind the black voxel and only the black voxel is displayed on the 2D volume rendering image. In depth-3-dimensional (D3D), both the gray and the black voxels are projected onto the imaging plane; furthermore, the user can distinguish that the gray voxel is farther than the black voxel because the D3D process displayed with AR/VR HDUs provides depth perception.

The interocular distance can of course be altered to change stereopsis. The rendering engine allows the operator to comfortably move the viewing position, zoom, and rotate about the pitch, roll or yaw axes. The angular field of view can be changed. The volume of interest (VOI) can be rotated. The viewing perspectives can also be rotated. See **Figure 14**. And to improve

**Figure 13.** Figure illustrates left eye viewing perspective (LEVP) and right eye viewing perspective (REVP). The angles from the LEVP through the volume of interest can converge to a convergence point "c" to enhance visualization of a

small volume of interest [26, 27].

**Figure 12.** This is an overview illustrating the fact that the angle from the left eye viewing perspective (LEVP) to voxel (0, 64, 64) is different from the angle from the right eye viewing perspective (REVP) to voxel (0, 64, 64). This voxel (0, 64, 64) will be displayed on a different spot on the left eye display as compared with the right eye display within the HDU.

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

17

**Figure 11.** This is an overview of the depth-3-dimensional (D3D) processing system for the left eye viewing perspective (LEVP). Reprinted from the Douglas et al. [12] with permission from Dove Medical Press Ltd.

The Left Eye Viewing Perspective (LEVP) would be at an observer/ radiologist selected point to inspect the volume. The Right Eye Viewing Perspective (REVP) would be the inter-ocular distance to the right of the LEVP and similar math would be applied to generate the slightly different image presented to the right eye display. The rendering engine generates the left and right eye views to provide a true 3D data visualization to be displayed in the AR/VR HDUs [26, 27]. This component relies heavily on the Graphics Processing Units (GPU) built into the graphics card. The rendering engine supports a convergence depth adjustment for fine-tuning the operator's focal point as shown in **Figure 13**.

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology http://dx.doi.org/10.5772/intechopen.74317 17

**Figure 12.** This is an overview illustrating the fact that the angle from the left eye viewing perspective (LEVP) to voxel (0, 64, 64) is different from the angle from the right eye viewing perspective (REVP) to voxel (0, 64, 64). This voxel (0, 64, 64) will be displayed on a different spot on the left eye display as compared with the right eye display within the HDU.

The interocular distance can of course be altered to change stereopsis. The rendering engine allows the operator to comfortably move the viewing position, zoom, and rotate about the pitch, roll or yaw axes. The angular field of view can be changed. The volume of interest (VOI) can be rotated. The viewing perspectives can also be rotated. See **Figure 14**. And to improve

The Left Eye Viewing Perspective (LEVP) would be at an observer/ radiologist selected point to inspect the volume. The Right Eye Viewing Perspective (REVP) would be the inter-ocular distance to the right of the LEVP and similar math would be applied to generate the slightly different image presented to the right eye display. The rendering engine generates the left and right eye views to provide a true 3D data visualization to be displayed in the AR/VR HDUs [26, 27]. This component relies heavily on the Graphics Processing Units (GPU) built into the graphics card. The rendering engine supports a convergence depth adjustment for fine-tuning

**Figure 11.** This is an overview of the depth-3-dimensional (D3D) processing system for the left eye viewing perspective

(LEVP). Reprinted from the Douglas et al. [12] with permission from Dove Medical Press Ltd.

**Figure 10.** The 3D volume shows a gray voxel and a black voxel. In volume rendering (A), the gray voxel is hidden behind the black voxel and only the black voxel is displayed on the 2D volume rendering image. In depth-3-dimensional (D3D), both the gray and the black voxels are projected onto the imaging plane; furthermore, the user can distinguish that the gray voxel is farther than the black voxel because the D3D process displayed with AR/VR HDUs provides depth

the operator's focal point as shown in **Figure 13**.

perception.

16 State of the Art Virtual Reality and Augmented Reality Knowhow

**Figure 13.** Figure illustrates left eye viewing perspective (LEVP) and right eye viewing perspective (REVP). The angles from the LEVP through the volume of interest can converge to a convergence point "c" to enhance visualization of a small volume of interest [26, 27].

**Figure 14.** Five examples of augmented reality/virtual reality viewing options with the D3D technology. Notes: (A) initial viewing angle into the volume of interest. (B) Illustrates increasing the interocular distance, which provides for increased binocular disparity. (C) Illustrates changing of the angular FOV, so rather than an α FOV of 40°, it changes to an α FOV of 10° and rather than a β FOV of 40°, it changes to a β FOV of 10°. This serves to focus on a particular region within the volume. (D) Illustrates rotation of the VOI, so that the radiologist can have a different viewing perspective. (E) Illustrates rotation of the viewing perspective, which is similar to the radiologist turning the head to see new features of the image, allowing for improved HMI. Note that in (A–E), the center pixels (i.e., α = 0° and β = 0°) for both LEVP and REVP converge at the center of the VOI (i.e., voxel [32, 32, 32]), such that the VOI is optimally presented to the user. Abbreviations: FOV, field of view; HMI, human machine interface; LEVP, left eye viewing perspective; REVP, right eye viewing perspective; VOI, volume of interest. Reprinted from the Douglas et al. [12] with permission from Dove Medical Press Ltd.

visibility, the operator can select any number of color palettes for different tissue classes. The rendering engine also incorporates a graphical overlay for image markup.

#### **3.2. AR/VR control systems used in diagnostic radiology**

The field of AR/VR is evolving with new and changing HDUs and gaming control systems coming on the market. In order to illustrate what we have used in our prior research, we illustrate example control systems shown and how various control buttons correspond with basic maneuvers and advanced maneuvers. See **Figures 15**–**17**.

**3.3. Initial results of AR/VR in radiology**

movements, which are defined as a single controls input at one time.

movements, which are defined as a single controls input at one time.

As previously discussed, there have been numerous applications of AR/VR in surgery to include pre-operative planning, education, and intra-operative assistance [6]. However, AR/ VR is not yet FDA approved in diagnostic radiology, but is being actively researched by DXC

**Figure 17.** This figure illustrates a graphical user interface (GUI) used by D3D technologies programmed with complex

**Figure 16.** This figure illustrates a graphical user interface (GUI) used by D3D technologies programmed with basic

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

19

Breast Cancer is one of the leading causes of death in women [28, 29]. Breast calcifications are extremely common and are present in up to 86% of mammograms [30]. The calcifications are classified by distribution with a linear and branching pattern suspicious for ductal carcinoma in situ (DCIS) [31, 32]. Standard mammographic views may not reveal the true linear and

Technologies/D3D Technologies with an initial focus in breast cancer imaging [11–13].

**Figure 15.** Control system used by depth-3-dimensional (D3D) technologies. See example buttons and functions.

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology http://dx.doi.org/10.5772/intechopen.74317 19

**Figure 16.** This figure illustrates a graphical user interface (GUI) used by D3D technologies programmed with basic movements, which are defined as a single controls input at one time.

**Figure 17.** This figure illustrates a graphical user interface (GUI) used by D3D technologies programmed with complex movements, which are defined as a single controls input at one time.

#### **3.3. Initial results of AR/VR in radiology**

**Figure 15.** Control system used by depth-3-dimensional (D3D) technologies. See example buttons and functions.

visibility, the operator can select any number of color palettes for different tissue classes. The

VOI, volume of interest. Reprinted from the Douglas et al. [12] with permission from Dove Medical Press Ltd.

**Figure 14.** Five examples of augmented reality/virtual reality viewing options with the D3D technology. Notes: (A) initial viewing angle into the volume of interest. (B) Illustrates increasing the interocular distance, which provides for increased binocular disparity. (C) Illustrates changing of the angular FOV, so rather than an α FOV of 40°, it changes to an α FOV of 10° and rather than a β FOV of 40°, it changes to a β FOV of 10°. This serves to focus on a particular region within the volume. (D) Illustrates rotation of the VOI, so that the radiologist can have a different viewing perspective. (E) Illustrates rotation of the viewing perspective, which is similar to the radiologist turning the head to see new features of the image, allowing for improved HMI. Note that in (A–E), the center pixels (i.e., α = 0° and β = 0°) for both LEVP and REVP converge at the center of the VOI (i.e., voxel [32, 32, 32]), such that the VOI is optimally presented to the user. Abbreviations: FOV, field of view; HMI, human machine interface; LEVP, left eye viewing perspective; REVP, right eye viewing perspective;

The field of AR/VR is evolving with new and changing HDUs and gaming control systems coming on the market. In order to illustrate what we have used in our prior research, we illustrate example control systems shown and how various control buttons correspond with basic

rendering engine also incorporates a graphical overlay for image markup.

**3.2. AR/VR control systems used in diagnostic radiology**

18 State of the Art Virtual Reality and Augmented Reality Knowhow

maneuvers and advanced maneuvers. See **Figures 15**–**17**.

As previously discussed, there have been numerous applications of AR/VR in surgery to include pre-operative planning, education, and intra-operative assistance [6]. However, AR/ VR is not yet FDA approved in diagnostic radiology, but is being actively researched by DXC Technologies/D3D Technologies with an initial focus in breast cancer imaging [11–13].

Breast Cancer is one of the leading causes of death in women [28, 29]. Breast calcifications are extremely common and are present in up to 86% of mammograms [30]. The calcifications are classified by distribution with a linear and branching pattern suspicious for ductal carcinoma in situ (DCIS) [31, 32]. Standard mammographic views may not reveal the true linear and branching pattern due to suboptimal view point and lack of depth perception. Therefore, the D3D AR imaging system was tested on a simulated set of microcalcifications. The radiologist who rated the AR system found that when the microcalcifications appeared as a cluster when viewed from a single perspective, but with rotation and the AR HDU appeared as a linear and branching pattern. See **Figure 18**.

In addition to microcalcifications, breast cancer can also present as a mass on imaging. Characterization of the shape and margins is important in determining whether the breast mass is malignant or benign. Dedicated breast CT provides high spatial resolution of breast masses [33]. Recent data has shown the importance of characterizing tumor morphology [34]. Therefore, the D3D AR system was also tested in viewing of a known breast cancer [13]. Malignant features of spiculations were noted to be more conspicuous on the D3D system than the native CT. See **Figure 19**.

**3.4. Future work of AR/VR in radiology**

haptic feedback.

**4. Conclusions**

Both the fields of AR/VR and diagnostic radiology are large and rapidly growing. One of the most common reasons for performing diagnostic imaging is for cancer. Early determination of whether a particular therapy regimen is working would be extremely helpful to improve survival and would save costs. We outline a flow chart below as a recommended process for

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

21

**Figure 20.** Overview of how D3D AR/VR system could be used to evaluate how a tumor at multiple time points.

It is foreseeable that AR/VR will one day play a major role in diagnostic radiology. Computer aided diagnosis (CAD) will help to identify the abnormalities. The role of the radiologist will include assessing an abnormality in great detail to appreciate subtle changes to diagnose accurately and assess treatment response. Features of AR/VR including depth perception, head tracking, improved GUIs create an overall immersive environment allowing for new opportunities in diagnostic radiology. Radiologists will interact with medical images in ways never before including voice commands, gestures or through handheld devices with

Both diagnostic radiology and AR/VR are large and rapidly growing fields. In this chapter, we reviewed diagnostic medical imaging equipment, data storage, conventional slice-by-slice analysis and advanced 3D rendering techniques including surface rendering and volume rendering. We then introduced D3D processing of images, so volumetric medical imaging can be displayed in AR/VR HDUs. We showed a variety of GUIs including controllers and joysticks

introducing AR/VR evaluating a cancer at multiple time points. See **Figure 20**.

**Figure 18.** Simulated breast microcalcifications as white dots on a black background viewed using the D3D system. Notes: The microcalcifications are shown in (A), (B), and (C), but in different projections. (A) Initial single projection viewed by the radiologist (EW) was thought to represent a cluster as indicated by the circle. Note that some of the dots are bigger, which is due to the fact that they are closer to the viewing perspective and the calcifications subtend a larger angular resolution from the viewing perspective. (B) The cluster is now rotated and viewed with both an LEVP and an REVP. The top portion (arrows) was thought to represent a linear pattern by the radiologist (EW) with the linear portion indicated by the arrows. (C) Same cluster as (A) and (B), but now rotated once again. The bottom portion (arrows) was thought to represent a branching pattern by the radiologist (EW), where two branches originate at the bottom of the image as indicated by the arrows. In retrospect, the viewing perspective (A) was closest to the top of the microcalcifications as if one were looking down on linear, branching pattern seen from the side view on (B) and (C). Abbreviations: D3D, depth 3-dimensional; LEVP, left eye viewing perspective; REVP, right eye viewing perspective. Reprinted from the, Douglas et al. [12] with permission.

**Figure 19.** (A) Contrast-enhanced breast CT demonstrates the mass with small spiculations extending from the margins. (B and C) same mass from breast CT exam as seen in (A), but viewed with D3D where (B) represents the left eye viewing perspective (LEVP) and (C) represents the right eye viewing perspective (REVP). The red box illustrates the 3D cursor used. (D&E) represent the same mass from the breast CT, but zoomed in and viewed from a different perspective with (D) representing the LEVP and (E) representing the REVP. Red arrows show spiculations extending from the margins of the mass. The red circle represents a spiculation sticking out toward the user, which was well seen when rotating with the D3D system. Reprinted from the Douglas et al., [13] with permission.

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology http://dx.doi.org/10.5772/intechopen.74317 21

**Figure 20.** Overview of how D3D AR/VR system could be used to evaluate how a tumor at multiple time points.

#### **3.4. Future work of AR/VR in radiology**

branching pattern due to suboptimal view point and lack of depth perception. Therefore, the D3D AR imaging system was tested on a simulated set of microcalcifications. The radiologist who rated the AR system found that when the microcalcifications appeared as a cluster when viewed from a single perspective, but with rotation and the AR HDU appeared as a linear and

In addition to microcalcifications, breast cancer can also present as a mass on imaging. Characterization of the shape and margins is important in determining whether the breast mass is malignant or benign. Dedicated breast CT provides high spatial resolution of breast masses [33]. Recent data has shown the importance of characterizing tumor morphology [34]. Therefore, the D3D AR system was also tested in viewing of a known breast cancer [13]. Malignant features of spiculations were noted

**Figure 18.** Simulated breast microcalcifications as white dots on a black background viewed using the D3D system. Notes: The microcalcifications are shown in (A), (B), and (C), but in different projections. (A) Initial single projection viewed by the radiologist (EW) was thought to represent a cluster as indicated by the circle. Note that some of the dots are bigger, which is due to the fact that they are closer to the viewing perspective and the calcifications subtend a larger angular resolution from the viewing perspective. (B) The cluster is now rotated and viewed with both an LEVP and an REVP. The top portion (arrows) was thought to represent a linear pattern by the radiologist (EW) with the linear portion indicated by the arrows. (C) Same cluster as (A) and (B), but now rotated once again. The bottom portion (arrows) was thought to represent a branching pattern by the radiologist (EW), where two branches originate at the bottom of the image as indicated by the arrows. In retrospect, the viewing perspective (A) was closest to the top of the microcalcifications as if one were looking down on linear, branching pattern seen from the side view on (B) and (C). Abbreviations: D3D, depth 3-dimensional; LEVP, left eye viewing perspective; REVP, right eye viewing perspective.

**Figure 19.** (A) Contrast-enhanced breast CT demonstrates the mass with small spiculations extending from the margins. (B and C) same mass from breast CT exam as seen in (A), but viewed with D3D where (B) represents the left eye viewing perspective (LEVP) and (C) represents the right eye viewing perspective (REVP). The red box illustrates the 3D cursor used. (D&E) represent the same mass from the breast CT, but zoomed in and viewed from a different perspective with (D) representing the LEVP and (E) representing the REVP. Red arrows show spiculations extending from the margins of the mass. The red circle represents a spiculation sticking out toward the user, which was well seen when rotating with

to be more conspicuous on the D3D system than the native CT. See **Figure 19**.

branching pattern. See **Figure 18**.

20 State of the Art Virtual Reality and Augmented Reality Knowhow

Reprinted from the, Douglas et al. [12] with permission.

the D3D system. Reprinted from the Douglas et al., [13] with permission.

Both the fields of AR/VR and diagnostic radiology are large and rapidly growing. One of the most common reasons for performing diagnostic imaging is for cancer. Early determination of whether a particular therapy regimen is working would be extremely helpful to improve survival and would save costs. We outline a flow chart below as a recommended process for introducing AR/VR evaluating a cancer at multiple time points. See **Figure 20**.

It is foreseeable that AR/VR will one day play a major role in diagnostic radiology. Computer aided diagnosis (CAD) will help to identify the abnormalities. The role of the radiologist will include assessing an abnormality in great detail to appreciate subtle changes to diagnose accurately and assess treatment response. Features of AR/VR including depth perception, head tracking, improved GUIs create an overall immersive environment allowing for new opportunities in diagnostic radiology. Radiologists will interact with medical images in ways never before including voice commands, gestures or through handheld devices with haptic feedback.

#### **4. Conclusions**

Both diagnostic radiology and AR/VR are large and rapidly growing fields. In this chapter, we reviewed diagnostic medical imaging equipment, data storage, conventional slice-by-slice analysis and advanced 3D rendering techniques including surface rendering and volume rendering. We then introduced D3D processing of images, so volumetric medical imaging can be displayed in AR/VR HDUs. We showed a variety of GUIs including controllers and joysticks with a variety of functions achieved by various functions. Imaging cases were illustrated with a specific focus on breast cancer. We concluded with a discussion on future techniques in comparing how a tumor changes appearance over multiple time points.

[8] Mettler FA Jr, Wiest PW, Locken JA, Kelsey CA. CT scanning: Patterns of use and dose.

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

23

[9] Mitchell DG, Parker L, Sunshine JH, Levin DC. Body MR imaging and CT volume: Variations and trends based on an analysis of medicare and fee-for-service health insur-

[10] Boone JM, Brunberg JA. Computed tomography use in a tertiary care university hospi-

[11] Douglas DB, Wilke CA, Gibson JD, Boone JM, Wintermark M. Augmented reality: Advances in diagnostic imaging. Multimodal Technologies and Interaction. 2017;**1**(4):29

[12] Douglas DB, Petricoin EF, Liotta L, Wilson E. D3D augmented reality imaging system: Proof of concept in mammography. Medical Devices (Auckland). 2016;**9**:277-283 [13] Douglas DB, Boone JM, Petricoin E, Liotta L, Wilson E. Augmented reality imaging system: 3D viewing of a breast cancer. Journal of Natural Sciences. 2016;**2**(9):215

[14] Douglas DB, Chaudhari R, Zhao JM, Gullo J, Kirkland J, Douglas PK, et al. Perfusion imaging in acute traumatic brain injury. Neuroimaging Clinics of North America;

[15] Douglas DB, Iv M, Douglas PK, Anderson A, Vos SB, Bammer R, et al. Diffusion tensor imaging of TBI: Potentials and challenges. Topics in Magnetic Resonance Imaging:

[16] Douglas D, Goubran M, Wilson E, Xu G, Tripathi P, Holley D, et al. Correlation between arterial spin labeling MRI and dynamic FDG on PET-MR in Alzheimer's disease and

[17] David Douglas MDDbcnnmppiC, USA. Dual Board Certified Neuroradiologist Nuclear

[18] Ochi T, Shimizu K, Yasuhara Y, Shigesawa T, Mochizuki T, Ikezoe J. Curved planar reformatted CT angiography: Usefulness for the evaluation of aneurysms at the carotid

[19] White CS, Salis AI, Meyer CA. Missed lung cancer on chest radiography and computed tomography: Imaging and medicolegal issues. Journal of Thoracic Imaging.

[20] Gurney JW. Missed lung cancer at CT: Imaging findings in nine patients. Radiology.

[21] Ferroli P, Tringali G, Acerbi F, Schiariti M, Broggi M, Aquino D, et al. Advanced 3-dimensional planning in neurosurgery. Neurosurgery. 2013;**72**(suppl\_1):A54-A62

[22] Hohne K-H, Bomans M, Tiede U, Riemer M, editors. Display of Multiple 3D–Objects Using the Generalized Voxel-Model. Medical Imaging II. Newport Beach: SPIE; 1988 [23] Calhoun PS, Kuszyk BS, Heath DG, Carley JC, Fishman EK. Three-dimensional volume rendering of spiral CT data: Theory and method. Radiographics: A Review Publication

non-Alzhiemer's disease patients. EJNMMI Physics. 2015;**2**(Suppl 1):83

siphon. American Journal of Neuroradiology. 1999;**20**(6):1025-1030

of the Radiological Society of North America, Inc. 1999;**19**(3):745-764

Medicine Physician Practicing in California USA; 2017

ance databases. American Journal of Roentgenology. 2002;**179**(1):27-31

tal. Journal of the American College of Radiology. 2008;**5**(2):132-138

Journal of Radiological Protection. 2000;**20**(4):353-359

2017;**28**:55-65

1999;**14**(1):63-68

1996;**199**(1):117-122

TMRI. 2015;**24**(5):241-251

#### **Conflict of interest**

All authors either have direct financial interest or are employees of D3D Technologies or DXC Technologies.

#### **Author details**

David B. Douglas1 \*, Demetri Venets2 , Cliff Wilke<sup>3</sup> , David Gibson3 , Lance Liotta<sup>2</sup> , Emanuel Petricoin<sup>2</sup> , Buddy Beck<sup>4</sup> and Robert Douglas4


#### **References**


[8] Mettler FA Jr, Wiest PW, Locken JA, Kelsey CA. CT scanning: Patterns of use and dose. Journal of Radiological Protection. 2000;**20**(4):353-359

with a variety of functions achieved by various functions. Imaging cases were illustrated with a specific focus on breast cancer. We concluded with a discussion on future techniques in

All authors either have direct financial interest or are employees of D3D Technologies or DXC

, David Gibson3

, Lance Liotta<sup>2</sup>

,

, Cliff Wilke<sup>3</sup>

and Robert Douglas4

[1] Lovo EE, Quintana JC, Puebla MC, Torrealba G, Santos JL, Lira IH, et al. A novel, inexpensive method of image coregistration for applications in image-guided surgery using

[2] Baus O, Bouchard S. Moving from virtual reality exposure-based therapy to augmented reality exposure-based therapy: A review. Frontiers in Human Neuroscience. 2014;**8**:2 [3] Merel T. The 7 Drivers of the \$150 billion AR/VR Industry. Aol Tech; 2015. https://tech-

[4] Statistics NCfH. Health, United States, 2016: With chartbook on long-term trends in

[5] Insider B. Available from: http://www.businessinsider.com/20-most-expensive-surgeries-in-

[6] Douglas DB, Wilke CA, Gibson D, Petricoin EF, Liotta L. Virtual reality and augmented

[7] Mitchell JM, LaGalia RR. Controlling the escalating use of advanced imaging: The role of radiology benefit management programs. Medical Care Research and Review.

augmented reality. Neurosurgery. 2007;**60**(4 Suppl 2):366-371; discussion 71-2

crunch.com/2015/07/08/the-7-drivers-of-the-150-billion-arvr-industry/

reality: Advances in surgery. O Biologico. 2017;**2**(5):1-8

comparing how a tumor changes appearance over multiple time points.

\*, Demetri Venets2

\*Address all correspondence to: ddouglas@stanford.edu

, Buddy Beck<sup>4</sup>

22 State of the Art Virtual Reality and Augmented Reality Knowhow

3 Hewlett-Packard/DXC Technologies, VA, USA

**Conflict of interest**

Technologies.

**Author details**

David B. Douglas1

Emanuel Petricoin<sup>2</sup>

**References**

health. 2017

the-us-2014-2

2009;**66**(3):339-351

1 Stanford University, CA, USA

4 D3D Technologies, VA, USA

2 George Mason University, VA, USA


[24] Fishman EK, Ney DR, Heath DG, Corl FM, Horton KM, Johnson PT. Volume rendering versus maximum intensity projection in CT angiography: What works best, when, and why. Radiographics: A Review Publication of the Radiological Society of North America, Inc. 2006;**26**(3):905-922

**Chapter 3**

**Provisional chapter**

**Connect Smart Cities and Heritage Through**

**Connect Smart Cities and Heritage Through** 

DOI: 10.5772/intechopen.74363

This chapter aims to connect the digital resources of knowledge with the historical and cultural heritage in the context of smart cities. Specifically, combining the joint intervention in the Real Sitio of El Pardo and Aranjuez, as well as the Foundation Square of the University of Alcalá, both in Spain. Through traditional historical research and the innovative use of new technologies like augmented reality (AR), a historical, biological and cultural heritage is conserved, consisting of forests, gardens, agricultural spaces, urban centers and palatial residences. Cultural and artistic heritage is a resource of the first magnitude for the sustainable development of smart cities. It evolves with time and society; it is this that determines what goods are to be conserved and protected for posterity, according to the values attributed to them. Hence, the importance of achieving an awareness in society plays an active part in the conservation, enjoyment and dissemination of heritage. In this context, the augmented reality is presented as a powerful tool for contextualizing and disseminating the heritage, as well as to make the resources created more accessible, making an innovative use of the new technologies applied to the transfer of knowledge

and the enhancement of a country's cultural and historical heritage.

by forests, gardens, agricultural spaces, urban centers and palatial residences.

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution,

© 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

and reproduction in any medium, provided the original work is properly cited.

This project, through the use of new technologies in the dissemination and transfer of knowledge to society, seeks to preserve and improve knowledge of these royal sites and the city of Alcala de Henares, of enormous environmental, cultural and educational value, constituted

**Keywords:** augmented reality, smart cities, heritage, royal sites, historical architecture

Félix Labrador Arroyo, Julián de la Fuente Prieto and

Félix Labrador Arroyo, Julián de la Fuente Prieto

Additional information is available at the end of the chapter

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.74363

**Augmented Reality**

**Augmented Reality**

Enrique Castaño Perea

**Abstract**

**1. Introduction**

and Enrique Castaño Perea


#### **Connect Smart Cities and Heritage Through Augmented Reality Connect Smart Cities and Heritage Through Augmented Reality**

DOI: 10.5772/intechopen.74363

Félix Labrador Arroyo, Julián de la Fuente Prieto and Enrique Castaño Perea Félix Labrador Arroyo, Julián de la Fuente Prieto and Enrique Castaño Perea

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.74363

#### **Abstract**

[24] Fishman EK, Ney DR, Heath DG, Corl FM, Horton KM, Johnson PT. Volume rendering versus maximum intensity projection in CT angiography: What works best, when, and why. Radiographics: A Review Publication of the Radiological Society of North

[25] Johnson PT, Heath DG, Kuszyk BS, Fishman EK. CT angiography with volume rendering: Advantages and applications in splanchnic vascular imaging. Radiology.

[26] Douglas D, inventor; US Patent Office, assignee. US 8,384,771 Method and Apparatus for

[27] Douglas D, inventor; US Patent Office, assignee. US 9,349,183 Method and Apparatus for

[28] DeSantis C, Ma J, Bryan L, Jemal A. Breast cancer statistics, 2013. CA: a Cancer Journal

[29] Ferlay J, Shin HR, Bray F, Forman D, Mathers C, Parkin DM. Estimates of worldwide burden of cancer in 2008: GLOBOCAN 2008. International Journal of Cancer Journal

[30] Kopans D. Analyzing the Mammogram-Calcifications. Philadelphia, PA: Lippincott-

[31] Lee KS, Han BH, Chun YK, Kim HS, Kim EE. Correlation between mammographic manifestations and averaged histopathologic nuclear grade using prognosis-predict scoring system for the prognosis of ductal carcinoma in situ. Clinical Imaging. 1999;**23**(6):339-346

[32] Burnside ES, Ochsner JE, Fowler KJ, Fine JP, Salkowski LR, Rubin DL, et al. Use of microcalcification descriptors in BI-RADS 4th edition to stratify risk of malignancy.

[33] Boone JM, Lindfors KK. Breast CT: Potential for breast cancer screening and diagnosis.

[34] Mukhtar RA, Yau C, Rosen M, Tandon VJ, I-Spy T, Investigators A, et al. Clinically meaningful tumor reduction rates vary by prechemotherapy MRI phenotype and tumor subtype in the I-SPY 1 TRIAL (CALGB 150007/150012; ACRIN 6657). Annals of Surgical

America, Inc. 2006;**26**(3):905-922

24 State of the Art Virtual Reality and Augmented Reality Knowhow

for Clinicians. 2014;**64**(1):52-62

Radiology. 2007;**242**(2):388-395

Future Oncology. 2006;**2**(3):351-356

Oncology. 2013;**20**(12):3823-3830

Three Dimensional Viewing of Images. USA; 2013

Three Dimensional Viewing of Images. USA; 2016

International du Cancer. 2010;**127**(12):2893-2917

1996;**200**(2):564-568

Raven; 1998

This chapter aims to connect the digital resources of knowledge with the historical and cultural heritage in the context of smart cities. Specifically, combining the joint intervention in the Real Sitio of El Pardo and Aranjuez, as well as the Foundation Square of the University of Alcalá, both in Spain. Through traditional historical research and the innovative use of new technologies like augmented reality (AR), a historical, biological and cultural heritage is conserved, consisting of forests, gardens, agricultural spaces, urban centers and palatial residences. Cultural and artistic heritage is a resource of the first magnitude for the sustainable development of smart cities. It evolves with time and society; it is this that determines what goods are to be conserved and protected for posterity, according to the values attributed to them. Hence, the importance of achieving an awareness in society plays an active part in the conservation, enjoyment and dissemination of heritage. In this context, the augmented reality is presented as a powerful tool for contextualizing and disseminating the heritage, as well as to make the resources created more accessible, making an innovative use of the new technologies applied to the transfer of knowledge and the enhancement of a country's cultural and historical heritage.

**Keywords:** augmented reality, smart cities, heritage, royal sites, historical architecture

#### **1. Introduction**

This project, through the use of new technologies in the dissemination and transfer of knowledge to society, seeks to preserve and improve knowledge of these royal sites and the city of Alcala de Henares, of enormous environmental, cultural and educational value, constituted by forests, gardens, agricultural spaces, urban centers and palatial residences.

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Thus Aranjuez or El Pardo is today the lung of Madrid, which, being a hunting ground of a royal site, escaped the relentless destruction of the natural environment that occurred as a result of the growth of the capital. It is a testimony to the landscape of yesteryear, a place where the original fauna and flora are preserved, where there is also an urban center, El Pardo, integrated into the natural space and a royal residence, El Pardo Palace, where an important architectural and artistic ensemble is preserved. Therefore, the study of the Royal Sites, in the specific case of this project of El Pardo, cannot do without considering the urban or natural environment in which these are inserted or to which they are opposed and this has to be reflected in the investigations and in the knowledge that society has of them.

Traditionally, those who have paid attention to these aspects, with great quantitative and qualitative richness, have been the art historians or architects, and the utility of them from tourism; what does not correspond, until the last years, is an equal interest on the part of other social sciences. The plurality of ways, elements and lexicons through which these enclaves, in all their components, or the spaces of power, economic development and landscape that develop in them are transformed into a multiform, polysemic sign system and modulated by different levels of perception.

Indeed, one of the great challenges we face is the impact of new technologies and their rapid evolution. According to data from the European Commission, "the global demand for ICTs is worth 2 trillion euros, but only a quarter of this amount is covered by European companies"<sup>1</sup> , which is why it is a priority to stimulate an innovative digital society that contributes economic, social and educational benefits. No one can deny the multitude of applications they have for development in the health, architectural, social or educational field, including, for some years, the cultural. But besides being set as a factor of development, these resources have emerged as a particularly suitable means for the interpretation of cultural heritage and to transfer this knowledge to citizens and get them to identify with their past (**Figure 1**).

The architectural constructions, the urban interventions diffused or circumscribed in the territory, the more or less organic projects and the ideal city (more or less translated into practice) are an integral part of the exercise of power; they are codices constituting the authority and not ephemeral representations of it. The act of building and political action is always intimately united.

Indicators such as the ecological footprint, water footprint or ecological debt show that climate change, loss of biodiversity, lack of social justice, the persistence of disadvantaged social sectors, deforestation, the increase of inequalities in the distribution of wealth and so on are currently the environmental problems, that is, sociological and ecological, more importantly. We are not faced with a set of diverse problems; on the contrary, we are facing a systemic problem that obliges not only to try to alleviate its negative consequences but also to analyze

Connect Smart Cities and Heritage Through Augmented Reality

http://dx.doi.org/10.5772/intechopen.74363

27

Education alone is not enough to achieve a more sustainable future. However, without education and learning of sustainable development, we will not be able to achieve this goal. Education for sustainable development allows each human being to acquire the knowledge, skills, attitudes and values necessary to forge a sustainable future. Likewise, this sustainable education requires participatory methods of teaching and learning that motivate students and give them autonomy in order to change their behavior and facilitate the adoption of measures

and act directly on its causes.

**Figure 1.** Past heritage in the Royal Site of Aranjuez using A.R.

**Figure 2.** Present heritage in the Royal Site of Aranjuez.

in favor of sustainable development.

The study, enhancement and recovery of these places, shown in **Figures 2** and **3**, allow the recovery of a common past, the economic improvement of these places and their surroundings, their recovery and so on; also, they are an unbeatable space to advance the principles of an education for sustainable development, through which people acquire knowledge and skills in the field of sustainable development, which will increase their capacities and their self-confidence and increase their opportunities to opt for a healthy and productive life in harmony with nature, respectful of social values, equality between the sexes and cultural diversity, which will result in intercultural and diverse societies.

<sup>1</sup> European Commission, "Europe 2020. A strategy for smart, sustainable and inclusive growth". Brussels: Publications Office of the European Union (2010). [Accessed on March 15, 2016]. Available at http://eur-lex.europa.eu/LexUriServ/ LexUriServ.do?uri=COM:2010:2020:FIN:ES:PDF, p. 12 and 15.

**Figure 1.** Past heritage in the Royal Site of Aranjuez using A.R.

Thus Aranjuez or El Pardo is today the lung of Madrid, which, being a hunting ground of a royal site, escaped the relentless destruction of the natural environment that occurred as a result of the growth of the capital. It is a testimony to the landscape of yesteryear, a place where the original fauna and flora are preserved, where there is also an urban center, El Pardo, integrated into the natural space and a royal residence, El Pardo Palace, where an important architectural and artistic ensemble is preserved. Therefore, the study of the Royal Sites, in the specific case of this project of El Pardo, cannot do without considering the urban or natural environment in which these are inserted or to which they are opposed and this has

Traditionally, those who have paid attention to these aspects, with great quantitative and qualitative richness, have been the art historians or architects, and the utility of them from tourism; what does not correspond, until the last years, is an equal interest on the part of other social sciences. The plurality of ways, elements and lexicons through which these enclaves, in all their components, or the spaces of power, economic development and landscape that develop in them are transformed into a multiform, polysemic sign system and modulated by

Indeed, one of the great challenges we face is the impact of new technologies and their rapid evolution. According to data from the European Commission, "the global demand for ICTs is worth 2 trillion euros, but only a quarter of this amount is covered by European companies"<sup>1</sup>

which is why it is a priority to stimulate an innovative digital society that contributes economic, social and educational benefits. No one can deny the multitude of applications they have for development in the health, architectural, social or educational field, including, for some years, the cultural. But besides being set as a factor of development, these resources have emerged as a particularly suitable means for the interpretation of cultural heritage and to transfer this knowledge to citizens and get them to identify with their past (**Figure 1**).

The architectural constructions, the urban interventions diffused or circumscribed in the territory, the more or less organic projects and the ideal city (more or less translated into practice) are an integral part of the exercise of power; they are codices constituting the authority and not ephemeral representations of it. The act of building and political action is always inti-

The study, enhancement and recovery of these places, shown in **Figures 2** and **3**, allow the recovery of a common past, the economic improvement of these places and their surroundings, their recovery and so on; also, they are an unbeatable space to advance the principles of an education for sustainable development, through which people acquire knowledge and skills in the field of sustainable development, which will increase their capacities and their self-confidence and increase their opportunities to opt for a healthy and productive life in harmony with nature, respectful of social values, equality between the sexes and cultural

European Commission, "Europe 2020. A strategy for smart, sustainable and inclusive growth". Brussels: Publications Office of the European Union (2010). [Accessed on March 15, 2016]. Available at http://eur-lex.europa.eu/LexUriServ/

diversity, which will result in intercultural and diverse societies.

LexUriServ.do?uri=COM:2010:2020:FIN:ES:PDF, p. 12 and 15.

,

to be reflected in the investigations and in the knowledge that society has of them.

different levels of perception.

26 State of the Art Virtual Reality and Augmented Reality Knowhow

mately united.

1

**Figure 2.** Present heritage in the Royal Site of Aranjuez.

Indicators such as the ecological footprint, water footprint or ecological debt show that climate change, loss of biodiversity, lack of social justice, the persistence of disadvantaged social sectors, deforestation, the increase of inequalities in the distribution of wealth and so on are currently the environmental problems, that is, sociological and ecological, more importantly. We are not faced with a set of diverse problems; on the contrary, we are facing a systemic problem that obliges not only to try to alleviate its negative consequences but also to analyze and act directly on its causes.

Education alone is not enough to achieve a more sustainable future. However, without education and learning of sustainable development, we will not be able to achieve this goal. Education for sustainable development allows each human being to acquire the knowledge, skills, attitudes and values necessary to forge a sustainable future. Likewise, this sustainable education requires participatory methods of teaching and learning that motivate students and give them autonomy in order to change their behavior and facilitate the adoption of measures in favor of sustainable development.

**d.** Development of accessible didactic and educational materials with new technologies for

Connect Smart Cities and Heritage Through Augmented Reality

http://dx.doi.org/10.5772/intechopen.74363

29

The theoretical framework justifies the innovative nature of the proposal. Starting from the concept of "Smart City," the technology of augmented reality is considered as the ideal instru-

The concept of "Smart Cities" [1–3] is associated with the principle of sustainable urban development that includes different areas such as infrastructures, technologies or democracy. The efficiency of smart cities has an important component of social development, and there is no better way to connect a city with its population than through the heritage that unites them. The new means of communication have generated a new field of dissemination and protection of cultural heritage [4–6] Since the declaration of UNESCO in 1972 on the preservation of material heritage and in the year 2003 on the intangible heritage, a series of processes, tools and especially networks have been generated that increase the accessibility and participation

use in Moocs or in the new realities of teaching and learning processes.

**2. Theoretical framework**

ment to develop a virtual heritage proposal.

of citizens in the management of cultural assets (**Figure 4**).

**2.1. The "smart cities" and their heritage**

**Figure 4.** Identifying interested places in a smart city.

**Figure 3.** Virtual heritage in the University of Alcalá.

Finally, indicate that in the elaboration of this project, we have taken into account the recommendations and suggestions emanating from the document Getting cultural heritage to work for Europe. Report of the Horizon 2020 Expert Group in Cultural Heritage (2015) of the Conclusions of the Council of Europe (Education, Youth, Culture and Sports) adopted on May 20, 2014.

#### **1.1. Goals**

The main objective of this chapter is to connect digital knowledge resources with historical and cultural heritage in the context of Smart Cities. Specifically, combining the joint intervention in the Real Sitio del Pardo and Aranjuez, as well as the Foundation Apple of the University of Alcalá.

This main aim is developed through five specific objectives:


**d.** Development of accessible didactic and educational materials with new technologies for use in Moocs or in the new realities of teaching and learning processes.

### **2. Theoretical framework**

Finally, indicate that in the elaboration of this project, we have taken into account the recommendations and suggestions emanating from the document Getting cultural heritage to work for Europe. Report of the Horizon 2020 Expert Group in Cultural Heritage (2015) of the Conclusions of the Council of Europe (Education, Youth, Culture and Sports) adopted on

The main objective of this chapter is to connect digital knowledge resources with historical and cultural heritage in the context of Smart Cities. Specifically, combining the joint intervention in the Real Sitio del Pardo and Aranjuez, as well as the Foundation Apple of the

**a.** Creation of an environment of augmented reality of free access, showing the historical evolution of these places and the different changes that have taken place in it, highlighting the historical and cultural heritage of it, as well as the relationship with the environment

**b.** 2D and 3D interpretation of the cartography and planimetry that analyzes the process of configuration, development and consolidation of these places to the present day, which

**c.** Create tourist and/or educational itineraries at various levels, accessible for the transmission of artistic, cultural and landscape heritage and value this heritage using new technologies.

This main aim is developed through five specific objectives:

allows their better transmission and contextualization.

May 20, 2014.

**Figure 3.** Virtual heritage in the University of Alcalá.

28 State of the Art Virtual Reality and Augmented Reality Knowhow

University of Alcalá.

and the landscape.

**1.1. Goals**

The theoretical framework justifies the innovative nature of the proposal. Starting from the concept of "Smart City," the technology of augmented reality is considered as the ideal instrument to develop a virtual heritage proposal.

#### **2.1. The "smart cities" and their heritage**

The concept of "Smart Cities" [1–3] is associated with the principle of sustainable urban development that includes different areas such as infrastructures, technologies or democracy. The efficiency of smart cities has an important component of social development, and there is no better way to connect a city with its population than through the heritage that unites them.

The new means of communication have generated a new field of dissemination and protection of cultural heritage [4–6] Since the declaration of UNESCO in 1972 on the preservation of material heritage and in the year 2003 on the intangible heritage, a series of processes, tools and especially networks have been generated that increase the accessibility and participation of citizens in the management of cultural assets (**Figure 4**).

**Figure 4.** Identifying interested places in a smart city.

The development of a "smart city" inevitably requires efficient management of heritage, which involves its integration into new information and communication technologies [7–9]. In this sense, we now have an exceptional resource when it comes to integrating heritage in the field of these new technologies: augmented reality.

#### **2.2. From virtual reality to augmented reality**

The scope of new technologies is truly extensive and, although all of them work together for the same purpose, the truth is that here we want to make special mention of the technology of augmented reality, which has been shaping up since the last quarter of the twentieth century as a tool with great potential to transmit our past, value and raise public awareness of our heritage ([10] 5).

**2.3. The applications of virtual heritage**

• Reconstruction of buildings in ruins or significantly altered.

**Table 1.** "Reality-virtuality continuum" based on Milgran and Kishino [12].

• Simulation of social or natural environments on historic sites.

material objects according to Gutiérrez and Hernández [17]:

• Speculate with different hypotheses about lost objects.

• Analyze an archeological object in its original environment.

• Test restoration techniques on synthetic models.

of our own reality ([17] 14).

systems shown in **Figures 5** and **6**.

• Recreation of lost or damaged archeological pieces.

The application of augmented reality to the field of historical heritage has generated a new concept that is virtual heritage. Following the discourse of several authors [13–16], we can synthesize three approaches when using augmented reality to disseminate historical heritage:

Connect Smart Cities and Heritage Through Augmented Reality

http://dx.doi.org/10.5772/intechopen.74363

31

To these informative functions, we must add other approaches that also consider useful in scientific contexts the generation of 3D images to simulate and investigate certain intangible

Therefore, it is necessary to propose new forms of content associated with this technology that go beyond the simple virtual reconstruction of historical buildings. Through augmented reality, it should be possible to enjoy new experiences that cannot occur in the real world, even in the one that has already disappeared, and that help to better understand what heritage means beyond its spatial analysis. In this sense, Gutiérrez and Hernández also defend that the incorporation of multimedia, multi-user exploration, telepresence and the possibility of showing worlds in ways that are not subject to the physical limitations of the world we live in will lead this technology to become no longer an emulation of what exists, but an expansion

This coincides perfectly with the objective of the augmented reality that according to Fernández Álvarez [14] is none other than overcoming the comprehension difficulties due to different levels of conceptual abstraction presented by the different traditional representation

Computer technology has facilitated the generation of 3D objects since the beginning of the first graphic interfaces. In fact, this computing capacity has allowed the creation of complex visual environments in three dimensions for all types of applications such as videogames, geographic information systems, as well as graphic, industrial or architectural design. However, these 3D images have always needed a computer interface to be able to represent themselves. As realistic as the three-dimensional reproduction was, they always referred us to a virtual reality.

Based on virtual reality, with which it was possible to generate a fictitious environment of 360°, vertically and horizontally, of great realism for the viewer, augmented reality is, broadly speaking, a computer system that manages to mix the virtual simulated environment with a real physical scenario. To this combination, you can also add texts, documents, 2D images, audios and so on, enriching it with the information that the visitor perceives. Both present today a wide range of possibilities for different fields of knowledge, but the main difference between them is that virtual reality can or cannot be based on a real environment ([11] 25), while augmented reality only recreates a real image and needs of the visitor's on-site presence to complete the image.

The first to try to overcome this antithetical opposition between the "real" and "virtual" world were Milgran and Kishino, when formulating their model of Real-Virtual Continuum [12]. His proposal was to try to integrate real and virtual elements into a mixed reality that could be experienced in direct continuity with both worlds, real and virtual.

Through this paradigm presented in **Table 1**, a new technology derived from the application to the real world of this virtual reality began to develop in the mid-1990s, and that unlike this, it does not consist of generating virtual environments but is characterized by inserting objects or virtual spaces in a real scenario ([13] 215). It is what we know today as augmented reality and allows us to visualize 3D elements through any type of device that reproduces a real image.

This condition can be attributed to any type of montage or photo retouching, but according to Fernández Álvarez, the key to this augmented Reality is that there is a direct correspondence between the real and the virtual in terms of scale, proportion, proximity, point of view, depth and so on, which allows the user in some applications to experience the space on a natural scale ([14] 3). Therefore, we are proposing a paradigm that does not intend to recreate a new virtual world but to create a unique visual world integrated by both real space and 3D images.

**Table 1.** "Reality-virtuality continuum" based on Milgran and Kishino [12].

#### **2.3. The applications of virtual heritage**

The development of a "smart city" inevitably requires efficient management of heritage, which involves its integration into new information and communication technologies [7–9]. In this sense, we now have an exceptional resource when it comes to integrating heritage in

The scope of new technologies is truly extensive and, although all of them work together for the same purpose, the truth is that here we want to make special mention of the technology of augmented reality, which has been shaping up since the last quarter of the twentieth century as a tool with great potential to transmit our past, value and raise public awareness of our

Computer technology has facilitated the generation of 3D objects since the beginning of the first graphic interfaces. In fact, this computing capacity has allowed the creation of complex visual environments in three dimensions for all types of applications such as videogames, geographic information systems, as well as graphic, industrial or architectural design. However, these 3D images have always needed a computer interface to be able to represent themselves. As realistic

Based on virtual reality, with which it was possible to generate a fictitious environment of 360°, vertically and horizontally, of great realism for the viewer, augmented reality is, broadly speaking, a computer system that manages to mix the virtual simulated environment with a real physical scenario. To this combination, you can also add texts, documents, 2D images, audios and so on, enriching it with the information that the visitor perceives. Both present today a wide range of possibilities for different fields of knowledge, but the main difference between them is that virtual reality can or cannot be based on a real environment ([11] 25), while augmented reality only recreates a real image and needs of the visitor's on-site presence to complete the image. The first to try to overcome this antithetical opposition between the "real" and "virtual" world were Milgran and Kishino, when formulating their model of Real-Virtual Continuum [12]. His proposal was to try to integrate real and virtual elements into a mixed reality that

Through this paradigm presented in **Table 1**, a new technology derived from the application to the real world of this virtual reality began to develop in the mid-1990s, and that unlike this, it does not consist of generating virtual environments but is characterized by inserting objects or virtual spaces in a real scenario ([13] 215). It is what we know today as augmented reality and allows us to visualize 3D elements through any type of device that reproduces a real image.

This condition can be attributed to any type of montage or photo retouching, but according to Fernández Álvarez, the key to this augmented Reality is that there is a direct correspondence between the real and the virtual in terms of scale, proportion, proximity, point of view, depth and so on, which allows the user in some applications to experience the space on a natural scale ([14] 3). Therefore, we are proposing a paradigm that does not intend to recreate a new virtual world but to create a unique visual world integrated by both real space and 3D images.

as the three-dimensional reproduction was, they always referred us to a virtual reality.

could be experienced in direct continuity with both worlds, real and virtual.

the field of these new technologies: augmented reality.

**2.2. From virtual reality to augmented reality**

30 State of the Art Virtual Reality and Augmented Reality Knowhow

heritage ([10] 5).

The application of augmented reality to the field of historical heritage has generated a new concept that is virtual heritage. Following the discourse of several authors [13–16], we can synthesize three approaches when using augmented reality to disseminate historical heritage:


To these informative functions, we must add other approaches that also consider useful in scientific contexts the generation of 3D images to simulate and investigate certain intangible material objects according to Gutiérrez and Hernández [17]:


Therefore, it is necessary to propose new forms of content associated with this technology that go beyond the simple virtual reconstruction of historical buildings. Through augmented reality, it should be possible to enjoy new experiences that cannot occur in the real world, even in the one that has already disappeared, and that help to better understand what heritage means beyond its spatial analysis. In this sense, Gutiérrez and Hernández also defend that the incorporation of multimedia, multi-user exploration, telepresence and the possibility of showing worlds in ways that are not subject to the physical limitations of the world we live in will lead this technology to become no longer an emulation of what exists, but an expansion of our own reality ([17] 14).

This coincides perfectly with the objective of the augmented reality that according to Fernández Álvarez [14] is none other than overcoming the comprehension difficulties due to different levels of conceptual abstraction presented by the different traditional representation systems shown in **Figures 5** and **6**.

in the research on heritage, focusing on this project on the evolution and its relationship with

Connect Smart Cities and Heritage Through Augmented Reality

http://dx.doi.org/10.5772/intechopen.74363

33

The RA together with this new work methodology will allow a selective adaptation of the information shown, in terms of quantity, rigor and complexity according to the purpose or audience to which it is shown, facilitating the understanding of the heritage in an individual and personalized way to age, level of specialization or interest [22]. This point is essential as a

To this main objective, a new research parameter is added: the graph. To make possible the understanding of the proposed information model, it is essential to search for a graphic language capable of giving a coherent and effective response to the proposed new order, facilitating its understanding. A language that allows the visualization of variable parameters and complex relationships between different layers of information, in an affordable way. A form of graphic expression halfway between representation and communication that naturally incorporates the advantages and opportunities offered by new technologies, especially the

In addition, will be made, in order to have the necessary documentation and combining and mixing new technologies with classical systems, a dump of published documentation on these places of different archives and libraries in order to agglutinate a documentary corpus that allows to visualize the evolution of this real site throughout history and the faithful reproduction in augmented reality. In this sense, we will work, fundamentally, in the General Archive of Simancas (Valladolid), General Palace Archive, National Library and National Geographic

Indicate, in this point, that the work developed by the research team of the URJC in recent years on the study of the court, houses and real sites from the fifteenth to the eighteenth century, allows to make feasible the work proposed by the work and analysis already done.

tool for disclosure and dissemination of heritage, as shown in **Figure 7**.

augmented reality, together with knowledge and classical historical research.

Institute and the Municipal Archives of Spain.

**Figure 7.** Visualization of virtual heritage through RA.

the surrounding territory.

**Figure 5.** Virtual heritage through AR in Aranjuez.

**Figure 6.** Real heritage (down) through AR in Aranjuez.

#### **3. Method**

In the context indicated earlier, the augmented reality (AR) is presented as a powerful tool for contextualizing, disseminating the heritage, as well as to make the resources created more accessible. A new methodology is proposed for the organization and storage of documentation, both original and processed, from different areas and digitized and georeferenced by categorized strata: time, originality of information, scope [20, 21]. A wicker on which to deposit the documentation in an orderly and rational way for optimal management and storage.

To do this, it is proposed to use the augmented reality technology as a hub and documentation connection vehicle (from different times, types and areas), georeferencing it to the place and allowing holistic, more extensive, deep and rigorous knowledge of the heritage. Thus, a model applicable to the different scales of approach (territorial, urban, architectural and interior or domestic) and transferable to different architectural models, which allows finding new relationships between different documents that will allow in some cases verify or discard hypotheses about states and previous times [23]. Also helping to advance in a rigorous way in the research on heritage, focusing on this project on the evolution and its relationship with the surrounding territory.

The RA together with this new work methodology will allow a selective adaptation of the information shown, in terms of quantity, rigor and complexity according to the purpose or audience to which it is shown, facilitating the understanding of the heritage in an individual and personalized way to age, level of specialization or interest [22]. This point is essential as a tool for disclosure and dissemination of heritage, as shown in **Figure 7**.

To this main objective, a new research parameter is added: the graph. To make possible the understanding of the proposed information model, it is essential to search for a graphic language capable of giving a coherent and effective response to the proposed new order, facilitating its understanding. A language that allows the visualization of variable parameters and complex relationships between different layers of information, in an affordable way. A form of graphic expression halfway between representation and communication that naturally incorporates the advantages and opportunities offered by new technologies, especially the augmented reality, together with knowledge and classical historical research.

In addition, will be made, in order to have the necessary documentation and combining and mixing new technologies with classical systems, a dump of published documentation on these places of different archives and libraries in order to agglutinate a documentary corpus that allows to visualize the evolution of this real site throughout history and the faithful reproduction in augmented reality. In this sense, we will work, fundamentally, in the General Archive of Simancas (Valladolid), General Palace Archive, National Library and National Geographic Institute and the Municipal Archives of Spain.

Indicate, in this point, that the work developed by the research team of the URJC in recent years on the study of the court, houses and real sites from the fifteenth to the eighteenth century, allows to make feasible the work proposed by the work and analysis already done.

**Figure 7.** Visualization of virtual heritage through RA.

**3. Method**

**Figure 6.** Real heritage (down) through AR in Aranjuez.

**Figure 5.** Virtual heritage through AR in Aranjuez.

32 State of the Art Virtual Reality and Augmented Reality Knowhow

ment and storage.

In the context indicated earlier, the augmented reality (AR) is presented as a powerful tool for contextualizing, disseminating the heritage, as well as to make the resources created more accessible. A new methodology is proposed for the organization and storage of documentation, both original and processed, from different areas and digitized and georeferenced by categorized strata: time, originality of information, scope [20, 21]. A wicker on which to deposit the documentation in an orderly and rational way for optimal manage-

To do this, it is proposed to use the augmented reality technology as a hub and documentation connection vehicle (from different times, types and areas), georeferencing it to the place and allowing holistic, more extensive, deep and rigorous knowledge of the heritage. Thus, a model applicable to the different scales of approach (territorial, urban, architectural and interior or domestic) and transferable to different architectural models, which allows finding new relationships between different documents that will allow in some cases verify or discard hypotheses about states and previous times [23]. Also helping to advance in a rigorous way

#### **4. Results**

The product and final result of the project is the elaboration of a portal of augmented reality, of free access, showing the historical evolution of these places and the different changes that have taken place in it, highlighting the historical heritage, cultural identity, as well as the relationship with the environment and the landscape, with a 2D and 3D interpretation of the cartography and planimetry of these places, analyzing the process of configuration, development and consolidation of the real site until our days, as well as the creation of an augmented reality environment that allows its better transmission and contextualization (**Figure 8**).

The dissemination plan of the augmented reality application is designed to offer an innovative tool that can be integrated into previous and subsequent strategies to disclosure heritage: Some of the activities proposed are the following:


• This web portal will incorporate a documentary and literary corpus on the royal site that helps to understand and contextualize its evolution. These digital documents will also be deposited in the corresponding digital repositories through the library services of both universities. • Given the uniqueness of the experience carried out, the results of the research project will be published and disseminated jointly through international congresses as well as impact

Connect Smart Cities and Heritage Through Augmented Reality

http://dx.doi.org/10.5772/intechopen.74363

35

**Figure 9.** Screenshot form the app designed for the Royal Sites (URJC-44creens.Fr).

**Figure 10.** Screenshot form the app designed for the Aranjuez (44creens.Fr).

academic journals.

**Figure 8.** Augmented reality app designed for the University of Alcalá.

**Figure 9.** Screenshot form the app designed for the Royal Sites (URJC-44creens.Fr).

**4. Results**

The product and final result of the project is the elaboration of a portal of augmented reality, of free access, showing the historical evolution of these places and the different changes that have taken place in it, highlighting the historical heritage, cultural identity, as well as the relationship with the environment and the landscape, with a 2D and 3D interpretation of the cartography and planimetry of these places, analyzing the process of configuration, development and consolidation of the real site until our days, as well as the creation of an augmented reality

The dissemination plan of the augmented reality application is designed to offer an innovative tool that can be integrated into previous and subsequent strategies to disclosure heritage:

• This application may be available to the society and to the university community in par-

• This web portal and the technology contained in it will also allow the development of materials and teaching resources in virtual environments that allow students of compulsory and

• The content will also be used for the development of tourist itineraries in virtual environments, accessible for people, which will give value to the transmission of artistic, cultural and landscape heritage. The Visitors Service of the UAH could also have the application to

• The graphic materials and 3D models generated can be used both by the publications service and by the communication service of each university. These contents may also be

environment that allows its better transmission and contextualization (**Figure 8**).

university education the acquisition of competences and transferable skills.

ticular through the services of applications for mobile devices.

improve the experience of tourists visiting the Rectorate building.

transferred to third parties for tourist or cultural use.

**Figure 8.** Augmented reality app designed for the University of Alcalá.

Some of the activities proposed are the following:

34 State of the Art Virtual Reality and Augmented Reality Knowhow

**Figure 10.** Screenshot form the app designed for the Aranjuez (44creens.Fr).


• This project also has a precedent in the use of augmented reality for the dissemination of heritage through several Madrid universities, which is the "Campus Husso Digital" project. The developed contents are susceptible to be included, completing this application.

Taking into account all these proposals and far from being configured as a closed activity, it is proposed that this project continues to develop through the synergies established between both research groups [18, 19] (**Figures 9** and **10**).

### **5. Conclusions**

The concept of heritage is something subjective and alive. It evolves with time and society, that it is, in short, who determines what goods are to be conserved and protected for posterity, based on the values attributed to them. Hence, the importance of achieving an awareness of society is an active part in the conservation, enjoyment and dissemination of heritage, and that can only be achieved through knowledge. The key is how to get that knowledge to society.

in **Figure 11**. This consideration greatly expands the scope and amount of assets to be conserved, protected and maintained. This fact, linked to the current economic context, makes the safeguarding of heritage an increasingly difficult task for which it is essential to have effective management and dissemination tools and of course support and help of

It is at this point, where education and communication play a fundamental role in the dissemination and understanding of heritage from a holistic view, within the historical and social context on which it bases its value. New ways of research and transdisciplinary work are opened around the way of contextualizing, communicating and disseminating heritage, pointing out interesting synergies between the different areas that affect the understanding of

In the same measure, the way to show that information to society is fundamental is through a clear, direct and rigorous graphic language. To do this, we must take into account the evolution of technology and its impact on the visual environment, taking as reference also the field of architecture, language and tools used in the world of communication, shown in **Figure 12** (user-based design, narrative, etc.). In this context, the augmented reality is presented as a powerful tool for contextualizing, disclosure and disseminating the heritage, as well as to

and Enrique Castaño Perea3

Connect Smart Cities and Heritage Through Augmented Reality

http://dx.doi.org/10.5772/intechopen.74363

37

that heritage: architecture, art, history, society, politics, geography and so on.

\*, Julián de la Fuente Prieto<sup>2</sup>

1 Dpto. CC. de la Educación, el Lenguaje, la Cultura y las Artes, CC. Histórico-Jurídicas y

Humanísticas y Lenguas Modernas, Universidad Rey Juan Carlos, Madrid, Spain

3 Dpto. de Arquitectura, Universidad de Alcalá, Alcalá de Henares, Madrid, Spain

2 Dpto. de Filología, Comunicación y Documentación, Universidad de Alcalá,

make the created resources more accessible.

\*Address all correspondence to: felix.labrador@urjc.es

**Figure 12.** Animated video for children about (Manuel Álvaro Mora).

the public.

**Author details**

Félix Labrador Arroyo<sup>1</sup>

Alcalá de Henares, Madrid, Spain

Although the definition of heritage in its historical and artistic sense dates back to the Renaissance from the perspective of its study, the current concept of heritage is only established from the twentieth century, in which no longer called monuments, only works built and of historical artistic value, to also incorporate other types of goods, material and immaterial that constitute the reflection of a culture, include digital heritage depicted

**Figure 11.** Screenshot of the three-dimensional model of the casa de Oficios y caballeros (David Beltrán Vizcaino).

**Figure 12.** Animated video for children about (Manuel Álvaro Mora).

in **Figure 11**. This consideration greatly expands the scope and amount of assets to be conserved, protected and maintained. This fact, linked to the current economic context, makes the safeguarding of heritage an increasingly difficult task for which it is essential to have effective management and dissemination tools and of course support and help of the public.

It is at this point, where education and communication play a fundamental role in the dissemination and understanding of heritage from a holistic view, within the historical and social context on which it bases its value. New ways of research and transdisciplinary work are opened around the way of contextualizing, communicating and disseminating heritage, pointing out interesting synergies between the different areas that affect the understanding of that heritage: architecture, art, history, society, politics, geography and so on.

In the same measure, the way to show that information to society is fundamental is through a clear, direct and rigorous graphic language. To do this, we must take into account the evolution of technology and its impact on the visual environment, taking as reference also the field of architecture, language and tools used in the world of communication, shown in **Figure 12** (user-based design, narrative, etc.). In this context, the augmented reality is presented as a powerful tool for contextualizing, disclosure and disseminating the heritage, as well as to make the created resources more accessible.

#### **Author details**

• This project also has a precedent in the use of augmented reality for the dissemination of heritage through several Madrid universities, which is the "Campus Husso Digital" project. The developed contents are susceptible to be included, completing this

Taking into account all these proposals and far from being configured as a closed activity, it is proposed that this project continues to develop through the synergies established between

The concept of heritage is something subjective and alive. It evolves with time and society, that it is, in short, who determines what goods are to be conserved and protected for posterity, based on the values attributed to them. Hence, the importance of achieving an awareness of society is an active part in the conservation, enjoyment and dissemination of heritage, and that can only be achieved through knowledge. The key is how to get that knowledge to

Although the definition of heritage in its historical and artistic sense dates back to the Renaissance from the perspective of its study, the current concept of heritage is only established from the twentieth century, in which no longer called monuments, only works built and of historical artistic value, to also incorporate other types of goods, material and immaterial that constitute the reflection of a culture, include digital heritage depicted

**Figure 11.** Screenshot of the three-dimensional model of the casa de Oficios y caballeros (David Beltrán Vizcaino).

application.

**5. Conclusions**

society.

both research groups [18, 19] (**Figures 9** and **10**).

36 State of the Art Virtual Reality and Augmented Reality Knowhow

Félix Labrador Arroyo<sup>1</sup> \*, Julián de la Fuente Prieto<sup>2</sup> and Enrique Castaño Perea3

\*Address all correspondence to: felix.labrador@urjc.es

1 Dpto. CC. de la Educación, el Lenguaje, la Cultura y las Artes, CC. Histórico-Jurídicas y Humanísticas y Lenguas Modernas, Universidad Rey Juan Carlos, Madrid, Spain

2 Dpto. de Filología, Comunicación y Documentación, Universidad de Alcalá, Alcalá de Henares, Madrid, Spain

3 Dpto. de Arquitectura, Universidad de Alcalá, Alcalá de Henares, Madrid, Spain

#### **References**

[1] Caragliu A, Del Bo C, Nijkamp P. Smart cities in Europe. Journal of urban technology. 2011;**18**(2):65-82

[15] Flores Gutierrez M. (2011) "Visor de Realidad Aumentada en Museos (RAM) para Exposiciones Situadas en Entornos Cerrados" en Virtual Archaeology Review. N° 3.

Connect Smart Cities and Heritage Through Augmented Reality

http://dx.doi.org/10.5772/intechopen.74363

39

[16] Gil T, Santacana J. De lo real a lo digital: la arqueología reconstructiva y la obtención de imágenes virtuales para la investigación en la didáctica del patrimonio. en HER&MES

[17] Gutiérrez D, Hernández LA. Potencial de la realidad virtual en el ámbito del Patrimonio.

[18] Castaño Perea E, de la Fuente Prieto J. The future of augmented reality in architecture: From real to virtual heritage. In: Castaño Perea E, Echeverria Valiente E, editors. Architectural Draughtsmanship: From Analog to Digital Narratives. Cham: Springer

[19] de la Fuente Prieto J, Castaño Perea E, Labrador Arroyo F. Augmented reality in architecture: Rebuilding archeological heritage. International Archives of the Photogrammetry Remote Sensing and Spatial Information Science. 2017;**XLII-2/W3**:311-315. DOI: 10.5194/

[20] Alonso N et al. Análisis de escenarios de futuro en realidad aumentada. Aplicación al yacimiento arqueológico de Els Vilars. En AA. VV. Actas del 2° Congreso Internacional

de Interacción Persona-Ordenador (Interacción'2001), Salamanca. 2001. pp. 16-18 [21] Blasco Senabre J et al. (2011) "Procesos de documentación arqueológica y generación de

[22] Redondo Domínguez E. (2012) "La ciudad como aula digital. Enseñando urbanismo y arquitectura mediante mobile learning y la realidad aumentada" en ACE: Arquitectura,

[23] Rivera Blanco J. Restauración Contemporanea. Ciudades universitarias, ciudades Patrimonio de la Humanidad. La manzana fundacional cisneriana de la Universidad de

modelos virtuales" en Virtual Archaeology Review. N° 3. Pág. 65-70

Madrid: Universidad de Alcalá. 2013. 652 p. ISBN: 978-84-15834-04-5

Pág. 87-91

N. 2013. 13. Págs. 16-35

Universidad de Zaragoza; 2003

isprs-archives-XLII-2-W3-311-2017

Ciudad y Entorno N°.19. Pág. 27-54

International Publishing; 2017. pp. 1745-1753


[15] Flores Gutierrez M. (2011) "Visor de Realidad Aumentada en Museos (RAM) para Exposiciones Situadas en Entornos Cerrados" en Virtual Archaeology Review. N° 3. Pág. 87-91

**References**

2012

2010

ceedings. 2014

Springer; 2012

Históric. 2009;**9**

Pág. 212-226

2011;**18**(2):65-82

net assembly. 2011;**6656**(31):431-446

38 State of the Art Virtual Reality and Augmented Reality Knowhow

Burlington, VT: Ashgate; 2012

on Information Systems. 1994;**E77-D**(12)

Expresión Gráfica aplicada a la Edificación. 2010

[1] Caragliu A, Del Bo C, Nijkamp P. Smart cities in Europe. Journal of urban technology.

[2] Chourabi H, Nam T, Walker S, Gil-Garcia JR, Mellouli S, Nahon K, Scholl HJ, et al. Understanding smart cities: An integrative framework. Paper presented at the system

[3] Schaffers H, Komninos N, Pallot M, Trousse B, Nilsson M, Oliveira A. Smart cities and the future internet: Towards cooperation frameworks for open innovation. Future inter-

[5] Grana C, Cucchiara R. Multimedia for cultural heritage : first international workshop, MM4CH 2011, Modena, Italy, May 3, 2011 : revised selected papers. New York: Springer;

[6] Ioannides M. Heritage in the Digital era. Brentwood, Essex: Multi-Science Publishing;

[7] Bentkowska-Kafel A, Denard H, Baker D. Paradata and Transparency in Virtual Heritage.

[8] Shumaker Re, Lackey Se. Virtual, augmented and mixed reality : designing and developing virtual and augmented environments : 6th international conference, VAMR 2014, Held As Part of HCI International 2014, Heraklion, Crete, Greece, June 22-27, 2014 : pro-

[9] Zhou M, Geng G, Wu Z. Digital Preservation Technology for Cultural Heritage. New York:

[10] Gómez Robles L, Quirosa García V. Nuevas tecnologías para difundir el Patrimonio Cultural: las reconstrucciones virtuales en España. en Revista Electrónica de Patrimonio

[11] López Pombo H. Análisis y Desarrollo de Sistemas de Realidad Aumentada. Madrid: Universidad Complutense. Proyecto Fin de Máster en Sistemas inteligentes; 2010 [12] Milgram P, Kishino F.A taxonomy of mixed reality visual displays. en IEICE Transactions

[13] Ruiz Torres D. Realidad Aumentada, Educación Y Museos. en Icono 14. Año 9. 2012;**2**:

[14] Fernández Álvarez AJ. De las arquitecturas virtuales a la realidad aumentada: un nuevo paradigma de visualización arquitectónica. en: Actas del X Congreso Internacional de

[4] Din H, Wu S. Digital Heritage and Culture. Hackensack, NJ: World Scientific; 2014

science (HICSS), 2012 45th Hawaii international conference on. 2012


**Chapter 4**

**Provisional chapter**

**Waveguide-Type Head-Mounted Display System for AR**

Currently, a lot of institutes and industries are working on the development of the virtual reality and augmented reality techniques, and these techniques have been recognized as the determination for the direction of the three-dimensional display development in the near future. In this chapter, we mainly discussed the design and application of several wearable head-mounted display (HMD) systems with the waveguide structure using the in- and out-couplers which are fabricated by the diffractive optical elements or holographic volume gratings. Although the structure is simple, the waveguide-type HMDs are very efficient, especially in the practical applications, especially in the augmented reality applications, which make the device light-weighted. In addition, we reviewed the

existing major head-mounted display and augmented reality systems.

**Keywords:** head-mounted display, AR application, see-through display, optical waveguide, holographic optical element, wedge-shaped holographic waveguide

Everything in the world has three spatial dimensions which are the width, height, and depth information; people look at the real objects directly, and they see them intact as three-dimensional (3D). But, most of the images displayed on the modern display devices such as TVs, monitors, and mobile displays are two-dimensional (2D); therefore, people have always been interested in 3D imaging systems on how to acquire and display the 3D images properly, especially, after the photographic method has been put to practical use. The 3D imaging technology

**Waveguide-Type Head-Mounted Display System for** 

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

© 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

DOI: 10.5772/intechopen.75172

**Application**

**AR Application**

**Abstract**

**1. Introduction**

Nam Kim

Nam Kim

Munkh-Uchral Erdenebat, Young-Tae Lim, Ki-Chul Kwon, Nyamsuren Darkhanbaatar and

Munkh-Uchral Erdenebat, Young-Tae Lim, Ki-Chul Kwon, Nyamsuren Darkhanbaatar and

Additional information is available at the end of the chapter

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.75172

#### **Waveguide-Type Head-Mounted Display System for AR Application Waveguide-Type Head-Mounted Display System for AR Application**

DOI: 10.5772/intechopen.75172

Munkh-Uchral Erdenebat, Young-Tae Lim, Ki-Chul Kwon, Nyamsuren Darkhanbaatar and Nam Kim Munkh-Uchral Erdenebat, Young-Tae Lim, Ki-Chul Kwon, Nyamsuren Darkhanbaatar and Nam Kim

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.75172

#### **Abstract**

Currently, a lot of institutes and industries are working on the development of the virtual reality and augmented reality techniques, and these techniques have been recognized as the determination for the direction of the three-dimensional display development in the near future. In this chapter, we mainly discussed the design and application of several wearable head-mounted display (HMD) systems with the waveguide structure using the in- and out-couplers which are fabricated by the diffractive optical elements or holographic volume gratings. Although the structure is simple, the waveguide-type HMDs are very efficient, especially in the practical applications, especially in the augmented reality applications, which make the device light-weighted. In addition, we reviewed the existing major head-mounted display and augmented reality systems.

**Keywords:** head-mounted display, AR application, see-through display, optical waveguide, holographic optical element, wedge-shaped holographic waveguide

#### **1. Introduction**

Everything in the world has three spatial dimensions which are the width, height, and depth information; people look at the real objects directly, and they see them intact as three-dimensional (3D). But, most of the images displayed on the modern display devices such as TVs, monitors, and mobile displays are two-dimensional (2D); therefore, people have always been interested in 3D imaging systems on how to acquire and display the 3D images properly, especially, after the photographic method has been put to practical use. The 3D imaging technology

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

has been developed since the 1840s, but since 2009, after the stereoscopic 3D movie "Avatar" has been released, it became a hot topic worldwide. Nowadays, the 3D imaging/display systems are widely used in various fields such as entertainment, education, training, biomedical science, and so on."

A head-mounted display (HMD) which has gained much popularity in recent years after the virtual reality (VR)/augmented reality (AR) application's boom is a display device that the user wears it on the head like a helmet [1]. Note that, in the application of the combat aircrafts, it is called as a helmet-mounted display that shows the situational awareness of that moment, and the combatant-pilot can control the weapons according to the head pointing direction [2]. Also, the terrestrial guardians and special combatants of special forces of various countries widely use the helmet-mounted displays. **Figure 1** shows the actual representation of the typical HMD and helmet-mounted display.

> and features can be viewed and controlled in the real-world environment, instead of the user that is completely sequestered from the real world. In other words, the AR technique is able to overlay the elements of the digital world into a real world. Here, the information of the digital elements can be added into the real world or masked. The user can control the digital elements without using any specific device such as keyboard or mouse, only by the face's actions such as blinking eyes, and see the text and video on any background where he/she wants. Most of the AP systems are applied through the glass-type monocular HMD or headup display (HUD). The HUD is a transparent display that is usually located in front of the users, and users observe the important data without looking up or down. Initially, it was developed for the military aviation-based applications, but currently, it is widely used in the

Waveguide-Type Head-Mounted Display System for AR Application

http://dx.doi.org/10.5772/intechopen.75172

43

**Figure 2.** General structure of the HMD systems: (a) binocular and (b) monocular.

The HMD-based VR and AR technology markets become extremely huge and rapidly growing continuously. For the VR, it has been applied extensively for educational and training purposes in recent years, not only in the entertainment fields such as 3D games and videos. Especially, for the drivers, pilots, and special combatants, it is the efficient and safe way for the training that the trainer can perform various activities in a virtual space similar to the actual environment. Also, VR systems are exploited in the dental clinics for children, in order

Based on the ability and development of the technology, the interest for the see-through AR systems is increasing acutely. Modern handheld smart mobile devices include a lot of functions such as the Internet use, navigation, image and video processing, high-resolution camera, games, and so on, except the communication, and it was a huge technological revolution. But, the size of the smart mobile devices is limited by the average hand size of the human; so, no matter how good the display resolution is, it gives a lot of inconvenience to users. For the see-through AR systems, the smart mobile devices can be replaced, because the AR systems have all of the functions of mobile devices basically and enough of computing power; additionally, they include the hand-free feature that does not require the interaction of the hands and large-screen experience, while the user wears the AR simply like the eyeglasses. Therefore, experts and developers believe that the AR technology will open the new generation of mobile

automobiles, aircrafts, and so on.

devices soon.

to reduce the children's fear during the treatment.

Most of the HMDs are based on the stereoscopic 3D display technique. The stereoscopic technique is a simple way to display 3D images through the binocular depth cues of human visual perception considering the interocular distance of the human eyes [3, 4]. It provides slightly different dual two-dimensional (2D) images to the left and right eyes, respectively, and makes sure that each eye gets the corresponding image only, by the use of the special spectacles. Then, both 2D offset images are combined in the brain to give the perception of 3D depth. The general structure of the typical HMD is that two micro displays, lenses, and semitransparent mirrors or prisms are installed in for each eye, as shown in **Figure 2(a)**. Some HMDs has single micro display and lens for one eye, and this structure is named by monocular HMD, as shown in **Figure 2(b)**, where the typical HMD is defined as binocular. Nowadays, a binocular-type HMD is usually applied in the VR-based applications, and monocular-type HMD is mainly used in AR applications.

Then, it can be seen that the AR can be the next generation of VR. Unlike the VR technique, the AR can be seen as a half reality in which any virtual computer-generated perceptual items

**Figure 1.** The actual representation of (a) the typical HMD (source: http://www.5dt.com/head-mounted-displays) and (b) helmet-mounted display (source: https://www.flickr.com/photos/48970996@N04/12682 497224).

**Figure 2.** General structure of the HMD systems: (a) binocular and (b) monocular.

has been developed since the 1840s, but since 2009, after the stereoscopic 3D movie "Avatar" has been released, it became a hot topic worldwide. Nowadays, the 3D imaging/display systems are widely used in various fields such as entertainment, education, training, biomedical

A head-mounted display (HMD) which has gained much popularity in recent years after the virtual reality (VR)/augmented reality (AR) application's boom is a display device that the user wears it on the head like a helmet [1]. Note that, in the application of the combat aircrafts, it is called as a helmet-mounted display that shows the situational awareness of that moment, and the combatant-pilot can control the weapons according to the head pointing direction [2]. Also, the terrestrial guardians and special combatants of special forces of various countries widely use the helmet-mounted displays. **Figure 1** shows the actual representation of the

Most of the HMDs are based on the stereoscopic 3D display technique. The stereoscopic technique is a simple way to display 3D images through the binocular depth cues of human visual perception considering the interocular distance of the human eyes [3, 4]. It provides slightly different dual two-dimensional (2D) images to the left and right eyes, respectively, and makes sure that each eye gets the corresponding image only, by the use of the special spectacles. Then, both 2D offset images are combined in the brain to give the perception of 3D depth. The general structure of the typical HMD is that two micro displays, lenses, and semitransparent mirrors or prisms are installed in for each eye, as shown in **Figure 2(a)**. Some HMDs has single micro display and lens for one eye, and this structure is named by monocular HMD, as shown in **Figure 2(b)**, where the typical HMD is defined as binocular. Nowadays, a binocular-type HMD is usually applied in the VR-based applications, and monocular-type HMD is mainly

Then, it can be seen that the AR can be the next generation of VR. Unlike the VR technique, the AR can be seen as a half reality in which any virtual computer-generated perceptual items

**Figure 1.** The actual representation of (a) the typical HMD (source: http://www.5dt.com/head-mounted-displays) and

(b) helmet-mounted display (source: https://www.flickr.com/photos/48970996@N04/12682 497224).

science, and so on."

used in AR applications.

typical HMD and helmet-mounted display.

42 State of the Art Virtual Reality and Augmented Reality Knowhow

and features can be viewed and controlled in the real-world environment, instead of the user that is completely sequestered from the real world. In other words, the AR technique is able to overlay the elements of the digital world into a real world. Here, the information of the digital elements can be added into the real world or masked. The user can control the digital elements without using any specific device such as keyboard or mouse, only by the face's actions such as blinking eyes, and see the text and video on any background where he/she wants. Most of the AP systems are applied through the glass-type monocular HMD or headup display (HUD). The HUD is a transparent display that is usually located in front of the users, and users observe the important data without looking up or down. Initially, it was developed for the military aviation-based applications, but currently, it is widely used in the automobiles, aircrafts, and so on.

The HMD-based VR and AR technology markets become extremely huge and rapidly growing continuously. For the VR, it has been applied extensively for educational and training purposes in recent years, not only in the entertainment fields such as 3D games and videos. Especially, for the drivers, pilots, and special combatants, it is the efficient and safe way for the training that the trainer can perform various activities in a virtual space similar to the actual environment. Also, VR systems are exploited in the dental clinics for children, in order to reduce the children's fear during the treatment.

Based on the ability and development of the technology, the interest for the see-through AR systems is increasing acutely. Modern handheld smart mobile devices include a lot of functions such as the Internet use, navigation, image and video processing, high-resolution camera, games, and so on, except the communication, and it was a huge technological revolution. But, the size of the smart mobile devices is limited by the average hand size of the human; so, no matter how good the display resolution is, it gives a lot of inconvenience to users. For the see-through AR systems, the smart mobile devices can be replaced, because the AR systems have all of the functions of mobile devices basically and enough of computing power; additionally, they include the hand-free feature that does not require the interaction of the hands and large-screen experience, while the user wears the AR simply like the eyeglasses. Therefore, experts and developers believe that the AR technology will open the new generation of mobile devices soon.

#### **1.1. Earlier AR systems**

The term "augmented reality" has been popularized by T. Caudell and D. Mizell who were the researchers at the Boeing, since 1990. However, a transparent electronic display that displays the virtual objects in the real-world environment has been mentioned the first time in the children novel *The Wonderful Wizard of Oz* written by L. F. Baum, in 1901, but the actual development of AR systems has begun quite later, since the 1980s.

In 2014, Sony released their 85% see-through wearable glass-type AR device "SmartEyeGlass." It includes the basic devices required in the modern AR systems such as microcomputer, micro display, and microphone, and it can be connected to the smart phones. The most important feature is that the system is based on holographic waveguide structure, and this structure

Waveguide-Type Head-Mounted Display System for AR Application

http://dx.doi.org/10.5772/intechopen.75172

45

Another AR technology-based see-through HMD system using the diffractive waveguide structure "HoloLens" which contains the multiple cameras and sensors is released by Microsoft in 2015. Here, they attributed another term "mixed reality (MR)," and this term is much popularized currently. The meaning of the MR is almost similar with the AR that vir-

In 2016, Niantic has released an AR-based mobile game "Pokemon GO," and it became the most popular smart phone game in short time. Accordingly, the people got more interests in

In 2017, Apple Inc. announced that their new operating system for iPhone and iPad products supports the AR contents. Also, in 2017, Microsoft Research presented the prototype of holographic near-eye display which is a see-through display system using holographic waveguide

As mentioned earlier, the AR and/or MR technology is the next generation of VR technology, and it expected to be the direction of the 3D display development in the near future. AR system has a lot more of functions than VR systems that it has been defined as a wearable computer, and the hardware structure is much complicated. Due to the multifunctions, the AR technology requires several electronic and optical devices such as microcomputer, projector, microphone, speaker, and in- and out-couplers; so, if the typical HMD structure is utilized, the system will become very bulky, and it is inconvenient to wear it. Therefore, the miniatur-

Recently released complete AR technologies such as Google Glass, Epson Moverio, Sony SmartEyeglass, and Microsoft HoloLens use the optical waveguide structure. The waveguide structure makes the AR systems small and light-weighted, and it is the biggest advantage for the users to use an ad hoc or day-to-day basis. This section mainly discusses about the principle of the optical waveguide and the various waveguide for the AR application-based

The waveguide structure is widely applied in various fields such as electromagnetics, optics, and sound synthesis [6]. It guides the electromagnetic or sound waves through the fibers and pipes. In the optics, it is used as a transmitter that transmits the input signal, the light wave, between two different fields by guiding the waves, without any loss of input information.

and *n*<sup>2</sup>

are the refractive

tual components can be mixed in real-world environment, half real and half virtual.

makes the device as light-weighted and small.

AR technology.

HMD devices.

for the AR/VR applications.

**2. Waveguide-type HMD system**

ized and light-weighted hardware structure is necessary.

**2.1. Overview of the conventional optical waveguide**

**Figure 3** shows the basic illustration of the optical waveguide where *n*<sup>1</sup>

Actually, the HUD has been developed since the 1940s, but in that time, it has not been commercialized and is only used in the military applications such as in the night fighting combat aircrafts [5]. It was advantageous during the air fight in the night or bad visibility, because the radar display has been projected to the window front of the pilot, and, later, the gunsight has been displayed additionally.

A HUD system for pilots' training application in the commercial aviation has been introduced by G. Lintern, from the University of Illinois, in 1980. Also, in 1980, S. Mann has invented a wearable computer which is a glass-type computer vision system, "EyeTap." It was a monocular HMD and the first example of the AR display system which includes the camera, computer, projector, and micro display in front of the human eye. It was allowed to the users that is observing the computer-generated data and real-world environment at the same time. Here, when the camera captures the images, the microcomputer processes the captured images as the computer-generated data, and the processed data is overlaid with the real-world environment through the projector and micro display. But in that time, it was quite difficult to demonstrate the full AR features through the system.

The AR technologies also have been in the TV applications such as in the weather broadcasting that the virtual symbols and information are mixed with the actual earth map. Also, the AR system has been presented precisely in the movie "The Terminator," in 1984.

The HUD has been applied in the astronomical telescope system that the images of the stars and celestial body images and any other specific information are displayed when the users observe the actual sky images through the eyepiece of the astronomical telescope.

In the 1990s, several popular AR systems have been introduced. The "virtual fixture" which is a first complete functional AR system for the aircraft has been proposed by L. Rosenberg, in 1992. The system provides the improvement of the human performance in both of actual and remotely manipulated processes, and this system was somehow VR technology.

Thereafter, various institutes and researchers developed the AR technologies in the 1990s and 2000s. For example, the ARToolKit has been created by HITLAB. in 1999 (it was allowed for web browser in 2009), an AR game "ARQuake" has been demonstrated in 2000, the Wikitude AR traveling guide for Android mobile devices has been released in 2008, and so on.

In 2013, Google demonstrated the beta version of complete wearable glass-type see-through AR system "Google Glass" which includes the microcomputer, camera, projector, microliquid crystal display (LCD), microphone, and speaker. Also, it was able to connect to the Internet and other devices through the wireless network and Bluetooth. It has many functions such that the user can send/receive the emails, see the maps, watch the images and videos, make the voice call, and use the Internet.

In 2014, Sony released their 85% see-through wearable glass-type AR device "SmartEyeGlass." It includes the basic devices required in the modern AR systems such as microcomputer, micro display, and microphone, and it can be connected to the smart phones. The most important feature is that the system is based on holographic waveguide structure, and this structure makes the device as light-weighted and small.

Another AR technology-based see-through HMD system using the diffractive waveguide structure "HoloLens" which contains the multiple cameras and sensors is released by Microsoft in 2015. Here, they attributed another term "mixed reality (MR)," and this term is much popularized currently. The meaning of the MR is almost similar with the AR that virtual components can be mixed in real-world environment, half real and half virtual.

In 2016, Niantic has released an AR-based mobile game "Pokemon GO," and it became the most popular smart phone game in short time. Accordingly, the people got more interests in AR technology.

In 2017, Apple Inc. announced that their new operating system for iPhone and iPad products supports the AR contents. Also, in 2017, Microsoft Research presented the prototype of holographic near-eye display which is a see-through display system using holographic waveguide for the AR/VR applications.

### **2. Waveguide-type HMD system**

**1.1. Earlier AR systems**

44 State of the Art Virtual Reality and Augmented Reality Knowhow

been displayed additionally.

onstrate the full AR features through the system.

make the voice call, and use the Internet.

The term "augmented reality" has been popularized by T. Caudell and D. Mizell who were the researchers at the Boeing, since 1990. However, a transparent electronic display that displays the virtual objects in the real-world environment has been mentioned the first time in the children novel *The Wonderful Wizard of Oz* written by L. F. Baum, in 1901, but the actual

Actually, the HUD has been developed since the 1940s, but in that time, it has not been commercialized and is only used in the military applications such as in the night fighting combat aircrafts [5]. It was advantageous during the air fight in the night or bad visibility, because the radar display has been projected to the window front of the pilot, and, later, the gunsight has

A HUD system for pilots' training application in the commercial aviation has been introduced by G. Lintern, from the University of Illinois, in 1980. Also, in 1980, S. Mann has invented a wearable computer which is a glass-type computer vision system, "EyeTap." It was a monocular HMD and the first example of the AR display system which includes the camera, computer, projector, and micro display in front of the human eye. It was allowed to the users that is observing the computer-generated data and real-world environment at the same time. Here, when the camera captures the images, the microcomputer processes the captured images as the computer-generated data, and the processed data is overlaid with the real-world environment through the projector and micro display. But in that time, it was quite difficult to dem-

The AR technologies also have been in the TV applications such as in the weather broadcasting that the virtual symbols and information are mixed with the actual earth map. Also, the

The HUD has been applied in the astronomical telescope system that the images of the stars and celestial body images and any other specific information are displayed when the users

In the 1990s, several popular AR systems have been introduced. The "virtual fixture" which is a first complete functional AR system for the aircraft has been proposed by L. Rosenberg, in 1992. The system provides the improvement of the human performance in both of actual and

Thereafter, various institutes and researchers developed the AR technologies in the 1990s and 2000s. For example, the ARToolKit has been created by HITLAB. in 1999 (it was allowed for web browser in 2009), an AR game "ARQuake" has been demonstrated in 2000, the Wikitude

In 2013, Google demonstrated the beta version of complete wearable glass-type see-through AR system "Google Glass" which includes the microcomputer, camera, projector, microliquid crystal display (LCD), microphone, and speaker. Also, it was able to connect to the Internet and other devices through the wireless network and Bluetooth. It has many functions such that the user can send/receive the emails, see the maps, watch the images and videos,

AR system has been presented precisely in the movie "The Terminator," in 1984.

observe the actual sky images through the eyepiece of the astronomical telescope.

remotely manipulated processes, and this system was somehow VR technology.

AR traveling guide for Android mobile devices has been released in 2008, and so on.

development of AR systems has begun quite later, since the 1980s.

As mentioned earlier, the AR and/or MR technology is the next generation of VR technology, and it expected to be the direction of the 3D display development in the near future. AR system has a lot more of functions than VR systems that it has been defined as a wearable computer, and the hardware structure is much complicated. Due to the multifunctions, the AR technology requires several electronic and optical devices such as microcomputer, projector, microphone, speaker, and in- and out-couplers; so, if the typical HMD structure is utilized, the system will become very bulky, and it is inconvenient to wear it. Therefore, the miniaturized and light-weighted hardware structure is necessary.

Recently released complete AR technologies such as Google Glass, Epson Moverio, Sony SmartEyeglass, and Microsoft HoloLens use the optical waveguide structure. The waveguide structure makes the AR systems small and light-weighted, and it is the biggest advantage for the users to use an ad hoc or day-to-day basis. This section mainly discusses about the principle of the optical waveguide and the various waveguide for the AR application-based HMD devices.

#### **2.1. Overview of the conventional optical waveguide**

The waveguide structure is widely applied in various fields such as electromagnetics, optics, and sound synthesis [6]. It guides the electromagnetic or sound waves through the fibers and pipes. In the optics, it is used as a transmitter that transmits the input signal, the light wave, between two different fields by guiding the waves, without any loss of input information. **Figure 3** shows the basic illustration of the optical waveguide where *n*<sup>1</sup> and *n*<sup>2</sup> are the refractive

waves and semi-reflective mirror in the front of an eye. When the images are displayed by the micro display, magnified by the collimating lens, the collimated light waves are transmitted to the semi-reflective mirror through the waveguide. Finally, the human eye sees the images reflected on the semi-reflective mirror and the real-world environment at the same time. Here, the light waves are guided through the pipe without any loss or degradation, and the images are reflected on the semi-reflective mirror without color nonuniformity. Also, the system provides the high optical efficiency and low cost. The main disadvantage of this structure is that field of view is proportional to the size of the reflector; therefore, in order to increase the field of view, the reflector should be larger, the waveguide should be thick, and, accordingly, the device becomes bulky. The thick waveguide affects to the image quality that the image can be observed as highly distorted. Google Glass and Epson's commercialized AR device "Moverio" use the reflective-type waveguide structure. Note that Google Glass does not use the total internal reflection phenomena, while Moverio uses it. The schemes of the Google Glass and Moverio are illustrated in **Figure 4**. In the case of Google Glass, some light waves are transmitted across through the reflector, then come back from the opposite side, and reflect again to the

Waveguide-Type Head-Mounted Display System for AR Application

http://dx.doi.org/10.5772/intechopen.75172

47

The polarized waveguide requires multiple layers of coating and polarized reflectors. The polarized reflectors should be parallelized and polished in order to guide the light waves. Some of them are stuck with each other, and each reflector needs to be coated by different amounts of coating. Although this structure has large field of view and large eye motion box, however, it has many drawbacks such as high cost due to the coating, parallelism, low optical efficiency, only up to 30%, color nonuniformity, and too much loss of the incident light waves.

The diffractive waveguide is one of the most widely used structures in the see-through AR displays. In this case, the incident light waves flow into the waveguide with certain angle by collimating by the first slanted gratings, in-coupler, pass through the waveguide, and extract to the exit pupil via the second slanted gratings, out-coupler. The most important parts of this structure, in- and out-couplers, are produced by the diffractive optical element (DOE) which has the deep slanted nanometric gratings. Currently, the DOE with deep slanted nanometric gratings can be fabricated cheaply anywhere through the common optical component manufacturing;

**Figure 4.** The structure of the AR systems (a) Google Glass [Source: https://www.pinterest.com/pin/180355160053768227/]

and (b) Moveiro [Source: http://global.epson.com/innovation/engineer/moverio.html].

exit pupil. So, it causes the light intensity loss issue.

Lumus uses this type of waveguide in their see-through AR products.

**Figure 3.** The basic structure of the optical waveguide.

indices of both mediums, the core (transmitter) and cladding (reflector), where *n*<sup>2</sup> is higher than *n*<sup>1</sup> in the usual cases. The optical waveguides are categorized in several types according to the geometrical structure, the allowance of the number of transmitting signals, refractive index distribution, material, and application.

The optical fiber is a typical example of the optical waveguide where the light waves are guided through the optical fiber according to the total internal reflection theory. The optical fiber consists of the basic layer of optical waveguide, the core, and the cladding with different refractive indices. Here, the light waves are transmitted through the core with higher refractive index by iteratively reflecting on the wall of the fiber, and the cladding which covered the core. The light waves are passed through the optical fiber without any loss in light quantity and emitted to the target or receiver which is located at the output field.

Also, the prism can be another example of the optical waveguide. Unlike the optical fiber, it transmits the incident light through the body, and the wavelengths of the output light wave leave the prism at different angles to each other, due to the dispersion phenomena caused by the refractive index of the material of prism that it breaks the input light into the complete spectrum of colors according to the wavelength. So, the output light appears as a rainbow.

#### **2.2. Waveguide in the AR-based HMD**

In order to implement the virtual components mixed with the real-world environment, it is necessary to design the optical system considering the pupil size, virtual image, optical distance to the eye (eye relief), image magnification, and field of view. Since the optical elements such as mirror and beam splitter are utilized, the overall device becomes bulky and heavy, and it is difficult to mount various elements in a narrow space.

Most of the AR-based HMDs use the waveguide structure in order to reduce the overall size and weight [7]. Actually, some AR devices use the curved mirror structure using the semireflective curved mirror as an image reflector in front of the eyes where the projector shoots the image to the curved mirror. However, the displayed image has low resolution and is highly distorted.

Several types of waveguide method are utilized in the modern HMD systems for AR application: the reflective-type, polarized, diffractive, and holographic waveguides. The reflective-type waveguide structure uses the molded plastic substrate as guide of the light waves and semi-reflective mirror in the front of an eye. When the images are displayed by the micro display, magnified by the collimating lens, the collimated light waves are transmitted to the semi-reflective mirror through the waveguide. Finally, the human eye sees the images reflected on the semi-reflective mirror and the real-world environment at the same time. Here, the light waves are guided through the pipe without any loss or degradation, and the images are reflected on the semi-reflective mirror without color nonuniformity. Also, the system provides the high optical efficiency and low cost. The main disadvantage of this structure is that field of view is proportional to the size of the reflector; therefore, in order to increase the field of view, the reflector should be larger, the waveguide should be thick, and, accordingly, the device becomes bulky. The thick waveguide affects to the image quality that the image can be observed as highly distorted. Google Glass and Epson's commercialized AR device "Moverio" use the reflective-type waveguide structure. Note that Google Glass does not use the total internal reflection phenomena, while Moverio uses it. The schemes of the Google Glass and Moverio are illustrated in **Figure 4**. In the case of Google Glass, some light waves are transmitted across through the reflector, then come back from the opposite side, and reflect again to the exit pupil. So, it causes the light intensity loss issue.

indices of both mediums, the core (transmitter) and cladding (reflector), where *n*<sup>2</sup>

and emitted to the target or receiver which is located at the output field.

and it is difficult to mount various elements in a narrow space.

 in the usual cases. The optical waveguides are categorized in several types according to the geometrical structure, the allowance of the number of transmitting signals, refractive

The optical fiber is a typical example of the optical waveguide where the light waves are guided through the optical fiber according to the total internal reflection theory. The optical fiber consists of the basic layer of optical waveguide, the core, and the cladding with different refractive indices. Here, the light waves are transmitted through the core with higher refractive index by iteratively reflecting on the wall of the fiber, and the cladding which covered the core. The light waves are passed through the optical fiber without any loss in light quantity

Also, the prism can be another example of the optical waveguide. Unlike the optical fiber, it transmits the incident light through the body, and the wavelengths of the output light wave leave the prism at different angles to each other, due to the dispersion phenomena caused by the refractive index of the material of prism that it breaks the input light into the complete spectrum of colors according to the wavelength. So, the output light appears as a rainbow.

In order to implement the virtual components mixed with the real-world environment, it is necessary to design the optical system considering the pupil size, virtual image, optical distance to the eye (eye relief), image magnification, and field of view. Since the optical elements such as mirror and beam splitter are utilized, the overall device becomes bulky and heavy,

Most of the AR-based HMDs use the waveguide structure in order to reduce the overall size and weight [7]. Actually, some AR devices use the curved mirror structure using the semireflective curved mirror as an image reflector in front of the eyes where the projector shoots the image to the curved mirror. However, the displayed image has low resolution and is

Several types of waveguide method are utilized in the modern HMD systems for AR application: the reflective-type, polarized, diffractive, and holographic waveguides. The reflective-type waveguide structure uses the molded plastic substrate as guide of the light

than *n*<sup>1</sup>

index distribution, material, and application.

**Figure 3.** The basic structure of the optical waveguide.

46 State of the Art Virtual Reality and Augmented Reality Knowhow

**2.2. Waveguide in the AR-based HMD**

highly distorted.

is higher

The polarized waveguide requires multiple layers of coating and polarized reflectors. The polarized reflectors should be parallelized and polished in order to guide the light waves. Some of them are stuck with each other, and each reflector needs to be coated by different amounts of coating. Although this structure has large field of view and large eye motion box, however, it has many drawbacks such as high cost due to the coating, parallelism, low optical efficiency, only up to 30%, color nonuniformity, and too much loss of the incident light waves. Lumus uses this type of waveguide in their see-through AR products.

The diffractive waveguide is one of the most widely used structures in the see-through AR displays. In this case, the incident light waves flow into the waveguide with certain angle by collimating by the first slanted gratings, in-coupler, pass through the waveguide, and extract to the exit pupil via the second slanted gratings, out-coupler. The most important parts of this structure, in- and out-couplers, are produced by the diffractive optical element (DOE) which has the deep slanted nanometric gratings. Currently, the DOE with deep slanted nanometric gratings can be fabricated cheaply anywhere through the common optical component manufacturing;

**Figure 4.** The structure of the AR systems (a) Google Glass [Source: https://www.pinterest.com/pin/180355160053768227/] and (b) Moveiro [Source: http://global.epson.com/innovation/engineer/moverio.html].

waveguide-based AR display, the light waves from the micro display are reflected on the in-coupler HMD with the incident angle, traveled within the waveguide surface, and out-coupler HOE turns the light back toward the eye of the user. Although, intrinsically, most of the holographic waveguides have few drawbacks such as the limited field of view and rainbow effect, due to the limited incident angle of HOE and three holographic volume gratings (HVGs) with different

Waveguide-Type Head-Mounted Display System for AR Application

http://dx.doi.org/10.5772/intechopen.75172

49

When the HOE is applied to the HMD system for AR application, the number of optical elements is remarkably reduced, and the overall device becomes considerably lighter than the conventional HMD systems. The biggest advantage of the holographic waveguide is that it does not require the glass substrate where other types of waveguide-based AR displays use the glass substrates in the in- and out-couplers, as shown in **Figure 6**. Here, *α* is the diffractive angle of the light waves which will be guided through the waveguide between in- and outcouplers, and it should be larger than the critical angle, because light waves are propagated

The HOE is an optical element (film) which obtains the desired wave front by recording a hologram through the analogical holographic recording process. HOE can be a lens or lens array, mirror, prism, and so on; so, currently, many conventional optical elements and devices are replaced by the HOEs [9]. The development of hologram recording material is of utmost importance for developing the HOE-based applications such as a see-through AR display.

The principle of the HOE depends on the hologram recording principle. By the use of the coherent illumination such as a laser, the light waves reflected from the object, the object beam, collide with the light waves coming from another direction, the reference beam, on the holographic material. When the object and reference beams are met on the holographic material, the interference pattern consisting of many bright and dark lines occurred according to the phase difference of the reflected object waves from each part of the object. The interference pattern contains both amplitude and phase information of the object. When the reference laser beam illuminated the recorded hologram, the virtual 3D image which is visualized very close

**Figure 6.** The schematic diagram of the holographic-type see-through HMD for AR application.

wavelengths, it is much an advantageous structure especially for the AR displays.

by total internal reflection.

to the original object is reconstructed.

**Figure 5.** (a) The illustration of the diffraction waveguide [source: https://uploadvr.com/waveguides-smartglasses] and the actual representation of (b) Nokia Vuzix [source: https://www.phonearena.com/ news/Vuzix-Blade-and-M300 hands-on\_id101589] and (c) Microsoft HoloLens [source: http://www.briteliteimmersive.com/blog/hololens-leadingthe-vr-to-ar-revolution].

accordingly, the price of AR products using DOE with deep slanted nanometric gratings is also low. However, when the incident light waves are traveled between the in-coupler and out-coupler through the waveguide, the output image is observed with the rainbow effect, because the wavelengths of the reflected light waves have the different amplitudes when they come across the diffraction pattern at the incidence angle. Therefore, it is quite difficult to be applied in the full color applications, and the monochromatic performance-based application is recommended. Also, the size of the display screen cannot be large enough, because of the variation of spectral reflectivity; accordingly, the field of view is quite narrow, averagely 20°. In order to increase the field of view, the incidence angle should be higher, but the color nonuniformity, the rainbow effect, is observed more severely to the user, due to the higher incidence angle. Currently, the diffractive waveguide structure is applied in the see-through AR-systems "Vuzix" manufactured by Nokia and Microsoft HoloLens. **Figure 5** shows the schematic illustration of the diffraction waveguide structure and actual representation of Vuzix and HoloLens AR systems.

#### **3. Holographic waveguide for AR-based HMD**

#### **3.1. Principle of HOE**

The holographic waveguide structure is almost similar with the diffraction waveguide, and the holographic optical element (HOE) is applied as the in- and out-couplers, instead of DOEs. The HOEs reflect the monochromatic (single wavelength) or polychromatic (three wavelengths) light waves, according to the fabrication method. The HOE is fabricated in the analogical hologram recording process with the incident angle using the laser illumination [8]. In the holographic waveguide-based AR display, the light waves from the micro display are reflected on the in-coupler HMD with the incident angle, traveled within the waveguide surface, and out-coupler HOE turns the light back toward the eye of the user. Although, intrinsically, most of the holographic waveguides have few drawbacks such as the limited field of view and rainbow effect, due to the limited incident angle of HOE and three holographic volume gratings (HVGs) with different wavelengths, it is much an advantageous structure especially for the AR displays.

When the HOE is applied to the HMD system for AR application, the number of optical elements is remarkably reduced, and the overall device becomes considerably lighter than the conventional HMD systems. The biggest advantage of the holographic waveguide is that it does not require the glass substrate where other types of waveguide-based AR displays use the glass substrates in the in- and out-couplers, as shown in **Figure 6**. Here, *α* is the diffractive angle of the light waves which will be guided through the waveguide between in- and outcouplers, and it should be larger than the critical angle, because light waves are propagated by total internal reflection.

The HOE is an optical element (film) which obtains the desired wave front by recording a hologram through the analogical holographic recording process. HOE can be a lens or lens array, mirror, prism, and so on; so, currently, many conventional optical elements and devices are replaced by the HOEs [9]. The development of hologram recording material is of utmost importance for developing the HOE-based applications such as a see-through AR display.

The principle of the HOE depends on the hologram recording principle. By the use of the coherent illumination such as a laser, the light waves reflected from the object, the object beam, collide with the light waves coming from another direction, the reference beam, on the holographic material. When the object and reference beams are met on the holographic material, the interference pattern consisting of many bright and dark lines occurred according to the phase difference of the reflected object waves from each part of the object. The interference pattern contains both amplitude and phase information of the object. When the reference laser beam illuminated the recorded hologram, the virtual 3D image which is visualized very close to the original object is reconstructed.

accordingly, the price of AR products using DOE with deep slanted nanometric gratings is also low. However, when the incident light waves are traveled between the in-coupler and out-coupler through the waveguide, the output image is observed with the rainbow effect, because the wavelengths of the reflected light waves have the different amplitudes when they come across the diffraction pattern at the incidence angle. Therefore, it is quite difficult to be applied in the full color applications, and the monochromatic performance-based application is recommended. Also, the size of the display screen cannot be large enough, because of the variation of spectral reflectivity; accordingly, the field of view is quite narrow, averagely 20°. In order to increase the field of view, the incidence angle should be higher, but the color nonuniformity, the rainbow effect, is observed more severely to the user, due to the higher incidence angle. Currently, the diffractive waveguide structure is applied in the see-through AR-systems "Vuzix" manufactured by Nokia and Microsoft HoloLens. **Figure 5** shows the schematic illustration of the diffraction

**Figure 5.** (a) The illustration of the diffraction waveguide [source: https://uploadvr.com/waveguides-smartglasses] and the actual representation of (b) Nokia Vuzix [source: https://www.phonearena.com/ news/Vuzix-Blade-and-M300 hands-on\_id101589] and (c) Microsoft HoloLens [source: http://www.briteliteimmersive.com/blog/hololens-leading-

waveguide structure and actual representation of Vuzix and HoloLens AR systems.

The holographic waveguide structure is almost similar with the diffraction waveguide, and the holographic optical element (HOE) is applied as the in- and out-couplers, instead of DOEs. The HOEs reflect the monochromatic (single wavelength) or polychromatic (three wavelengths) light waves, according to the fabrication method. The HOE is fabricated in the analogical hologram recording process with the incident angle using the laser illumination [8]. In the holographic

**3. Holographic waveguide for AR-based HMD**

**3.1. Principle of HOE**

the-vr-to-ar-revolution].

48 State of the Art Virtual Reality and Augmented Reality Knowhow

**Figure 6.** The schematic diagram of the holographic-type see-through HMD for AR application.

The HOEs are classified into the reflection and transmission types, depending on the reconstruction method. The transmission-type HOE transmits the beam through the hologram during the reconstruction. In the reflection-type HOE, the reference beam reflects on the HOE which causes the direction of the object and reference beams to cross with each other in the opposite direction of the recording medium during the hologram recording, as shown in **Figure 7(a)**. The reflection-type HOE transmits the imaging light to the other field by reflecting the band of specific wavelength of light waves; therefore, it is advantageous to apply the reflection-type HOEs in the see-through AR display systems.

optical power can be diffracted into a waveguide compared to incident power onto the DOEs or HOEs. In the holographic waveguide structure, the diffraction efficiency of the HOE has a great influence on the final displayed image quality; so, the diffraction efficiency must be well

Recently, Piao et al. fully analyzed and measured the diffraction efficiency of the reflectiontype photopolymer intrinsically, for the monochromatic and polychromatic cases [9]. In order to measure the diffraction efficiency of the HOE, i.e., the photopolymer, two photodetectors connected to the power meter are used to detect the diffraction and transmission beams, respectively, where the incident angle is set to 45°, in order to prevent the occurrence of the multi-spot by total internal reflection. As the principle of the holographic recording, the HVGs are recorded onto the HOE via laser illumination. Note that most of the laser illuminations have fixed wavelengths, 633 nm for red, 532 nm for green, and 473 nm for blue-colored lasers. **Figure 8(a)** shows the diffraction efficiency according to the exposure energy. Here, the dif-

. Here, the optimum case of diffraction efficiency is measured that maximum value

**Figure 8(b)** shows the diffraction efficiency analysis for the monochromatic cases, for red, green, and blue laser illuminations, respectively, according to the incident angle where it is set

more than 95% for the incident angle between 30 and 55°. For the time duration, the diffraction efficiency is almost saturated with more than 95% after 10 seconds, as illustrated in **Figure 8(c)**. Basically, the diffraction efficiency of the HOE according to the angular selectivity is calculated

, where *<sup>ν</sup>* <sup>=</sup> \_\_\_\_\_\_\_\_\_\_\_ *<sup>d</sup>* <sup>⋅</sup> <sup>Δ</sup>*<sup>n</sup> <sup>λ</sup>* <sup>√</sup>

where Δ*n* and *d* are the refractive index modulation and thickness of HOE, *λ* is the wavelength of laser illumination, *θO* and *θR* are the incident angle of object and reference beams, *K*

**Figure 8.** The diffraction efficiency according to (a) the exposure energy, (b) incident angle, and (c) exposure time.

\_\_\_\_\_\_\_\_\_\_\_\_ cos*<sup>θ</sup> <sup>R</sup>* cos*θ<sup>O</sup>*

(red and green lasers) and 97% at 200 mJ/cm<sup>2</sup>

Waveguide-Type Head-Mounted Display System for AR Application

http://dx.doi.org/10.5772/intechopen.75172

for blue laser, the diffraction efficiency is measured approximately

and *<sup>ξ</sup>* <sup>=</sup> *Kd* <sup>⋅</sup> <sup>Δ</sup>*<sup>θ</sup>* cos(*<sup>ϕ</sup>* <sup>−</sup> *<sup>θ</sup>B*)

 \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ 2 cos*θ<sup>O</sup>*

at an interval of

(blue laser).

51

for red and

(1)

fraction efficiency is analyzed when the exposure energy is set to 700 mJ/cm<sup>2</sup>

between 10 and 70° at 5° interval. When the exposure energies are set as 150 mJ/cm<sup>2</sup>

calculated and measured.

is approximately 97% at 150 mJ/cm<sup>2</sup>

\_\_\_\_\_ *ν* <sup>2</sup> − *ξ*2) 2 \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

green lasers and 200 mJ/cm<sup>2</sup>

*<sup>η</sup>* <sup>=</sup> (*sh* <sup>√</sup>

(*sh* √

\_\_\_\_\_ *ν* <sup>2</sup> − *ξ*2) 2 + (1 − (\_\_ *ξ ν*) 2 )

100 mJ/cm<sup>2</sup>

by

Usually, the holograms are recorded on the reflective-type HOEs with an asymmetrical grating structure, as illustrated in **Figure 7(b)**. Here, the incident beam is perpendicular to the HOE, in order to reduce the loss of light wave which can be caused by the surface reflection. The object beam is refracted at *β* angle during the pass through the material due to the difference of refractive indices between air and HOE material.

The most commonly used HOEs are the dichromatic gelatin, silver halide, photoresist, photopolymer, etc. Among them, the photopolymer has a high diffraction efficiency in hologram recording due to a change of the refractive index according to the intensity of illumination. Also, it has the merit of recording the hologram easily by only drying treatment, without requiring the chemical treatment, and it has advantages of high reliability and high resolution. Currently, the photopolymer is widely applied to HOE-based applications, optical filters, holographic memories, and many more.

#### **3.2. Optical characteristics of the HOE**

In the see-through AR display systems, the optical characteristics of HOEs, in- and out-couplers, have a significant role. Here, we mainly discussed the optical characteristics of the photopolymer such that it has been recognized as the most effective HOE material especially in the application of holographic waveguides for see-through AR displays.

The diffraction efficiency is the performance of the power throughput of DOEs or HOEs and is the main characteristic of the reflection-type HOE materials. In order to apply the HOEs in the see-through AR displays, the diffraction efficiency should be analyzed on how much

**Figure 7.** The basic scheme of (a) the hologram recording and (b) optical reconstruction processes.

optical power can be diffracted into a waveguide compared to incident power onto the DOEs or HOEs. In the holographic waveguide structure, the diffraction efficiency of the HOE has a great influence on the final displayed image quality; so, the diffraction efficiency must be well calculated and measured.

The HOEs are classified into the reflection and transmission types, depending on the reconstruction method. The transmission-type HOE transmits the beam through the hologram during the reconstruction. In the reflection-type HOE, the reference beam reflects on the HOE which causes the direction of the object and reference beams to cross with each other in the opposite direction of the recording medium during the hologram recording, as shown in **Figure 7(a)**. The reflection-type HOE transmits the imaging light to the other field by reflecting the band of specific wavelength of light waves; therefore, it is advantageous to apply the

Usually, the holograms are recorded on the reflective-type HOEs with an asymmetrical grating structure, as illustrated in **Figure 7(b)**. Here, the incident beam is perpendicular to the HOE, in order to reduce the loss of light wave which can be caused by the surface reflection. The object beam is refracted at *β* angle during the pass through the material due to the differ-

The most commonly used HOEs are the dichromatic gelatin, silver halide, photoresist, photopolymer, etc. Among them, the photopolymer has a high diffraction efficiency in hologram recording due to a change of the refractive index according to the intensity of illumination. Also, it has the merit of recording the hologram easily by only drying treatment, without requiring the chemical treatment, and it has advantages of high reliability and high resolution. Currently, the photopolymer is widely applied to HOE-based applications, optical filters,

In the see-through AR display systems, the optical characteristics of HOEs, in- and out-couplers, have a significant role. Here, we mainly discussed the optical characteristics of the photopolymer such that it has been recognized as the most effective HOE material especially in

The diffraction efficiency is the performance of the power throughput of DOEs or HOEs and is the main characteristic of the reflection-type HOE materials. In order to apply the HOEs in the see-through AR displays, the diffraction efficiency should be analyzed on how much

the application of holographic waveguides for see-through AR displays.

**Figure 7.** The basic scheme of (a) the hologram recording and (b) optical reconstruction processes.

reflection-type HOEs in the see-through AR display systems.

50 State of the Art Virtual Reality and Augmented Reality Knowhow

ence of refractive indices between air and HOE material.

holographic memories, and many more.

**3.2. Optical characteristics of the HOE**

Recently, Piao et al. fully analyzed and measured the diffraction efficiency of the reflectiontype photopolymer intrinsically, for the monochromatic and polychromatic cases [9]. In order to measure the diffraction efficiency of the HOE, i.e., the photopolymer, two photodetectors connected to the power meter are used to detect the diffraction and transmission beams, respectively, where the incident angle is set to 45°, in order to prevent the occurrence of the multi-spot by total internal reflection. As the principle of the holographic recording, the HVGs are recorded onto the HOE via laser illumination. Note that most of the laser illuminations have fixed wavelengths, 633 nm for red, 532 nm for green, and 473 nm for blue-colored lasers. **Figure 8(a)** shows the diffraction efficiency according to the exposure energy. Here, the diffraction efficiency is analyzed when the exposure energy is set to 700 mJ/cm<sup>2</sup> at an interval of 100 mJ/cm<sup>2</sup> . Here, the optimum case of diffraction efficiency is measured that maximum value is approximately 97% at 150 mJ/cm<sup>2</sup> (red and green lasers) and 97% at 200 mJ/cm<sup>2</sup> (blue laser). **Figure 8(b)** shows the diffraction efficiency analysis for the monochromatic cases, for red, green, and blue laser illuminations, respectively, according to the incident angle where it is set between 10 and 70° at 5° interval. When the exposure energies are set as 150 mJ/cm<sup>2</sup> for red and green lasers and 200 mJ/cm<sup>2</sup> for blue laser, the diffraction efficiency is measured approximately more than 95% for the incident angle between 30 and 55°. For the time duration, the diffraction efficiency is almost saturated with more than 95% after 10 seconds, as illustrated in **Figure 8(c)**.

Basically, the diffraction efficiency of the HOE according to the angular selectivity is calculated by

$$\eta = \frac{\left(sh\sqrt{\nu^2 - \xi^2}\right)^2}{\left(sh\sqrt{\nu^2 - \xi^2}\right)^2 + \left(1 - \left(\frac{\xi}{\nu}\right)^2\right)}\quad\text{where}\quad\nu = \frac{\pi d \cdot \Delta n}{\lambda\sqrt{\cos\theta\_\lambda \cos\theta\_o}}\text{ and }\ \xi = \frac{\text{Kd} \cdot \Delta \theta \cos\left(\theta - \theta\_o\right)}{2\cos\theta\_o}\tag{1}$$

where Δ*n* and *d* are the refractive index modulation and thickness of HOE, *λ* is the wavelength of laser illumination, *θO* and *θR* are the incident angle of object and reference beams, *K*

**Figure 8.** The diffraction efficiency according to (a) the exposure energy, (b) incident angle, and (c) exposure time.

is the magnitude of grating vector, Δ*θ* is the angular deviation, *ϕ* is the angle of inside grating, and *θB* is the Bragg angle. **Figure 9** shows the diffraction efficiencies of actual and calculated case, according to the angular selectivity. Here, it can be seen that the photopolymer provides wide angular selectivity.

But in the actual measurement of photopolymer, the diffraction efficiencies of monochromatic cases were measured as 55% for red, 54% for green, and 50% for blue laser, due to the loss of the light waves during the passing through the prism. By considering the absorption factor for each laser that 15, 25, and 28% for red, green, and blue lasers, respectively, the photopolymer provides high diffraction efficiency, similar like the analyzed result. **Figure 10(a)** shows the actual appearance of HOEs of in- and out-couplers which are the photopolymers and the displayed image from the micro display, and **Figure 10(b)** shows the final guided images in each monochromatic case of red, green, and blue lasers.

In order to display the full-color image in the holographic waveguide-type AR displays, the three HVGs are recorded on the HOE film by red, green, and blue lasers simultaneously. But its much low diffraction efficiency is measured when the three HVGs are recorded simply through red, green, and blue lasers simultaneously that only 16%, 20%, and 10%, respectively, for each wavelength, so the HOE is useless in the see-though holographic waveguide-type AR display. In order to obtain the higher diffraction efficiency for full-color application in a holographic waveguide, a composite structure of HVG recording should be configured. Here, two-layered photopolymers are laminated such that the first photopolymer for red laser and second HOE for green and blue lasers are combined, as shown in **Figure 11**. When the fabricated HOE film is applied as in- and out-couplers, the diffraction efficiency is measured approximately 40, 44, and 42%, respectively, and it does not seem to be a problem when applied in the see-through AR displays. The example of the full-color guided images within the holographic waveguide is also presented in **Figure 11**.

Due to the high diffraction efficiency and light weight of the holographic waveguide structure, currently, it is applied in several see-through AR display systems such as Sony SmartEyeglass, BAE Systems AR glasses, DigiLens's HUD systems, etc. Also, Microsoft Research uses a holographic waveguide in their new research of near-eye see-through holographic display.

**Figure 10.** (a) The actual appearance of the fabricated in- and out-couplers of holographic waveguide and the originally

displayed image and (b) the guided images in the monochromatic case: red, green, and blue.

Actually, the composite structure for full-color HOE cannot solve the issues of holographic waveguide such as the thickness, weight, color nonuniformity, and field of view, fully. Then, the wedge-shaped structure of holographic waveguide has a great advantage that obtains the improved field of view and color uniformity through the HOE with high diffraction efficiency [10]. **Figure 12(a)** shows the schematic configuration of wedge-shaped structure of holographic waveguide. Here, the in- and out-couplers of HOE films are located at the certain angles; therefore, the thickness of the waveguide can be reduced by a large angle of total internal reflection, and field of view can be widened according to the wide angular selectivity of the HVGs. For the angle of the light path inside the wedge-shaped holographic waveguide, as shown in **Figure 12(b)**, the sum of the incident angles on the outside of the HOE films, *θ*<sup>1</sup>

where *n* is the average refractive index of the HOE (1.58 for photopolymer), *φ*, is the grating

is the incident angle of the inside the HOE.

. Here, the spatial frequency of the in- and out-coupler HVGs should be

, where the slope angle of wedge,

*<sup>λ</sup>* (2)

Waveguide-Type Head-Mounted Display System for AR Application

http://dx.doi.org/10.5772/intechopen.75172

53

**3.3. Wedge-shaped holographic waveguide**

, are identical to the angle of total internal reflection, *θ<sup>t</sup>*

*<sup>f</sup>* <sup>=</sup> <sup>2</sup>*<sup>n</sup>* cos(*<sup>φ</sup>* <sup>−</sup> *<sup>θ</sup>*0) \_\_\_\_\_\_\_\_\_\_

and *θ*<sup>2</sup>

, is same with *θ*<sup>2</sup>

slated angle and *θ*<sup>0</sup>

considered, and it can be calculated as

*θt*

Additionally, if three-layered photopolymer is utilized, most exterior layer loses too much of the light beams. When three photopolymers are located in order for red, green, and blue lasers, the diffraction efficiencies are only 10, 23, and 30% that is impossible to be applied.

**Figure 9.** The normalized diffraction efficiency according to the Bragg angle for (a) red, (b) green, and (c) blue laser illuminations.

Waveguide-Type Head-Mounted Display System for AR Application http://dx.doi.org/10.5772/intechopen.75172 53

**Figure 10.** (a) The actual appearance of the fabricated in- and out-couplers of holographic waveguide and the originally displayed image and (b) the guided images in the monochromatic case: red, green, and blue.

Due to the high diffraction efficiency and light weight of the holographic waveguide structure, currently, it is applied in several see-through AR display systems such as Sony SmartEyeglass, BAE Systems AR glasses, DigiLens's HUD systems, etc. Also, Microsoft Research uses a holographic waveguide in their new research of near-eye see-through holographic display.

#### **3.3. Wedge-shaped holographic waveguide**

is the magnitude of grating vector, Δ*θ* is the angular deviation, *ϕ* is the angle of inside grating, and *θB* is the Bragg angle. **Figure 9** shows the diffraction efficiencies of actual and calculated case, according to the angular selectivity. Here, it can be seen that the photopolymer provides

But in the actual measurement of photopolymer, the diffraction efficiencies of monochromatic cases were measured as 55% for red, 54% for green, and 50% for blue laser, due to the loss of the light waves during the passing through the prism. By considering the absorption factor for each laser that 15, 25, and 28% for red, green, and blue lasers, respectively, the photopolymer provides high diffraction efficiency, similar like the analyzed result. **Figure 10(a)** shows the actual appearance of HOEs of in- and out-couplers which are the photopolymers and the displayed image from the micro display, and **Figure 10(b)** shows the final guided images in

In order to display the full-color image in the holographic waveguide-type AR displays, the three HVGs are recorded on the HOE film by red, green, and blue lasers simultaneously. But its much low diffraction efficiency is measured when the three HVGs are recorded simply through red, green, and blue lasers simultaneously that only 16%, 20%, and 10%, respectively, for each wavelength, so the HOE is useless in the see-though holographic waveguide-type AR display. In order to obtain the higher diffraction efficiency for full-color application in a holographic waveguide, a composite structure of HVG recording should be configured. Here, two-layered photopolymers are laminated such that the first photopolymer for red laser and second HOE for green and blue lasers are combined, as shown in **Figure 11**. When the fabricated HOE film is applied as in- and out-couplers, the diffraction efficiency is measured approximately 40, 44, and 42%, respectively, and it does not seem to be a problem when applied in the see-through AR displays. The example of the full-color guided images within the holographic waveguide is also presented in

Additionally, if three-layered photopolymer is utilized, most exterior layer loses too much of the light beams. When three photopolymers are located in order for red, green, and blue lasers, the diffraction efficiencies are only 10, 23, and 30% that is impossible to be applied.

**Figure 9.** The normalized diffraction efficiency according to the Bragg angle for (a) red, (b) green, and (c) blue laser

wide angular selectivity.

**Figure 11**.

illuminations.

each monochromatic case of red, green, and blue lasers.

52 State of the Art Virtual Reality and Augmented Reality Knowhow

Actually, the composite structure for full-color HOE cannot solve the issues of holographic waveguide such as the thickness, weight, color nonuniformity, and field of view, fully. Then, the wedge-shaped structure of holographic waveguide has a great advantage that obtains the improved field of view and color uniformity through the HOE with high diffraction efficiency [10]. **Figure 12(a)** shows the schematic configuration of wedge-shaped structure of holographic waveguide. Here, the in- and out-couplers of HOE films are located at the certain angles; therefore, the thickness of the waveguide can be reduced by a large angle of total internal reflection, and field of view can be widened according to the wide angular selectivity of the HVGs. For the angle of the light path inside the wedge-shaped holographic waveguide, as shown in **Figure 12(b)**, the sum of the incident angles on the outside of the HOE films, *θ*<sup>1</sup> and *θ*<sup>2</sup> , are identical to the angle of total internal reflection, *θ<sup>t</sup>* , where the slope angle of wedge, *θt* , is same with *θ*<sup>2</sup> . Here, the spatial frequency of the in- and out-coupler HVGs should be considered, and it can be calculated as

$$f = \frac{2n\cos(\phi - \theta\_o)}{\lambda} \tag{2}$$

where *n* is the average refractive index of the HOE (1.58 for photopolymer), *φ*, is the grating slated angle and *θ*<sup>0</sup> is the incident angle of the inside the HOE.

**Figure 11.** The schematic configuration of a composite structure and the example of the guided full-color image.

According to the above calculation, when *θw* is designed by 30° (angle of total internal reflection should be larger than the critical angle), *θ*<sup>1</sup> can be 20, 30, 40, or 50°; the total internal reflection occurred at all angles of incidence *θ*<sup>1</sup> ; and the spatial frequencies are corresponded to each incident angle as 5930, 5617, 5931, and 5904 lines/mm, respectively. When considering the photopolymer as a HOE that the thickness is 16.8 µm, the angular selectivity for HVG recordings become larger at higher spatial frequencies; but, the wavelength selectivity exhibits the similar tendency, as shown in **Figure 13(a)** and **(b)**. So, it can be seen that the recording incident angle at 30/20 and 30/40° on the outside of HOE achieves very wide angular selectivity and narrow spectral selectivity. Therefore, it is suitable for the incident angle of HVGs that are set as *θ*<sup>1</sup> = 40 and *θ*<sup>2</sup> = 30°, because larger angle of total internal reflection leads to a thinner holographic waveguide.

For the field of view, it can be calculated as

$$\theta\_{\rm tOV} = \arcsin\left(\mu \sin\left(\phi - \arccos\left(\frac{\lambda\_i \cos\left(\phi - \theta\_o\right)}{\lambda\_o}\right)\right)\right) \tag{3}$$

where *λ*<sup>0</sup>

and *λ*<sup>1</sup>

of each incident angle.

are the wavelengths of the recording and readout beams. According to Eq. (3),

Waveguide-Type Head-Mounted Display System for AR Application

http://dx.doi.org/10.5772/intechopen.75172

55

the field of view can be analyzed for the different wavelengths of illumination that between

**Figure 13.** Analysis for (a) the angular selectivity and (b) the spectral selectivity, for the corresponding spatial frequencies

The wedge-shaped holographic waveguide structure requires the thin films for in- and outcouplers, so it is necessary to record the full-color HVGs onto a single HOE film. As mentioned earlier, the simple recording method using three lasers is difficult to apply because of very low diffraction efficiency. Therefore, the time scheduling method must be utilized that the shutters control the exposure duration of each laser illumination. Here, when the HVG is recorded onto the HOE film by any laser illumination, other two lasers are blocked by the corresponding shutters. The saturation grating for each HVG should be within 5 seconds, because it will be difficult to form the grating on the HOE film after 5 seconds. The biggest advantage of this recording method is that the full-color HVG can be recorded onto a single HOE film with good uniformity of color and short recording time, and the maximum unifor-

But in the actual case, especially for the photopolymer, it has the absorption and transmittance for each color of laser. Piao et al. measured the diffraction efficiencies of full-color single HOE in the six different cases for the sequence of three laser illuminations: red-green-blue, red-blue-green, green-red-blue, green-blue-red, blue-red-green, and blue-green-red. The optimum case is green-blue-red sequence, because this case exhibits much higher optical efficiency and more uniform distribution than other sequential recording processes and has a high potential for obtaining high image quality that the diffraction efficiencies are measured as 49, 47, and 44% for each wavelength. **Figure 14** shows the color analysis for the different sequential exposures. Here, it can be seen that the green-blue-red sequence has successful

When the HVGs are recorded onto the photopolymer films by green-blue-red sequential beam exposure, and applied into the see-through AR display system, the guided image

±17° in 430–500 and 580–670 nm wavelengths and ±15° in 500–560 nm.

mity can be with only 10% loss in diffraction efficiency.

higher color uniformity than other cases.

**Figure 12.** The schematic configuration of a wedge-shaped holographic waveguide structure and (b) angle of the light path inside the waveguide.

Waveguide-Type Head-Mounted Display System for AR Application http://dx.doi.org/10.5772/intechopen.75172 55

**Figure 13.** Analysis for (a) the angular selectivity and (b) the spectral selectivity, for the corresponding spatial frequencies of each incident angle.

According to the above calculation, when *θw* is designed by 30° (angle of total internal reflec-

**Figure 11.** The schematic configuration of a composite structure and the example of the guided full-color image.

to each incident angle as 5930, 5617, 5931, and 5904 lines/mm, respectively. When considering the photopolymer as a HOE that the thickness is 16.8 µm, the angular selectivity for HVG recordings become larger at higher spatial frequencies; but, the wavelength selectivity exhibits the similar tendency, as shown in **Figure 13(a)** and **(b)**. So, it can be seen that the recording incident angle at 30/20 and 30/40° on the outside of HOE achieves very wide angular selectivity and narrow spectral selectivity. Therefore, it is suitable for the incident angle of HVGs that are set as *θ*<sup>1</sup> = 40 and *θ*<sup>2</sup> = 30°, because larger angle of total internal reflection leads to a thinner

**Figure 12.** The schematic configuration of a wedge-shaped holographic waveguide structure and (b) angle of the light

can be 20, 30, 40, or 50°; the total internal

*<sup>λ</sup>*<sup>0</sup> ))) (3)

; and the spatial frequencies are corresponded

*<sup>λ</sup>*<sup>1</sup> cos(*<sup>φ</sup>* <sup>−</sup> *<sup>θ</sup>*0) \_\_\_\_\_\_\_\_\_\_

tion should be larger than the critical angle), *θ*<sup>1</sup>

reflection occurred at all angles of incidence *θ*<sup>1</sup>

54 State of the Art Virtual Reality and Augmented Reality Knowhow

For the field of view, it can be calculated as

*<sup>θ</sup>FOV* <sup>=</sup> arcsin(*<sup>n</sup>* sin(*<sup>φ</sup>* <sup>−</sup> arccos(

holographic waveguide.

path inside the waveguide.

where *λ*<sup>0</sup> and *λ*<sup>1</sup> are the wavelengths of the recording and readout beams. According to Eq. (3), the field of view can be analyzed for the different wavelengths of illumination that between ±17° in 430–500 and 580–670 nm wavelengths and ±15° in 500–560 nm.

The wedge-shaped holographic waveguide structure requires the thin films for in- and outcouplers, so it is necessary to record the full-color HVGs onto a single HOE film. As mentioned earlier, the simple recording method using three lasers is difficult to apply because of very low diffraction efficiency. Therefore, the time scheduling method must be utilized that the shutters control the exposure duration of each laser illumination. Here, when the HVG is recorded onto the HOE film by any laser illumination, other two lasers are blocked by the corresponding shutters. The saturation grating for each HVG should be within 5 seconds, because it will be difficult to form the grating on the HOE film after 5 seconds. The biggest advantage of this recording method is that the full-color HVG can be recorded onto a single HOE film with good uniformity of color and short recording time, and the maximum uniformity can be with only 10% loss in diffraction efficiency.

But in the actual case, especially for the photopolymer, it has the absorption and transmittance for each color of laser. Piao et al. measured the diffraction efficiencies of full-color single HOE in the six different cases for the sequence of three laser illuminations: red-green-blue, red-blue-green, green-red-blue, green-blue-red, blue-red-green, and blue-green-red. The optimum case is green-blue-red sequence, because this case exhibits much higher optical efficiency and more uniform distribution than other sequential recording processes and has a high potential for obtaining high image quality that the diffraction efficiencies are measured as 49, 47, and 44% for each wavelength. **Figure 14** shows the color analysis for the different sequential exposures. Here, it can be seen that the green-blue-red sequence has successful higher color uniformity than other cases.

When the HVGs are recorded onto the photopolymer films by green-blue-red sequential beam exposure, and applied into the see-through AR display system, the guided image

**4. Conclusions**

**Acknowledgements**

(No. NRF-2017R1A2B4012096).

Cheongju, Chungbuk, South Korea

\*Address all correspondence to: namkim@chungbuk.ac.kr

**Author details**

Nam Kim\*

Nowadays, the general consumers and developers are widely interested in AR/VR technology. Especially, AR technology has been recognized that it is a technology that has a much big change in mobile technology fields. Modern smart mobile devices have limited size, and it must be touched by the user's hand, but the see-through AR systems are hand-free and controlled only by the eye of the user. Also, it has almost every function of mobile devices such as voice call, Internet connection, video playback, virtual game, and so on and mixes the virtual objects with the real-world environment. It has been developed since the 1940s, and there is a long way to go for the development. In order to make the AR displays light-weight, the optical waveguide structure is usually utilized. Several types are existed such as reflective, polarized, diffractive, and holographic waveguides; among them, the holographic waveguide structured is an efficient way to display the clear images on the large screen with wide field of view for the comfortable viewing and make the overall AR device as small and light weight. The holographic waveguide structure requires the HOEs as the in- and out-couplers with the incident diffracting angle. Among the HOE materials, the photopolymer is advantageous that it does not require the post-processing, and it provides high diffraction, high reliability, and high resolution; it is the most suitable HOE material in the see-through AR display system. Especially, the diffraction efficiency is quite high for the both monochromatic and full-color visualizations. Also, when the full-color HVGs are recorded onto a single HOE film and applied into the wedge-shaped holographic waveguide, the entire AR display system can be more lightweighted. Currently, the technological companies such as Sony, Konica-Minolta, DigiLens, and BAE Systems use the holographic-type waveguide structure in their see-through AR application-based HMDs and HUDs. Also, Microsoft Research is working hard to release their new near-eye see-through holographic display using a holographic waveguide in the near future.

Waveguide-Type Head-Mounted Display System for AR Application

http://dx.doi.org/10.5772/intechopen.75172

57

This research was supported by the Government of Korea, under the ITRC (Information Technology Research Center) support program (IITP-2017-2015-0-00448) supervised by the IITP (Institute for Information and Communications Technology Promotion) and supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government

Munkh-Uchral Erdenebat, Young-Tae Lim, Ki-Chul Kwon, Nyamsuren Darkhanbaatar and

School of Information and Communication Engineering, Chungbuk National University,

**Figure 14.** (a) For the given input image: The detected color uniformity for (b) red-green-blue, (c) red-blue-green, (d) greenred-blue, (e) green-blue-red, (f) blue-red-green, and (g) blue-green-red sequential beam exposures.

through the wedge-shaped waveguide is observed very accurately, as shown in **Figure 15**. Here, the diffraction efficiency was measured by 30, 34, and 28% for the monochromatic cases of red, green, and blue lasers and 31% for the full-color visualization using the green-blue-red sequence. The diffraction efficiency is fine, even though full-color HVGs are recorded onto a single HOE. **Figure 15** shows the full-color visualizations for the given test image when the HOEs are fabricated by green-blue-red sequential exposure and applied into the AR display system using the wedge-shaped holographic waveguide. Note that micro display using a light emitting diode source was laid at the focal plane of the collimating lens, and the plane waves formed, in the AR display system.

**Figure 15.** (a) The original test image, (b) a full-color visualization, and (c) the output image illuminated by the light emitting diode source.

#### **4. Conclusions**

Nowadays, the general consumers and developers are widely interested in AR/VR technology. Especially, AR technology has been recognized that it is a technology that has a much big change in mobile technology fields. Modern smart mobile devices have limited size, and it must be touched by the user's hand, but the see-through AR systems are hand-free and controlled only by the eye of the user. Also, it has almost every function of mobile devices such as voice call, Internet connection, video playback, virtual game, and so on and mixes the virtual objects with the real-world environment. It has been developed since the 1940s, and there is a long way to go for the development. In order to make the AR displays light-weight, the optical waveguide structure is usually utilized. Several types are existed such as reflective, polarized, diffractive, and holographic waveguides; among them, the holographic waveguide structured is an efficient way to display the clear images on the large screen with wide field of view for the comfortable viewing and make the overall AR device as small and light weight. The holographic waveguide structure requires the HOEs as the in- and out-couplers with the incident diffracting angle. Among the HOE materials, the photopolymer is advantageous that it does not require the post-processing, and it provides high diffraction, high reliability, and high resolution; it is the most suitable HOE material in the see-through AR display system. Especially, the diffraction efficiency is quite high for the both monochromatic and full-color visualizations. Also, when the full-color HVGs are recorded onto a single HOE film and applied into the wedge-shaped holographic waveguide, the entire AR display system can be more lightweighted. Currently, the technological companies such as Sony, Konica-Minolta, DigiLens, and BAE Systems use the holographic-type waveguide structure in their see-through AR application-based HMDs and HUDs. Also, Microsoft Research is working hard to release their new near-eye see-through holographic display using a holographic waveguide in the near future.

#### **Acknowledgements**

through the wedge-shaped waveguide is observed very accurately, as shown in **Figure 15**. Here, the diffraction efficiency was measured by 30, 34, and 28% for the monochromatic cases of red, green, and blue lasers and 31% for the full-color visualization using the green-blue-red sequence. The diffraction efficiency is fine, even though full-color HVGs are recorded onto a single HOE. **Figure 15** shows the full-color visualizations for the given test image when the HOEs are fabricated by green-blue-red sequential exposure and applied into the AR display system using the wedge-shaped holographic waveguide. Note that micro display using a light emitting diode source was laid at the focal plane of the collimating lens, and the plane

**Figure 15.** (a) The original test image, (b) a full-color visualization, and (c) the output image illuminated by the light

**Figure 14.** (a) For the given input image: The detected color uniformity for (b) red-green-blue, (c) red-blue-green, (d) green-

red-blue, (e) green-blue-red, (f) blue-red-green, and (g) blue-green-red sequential beam exposures.

waves formed, in the AR display system.

56 State of the Art Virtual Reality and Augmented Reality Knowhow

emitting diode source.

This research was supported by the Government of Korea, under the ITRC (Information Technology Research Center) support program (IITP-2017-2015-0-00448) supervised by the IITP (Institute for Information and Communications Technology Promotion) and supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (No. NRF-2017R1A2B4012096).

#### **Author details**

Munkh-Uchral Erdenebat, Young-Tae Lim, Ki-Chul Kwon, Nyamsuren Darkhanbaatar and Nam Kim\*

\*Address all correspondence to: namkim@chungbuk.ac.kr

School of Information and Communication Engineering, Chungbuk National University, Cheongju, Chungbuk, South Korea

### **References**

[1] Melzer J, Moffitt K. Head-Mounted Displays: Designing for the User. Columbus, United States: McGraw-Hill Education; 1997

**Chapter 5**

**Provisional chapter**

**Enhancing BIM Methodology with VR Technology**

**Enhancing BIM Methodology with VR Technology**

DOI: 10.5772/intechopen.74070

Building information modeling (BIM) is defined as the process of generating, storing, managing, exchanging, and sharing building information. In the construction industry, the processes and technologies that support BIM are constantly evolving, making the BIM even more attractive. A current topic that requires attention is the integration of BIM with virtual reality (VR) where the user visualizes a virtual world and can interact with it. By adding VR, the BIM solution can address retrieving and presenting information and increasing efficiency on communication and problem solving in an interactive and collaborative project. The objective of this chapter is to report the improvement of BIM uses with the addition of interactive capacities allowed by VR technology. A bibliographic and

**Keywords:** BIM, VR, VR devices, VR + BIM applications, construction, facility

Building information modeling (BIM) methodology enables project stakeholders to create information-rich virtual models that help better to visualize building projects. There is currently a shift in the AEC (Architecture, Engineering, and Construction) industry to embrace BIM as a tool that can assist in integrating the fragmented industry by eliminating inefficiencies and redundancies, improving collaboration and communication, and enhancing overall productivity [1]. BIM is defined as the process of creating building information, stored in a centralized virtual model that can be managed allowing exchanging and sharing building data in an interoperable and reusable way [2]. BIM concerns the process of development of a computergenerated model representing all disciplines of a building design, and its use in several tasks as design simulation, construction planning, or facilities management. The resulting product,

> © 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

© 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

Alcínia Zita Sampaio

Alcínia Zita Sampaio

**Abstract**

management

**1. Introduction**

Additional information is available at the end of the chapter

Additional information is available at the end of the chapter

software research was made to support the study.

http://dx.doi.org/10.5772/intechopen.74070


#### **Enhancing BIM Methodology with VR Technology Enhancing BIM Methodology with VR Technology**

DOI: 10.5772/intechopen.74070

#### Alcínia Zita Sampaio Alcínia Zita Sampaio

**References**

jbio.201600268

org/10.5772/67297

Society of Korea. 2013;**17**:242-248

Optics. 2014;**53**:2180-2186

up\_display

States: McGraw-Hill Education; 1997

58 State of the Art Virtual Reality and Augmented Reality Knowhow

Bellingham, United States: SPIE Press; 2001

and Hall, Taylor & Francis Group; 1983

Transactions. Royal Society of London. 1838;**128**:371-394

[1] Melzer J, Moffitt K. Head-Mounted Displays: Designing for the User. Columbus, United

[2] Rash CE. Helmet-Mounted Displays: Design Issues for Rotary-Wing Aircraft. 2nd ed.

[3] Wheatstone C. Contributions to the physiology of vision. Part the first. On some remarkable, and hitherto unobserved, phenomena of binocular vision. Philosophical

[4] Kwon K-C, Im CY, Seo KS, Nam SM, Erdenebat M-U, Shim YB, Han Y-G, Kim N. Three-dimensional visualization system for ophthalmic microscopes using visible light and near-infrared illumination. Journal of Biophotonics. 2018:53. DOI: http://10.1002/

[5] Wikipedia. Head-up Display. 2009. Available from: https://en.wikipedia.org/wiki/Head-

[6] Snyder AW, Love JD. Optical Waveguide Theory. Abingdon, United Kingdom: Chapman

[7] Sarayeddine K, Mirza K. Key challenges to affordable see-through wearable displays: The missing link for mobile AR mass deployment. In: Proceedings of the SPIE Defense,

[8] Kim N, Piao Y-L, Wu H-Y. Holographic optical elements and application. In: Naydenova I, Nazarova D, Babeva T, editors. Holographic Materials and Optical Systems. Rijeka, Croatia: InTech, Prolaz Marije Krucifikse Kozulić; 2017. pp. 99-131. DOI: http://dx.doi.

[9] Piao J-A, Li G, Piao M-L, Kim N. Full color holographic optical element fabrication for waveguide-type head mounted display using photopolymer. Journal of the Optical

[10] Piao M-L, Kim N. Achieving high levels of color uniformity and optical efficiency for a wedge-shaped waveguide head-mounted display using a photopolymer. Applied

Security, and Sensing, April 29–May 3, 2013. United States: SPIE; 2013. p. 87200D

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.74070

**Abstract**

Building information modeling (BIM) is defined as the process of generating, storing, managing, exchanging, and sharing building information. In the construction industry, the processes and technologies that support BIM are constantly evolving, making the BIM even more attractive. A current topic that requires attention is the integration of BIM with virtual reality (VR) where the user visualizes a virtual world and can interact with it. By adding VR, the BIM solution can address retrieving and presenting information and increasing efficiency on communication and problem solving in an interactive and collaborative project. The objective of this chapter is to report the improvement of BIM uses with the addition of interactive capacities allowed by VR technology. A bibliographic and software research was made to support the study.

**Keywords:** BIM, VR, VR devices, VR + BIM applications, construction, facility management

#### **1. Introduction**

Building information modeling (BIM) methodology enables project stakeholders to create information-rich virtual models that help better to visualize building projects. There is currently a shift in the AEC (Architecture, Engineering, and Construction) industry to embrace BIM as a tool that can assist in integrating the fragmented industry by eliminating inefficiencies and redundancies, improving collaboration and communication, and enhancing overall productivity [1]. BIM is defined as the process of creating building information, stored in a centralized virtual model that can be managed allowing exchanging and sharing building data in an interoperable and reusable way [2]. BIM concerns the process of development of a computergenerated model representing all disciplines of a building design, and its use in several tasks as design simulation, construction planning, or facilities management. The resulting product,

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

a Building Information Model, is a data-rich, intelligent and parametric digital representation of the building project. So, as a first perspective, the result of BIM methodology can not only be considered as a virtual representation of a building, an object-oriented three-dimensional (3D) model, but is also a repository of project data [3]. The BIM data-model can be easily accessed, facilitating interoperability and exchange of project data with related software applications.

in its ability to present multi-dimensional data concerning design data. The use of BIM data in the development of several tasks is considered for the generation of nD/BIM applications [6].

Enhancing BIM Methodology with VR Technology http://dx.doi.org/10.5772/intechopen.74070 61

• 3D/BIM model concerns the modeling process of all disciplines (architecture, structures an MEP - mechanical, electrical, and piping) and as it is based in parametric objects, the rela-

• 4D/BIM model refers to the construction planning supporting the visual simulation of the building construction and its control. The 4D application links 3D elements, organized in sets, to the correspondent time schedule, allowing the comparison between the real work and the planned sequence of the process (supports eventual changes and allows a perma-

• 5D/BIM model is associated with cost estimation (control of the financing plan based on

• 6D/BIM model supports the performance of energy and sustainability studies (energy consumption estimate along the project and subsequent monitoring, measurement and verifi-

• 7D/BIM model is generated to support the maintenance activity and distinct management

The use of design data, centralized in a BIM model, in several activities allows a high level of communication among the team members, but to achieve an adequate collaborative work it is required an efficient interoperability between specific BIM tools. As per the reports, companies began seeing faster project approvals, increased positive team interactions, and higher product quality [11]. Unlimited extension of the use of BIM information combines BIM-based software with other advanced technologies, such as VR for quality defect management [12]. In the context of linking VR with BIM, the application of VR tools seems to be the next natural step for BIM as it can seamlessly blend into the design process, ultimately improving archi-

The VR field is interrelated with other domains that can make use of the visualization allowed by the BIM model, such as facilities management related with the visualization of data included in a BIM model, in real time following the interaction made possible by VR technology. As such, it is expected to be further explored in the near future. This text reports the main improvement of BIM uses when interactive capacities allowed by VR technology are added. The focus is how to explore BIM data in an interactive VR environment. As so, the study analyses the degree of achievements allowed by the actual software to perform several tasks combining BIM and VR, mainly in two important applications: 4D model supporting

Virtual reality (VR) technology leads to a better communication for key players across the building sector, based on greater design visualization, contributing to a better understanding of the

tionships (spatial geometry) and the properties (physical and mechanic);

nent actualization of schedules, estimates, and logistic site management) [7];

take-off material quantities in each constructive step) [8];

facilities needed in the building during its occupancy [10].

tecture and engineers and the entire built environment [13].

construction activity and 7D model concerning maintenance.

**3. VR and BIM capacities**

cation of consumption during the occupation of the building) [9];

An actual BIM subject that requires research, practical application and software adaptation, is the integration of BIM with virtual reality (VR) technology. Current VR devices allow the user to visualize a virtual world and to interact with the virtual space and its components. The main objective, that has been reported, concerns the use of VR system combined with plugins in order to support BIM multiuse throughout the building lifecycle. As so, the VR environment integrates information systems and immersive environments. For that, VR requires devices for interaction with the virtual environment of the building and the construction space. The current target, concerning integrating VR and BIM, is the study of how to use or adapt those VR devices and how to establish links for the presentation of information contained in a BIM model. This is bringing BIM data into a VR environment. Visualizing virtual worlds and interacting with it, is essentially a VR aspect, and creating and manipulating design data, is the essence of BIM, so the link of these two strong capacities should be known and explored. Following this perspective, commercial software houses have been developing advanced BIM + VR products. The use of BIM technology on construction projects has the potential to improve the process by allowing all team members to collaborate in an accurate and efficient way. Unlimited extension of the use of BIM information combines BIM-based software with other advanced technologies [4]. These advanced technologies link the digital to the physical entities and research has already been conducted to investigate the combination of BIM and other technologies. The present reflection involves the link of BIM to VR.

Concerning BIM training, the school has an important role in the education of young entrants in the industry and to equip them with BIM skills, including the additional knowledge about VR + BIM. Regarding current digital 3D modeling technologies, added to design information and provided in mobile equipment, students and professionals can use advanced capabilities of interaction like VR technology, improving the use BIM methodology. The objective of this text is to report some of the most important enhances of BIM uses, when interactive capacity allowed by VR technology is added. A research concerning technologic advances in VR software, as a complement to improve BIM environment's management, was made.

#### **2. nD/BIM models**

BIM methodology can generate and maintain information produced during the whole lifecycle of a building project, from design to maintenance, in a centralized model, the BIM model. As so, the BIM data-rich model supports the application of the rich-data in various fields [5]. The model allows views and data, appropriate to various users' needs. Distinct type of data can be extracted from the model, analyzed, and applied to generate new information. The data-rich model supports decisions making and improves the process of developing and delivering the design. Due to the consistency of design data with quality data and construction modeling process with value control, the potential of BIM implementation in improving management lies in its ability to present multi-dimensional data concerning design data. The use of BIM data in the development of several tasks is considered for the generation of nD/BIM applications [6].


The use of design data, centralized in a BIM model, in several activities allows a high level of communication among the team members, but to achieve an adequate collaborative work it is required an efficient interoperability between specific BIM tools. As per the reports, companies began seeing faster project approvals, increased positive team interactions, and higher product quality [11]. Unlimited extension of the use of BIM information combines BIM-based software with other advanced technologies, such as VR for quality defect management [12]. In the context of linking VR with BIM, the application of VR tools seems to be the next natural step for BIM as it can seamlessly blend into the design process, ultimately improving architecture and engineers and the entire built environment [13].

The VR field is interrelated with other domains that can make use of the visualization allowed by the BIM model, such as facilities management related with the visualization of data included in a BIM model, in real time following the interaction made possible by VR technology. As such, it is expected to be further explored in the near future. This text reports the main improvement of BIM uses when interactive capacities allowed by VR technology are added. The focus is how to explore BIM data in an interactive VR environment. As so, the study analyses the degree of achievements allowed by the actual software to perform several tasks combining BIM and VR, mainly in two important applications: 4D model supporting construction activity and 7D model concerning maintenance.

### **3. VR and BIM capacities**

a Building Information Model, is a data-rich, intelligent and parametric digital representation of the building project. So, as a first perspective, the result of BIM methodology can not only be considered as a virtual representation of a building, an object-oriented three-dimensional (3D) model, but is also a repository of project data [3]. The BIM data-model can be easily accessed, facilitating interoperability and exchange of project data with related software applications.

60 State of the Art Virtual Reality and Augmented Reality Knowhow

An actual BIM subject that requires research, practical application and software adaptation, is the integration of BIM with virtual reality (VR) technology. Current VR devices allow the user to visualize a virtual world and to interact with the virtual space and its components. The main objective, that has been reported, concerns the use of VR system combined with plugins in order to support BIM multiuse throughout the building lifecycle. As so, the VR environment integrates information systems and immersive environments. For that, VR requires devices for interaction with the virtual environment of the building and the construction space. The current target, concerning integrating VR and BIM, is the study of how to use or adapt those VR devices and how to establish links for the presentation of information contained in a BIM model. This is bringing BIM data into a VR environment. Visualizing virtual worlds and interacting with it, is essentially a VR aspect, and creating and manipulating design data, is the essence of BIM, so the link of these two strong capacities should be known and explored. Following this perspective, commercial software houses have been developing advanced BIM + VR products. The use of BIM technology on construction projects has the potential to improve the process by allowing all team members to collaborate in an accurate and efficient way. Unlimited extension of the use of BIM information combines BIM-based software with other advanced technologies [4]. These advanced technologies link the digital to the physical entities and research has already been conducted to investigate the combination of BIM and other technologies. The present reflection involves the link of BIM to VR.

Concerning BIM training, the school has an important role in the education of young entrants in the industry and to equip them with BIM skills, including the additional knowledge about VR + BIM. Regarding current digital 3D modeling technologies, added to design information and provided in mobile equipment, students and professionals can use advanced capabilities of interaction like VR technology, improving the use BIM methodology. The objective of this text is to report some of the most important enhances of BIM uses, when interactive capacity allowed by VR technology is added. A research concerning technologic advances in VR soft-

BIM methodology can generate and maintain information produced during the whole lifecycle of a building project, from design to maintenance, in a centralized model, the BIM model. As so, the BIM data-rich model supports the application of the rich-data in various fields [5]. The model allows views and data, appropriate to various users' needs. Distinct type of data can be extracted from the model, analyzed, and applied to generate new information. The data-rich model supports decisions making and improves the process of developing and delivering the design. Due to the consistency of design data with quality data and construction modeling process with value control, the potential of BIM implementation in improving management lies

ware, as a complement to improve BIM environment's management, was made.

**2. nD/BIM models**

Virtual reality (VR) technology leads to a better communication for key players across the building sector, based on greater design visualization, contributing to a better understanding of the project [14]. The knowledge about the different type of data that can be associated to the set of parameters that identifies each parametric object, base of the BIM modeling process, is of great importance when the team members wants to analyze problems and to discuss alternative solutions. Interacting with the BIM model, in order to visualize the geometry of the elements and to consult the parametric data, is an improvement in the development of a collaborative project.

The acceptance of VR use in construction field has been growing. VR enables the user to fully immerse himself in a 1:1-scale, 3D/BIM model which can be manipulated, providing an immersive accurate sense of presence in a space that is yet to be built [15]. Architectural, engineering, and construction management professionals recognize that VR applications are making it easier for clients to visualize designs earlier, shaving material costs off budgets and reducing the amount of workers needed for projects. For instance, a VR tour can be applied over a BIM model in order to check, from a facility management perspective, the maintenance schedule (7D/BIM model), or from a project team view, the constructability review (4D/BIM model), supporting decisions making. VR technology can improve BIM methodology, as it allows interaction with 3D/BIM models in two essential ways:

software and hardware tools available right now that can redefine the way BIM is imple-

Enhancing BIM Methodology with VR Technology http://dx.doi.org/10.5772/intechopen.74070 63

**Figure 1.** Virtual reality tour inside and outside a BIM model and supporting a collaborative meeting.

Experience VR content can be non-immersive, by using just a desktop or tablet PC, or immersive, by using a head-mounted display (HMD) like Gear VR or Oculus Rift. VR head-mounted displays have the capacity to improve the way designers communicate before buildings are built. HMD device allows the user to be immersed in the project model supporting to understand and assess new design concepts and alternatives created with BIM modeler tools. Currently, there are various applications of virtual reality tools that can be applied in the construction activity as a support to engineers' and architects'

• **Samsung Gear VR** is a virtual reality device that allows exploring virtual worlds at the construction site or during meetings (**Figure 2**). To accomplish virtual reality by utilizing Gear VR during facility management purposes, a BIM model is needed and also construction site pictures are needed to follow construction phases [18]. The user should be familiar with BIM software, such as Revit for visualizing, and 3ds Max for rendering. In order to navigate inside and outside the BIM model in a virtual world, the use of game engines like Unity3D with Android Studio are necessary and regarding exploring panorama pictures, equipment for panoramic photographs and editing tools that will help convert those pho-

• **Oculus Rift** is a real-time viewer application with interactive capacities, developed by the Oculus Company, and can be implemented as a plugin in Revit [20]. With the right technical know-how, Oculus can be used to represent an entire building and give a better perspective of the actual scale of the project, which is not allowed so realistically using just traditional monitor oriented systems. It offers a wide viewing angle of up to 110°. These VR glasses are handling to visualize and experience 3D model, 360° panorama picture, and virtual mock-up over a BIM model [21]. For Oculus Rift, the minimum interactivity demands are typically higher, as physical interaction and display update becomes much more integrated. The device provides stereoscopic 3D view and includes a gyroscope, an accelerometer, and a magnetometer to determine the orientation of the user's head in the real world. In the case of the Rift, the stereoscopic 3D view is implemented by means of split-screen rendering, where the left half of the screen corresponds to the left eye, and vice

mented and interpreted [17].

tos to the 3D world will be required as well [19];

work:

versa (**Figure 3**);


#### **3.1. Walkthrough**

Virtual reality can improve the development and analyses of any detail of the project, along distinct design steps, as it easily involves all team and a justifiable decision can be worked out concerning alternative solutions or problems resolution. The viability of a building could be tested through scale digital models and human judgment, but these models cannot completely simulate the environmental factors that a building is subjected to and human analyses can be inaccurate. VR offers more realistic possibilities as the final model can be rendered in 3D and the team member can experience and explore the space as a real space.

VR technology allows the user to observe artificial world, by walking through outside and inside and see everything around, and also reach out touching objects in real time. To create an immersive VR environment computer graphics technology and a selective equipment with advanced viewing and interaction capabilities are needed [16]. The 3D virtual world used to support the distinct aspects of the construction activity is permanently increasing nowadays. The current perspective, reported in research works and in commercial web sites, is the use of VR applied over the BIM model, as a positive and an innovative contribute, to improve the construction industry efficiency. VR technology has seen developments in recent years and has been applied in architectural, engineering, and construction fields. The checklist can be prepared smoothly so as to save valuable time and money for the company as well as the client.

Currently architects, engineers, and other construction specialists are already exploring the length and breadth of the joint RV + BIM technology (**Figure 1**). There are almost 50+ VR

**Figure 1.** Virtual reality tour inside and outside a BIM model and supporting a collaborative meeting.

project [14]. The knowledge about the different type of data that can be associated to the set of parameters that identifies each parametric object, base of the BIM modeling process, is of great importance when the team members wants to analyze problems and to discuss alternative solutions. Interacting with the BIM model, in order to visualize the geometry of the elements and to consult the parametric data, is an improvement in the development of a collaborative project.

The acceptance of VR use in construction field has been growing. VR enables the user to fully immerse himself in a 1:1-scale, 3D/BIM model which can be manipulated, providing an immersive accurate sense of presence in a space that is yet to be built [15]. Architectural, engineering, and construction management professionals recognize that VR applications are making it easier for clients to visualize designs earlier, shaving material costs off budgets and reducing the amount of workers needed for projects. For instance, a VR tour can be applied over a BIM model in order to check, from a facility management perspective, the maintenance schedule (7D/BIM model), or from a project team view, the constructability review (4D/BIM model), supporting decisions making. VR technology can improve BIM methodology, as it

• **Walkthrough** is the most popular as the user can view the 3D model in a virtual environment in real time from multiple perspectives of the building, from inside or outside;

• **Consulting data** concerns the possibility to retrieve information centralized in a BIM model, namely, data associated with parameters that compose the parametric objects used in

Virtual reality can improve the development and analyses of any detail of the project, along distinct design steps, as it easily involves all team and a justifiable decision can be worked out concerning alternative solutions or problems resolution. The viability of a building could be tested through scale digital models and human judgment, but these models cannot completely simulate the environmental factors that a building is subjected to and human analyses can be inaccurate. VR offers more realistic possibilities as the final model can be rendered in

VR technology allows the user to observe artificial world, by walking through outside and inside and see everything around, and also reach out touching objects in real time. To create an immersive VR environment computer graphics technology and a selective equipment with advanced viewing and interaction capabilities are needed [16]. The 3D virtual world used to support the distinct aspects of the construction activity is permanently increasing nowadays. The current perspective, reported in research works and in commercial web sites, is the use of VR applied over the BIM model, as a positive and an innovative contribute, to improve the construction industry efficiency. VR technology has seen developments in recent years and has been applied in architectural, engineering, and construction fields. The checklist can be prepared smoothly so as to save valuable time and money for the company as well as the client. Currently architects, engineers, and other construction specialists are already exploring the length and breadth of the joint RV + BIM technology (**Figure 1**). There are almost 50+ VR

3D and the team member can experience and explore the space as a real space.

allows interaction with 3D/BIM models in two essential ways:

62 State of the Art Virtual Reality and Augmented Reality Knowhow

the modeling process.

**3.1. Walkthrough**

software and hardware tools available right now that can redefine the way BIM is implemented and interpreted [17].

Experience VR content can be non-immersive, by using just a desktop or tablet PC, or immersive, by using a head-mounted display (HMD) like Gear VR or Oculus Rift. VR head-mounted displays have the capacity to improve the way designers communicate before buildings are built. HMD device allows the user to be immersed in the project model supporting to understand and assess new design concepts and alternatives created with BIM modeler tools. Currently, there are various applications of virtual reality tools that can be applied in the construction activity as a support to engineers' and architects' work:


**Figure 2.** Improving BIM with gear VR glasses in a desktop and in a work place [18].

• The rapid advancement in engineering brings VR to expand vision capacities. The use of **hologram projections** in the construction activity, enables the end user to experience different textures or choose from a variety of in-built texture formats the required aspect for the project [22] (**Figure 4**). The Revit model can be manipulated using hologram projections supporting a building collaborative design;

• **Smart Reality** app brings a new realism of tools to interact with clients, engineers, contractors, and other professionals in the building industry [24]. The user is instantly immersed in a true 3D environment that gives a sense of scale, depth, and spatial awareness adding a more realistic perspective of the model and its environment. The VR mobile app was developed for builders and combines 2D plans and 3D project models for an enhanced, mobile visualization of construction projects. The VR capacity of this app is beginning to introduce a new level of certainty for the design and construction process as well, helping stakeholders to get a more full experience of the BIM model, offering an unobstructed, real-world perspective of projects for planning, design, and construction management (**Figure 6**). It is designed for the AEC sector and works with BIM tools like Autodesk's Revit. As the design process is slow, Smart Reality can be used to help the owner or architect to get a better feel of spatial recognition of the intended product in advance of construction beginning. The Smart Reality app can be downloaded for iOS and Android and used to check and analyze distinct aspects of the project in any place and with any member of the team, in an easy and collaborative way. The viewing experience can further be enhanced by using VR headsets

Enhancing BIM Methodology with VR Technology http://dx.doi.org/10.5772/intechopen.74070 65

• The **Visidraft** app actually builds a 3D world around the user. Visidraft instantly converts 2D elevations as well plans into corresponding 3D projections (when viewed through a mobile camera). When the user places virtual objects into the environment, the app automatically calculates the relative distances of all items present in that room allowing the

like Oculus Rift and Epson's Moverio BT-200 smart glasses [25];

**Figure 5.** Walkthrough capacity using Enscape Revit plugin [23].

**Figure 4.** Touchable holograms [22].

• With the **Enscape** Revit plugin, the user is able to quickly explore different design options and present projects to clients and so it has become a standard application in projects worldwide [23]. A walkthrough using Enscape usually runs alongside Revit or SketchUp, reflecting all the perspectives, geometric details, and eventual adjustments to the project. Additionally, in a real time rendering in SketchUp, available by Enscape, a programed walkthrough can be programed, allowing the creation of a series of walkthrough movies, which can be exported as a stand-alone executable file to be distributed to clients or colleagues. Using Enscape for real-time visualization and rendering enables the project team to accurately communicate with engineering design, clients, and stakeholders, improving the collaborative design. The ability to walkthrough inside BIM models and to regenerate design changes without leaving Revit is of a great importance and time saver. So Enscape add a significant value to communication in a team, as a real-time rendered walkthroughs, is able to be generated straight from Revit models. In this immersive and highly realistic way, consultants or designers better understand the project, as they can judge not just the technical aspect but also the realistic visual impact (**Figure 5**);

**Figure 3.** Left and right half of the screen corresponds to each eye [20].

**Figure 4.** Touchable holograms [22].

• The rapid advancement in engineering brings VR to expand vision capacities. The use of **hologram projections** in the construction activity, enables the end user to experience different textures or choose from a variety of in-built texture formats the required aspect for the project [22] (**Figure 4**). The Revit model can be manipulated using hologram projections

• With the **Enscape** Revit plugin, the user is able to quickly explore different design options and present projects to clients and so it has become a standard application in projects worldwide [23]. A walkthrough using Enscape usually runs alongside Revit or SketchUp, reflecting all the perspectives, geometric details, and eventual adjustments to the project. Additionally, in a real time rendering in SketchUp, available by Enscape, a programed walkthrough can be programed, allowing the creation of a series of walkthrough movies, which can be exported as a stand-alone executable file to be distributed to clients or colleagues. Using Enscape for real-time visualization and rendering enables the project team to accurately communicate with engineering design, clients, and stakeholders, improving the collaborative design. The ability to walkthrough inside BIM models and to regenerate design changes without leaving Revit is of a great importance and time saver. So Enscape add a significant value to communication in a team, as a real-time rendered walkthroughs, is able to be generated straight from Revit models. In this immersive and highly realistic way, consultants or designers better understand the project, as they can judge not just the

supporting a building collaborative design;

64 State of the Art Virtual Reality and Augmented Reality Knowhow

**Figure 2.** Improving BIM with gear VR glasses in a desktop and in a work place [18].

technical aspect but also the realistic visual impact (**Figure 5**);

**Figure 3.** Left and right half of the screen corresponds to each eye [20].


**Figure 5.** Walkthrough capacity using Enscape Revit plugin [23].

**Figure 6.** Smart reality application [20].

user to walk around the room physically with the device and visualize and re-arrange the items conveniently (**Figure 7**). The app comes out with Revit, ArchiCAD, and Nemetschek Vectorworks plugins [26];

sensation of the user being immersed in a real place is the most convincing. As the team member better communicate between them and the client better understand spatial relationships and scale of the project. VR can induce a quit realist analyses as the physical construction seems to be present. VR plays an important role at all stages of the designto-construction process. Igloo BIM cave has space constraints; there are limitations on where an environment can be placed and how easy it is to bring people to it (**Figure 9**).

Enhancing BIM Methodology with VR Technology http://dx.doi.org/10.5772/intechopen.74070 67

Once the model is inside the VR environment, things like materials, lighting, furniture, and other small details that make the VR experience feel real, are added. This is the concept of walkthrough, but BIM contains data and it is convenient to explore BIM data while walking inside a virtual model. VR can play an important role at all stages of the design-to-construction process, from evaluating design options and showcasing proposals, to designing out errors and ironing out construction and serviceability issues before breaking ground on site. Fully interactive VR software has extremely high performance demands, during a tour, but additional technology capacities concerning model-data retrieving is required when bringing BIM

Collaboration for BIM tools is designed to help project teams overcome the barriers of corporate firewalls and physical location by enabling centralized access to BIM models. Team members in

**3.2. Consulting data**

**Figure 8.** PrioVR capture system applied in construction [22].

data into a VR environment.

**Figure 9.** An igloo BIM cave [27].


**Figure 7.** Visualizing, in mobile device, a 3D model using Visidraft app [26].

**Figure 8.** PrioVR capture system applied in construction [22].

sensation of the user being immersed in a real place is the most convincing. As the team member better communicate between them and the client better understand spatial relationships and scale of the project. VR can induce a quit realist analyses as the physical construction seems to be present. VR plays an important role at all stages of the designto-construction process. Igloo BIM cave has space constraints; there are limitations on where an environment can be placed and how easy it is to bring people to it (**Figure 9**).

#### **3.2. Consulting data**

Once the model is inside the VR environment, things like materials, lighting, furniture, and other small details that make the VR experience feel real, are added. This is the concept of walkthrough, but BIM contains data and it is convenient to explore BIM data while walking inside a virtual model. VR can play an important role at all stages of the design-to-construction process, from evaluating design options and showcasing proposals, to designing out errors and ironing out construction and serviceability issues before breaking ground on site. Fully interactive VR software has extremely high performance demands, during a tour, but additional technology capacities concerning model-data retrieving is required when bringing BIM data into a VR environment.

Collaboration for BIM tools is designed to help project teams overcome the barriers of corporate firewalls and physical location by enabling centralized access to BIM models. Team members in

**Figure 9.** An igloo BIM cave [27].

**Figure 7.** Visualizing, in mobile device, a 3D model using Visidraft app [26].

a 360° viewing angle around the virtual place;

user to walk around the room physically with the device and visualize and re-arrange the items conveniently (**Figure 7**). The app comes out with Revit, ArchiCAD, and Nemetschek

• **PrioVR** is a real-time motion capture system for virtual reality that can be also used in construction (**Figure 8**). The motion capture device PrioVR helps the user to experience a VR environment through natural body movements instead of using a keyboard and a mouse [22]. PrioVR can be used for virtually opening doors, switch the lights on and off and reposition objects, and also change materials such as wall paper and flooring and experiment with all kinds of environmental elements such as lighting, weather, and surroundings in real time. The device works via body sensors that transmit information about the body's movements back to the VR software. The user is equipped with a full body suit, which has 12 internal sensors attached to it, or just with an upper body, tracking with 8 sensors. To create a VR environment of a construction place and to allow a useful interaction with the design components, the Prio VR sensors must adequately be calibrated. This adjustment allows user to capture accurately movements and modeled objects in the VR world. It allows

• **CAVE** (Cave Automatic Virtual Environment) platforms have been developed for immersive VR experience as they track user's head and control stick usually with 6 degrees of freedom, to navigate inside the virtual environment and interact with the contents of a 3D model including a 3D BIM model [27]. A CAVE takes normally the form of a cubelike space in which images are displayed by a series of projectors. Due to its immersive experience and intuitive manipulation capability, it quickly gained popularity in both research and construction industry community. The sense of presence is overawed and the

Vectorworks plugins [26];

**Figure 6.** Smart reality application [20].

66 State of the Art Virtual Reality and Augmented Reality Knowhow

all disciplines from multiple companies or sites worldwide are able to access and work in models hosted on cloud servers. The model is centrally hosted and access to project data is permissions based, and it can be accessed by project teams anywhere. So, when collaborative projects are discussed between firms, they consult BIM data in models hosted on clouds. In addition, the capacity of consulting BIM data can be improved with VR devises applied directly over the BIM model.

Concerning a 7D/BIM application, a VR tour can be applied, from a facility management or maintenance perspective to support decision making. Combining VR devices applied over the BIM model and the real inspection in the place, it offers a new perspective for facility management professionals. VR has the capacity to improve how managers and construction workers access virtual information of the established project. All data-rich virtual information is available from the BIM model, and can be retrieved, consulted, and analyzed during the real inspection to the site. Construction issues can be addressed in real-time and internal assets can be illustrated to facilitate installation and layout, saving time, and resources. Several plugins or BIM viewer allows consulting data from the 3D/BIM model and now there is a new class of mobile apps available that let project teams experience building plans in 3D:

and in addition, with the Revit open the user can consult the database of the model or the parametric object used in the modeling process. Every time a unique geometric representa-

Enhancing BIM Methodology with VR Technology http://dx.doi.org/10.5772/intechopen.74070 69

• An accessible high end visualization and virtual reality of a BIM model is obtain using the VR plugin of Revit, the **Enscape** [23]. It was previously mentioned just the walkthrough capacity, but Enscape also allows, together with Revit, to consult data. Inside Revit is possible to access the plugin Enscape and the user can observe both models in Revit and in Enscape (**Figure 12**). In Revit, the user can consult the rich-data incorporated in the 3D/BIM model and in Enscape, can interact in a virtual space. So, all changes in Revit are immediately available to evaluate in Enscape. As the Revit allows the user to work over the model, it is possible to retrieve the information linked to each parametric object used in the BIM model. In a 7D/BIM domain (**Figure 12**), a virtual tour together with Enscape allows the facility manager to look around the facility and check the conditions of equipment and obtain relevant information from the BIM model, using Revit interface. For instances, visualizing the BIM model of a MEP system, in a virtual tour and using on a tablet PC, helps the facility manager to understand what is installed behind the ceiling tile. So the aspect of linking the consulting capacity and the VR ability of walking around is a very important improvement in the use of BIM methodology. To improve VR experience, Enscape can also be used together with Oculus Rift. With Enscape, the user is able to quickly explore different design

tion is encountered for the first time, all of its data can be retrieved;

**Figure 11.** The viewer plugin interface in Revit [28].

**Figure 12.** Enscape plugin of Revit to access data [23].


**Figure 10.** Using Autodesk 360 in a collaborative project [28].

**Figure 11.** The viewer plugin interface in Revit [28].

and in addition, with the Revit open the user can consult the database of the model or the parametric object used in the modeling process. Every time a unique geometric representation is encountered for the first time, all of its data can be retrieved;

• An accessible high end visualization and virtual reality of a BIM model is obtain using the VR plugin of Revit, the **Enscape** [23]. It was previously mentioned just the walkthrough capacity, but Enscape also allows, together with Revit, to consult data. Inside Revit is possible to access the plugin Enscape and the user can observe both models in Revit and in Enscape (**Figure 12**). In Revit, the user can consult the rich-data incorporated in the 3D/BIM model and in Enscape, can interact in a virtual space. So, all changes in Revit are immediately available to evaluate in Enscape. As the Revit allows the user to work over the model, it is possible to retrieve the information linked to each parametric object used in the BIM model. In a 7D/BIM domain (**Figure 12**), a virtual tour together with Enscape allows the facility manager to look around the facility and check the conditions of equipment and obtain relevant information from the BIM model, using Revit interface. For instances, visualizing the BIM model of a MEP system, in a virtual tour and using on a tablet PC, helps the facility manager to understand what is installed behind the ceiling tile. So the aspect of linking the consulting capacity and the VR ability of walking around is a very important improvement in the use of BIM methodology. To improve VR experience, Enscape can also be used together with Oculus Rift. With Enscape, the user is able to quickly explore different design

**Figure 12.** Enscape plugin of Revit to access data [23].

**Figure 10.** Using Autodesk 360 in a collaborative project [28].

in a collaborative environment;

68 State of the Art Virtual Reality and Augmented Reality Knowhow

all disciplines from multiple companies or sites worldwide are able to access and work in models hosted on cloud servers. The model is centrally hosted and access to project data is permissions based, and it can be accessed by project teams anywhere. So, when collaborative projects are discussed between firms, they consult BIM data in models hosted on clouds. In addition, the capacity of consulting BIM data can be improved with VR devises applied directly over the BIM model.

Concerning a 7D/BIM application, a VR tour can be applied, from a facility management or maintenance perspective to support decision making. Combining VR devices applied over the BIM model and the real inspection in the place, it offers a new perspective for facility management professionals. VR has the capacity to improve how managers and construction workers access virtual information of the established project. All data-rich virtual information is available from the BIM model, and can be retrieved, consulted, and analyzed during the real inspection to the site. Construction issues can be addressed in real-time and internal assets can be illustrated to facilitate installation and layout, saving time, and resources. Several plugins or BIM viewer allows consulting data from the 3D/BIM model and now there is a new class of mobile apps available that let project teams experience building plans in 3D:

• **Autodesk 360** allows the integration of the team in collaborative projects, as every modification in a BIM model is displayed in a handy timeline for everyone to see [28]. Autodesk produces a complete web-based solution for reviewing BIM content. To work with it, all Autodesk 360 involved members must be invited to the project and so every members will be able to see and download files uploaded in their project discipline. Along with showing the 3D model, Autodesk 360 displays every sheet embedded in the uploaded Revit model (**Figure 10**). The 3D viewer allows users to isolate elements by categories and display every elements property. So all data can be consulted, analyzed, and changed, refining the project

• To support an integrated design environment, the Autodesk Revit allows a **viewer Revit plugin** [28]. In Revit the user initializes the Add-Ins tab and a 3D visualization representation results visible in a new window (**Figure 11**). The user can navigate over the model, retrieving the desired data in a desktop using mouse and keyboard, or improving the VR experience connecting with an Oculus HMD device. The immersion in the model exposes the user to the entire underlying BIM database initializing a data consulting process. The model is visualized in an overlapped window, where the user can interact as a VR user, options supporting a collaborative project team. Enscape will immediately show the alternative solutions or changes in the design, the designer makes in the project, using Revit;


• Some progress has been made on VR techniques but only recently the link to BIM methodology has being made. A VR system should be combined with in-use applications to support multi-disciplinary users throughout construction lifecycle. Jiao et al. [14] presents a pilot cloud framework regarding an environment utilizing **web3D**, **BIM,** and **BSNS** (Business Social Networking Services). Web3D can be applied in several BIM that uses supporting the AEC industry, namely, in system coordination and constructability review tasks (**Figure 16**). So technical solutions to key issues such as authoring, publishing, and composition are designed. The proposed environment is seamlessly integrated into in-use

Enhancing BIM Methodology with VR Technology http://dx.doi.org/10.5772/intechopen.74070 71

information systems and therefore enjoys greater usability.

**Figure 16.** System coordination and constructability review using Web3D [1].

**Figure 14.** Interacting with Autodesk stingray [30].

**Figure 15.** Interacting with Covise software [31].

**Figure 13.** Consulting data to support maintenance using augment [29].

Enhancing BIM Methodology with VR Technology http://dx.doi.org/10.5772/intechopen.74070 71

**Figure 14.** Interacting with Autodesk stingray [30].

options supporting a collaborative project team. Enscape will immediately show the alternative solutions or changes in the design, the designer makes in the project, using Revit; • **Augment** is augmented reality (AR) software that allows user to view the 3D/BIM model in a realist environment and can be downloaded in a smartphone or tablet [29]. AR plus BIM technologies allow a real-time tour inside the building and also to consult BIM information. Augment allows the visualization of a 3D construct space from a set of 2D drawings of the project (**Figure 13**). The design team can upload the 2D design to the Augment app's website, and the client can view the 2D drawing in a 3D way, improving the understanding of the project details. In addition, AutoCAD, Revit, and SolidWorks software are compatible with Augment app. In Revit, the user can consult data and in augment the user can interact in a VR environment. • **Autodesk Stingray** allows the user, by clicking over objects of a Revit model, to view the underlying attribute information [30]. The VR device allows the user to create high-quality, interactive estimates, and schedules in VR by linking external BIM data sources into Autodesk's Stingray interactive engine. For that BIM data is exported from Revit software into an external data source, to be organized and sorted, and then this information is used in Stingray in various configurations to create interactive, immersive environments (**Figure 14**). So, clients can visualize different design, cost, and scheduling options in VR. Members' team will be able to visualize 4D site logistics plans and 5D estimates, and so, to choose from various design and material options and see the costs associated with them. In addition, BIM information can be easily exported from Stingray to an external data source. Benefits of estimating and scheduling in VR include realistic design visualization and real-time decision; • Since BIM software is normally run on single workstations, the potential for direct collaboration is somehow limited. The focus of research conducted by Kieferle and Woessner [12] was to provide a platform for the development and optimization by combining BIM and VR, based in linking Revit with **Covise**, VR software [31]. They have been able to implement a range of interactions but however they detected some limitation in the process. The project 3D model and the associated data can be visualized and consulted using Covise, improving collaboration even in remote team conferences situation. When changes are made in Revit, they directly reflected in Covise software and vice versa. This new approach as the capacity to promote bidirectional data exchange between the systems. Compared to other software this can be considered an advance in VR + BIM technology (**Figure 15**).

70 State of the Art Virtual Reality and Augmented Reality Knowhow

**Figure 13.** Consulting data to support maintenance using augment [29].

**Figure 15.** Interacting with Covise software [31].

• Some progress has been made on VR techniques but only recently the link to BIM methodology has being made. A VR system should be combined with in-use applications to support multi-disciplinary users throughout construction lifecycle. Jiao et al. [14] presents a pilot cloud framework regarding an environment utilizing **web3D**, **BIM,** and **BSNS** (Business Social Networking Services). Web3D can be applied in several BIM that uses supporting the AEC industry, namely, in system coordination and constructability review tasks (**Figure 16**). So technical solutions to key issues such as authoring, publishing, and composition are designed. The proposed environment is seamlessly integrated into in-use information systems and therefore enjoys greater usability.

**Figure 16.** System coordination and constructability review using Web3D [1].

Some VR + BIM tools were described in order to present the state-of-the-art and the main progress concerning the current topic involving BIM tendency. Next items regard two nD tasks where the use of VR over BIM has been applied.

In the modeling BIM process some elements, that were created as a unique object (but composed of several layers of materials), for a 4D/BIM model propose, must be decomposed in their essential elements. For that it is important that the BIM modeling software in use, can allows dividing elements, like a wall or floor, as needed to support the 4D construction modeling workflows. In an inverse way, it must also be able to assemble elements. In this context,

Enhancing BIM Methodology with VR Technology http://dx.doi.org/10.5772/intechopen.74070 73

• It allows **dividing** a model element into discrete parts that can be independently sched-

• It allows selecting any number of element instances to create an **assembly** and then the assembly can be manipulated as a single unit, which can be independently scheduled,

A part can be divided into smaller parts and each one is automatically updated to reflect any changes to the element from which it is are derived. So modifying a part has no effect on the original element. So each part, for instance, the finishing layer of a floor or a wall system (**Figure 18**), can be linked to the respective MS Project activity. This capacity allowed by the BIM tool is important, as normally the construction of a wall or a floor needs to be defined in a sequence of operations corresponding to distinct periods of time in a construction schedule planning. These two capacities, diving and assembling, allowed by the BIM modeling software can be improved with the use of a VR software that can operate over a 4D/BIM model. In the construction planning stage, VR and BIM can improve building design and construction management, as the boundary-pushing concepts can be tested and refined at an interactive

Normally, concrete conflicts became a reality in the construction site, when the construction process is in progress. At that moment any eventual error reverts in additional cost and time delay. A 4D/BIM following each construction step helps the team to analyze each eventual conflict and over the BIM model, it is possible to do the necessary corrections. By adding VR capacities the analyses performance in of BIM models can be improved. So, BIM + VR applications can contribute to reduce costs due to the construction of a real work, and to avoid

two Revit categories were designed to support construction modeling workflows [28]:

uled, tagged, filtered, and exported;

virtual world, before committing to real-world construction.

mistakes on site that generates material wastes.

**Figure 18.** Parametric objects of a wall solution and a floor system [6].

tagged, and filtered.

#### **4. 4D/VR/BIM MODEL: construction planning**

4D CAD models, that integrate physical 3D elements with time, have been used to visualize construction processes in several projects worldwide [32]. 4D models have been used during the construction phase and have been shown to have benefits over processes that span the entire lifecycle of a project such as collaboration with stakeholders, making design decisions, assessing project constructability, and identifying spatial conflicts in construction. Concerning the creation of 4D applications, as a support to follow construction planning, several studies can be found, linking VR technology to 3D geometric models. Two examples of 4D models are listed, using used only VR technology and the second model uses BIM tools without VR capacities (**Figure 17**):


**Figure 17.** Interfaces of 4D/VR model and of 4D/BIM model.

In the modeling BIM process some elements, that were created as a unique object (but composed of several layers of materials), for a 4D/BIM model propose, must be decomposed in their essential elements. For that it is important that the BIM modeling software in use, can allows dividing elements, like a wall or floor, as needed to support the 4D construction modeling workflows. In an inverse way, it must also be able to assemble elements. In this context, two Revit categories were designed to support construction modeling workflows [28]:

Some VR + BIM tools were described in order to present the state-of-the-art and the main progress concerning the current topic involving BIM tendency. Next items regard two nD

4D CAD models, that integrate physical 3D elements with time, have been used to visualize construction processes in several projects worldwide [32]. 4D models have been used during the construction phase and have been shown to have benefits over processes that span the entire lifecycle of a project such as collaboration with stakeholders, making design decisions, assessing project constructability, and identifying spatial conflicts in construction. Concerning the creation of 4D applications, as a support to follow construction planning, several studies can be found, linking VR technology to 3D geometric models. Two examples of 4D models are listed, using used only

• Sampaio et al. [33] implemented a prototype based on VR technology applied on construction planning. The geometrical AutoCAD 3D model of distinct steps of the construction activity is linked to the construction planning schedule, defining a 4D model. VR technology allows the visualization of different stages of the construction, and the interaction with the construction activity, resulting in a valuable asset in monitoring the development of the construction activity. The prototype makes use of MS Project, AutoCAD, and **EON Studio** VR software. The 4D/ VR application clearly shows the constructive process, avoiding inaccuracies, and building errors, and so improving the communication between partners in the construction process; • Sampaio and Mota [7] created a 4D/BIM model using Revit, MS Project, and Navisworks. The **Navisworks** software allows the interconnections among the 3D models, grouped in sets, and the related task following the activity sequence for the construction, stablished in MS Project. The created 4D model adds the time factor to 3D/BIM set components, allowing the visual simulation of the construction work. The ability to navigate through the model allows to analyze each corner and each location of the model, and to obtain quantities take-off of materials over each wanted construction step. Navisworks also has the potential to provide support in the analysis and detection of conflicts among the spe-

VR technology and the second model uses BIM tools without VR capacities (**Figure 17**):

tasks where the use of VR over BIM has been applied.

72 State of the Art Virtual Reality and Augmented Reality Knowhow

cialty projects;

**Figure 17.** Interfaces of 4D/VR model and of 4D/BIM model.

**4. 4D/VR/BIM MODEL: construction planning**


A part can be divided into smaller parts and each one is automatically updated to reflect any changes to the element from which it is are derived. So modifying a part has no effect on the original element. So each part, for instance, the finishing layer of a floor or a wall system (**Figure 18**), can be linked to the respective MS Project activity. This capacity allowed by the BIM tool is important, as normally the construction of a wall or a floor needs to be defined in a sequence of operations corresponding to distinct periods of time in a construction schedule planning. These two capacities, diving and assembling, allowed by the BIM modeling software can be improved with the use of a VR software that can operate over a 4D/BIM model. In the construction planning stage, VR and BIM can improve building design and construction management, as the boundary-pushing concepts can be tested and refined at an interactive virtual world, before committing to real-world construction.

Normally, concrete conflicts became a reality in the construction site, when the construction process is in progress. At that moment any eventual error reverts in additional cost and time delay. A 4D/BIM following each construction step helps the team to analyze each eventual conflict and over the BIM model, it is possible to do the necessary corrections. By adding VR capacities the analyses performance in of BIM models can be improved. So, BIM + VR applications can contribute to reduce costs due to the construction of a real work, and to avoid mistakes on site that generates material wastes.

**Figure 18.** Parametric objects of a wall solution and a floor system [6].

Recent developments in VR have encouraged the utilization of interactive visualization in the construction of building BIM projects. Du [5] finds that the interpersonal interaction in the VR environment is more critical to the effective communication in a building project, as it creates a shared immersive experience, and developed a BIM-enabled VR environment to realize multiplayer walkthrough in virtual buildings [11]. The multiplayer virtual walk allows real-time interactions of remotely located project stakeholders in the same environment, with a shared immersive walkthrough experience (**Figure 19**).

planned 4D/BIM model. BIM with VR enables project stakeholders to analyze informationrich virtual models that help better visualize the building construction project [34]. Nowadays, it is essential to build trust between project parties and give the owners and users the possibility to analyze and give feedback of the construction process plans, and using BIM

Enhancing BIM Methodology with VR Technology http://dx.doi.org/10.5772/intechopen.74070 75

• The traditional EON software has been used in 4D/VR models to support the construction activity. Currently, the **EON Icube** is a multi-walled system where the user is completely surrounded by images and sound (**Figure 21**). The EON Icube provides users a realistic aspect of the interior of the building namely concerning lighting and texture of wall materials. EON Icube is compatible with several VR devices (data-gloves, joysticks, or motion tracking systems) [35].

with VR this intention is easily achieved (**Figure 20**).

**5. 7D/VR/model: management and maintenance**

**Figure 20.** Screen shots of augment software [34].

**Figure 21.** EON Icube environment [35].

Combining BIM with VR is predictable to improve efficient collaboration, as the data can be visualized, consulted, and even changed while the team and client are inside an apparently construct build. In a real place, the equipment that cannot be observed because they are hidden over a false ceiling, such as HVAC devices or net systems (electrical, water, or gas supply or elements of acoustic protection), can be easily visualized using BIM + VR tools. So the management and maintenance facilities can be facilitated in a 7D/VR + BIM/model context. VR will enable the engineer to carry out a physical survey of the property or the building site (**Figure 22**). The VR technology applied to BIM tools has been enhancing a high level of integration but it is still necessary to do more in order to obtain an even more real platform, in a retrieving data perspective.

As so, the importance of adding VR benefits should be a constant research point [36].

**Figure 19.** Screen shots of 4D/VR/BIM simulation processes.

BIM supports 4D analysis, where activities from the project schedule can be simulated and studied to optimize the sequence of construction. A 4D model allows users to visualize the schedule and to optimize sequencing on the construction site. BIM can also be integrated with the cost factor, to generate a 5D/BIM simulation, in any stage of the building construction. The BIM is used to facilitate a quantity survey of building materials and components, and these quantities are linked directly to cost databases.

As mentioned, VR combines a device for interaction creating virtual environment and following that perspective, software houses have been developing advanced BIM/VR products applied also to improve 4D/BIM models. Some of the discussed VR software is also applied in construction planning:


planned 4D/BIM model. BIM with VR enables project stakeholders to analyze informationrich virtual models that help better visualize the building construction project [34]. Nowadays, it is essential to build trust between project parties and give the owners and users the possibility to analyze and give feedback of the construction process plans, and using BIM with VR this intention is easily achieved (**Figure 20**).

• The traditional EON software has been used in 4D/VR models to support the construction activity. Currently, the **EON Icube** is a multi-walled system where the user is completely surrounded by images and sound (**Figure 21**). The EON Icube provides users a realistic aspect of the interior of the building namely concerning lighting and texture of wall materials. EON Icube is compatible with several VR devices (data-gloves, joysticks, or motion tracking systems) [35].

**Figure 20.** Screen shots of augment software [34].

**Figure 19.** Screen shots of 4D/VR/BIM simulation processes.

quantities are linked directly to cost databases.

in construction planning:

construction place;

a shared immersive walkthrough experience (**Figure 19**).

74 State of the Art Virtual Reality and Augmented Reality Knowhow

Recent developments in VR have encouraged the utilization of interactive visualization in the construction of building BIM projects. Du [5] finds that the interpersonal interaction in the VR environment is more critical to the effective communication in a building project, as it creates a shared immersive experience, and developed a BIM-enabled VR environment to realize multiplayer walkthrough in virtual buildings [11]. The multiplayer virtual walk allows real-time interactions of remotely located project stakeholders in the same environment, with

BIM supports 4D analysis, where activities from the project schedule can be simulated and studied to optimize the sequence of construction. A 4D model allows users to visualize the schedule and to optimize sequencing on the construction site. BIM can also be integrated with the cost factor, to generate a 5D/BIM simulation, in any stage of the building construction. The BIM is used to facilitate a quantity survey of building materials and components, and these

As mentioned, VR combines a device for interaction creating virtual environment and following that perspective, software houses have been developing advanced BIM/VR products applied also to improve 4D/BIM models. Some of the discussed VR software is also applied

• **Enscape** is Revit plugin that creates a VR walkthrough with one click, based on BIM data and can be also used in a 4D context [23]. All materials, geometry, and entourage come from the Revit construction project and can be changed during the VR simulation. This flexibility allows spontaneous presentations with a real-time rendering quality within the construction design workflow. Enscape has become an useful construction planning support;

• Combined with the **Oculus Rif**t, customers can virtually walk through the Enscape project and experience it as if it were already built. In the construction process, the user visualize the correspondent building 4D/BIM step and consult all the information the user needs in order to compare the progress building stage and the delay or the advance realized in the

• Using **Augment** software, data-rich virtual information is available on site, through connected devices, like smartphone or tablet, in real time. It allows the user to follow the real construction evolution of the work, from multiple perspectives, and to compare it to the

**Figure 21.** EON Icube environment [35].

#### **5. 7D/VR/model: management and maintenance**

Combining BIM with VR is predictable to improve efficient collaboration, as the data can be visualized, consulted, and even changed while the team and client are inside an apparently construct build. In a real place, the equipment that cannot be observed because they are hidden over a false ceiling, such as HVAC devices or net systems (electrical, water, or gas supply or elements of acoustic protection), can be easily visualized using BIM + VR tools. So the management and maintenance facilities can be facilitated in a 7D/VR + BIM/model context. VR will enable the engineer to carry out a physical survey of the property or the building site (**Figure 22**). The VR technology applied to BIM tools has been enhancing a high level of integration but it is still necessary to do more in order to obtain an even more real platform, in a retrieving data perspective. As so, the importance of adding VR benefits should be a constant research point [36].

360; viewer Revit plugin; Enscape Revit plugin; Augment; Stingray; Covise; web3D; EON Icube. From the analyses of the applicability of each tool, the two most relevant combinations of tools are Enscape and EON Icube that operate over Revit. With these tools the user can visualize the building, with more or less immersive capacity, and can consult data in a Revit environment. So, the user sees final aspect of building in Enscape and EON and, in addition, retrieve the correspondent Revit component (with materials constitution and mechanic-

Enhancing BIM Methodology with VR Technology http://dx.doi.org/10.5772/intechopen.74070 77

Software houses have been demanding the integration of BIM with VR plugins in order to support the development of nD/BIM tasks. The fundamental base of BIM + VR concerns the collaboration as a first step, but the possibility of consult data while walking through the building, improves significantly the use of BIM in design, construction, maintenance, and management. Introducing VR interactive capabilities into 3D/BIM models in the construction process is a main way to test virtually and correct a construction project before the realization, as the walkthrough is available as well the visualization of data linked to each parametric object, improving the necessity collaboration within the design team. So, BIM/VR applications can contribute to reduce costs due to the construction of a real mock-up, and to avoid mistakes on site that generates material

[1] Campbell DA. Building Information Modeling: The Web3D Application for AEC. Web3D 2007, Italy, April 15-18, 2007. https://www.researchgate.net/scientific-contributions/69

[2] Azhar S, Hein M, Sketo B. Building Information Modeling (BIM): Benefits, Risks and

[3] Eadie R, Browne M, Odeyinka H, McKeown, McNiff S. BIM implementation throughout the UK construction project lifecycle: An analysis. Automation in Construction.

[4] Chen LJ, Luo H. A BIM-based construction quality management model and its applications. Automation in Construction. 2014;**46**:64-73. DOI: 10.1016/j.autcon.2014.05.009 [5] Du J, Shi Y, Mei C, Quarles J, Yan W. Communication by interaction: A multiplayer VR environment for building walkthroughs. Construction Research Congress. 2016:

Challenges. 2008. http://ascpro.ascweb.org/chair/paper/CPGT182002008.pdf

physic parameters).

wastes.

**Author details**

**References**

2281-2290

864924\_Dace\_A\_Campbell

Alcínia Zita Sampaio

Address all correspondence to: zita@civil.ist.utl.pt

Department of Civil Engineering, University of Lisbon, Portugal

2013;**36**:145-151. DOI: 10.1016/j.autcon.2013.09.001

**Figure 22.** Screen shots of 7D/VR/BIM applications.

The maintenance aspect needs the user to free consult all components of the real place through the correspondent BIM model. The model can include several parameters concerning a wall, roof and floor composition (layers of material and respective physical proprieties) and can also include maintenance plans, information on the date of replacement of equipment, certifications, and guaranties. This information should be able to be consulted from the model, and the access to this information could be easily provided. Any BIM tool allows the consult of BIM data and so it support management and maintenance within a collaborative team.

#### **6. Conclusions**

VR+ BIM enable architects and designers to better communicate design within the team members and with the client. In construction activity, the use of VR capacities brings interesting potentials. Presenting BIM models of projects in VR environment redefines communication and collaboration in the field and in the office. The advantage to using VR is in the communication of ideas, concepts, and the vision for the building. This enables all the parties to more quickly reach a full appreciation of the building plan and the distinct discipline projects. The BIM + VR topic require dissemination; application in real cases and pointed out, in reports, achievements, and limitation; following the technologic advances that supports the BIM use and the visualization of data, in real time while the interacting with the model made possible by VR technology. BIM + VR provide an opportunity to analyze and explore BIM models within virtual environments.

BIM technology and VR have the ability to innovate the building industry. At a first perspective, many feel the benefits of using a BIM model with VR are only for improving the 3D model for visual aids, but taking a second perception at a BIM model it reveals diverse reasons to adopt BIM with VR. Collaboration in VR can be the future of VR BIM. The text follows the principal technological advances that have been achieved so far. The so appellative 50 + RV tools applied in construction become from the entertainment world, but its application is almost in the perspective of visualizing stunning immersive, namely: headmounted display (Gear VR, Oculus Rift); hologram projections; Enscape Revit plugin; Smart Reality and Visidraft apps; PrioVR full or upper body suit; and CAVE platform. The construction field needs an additional and a very important complement, concerning the aim of BIM, which is retrieving, using, manipulating, and incrementing information. To cover this perspective, just a few VR software combined with BIM modeler tools, were found: Autodesk 360; viewer Revit plugin; Enscape Revit plugin; Augment; Stingray; Covise; web3D; EON Icube. From the analyses of the applicability of each tool, the two most relevant combinations of tools are Enscape and EON Icube that operate over Revit. With these tools the user can visualize the building, with more or less immersive capacity, and can consult data in a Revit environment. So, the user sees final aspect of building in Enscape and EON and, in addition, retrieve the correspondent Revit component (with materials constitution and mechanicphysic parameters).

Software houses have been demanding the integration of BIM with VR plugins in order to support the development of nD/BIM tasks. The fundamental base of BIM + VR concerns the collaboration as a first step, but the possibility of consult data while walking through the building, improves significantly the use of BIM in design, construction, maintenance, and management. Introducing VR interactive capabilities into 3D/BIM models in the construction process is a main way to test virtually and correct a construction project before the realization, as the walkthrough is available as well the visualization of data linked to each parametric object, improving the necessity collaboration within the design team. So, BIM/VR applications can contribute to reduce costs due to the construction of a real mock-up, and to avoid mistakes on site that generates material wastes.

## **Author details**

**Figure 22.** Screen shots of 7D/VR/BIM applications.

76 State of the Art Virtual Reality and Augmented Reality Knowhow

**6. Conclusions**

The maintenance aspect needs the user to free consult all components of the real place through the correspondent BIM model. The model can include several parameters concerning a wall, roof and floor composition (layers of material and respective physical proprieties) and can also include maintenance plans, information on the date of replacement of equipment, certifications, and guaranties. This information should be able to be consulted from the model, and the access to this information could be easily provided. Any BIM tool allows the consult of BIM data and so it support management and maintenance within a collaborative team.

VR+ BIM enable architects and designers to better communicate design within the team members and with the client. In construction activity, the use of VR capacities brings interesting potentials. Presenting BIM models of projects in VR environment redefines communication and collaboration in the field and in the office. The advantage to using VR is in the communication of ideas, concepts, and the vision for the building. This enables all the parties to more quickly reach a full appreciation of the building plan and the distinct discipline projects. The BIM + VR topic require dissemination; application in real cases and pointed out, in reports, achievements, and limitation; following the technologic advances that supports the BIM use and the visualization of data, in real time while the interacting with the model made possible by VR technology. BIM + VR pro-

vide an opportunity to analyze and explore BIM models within virtual environments.

BIM technology and VR have the ability to innovate the building industry. At a first perspective, many feel the benefits of using a BIM model with VR are only for improving the 3D model for visual aids, but taking a second perception at a BIM model it reveals diverse reasons to adopt BIM with VR. Collaboration in VR can be the future of VR BIM. The text follows the principal technological advances that have been achieved so far. The so appellative 50 + RV tools applied in construction become from the entertainment world, but its application is almost in the perspective of visualizing stunning immersive, namely: headmounted display (Gear VR, Oculus Rift); hologram projections; Enscape Revit plugin; Smart Reality and Visidraft apps; PrioVR full or upper body suit; and CAVE platform. The construction field needs an additional and a very important complement, concerning the aim of BIM, which is retrieving, using, manipulating, and incrementing information. To cover this perspective, just a few VR software combined with BIM modeler tools, were found: Autodesk Alcínia Zita Sampaio

Address all correspondence to: zita@civil.ist.utl.pt

Department of Civil Engineering, University of Lisbon, Portugal

### **References**


[6] Sampaio AZ. "Building Information Modelling: nD/BIM Models" (in Portuguese), FUNDEC Workshop. Lisbon, Portugal: University of Lisbon; Apr 2017

[20] Rift. Oculus Rift. 2017. https://www.oculus.com/experiences/rift/

[24] Smart Reality. The Ultimate BIM interface. 2017. http://smartreality.co/

[28] Autodesk360. 2017. https://bim42.com/2014/07/trying-out-autodesk-360/

maintenance-planning-supported-on-virtual-environments

[35] EON icube. 2017. https://www.eonreality.com/eon-icube/

thority.com/epson-moverio-bt-200-review-636996/

hal-00981165/document

com/

sort=score

Wh8IvamqmaQ

autcon.2009.11.015

4D. http://cic.vtt.fi/4D/4d.htm

[21] Mirzaei MA, Chardonnet J-R, Mérienne F, Genty A. Navigation and interaction in a real-scale digital mock-up using natural language and user gesture. Laval Virtual VRIC '14, April 9-11, 2014, Laval, France. Article No. 21. https://hal.archives-ouvertes.fr/

Enhancing BIM Methodology with VR Technology http://dx.doi.org/10.5772/intechopen.74070 79

[22] Cherian S. Integrating BIM and Virtual Reality – The Next Big Thing? 2016. https:// www.linkedin.com/pulse/integrating-bim-virtual-reality-next-big-thing-sarita-cherian

[23] Enscape3D. Real-time rendering for Revit, Sketchup and Rhino. 2017. https://enscape3d.

[25] Sims G. Epson Moverio BT-200 Smart Glasses Review. 2015. https://www.androidau-

[26] Visidraft. Visidraft: Augmented Reality for Visualizing CAD Designs. 2017. http://www. architectmagazine.com/videos/visidraft-augmented-reality-for-visualizing-cad-designs

[27] Nikolic D. Headsets, Caves, AR and VR – What's the Reality for Construction? 2017. http://www.bimplus.co.uk/technology/hea8dsets-cav6es-ar-and-vr-w7hats-reality/

[29] Augment 2017. http://www.augment.com/blog/enhancing-bim-with-augmented-reality/ [30] Stingray. 2017. https://knowledge.autodesk.com/support/stingray/getting-started?

[31] Covise. BIM Goes Virtual. 2017. https://thebimhub.com/2016/07/14/bim-goes-virtual/#.

[32] Mahalingam A, Kashyap R, Mahajan C. An evaluation of the applicability of 4D CAD on construction projects. Automation in Construction. 2010;**19**:148-159. DOI: 10.1016/j.

[33] Sampaio AZ, Santos JP, Gomes AR, Rosário DP. Construction and maintenance planning supported on virtual environments. In: Tang X-X, editor. Virtual Reality - Human Computer Interaction. Rijeka., ISBN: 978-953-51-0721-7: InTech; 2012. pp. 125-152. DOI: 10.5772/46409, ch. 07. http://www.intechopen.com/articles/show/title/construction-and-

[34] VTT. 2017. Modelling and Simulation of Construction Processes using VR Technology -

[36] Advenser. Integrating BIM and Virtual Reality – The Next Big Thing? 2016. https:// www.advenser.com/2016/11/23/integrating-bim-and-virtual-reality-the-next-big-thing/


[20] Rift. Oculus Rift. 2017. https://www.oculus.com/experiences/rift/

[6] Sampaio AZ. "Building Information Modelling: nD/BIM Models" (in Portuguese),

[7] Sampaio AZ, Mota C. BIM model of structures used in construction planning. National meeting of Structural Concrete, Coimbra, Portugal, 2-4 November 2016. ID: 4, 10 p.

[8] Silva B. BIM Methodology to Support Take-off of Quantities (in Portuguese), MSc in

[9] Araújo L. Development of Library of Walls for BIM Methodology. (in Portuguese), MSc

[10] Sampaio AZ, Simoes D. Maintenance of buildings supported on BIM methodology. Proceedings of XIV DBMC - Durability of Building Materials and Components, Ghent,

[11] Gu N, London K. Understanding and facilitating BIM adoption in the AEC industry. Automation in Construction. 2010;**19**:988-999. DOI: 10.1016/j.autcon.2010.09.002

[12] Kieferle J, Woessner U. BIM interactive – About combining BIM and virtual reality, A bidirectional interaction method for BIM models in different environments. Proceedings of the eCAADe 33, Vienna, Austria, September 2015, pp. 69-75. http://papers.cumincad.

[13] Brouchoud J. BIM Goes Virtual: Oculus Rift and virtual reality take architectural visualization to the next level. 2014. http://archvirtual.com/2014/01/19/bim-goes-virtual-ocu-

[14] Jiao Y, Zhang S, Li Y, Wang Y, Yang BM. Towards cloud augmented reality for construction application by BIM and SNS integration. Automation in Construction. 2013;**33**:37-

[15] Poussard B, Loup G, Christmann O, Eynard R, Pallot M, Richir S. Investigating the Main Characteristics of 3D Real Time Tele-Immersive Environments through the Example of a Computer Augmented Golf Platform", Laval Virtual VRIC '14, April 9-11, 2014, Laval,

[16] VRS, Virtual Reality Society. How do Virtual Reality Glasses Work? 2017. https://www.

[17] Viatechnik. 50 Virtual Reality Technologies in Architecture and Engineering. 2017, https://www.viatechnik.com/resources/50-virtual-reality-technologies-in-architecture-

[18] Gear VR. 2017, https://www.samsung.com/us/mobile/virtual-reality/gear-vr/gear-vr-

[19] Rho J, Kim J. CS10427: Virtual-Augmented Reality for Construction, Utilizing Gear VR. Autodesk University; 2015 http://au.autodesk.com/au-online/classes-on-demand/

lus-rift-and-virtual-reality-take-architectural-visualization-to-the-next-level/

FUNDEC Workshop. Lisbon, Portugal: University of Lisbon; Apr 2017

Civil Engineering. Lisbon, Portugal: University of Lisbon; 2016

in Civil Engineering. Lisbon, Portugal: University of Lisbon; 2016

Belgian, May 2017, abstract pp 83-84, e-ISBN: 978-2-35158-159-9

org/data/works/att/ecaade2015\_329.content.pdf

France. Article No. 31. DOI: 10.1145/2617841.2620720

vrs.org.uk/virtual-reality-gear/glasses/how-do-they-work.html

class-catalog/2015/revit-for-architects/cs10427#chapter=0

47. DOI: 10.1016/j.autcon.2012.09.018

engineering-and-construction/

with-controller-sm-r324nzaaxar/

http://be2016.dec.uc.pt/

78 State of the Art Virtual Reality and Augmented Reality Knowhow


**Chapter 6**

**Provisional chapter**

**Augmented Reality Trends in Education between 2016**

The aim of this chapter is to review literature regarding using augmented reality (AR) in education articles published in between 2016 and 2017 years. The literature source was Web of Science and SSCI, SCI-EXPANDED, A&HCI, CPCI-S, CPCI-SSH, and ESCI indexes. Fifty-two articles were reviewed; however, 14 of them were not been included in the study. As a result, 38 articles were examined. Level of education, field of education, and material types of AR used in education and reported educational advantages of AR have been investigated. All articles are categorized according to target groups, which are early childhood education, primary education, secondary education, high school education, graduate education, and others. AR technology has been mostly carried out in primary and graduate education. "Science education" is the most explored field of education. Mobile applications and marker-based materials on paper have been mostly preferred. The major advantages indicated in the articles are "Learning/Academic

**Keywords:** augmented reality, education, review, web of science, educational target

Augmented reality (AR) is a new technology that acts as a bridge between real world and virtual environment by providing synchronous interaction. Virtual objects can be added on real world through this technology. In other words, during recording of the real world with the camera, AR uses pre-determined target points in real world by connecting virtual objects and interpreting the results through certain programs [1]. AR can distinguish itself

**Augmented Reality Trends in Education between 2016** 

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

© 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

DOI: 10.5772/intechopen.74943

**and 2017 Years**

**and 2017 Years**

Additional information is available at the end of the chapter

Additional information is available at the end of the chapter

Achievement," "Motivation," and "Attitude".

http://dx.doi.org/10.5772/intechopen.74943

Rabia M. Yilmaz

Rabia M. Yilmaz

**Abstract**

group

**1. Introduction**

#### **Augmented Reality Trends in Education between 2016 and 2017 Years Augmented Reality Trends in Education between 2016 and 2017 Years**

DOI: 10.5772/intechopen.74943

Rabia M. Yilmaz Rabia M. Yilmaz

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.74943

#### **Abstract**

The aim of this chapter is to review literature regarding using augmented reality (AR) in education articles published in between 2016 and 2017 years. The literature source was Web of Science and SSCI, SCI-EXPANDED, A&HCI, CPCI-S, CPCI-SSH, and ESCI indexes. Fifty-two articles were reviewed; however, 14 of them were not been included in the study. As a result, 38 articles were examined. Level of education, field of education, and material types of AR used in education and reported educational advantages of AR have been investigated. All articles are categorized according to target groups, which are early childhood education, primary education, secondary education, high school education, graduate education, and others. AR technology has been mostly carried out in primary and graduate education. "Science education" is the most explored field of education. Mobile applications and marker-based materials on paper have been mostly preferred. The major advantages indicated in the articles are "Learning/Academic Achievement," "Motivation," and "Attitude".

**Keywords:** augmented reality, education, review, web of science, educational target group

#### **1. Introduction**

Augmented reality (AR) is a new technology that acts as a bridge between real world and virtual environment by providing synchronous interaction. Virtual objects can be added on real world through this technology. In other words, during recording of the real world with the camera, AR uses pre-determined target points in real world by connecting virtual objects and interpreting the results through certain programs [1]. AR can distinguish itself

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

from other technologies by combining virtual and real objects, by providing real-time interaction, and by involving 3D objects [1, 2]. AR is an increasingly popular technology that can be used on different platforms such as desktop, laptop computers, portable devices, and smartphones [3]. Applications developed with AR technology allow use of 3D objects, texts, images, videos, and animations, and this technology provides the use of them together at the same time [4]. Therefore, users can naturally interact with events, information, and objects [5, 6].

Advantages of AR in education Providing a sense of reality [21]

Supported learning approaches Authentic learning environments [18]

problems

**Table 1.** Advantages of AR, gains achieved, and learning approaches it supports.

Gains of using augmented reality in

education

Presenting a natural experience [11, 14] Visualize complex relationships [19, 22] Offer experiences that cannot be done in real life [6, 19] Concrete abstract concepts [6] Having fun learning [23] Presenting safe learning environment [6, 24, 25] Saving time and space [24, 25] Increasing student participation [6] Providing flexibility [16]

Augmented Reality Trends in Education between 2016 and 2017 Years

http://dx.doi.org/10.5772/intechopen.74943

83

Situational learning environments [6, 26] Constructivist learning environments [3] Learning by doing environments [6] Inquiry-based learning environments [6, 27] Research-based learning environments [18]

Increasing attention [11, 14, 25] Making learning attractive and effective [6, 28–31] Providing motivation [11, 14, 25, 32] Providing interaction [6, 13, 17, 19, 33–35]

Facilitating understanding [17, 36, 37]

Creating contextual awareness [17] Increasing engagement [17, 34] Ensuring permanent learning [17] Improving communication [17] Increasing collaboration [18, 39] Triggering creativity [18, 31, 40] Developed imagination [18, 40] Controlling self-learning [18, 34] Increasing spatial ability [6, 27, 34] Enhancing problem-solving skills [16] Improving interpretation skills [16]

[38]

Connecting with real world experiences and

With the development of software and hardware used in AR, this technology has begun to be used in many areas such as entertainment, marketing, military, medicine, engineering, psychology, and advertising [1, 3, 7]. Thanks to AR's advanced technology, related applications have been transformed from a blank area into a rich learning experience [8]. This situation leads AR technology to be used in education area. It is known that technological tools used in education provide new opportunities to increase interaction of learners and to learn by enjoying, making the learning process more active, effective, meaningful, and motivating [9–11]. AR technology has also attracted attention in education field with its ability to allow interaction with virtual and real objects, to learn by doing, and to increase attention and motivation [12]. AR offers many advantages when used in education and provides educational gains. While enabling rich interaction [13], AR provides a natural experience, enhancing attention and motivation [14, 15]. In addition, it enhances interpreting, problem solving [16], and creative thinking skills [17] and provides flexibility for students [16]. Although AR technology provides significant contributions, there are still many problems that need to be addressed. The first of these problems is the difficulty of developing and implementing content for AR applications. It is known that many teachers and learners are biased to use AR, especially because of technical knowledge requirement in 3D object development [18]. In addition, external factors such as light, output, and image quality negatively affect the applications. Students who use AR may have to use more than one device. It requires them to have spatial orientation, collaborative skills, problem-solving skills, and ability to intervene in this technology [19]. Besides the technical and pedagogical problems that AR brought with, the potential of having educational applications has led researchers to this field.

When literature is examined, it is observed that many studies have been carried out for the use of AR applications in education. Yilmaz [20] categorized advantages of AR, gains achieved, and learning approaches it supports and they are presented in **Table 1** in detail.

The aim of this chapter is to review literature regarding using AR in education. The literature source for this review was Web of Science and SSCI, SCI-EXPANDED, A&HCI, CPCI-S, CPCI-SSH, and ESCI indexes. The time span was limited from 2016 to 2017 and the document type was determined as "journal articles" in order to review studies regarding using AR in education of more consistent quality. The keyword was "Augmented reality" in title section and results were refined by "Education Educational Research" categories, one of the Web of Science categories. For this chapter, 52 articles regarding using AR in education were reviewed (*Reached date: October, 2017*). However, 14 of them were excluded in the study since 7 of them were review articles, 4 of them were technical articles, and 3 articles were not reached. Finally, 38 articles were examined for this study.


**Table 1.** Advantages of AR, gains achieved, and learning approaches it supports.

from other technologies by combining virtual and real objects, by providing real-time interaction, and by involving 3D objects [1, 2]. AR is an increasingly popular technology that can be used on different platforms such as desktop, laptop computers, portable devices, and smartphones [3]. Applications developed with AR technology allow use of 3D objects, texts, images, videos, and animations, and this technology provides the use of them together at the same time [4]. Therefore, users can naturally interact with events, information, and

With the development of software and hardware used in AR, this technology has begun to be used in many areas such as entertainment, marketing, military, medicine, engineering, psychology, and advertising [1, 3, 7]. Thanks to AR's advanced technology, related applications have been transformed from a blank area into a rich learning experience [8]. This situation leads AR technology to be used in education area. It is known that technological tools used in education provide new opportunities to increase interaction of learners and to learn by enjoying, making the learning process more active, effective, meaningful, and motivating [9–11]. AR technology has also attracted attention in education field with its ability to allow interaction with virtual and real objects, to learn by doing, and to increase attention and motivation [12]. AR offers many advantages when used in education and provides educational gains. While enabling rich interaction [13], AR provides a natural experience, enhancing attention and motivation [14, 15]. In addition, it enhances interpreting, problem solving [16], and creative thinking skills [17] and provides flexibility for students [16]. Although AR technology provides significant contributions, there are still many problems that need to be addressed. The first of these problems is the difficulty of developing and implementing content for AR applications. It is known that many teachers and learners are biased to use AR, especially because of technical knowledge requirement in 3D object development [18]. In addition, external factors such as light, output, and image quality negatively affect the applications. Students who use AR may have to use more than one device. It requires them to have spatial orientation, collaborative skills, problem-solving skills, and ability to intervene in this technology [19]. Besides the technical and pedagogical problems that AR brought with, the potential of having educational applications has led

When literature is examined, it is observed that many studies have been carried out for the use of AR applications in education. Yilmaz [20] categorized advantages of AR, gains achieved,

The aim of this chapter is to review literature regarding using AR in education. The literature source for this review was Web of Science and SSCI, SCI-EXPANDED, A&HCI, CPCI-S, CPCI-SSH, and ESCI indexes. The time span was limited from 2016 to 2017 and the document type was determined as "journal articles" in order to review studies regarding using AR in education of more consistent quality. The keyword was "Augmented reality" in title section and results were refined by "Education Educational Research" categories, one of the Web of Science categories. For this chapter, 52 articles regarding using AR in education were reviewed (*Reached date: October, 2017*). However, 14 of them were excluded in the study since 7 of them were review articles, 4 of them were technical articles, and 3 articles were not reached.

and learning approaches it supports and they are presented in **Table 1** in detail.

objects [5, 6].

82 State of the Art Virtual Reality and Augmented Reality Knowhow

researchers to this field.

Finally, 38 articles were examined for this study.

In total, 38 articles were analyzed from many different journals, including EURASIA Journal of Mathematics Science and Technology Education (5), Interactive Learning Environments (5), Computers & Education (4), British Journal Of Educational Technology (4), Journal of Computer Assisted Learning (2), Education and Science (2), Education Tech Research Dev (2), Asia-Pacific Edu Res, IEEE Transactions on Learning Technologies, Australasian Journal of Educational Technology, BMC Medical Education, Croatian Journal of Education, Digital Education Review, Early Child Development and Care, Innovations in Education and Teaching International, International Journal of Science Education Part B Communication and Public Engagement, International Journal of Serious Games, International Journal of Emerging Technologies in Learning, Journal of Geography in Higher Education, Revista Iberoamericana de Educación a Distancia, and Journal of Research on Technology in Education.

Firstly, level of education used in these reviewed studies has been investigated. Results show that AR technology has been mostly carried out in *primary* and *graduate education*. On the other hand, early childhood education and secondary school education need further research regarding using AR in education. All results are summarized in **Figure 1**.

When analyzed using AR by field of education, "*science education* (f = 16)" is the most explored field of education. Reviewed studies have generally focused on "mathematic education," "storytelling," "foreign language education," "culture education," and "health education." **Figure 2** shows all results regarding using AR by field of education.

indicate more than one advantage, total frequency of them is high. The results show that major advantages indicated in the articles are: "Learning/Academic Achievement" (f = 23), "Motivation" (f = 9), and "Attitude" (f = 6). Also, many variables, such as academic procrastination; writing skills; cognitive aspect; the number of errors they made; ability to remember the content; page design; teachers' acceptance and views; potential uses of augmented reality in education; social, emotional, and cognitive improvement; story comprehension performance; time spent; acquired learnings; metacognitive perception; experiential activity; and prospects of the use of AR for study, have been focused in reviewed articles by researchers.

Augmented Reality Trends in Education between 2016 and 2017 Years

http://dx.doi.org/10.5772/intechopen.74943

85

**Figure 2.** Using AR by field of education in reviewed articles.

**Figure 3.** Material types of AR used in education.

Material types of AR used in education have been examined in reviewed articles. Mobile applications (f = 16) and marker-based materials on paper (f = 12) have been mostly preferred in them. Besides, some of the studies have used AR picture books and AR game systems. Related results are stated in **Figure 3**.

**Figure 1.** Target group of reviewed articles.

Augmented Reality Trends in Education between 2016 and 2017 Years http://dx.doi.org/10.5772/intechopen.74943 85

**Figure 2.** Using AR by field of education in reviewed articles.

In total, 38 articles were analyzed from many different journals, including EURASIA Journal of Mathematics Science and Technology Education (5), Interactive Learning Environments (5), Computers & Education (4), British Journal Of Educational Technology (4), Journal of Computer Assisted Learning (2), Education and Science (2), Education Tech Research Dev (2), Asia-Pacific Edu Res, IEEE Transactions on Learning Technologies, Australasian Journal of Educational Technology, BMC Medical Education, Croatian Journal of Education, Digital Education Review, Early Child Development and Care, Innovations in Education and Teaching International, International Journal of Science Education Part B Communication and Public Engagement, International Journal of Serious Games, International Journal of Emerging Technologies in Learning, Journal of Geography in Higher Education, Revista Iberoamericana

Firstly, level of education used in these reviewed studies has been investigated. Results show that AR technology has been mostly carried out in *primary* and *graduate education*. On the other hand, early childhood education and secondary school education need further research

When analyzed using AR by field of education, "*science education* (f = 16)" is the most explored field of education. Reviewed studies have generally focused on "mathematic education," "storytelling," "foreign language education," "culture education," and "health education."

Material types of AR used in education have been examined in reviewed articles. Mobile applications (f = 16) and marker-based materials on paper (f = 12) have been mostly preferred in them. Besides, some of the studies have used AR picture books and AR game systems.

"Reported Educational Advantages of AR" has been explored as another category in this review. **Table 2** shows the results regarding them in the articles analyzed. As one study can

de Educación a Distancia, and Journal of Research on Technology in Education.

regarding using AR in education. All results are summarized in **Figure 1**.

**Figure 2** shows all results regarding using AR by field of education.

Related results are stated in **Figure 3**.

84 State of the Art Virtual Reality and Augmented Reality Knowhow

**Figure 1.** Target group of reviewed articles.

indicate more than one advantage, total frequency of them is high. The results show that major advantages indicated in the articles are: "Learning/Academic Achievement" (f = 23), "Motivation" (f = 9), and "Attitude" (f = 6). Also, many variables, such as academic procrastination; writing skills; cognitive aspect; the number of errors they made; ability to remember the content; page design; teachers' acceptance and views; potential uses of augmented reality in education; social, emotional, and cognitive improvement; story comprehension performance; time spent; acquired learnings; metacognitive perception; experiential activity; and prospects of the use of AR for study, have been focused in reviewed articles by researchers.

**Figure 3.** Material types of AR used in education.


Thirdly, Huang et al. [43] explored the effectiveness of coloring activities with AR technology in early art education. ColAR mobile application was used for coloring activities. This study showed that children could control, interact, and design with AR application, and this technology was seen as pedagogical innovation. All children enjoyed playing with them and loved it so much. In addition, teachers believed that AR promoted children's development. Lastly, Cheng et al. [44] focused on how the children and their parents behaved when using AR picture books. They consisted of four groups such as "parent as dominator," "children as dominator," "communicative child-parent pair," and "low communicative child-parent pair." In parent as dominator group, parents tent to narrate story to children. In children as dominator group, children had a tendency to operate AR book and engage them. Communicative child-parent group's parents guided their children to find virtual information in the book. In low communicative child-parent pair, parents did not continually inter-

Augmented Reality Trends in Education between 2016 and 2017 Years

http://dx.doi.org/10.5772/intechopen.74943

87

When analyzed the articles related to using AR in primary education, 11 studies were found. Joo-Nagata et al. [45] aimed to design a navigation mobile application in Geography field and evaluate effectiveness of the software. Their results revealed that experimental group using the mobile application had better learning score than control group. Besides, AR technology enhanced the effectiveness of learning process, contributed to studentcontent interaction, and increased students' performance. Gun et al. [46] examined the effect of AR technology on elementary students' spatial ability and academic achievement and views of students and their teachers. They studied on mathematics education and used marker-based material on paper. This study showed that there was a significant increase on spatial ability of experimental and control groups. However, there was no significant difference between mean scores of spatial ability and academic achievement in both groups. Academic achievement scores had an increase in experimental group. Hung et al. [47] focused on determining advantages of AR in biology teaching. In addition, they examined which material enabled students' learning the most. They compared AR graphic book, a picture book, or physical interactions with each other. Their results showed that AR graphic book provided a practical way to examine the lesson. In addition, students enjoyed

Nadolny [48] studied on over 13,000 data points in two different interactive print activities in order to reveal trends and patterns in user engagement. Marker-based material on paper was used in mathematics and informal learning field. This study revealed that user interaction was impressed by instructional design, pedagogical strategies, and digital interactions. Besides, page design, feedback, and cognitive tasks were important elements to engage users. Hsu [49] developed two different AR educational game systems (self-directed and task-based) for teaching English vocabulary. He evaluated students' flow experience, English learning anxiety, cognitive load, and learning effectiveness. Students using both game had similar learning performance; however, self-directed system provided higher flow experience for them. In addition, he stated that a little learning anxiety and mental effort were helpful and important

act with AR books.

the book and preferred it.

factors for learning process.

**1.2. Using augmented reality in primary education**

**Table 2.** Reported educational advantages of AR.

In this chapter, reviewed articles are categorized according to target groups, which are early childhood education, primary education, secondary education, high school education, graduate education, and others. Related studies with each title are stated below in detail.

#### **1.1. Using augmented reality in early childhood education**

When analyzed the articles related to using AR in early childhood education, four studies were found. Firstly, Yilmaz et al. [41] examined preschool children's attitudes, enjoyment, and story comprehension performance. They used AR picture books for storytelling activities. This study showed that most of the children felt "very happy" during this activity, found them interesting, and enjoyed them. Their story comprehension performance was good. Besides, AR picture books were attractive for children and they perceived them as magical and enjoyable. Secondly, Safar et al. [42] investigated effectiveness of using AR for teaching English alphabet lesson with flash cards to kindergarten children. A mobile application was developed for using flash cards. The results revealed that there were significant differences between control (traditional activities) and experimental group (AR activities) in terms of interaction and academic achievement in favor of experimental group. In addition, there has been a strong positive correlation between their interaction and academic achievement in experimental AR group.

Thirdly, Huang et al. [43] explored the effectiveness of coloring activities with AR technology in early art education. ColAR mobile application was used for coloring activities. This study showed that children could control, interact, and design with AR application, and this technology was seen as pedagogical innovation. All children enjoyed playing with them and loved it so much. In addition, teachers believed that AR promoted children's development. Lastly, Cheng et al. [44] focused on how the children and their parents behaved when using AR picture books. They consisted of four groups such as "parent as dominator," "children as dominator," "communicative child-parent pair," and "low communicative child-parent pair." In parent as dominator group, parents tent to narrate story to children. In children as dominator group, children had a tendency to operate AR book and engage them. Communicative child-parent group's parents guided their children to find virtual information in the book. In low communicative child-parent pair, parents did not continually interact with AR books.

#### **1.2. Using augmented reality in primary education**

In this chapter, reviewed articles are categorized according to target groups, which are early childhood education, primary education, secondary education, high school education, gradu-

When analyzed the articles related to using AR in early childhood education, four studies were found. Firstly, Yilmaz et al. [41] examined preschool children's attitudes, enjoyment, and story comprehension performance. They used AR picture books for storytelling activities. This study showed that most of the children felt "very happy" during this activity, found them interesting, and enjoyed them. Their story comprehension performance was good. Besides, AR picture books were attractive for children and they perceived them as magical and enjoyable. Secondly, Safar et al. [42] investigated effectiveness of using AR for teaching English alphabet lesson with flash cards to kindergarten children. A mobile application was developed for using flash cards. The results revealed that there were significant differences between control (traditional activities) and experimental group (AR activities) in terms of interaction and academic achievement in favor of experimental group. In addition, there has been a strong positive correlation between their interaction and academic achievement in

ate education, and others. Related studies with each title are stated below in detail.

**Advantages Number of studies Percentage (%)**

Learning/academic achievement 23 34.33 Motivation 9 13.43 Attitude 6 8.96 Satisfaction 4 5.97 Usability 4 5.97 Interaction 3 4.48 Spatial ability 3 4.48 Emotions 2 2.99 Cognitive load 2 2.99 Learning anxiety 2 2.99 Perception 2 2.99 Retention 2 2.99 Behaviors 2 2.99 Enjoyment 1 1.49 Flow experience 1 1.49 Communication 1 1.49

86 State of the Art Virtual Reality and Augmented Reality Knowhow

**1.1. Using augmented reality in early childhood education**

**Table 2.** Reported educational advantages of AR.

experimental AR group.

When analyzed the articles related to using AR in primary education, 11 studies were found. Joo-Nagata et al. [45] aimed to design a navigation mobile application in Geography field and evaluate effectiveness of the software. Their results revealed that experimental group using the mobile application had better learning score than control group. Besides, AR technology enhanced the effectiveness of learning process, contributed to studentcontent interaction, and increased students' performance. Gun et al. [46] examined the effect of AR technology on elementary students' spatial ability and academic achievement and views of students and their teachers. They studied on mathematics education and used marker-based material on paper. This study showed that there was a significant increase on spatial ability of experimental and control groups. However, there was no significant difference between mean scores of spatial ability and academic achievement in both groups. Academic achievement scores had an increase in experimental group. Hung et al. [47] focused on determining advantages of AR in biology teaching. In addition, they examined which material enabled students' learning the most. They compared AR graphic book, a picture book, or physical interactions with each other. Their results showed that AR graphic book provided a practical way to examine the lesson. In addition, students enjoyed the book and preferred it.

Nadolny [48] studied on over 13,000 data points in two different interactive print activities in order to reveal trends and patterns in user engagement. Marker-based material on paper was used in mathematics and informal learning field. This study revealed that user interaction was impressed by instructional design, pedagogical strategies, and digital interactions. Besides, page design, feedback, and cognitive tasks were important elements to engage users. Hsu [49] developed two different AR educational game systems (self-directed and task-based) for teaching English vocabulary. He evaluated students' flow experience, English learning anxiety, cognitive load, and learning effectiveness. Students using both game had similar learning performance; however, self-directed system provided higher flow experience for them. In addition, he stated that a little learning anxiety and mental effort were helpful and important factors for learning process.

Cascales-Martínez et al. [50] used a multi-touch tabletop system for mathematics learning with special needs. They focused on feasibility of it and found that this tabletop was feasible, attractive, and motivating. In addition, it contributed to improve the knowledge acquired and motivation of students. Hwang et al. [51] used an AR mobile game application for biology learning to improve students' academic achievement and attitude. The results showed that students' academic achievement and attitude enhanced with AR game. Chen et al. [52] investigated students' motivation, attitude, and learning outcomes when using concept map with AR picture book. Students using concept mapped AR technology were better than other students using only AR technology. Besides, they stated that using concept map provided them to organize their learning process.

**1.4. Using augmented reality in high school education**

on screen, and learning environment into consideration.

**1.5. Using augmented reality in graduate education**

reduced students' academic procrastination.

When analyzed the articles related to using AR in high school education, five studies were found. Wang [60] investigated the effectiveness of AR-based writing support system for Chinese writing education. The results indicated that this system helped students to improve their writing performance. Besides, it provided students to start writing quickly their first paragraph and to enrich their ideas. Wang [61] compared effectiveness of AR-based material and YouTube videos in software editing course. AR-based content provided positive contribution to support blended learning. Besides, it helped to increase students' interest and made them more active in the case that designers take proper content design, information displayed

Augmented Reality Trends in Education between 2016 and 2017 Years

http://dx.doi.org/10.5772/intechopen.74943

89

Mumtaz et al. [62] focused on students' learning and motivation. They used AR mobile application for science field in blended and traditional classroom learning. Students using AR application had better performance and this application affected their learning positively. In addition, students' motivation was high during learning process. Chang et al. [63] compared AR technology and interactive simulations in terms of students' learning and attitude. They used a mobile application for AR in science field. They found that there was no significant difference between them in learning and attitude. However, there was a significant difference in terms of perception in favor of AR technology. Ibáñez et al. [64] used AR-based simulation system for science learning. They consisted experimental and control group randomly. While experimental group had an extra support, control group had no support for activities. They examined students' behavioral patterns and learning performance in both groups. The results showed that experimental group outperformed in learning performance compared with con-

trol group. Students in control group were more willing to search information.

When analyzed the articles related to using AR in graduate education, 10 studies were found. Redondo Domínguez et al. [65] examined architecture students' experiences. They formed experimental and control groups. Students in experimental group used AR technology with a mobile application. Their results showed that using AR technology in course provided students to increase their academic performance, motivation, and satisfaction. Montoya et al. [66] developed an AR mobile application and they compared effect of static and dynamic content on students' science learning. Their results revealed that using dynamic content in AR applications helped students to enhance learning perception and performance. Bendicho et al. [67] focused on effect of AR technology on engineering students' academic procrastination. They used marker-based material on paper with mobile application. Results showed that using AR

Salinas et al. [68] developed an AR application in mathematics education. According to their results, students' anxiety towards mathematics could be reduced by understanding mathematical knowledge. In addition, using AR helped to develop students' spatial ability. Carbonell Carrera et al. [69] examined students' spatial orientation when using AR technology. They used marker-based material on paper in engineering field. Their results revealed that using AR enhanced students' spatial orientation skills. Cheng [70] focused on effects of

Laine et al. [53] aimed to develop an AR game system called as Leometry in science field. They produced a prototype and evaluated its effectiveness. The results revealed that it was feasible and had a valuable potential for science learning. In addition, they stated that using AR technology in game was important motivator for students. Solak et al. [54] examined current applications in literature about AR in education and tested effectiveness of AR application for English teaching. They used marker-based material on paper and their results revealed that students using AR application had better learning scores than those in control group. They stated that using AR technology in English teaching was affective way for improving learning performance. Chang et al. [55] tested the effectiveness of ARFlora system, which is marker-based material on paper about biology filed. They compared this system with digital video in terms of students' learning. Results revealed that both materials had same effect on students' learning. However, AR system was more effective and useful in terms of retaining knowledge and motivating students.

#### **1.3. Using augmented reality in secondary education**

When analyzed the articles related to using AR in secondary education, four studies were found. Cai et al. [56] examined the effectiveness of AR-based natural interaction in terms of learning performance, attitudes, and deep understanding. They compared it with traditional learning tools for physics learning. They used AR-based motion-sensing software and results showed that it could increase their learning and attitude. Salmi et al. [57] investigated motivational and cognitive aspects of informal learning while using AR technology in learning process and made a SEM path analysis. They focused on science education and found that AR was effective especially in lowest achieving group and for girls. They stated that AR was a promising method for teaching abstract phenomena.

Huang et al. [58] used AR-based learning system for biology field. They consisted of three groups: (1) self-directed AR group, (2) AR system with commenter guidance, and (3) traditional group. They compared them in terms of learning outcomes, emotions, and experiential learning activity. The results showed that AR system with commenter guidance had better learning performance than the traditional group. In addition, self-directed AR group has more positive emotion than the others do. Hsiao et al. [59] compared to AR technology with multimedia tools in inquiry-based learning environment for science education. They developed a manipulative AR system and used it in experimental group and control group used multimedia tools for learning activities. This study revealed that integrating AR technology in learning activities had positive effect on students' academic achievement and motivation in experimental group.

#### **1.4. Using augmented reality in high school education**

Cascales-Martínez et al. [50] used a multi-touch tabletop system for mathematics learning with special needs. They focused on feasibility of it and found that this tabletop was feasible, attractive, and motivating. In addition, it contributed to improve the knowledge acquired and motivation of students. Hwang et al. [51] used an AR mobile game application for biology learning to improve students' academic achievement and attitude. The results showed that students' academic achievement and attitude enhanced with AR game. Chen et al. [52] investigated students' motivation, attitude, and learning outcomes when using concept map with AR picture book. Students using concept mapped AR technology were better than other students using only AR technology. Besides, they stated that using concept map provided

Laine et al. [53] aimed to develop an AR game system called as Leometry in science field. They produced a prototype and evaluated its effectiveness. The results revealed that it was feasible and had a valuable potential for science learning. In addition, they stated that using AR technology in game was important motivator for students. Solak et al. [54] examined current applications in literature about AR in education and tested effectiveness of AR application for English teaching. They used marker-based material on paper and their results revealed that students using AR application had better learning scores than those in control group. They stated that using AR technology in English teaching was affective way for improving learning performance. Chang et al. [55] tested the effectiveness of ARFlora system, which is marker-based material on paper about biology filed. They compared this system with digital video in terms of students' learning. Results revealed that both materials had same effect on students' learning. However, AR system

was more effective and useful in terms of retaining knowledge and motivating students.

When analyzed the articles related to using AR in secondary education, four studies were found. Cai et al. [56] examined the effectiveness of AR-based natural interaction in terms of learning performance, attitudes, and deep understanding. They compared it with traditional learning tools for physics learning. They used AR-based motion-sensing software and results showed that it could increase their learning and attitude. Salmi et al. [57] investigated motivational and cognitive aspects of informal learning while using AR technology in learning process and made a SEM path analysis. They focused on science education and found that AR was effective especially in lowest achieving group and for girls. They stated that AR was

Huang et al. [58] used AR-based learning system for biology field. They consisted of three groups: (1) self-directed AR group, (2) AR system with commenter guidance, and (3) traditional group. They compared them in terms of learning outcomes, emotions, and experiential learning activity. The results showed that AR system with commenter guidance had better learning performance than the traditional group. In addition, self-directed AR group has more positive emotion than the others do. Hsiao et al. [59] compared to AR technology with multimedia tools in inquiry-based learning environment for science education. They developed a manipulative AR system and used it in experimental group and control group used multimedia tools for learning activities. This study revealed that integrating AR technology in learning activities had positive effect on students' academic achievement and motivation in experimental group.

them to organize their learning process.

88 State of the Art Virtual Reality and Augmented Reality Knowhow

**1.3. Using augmented reality in secondary education**

a promising method for teaching abstract phenomena.

When analyzed the articles related to using AR in high school education, five studies were found. Wang [60] investigated the effectiveness of AR-based writing support system for Chinese writing education. The results indicated that this system helped students to improve their writing performance. Besides, it provided students to start writing quickly their first paragraph and to enrich their ideas. Wang [61] compared effectiveness of AR-based material and YouTube videos in software editing course. AR-based content provided positive contribution to support blended learning. Besides, it helped to increase students' interest and made them more active in the case that designers take proper content design, information displayed on screen, and learning environment into consideration.

Mumtaz et al. [62] focused on students' learning and motivation. They used AR mobile application for science field in blended and traditional classroom learning. Students using AR application had better performance and this application affected their learning positively. In addition, students' motivation was high during learning process. Chang et al. [63] compared AR technology and interactive simulations in terms of students' learning and attitude. They used a mobile application for AR in science field. They found that there was no significant difference between them in learning and attitude. However, there was a significant difference in terms of perception in favor of AR technology. Ibáñez et al. [64] used AR-based simulation system for science learning. They consisted experimental and control group randomly. While experimental group had an extra support, control group had no support for activities. They examined students' behavioral patterns and learning performance in both groups. The results showed that experimental group outperformed in learning performance compared with control group. Students in control group were more willing to search information.

#### **1.5. Using augmented reality in graduate education**

When analyzed the articles related to using AR in graduate education, 10 studies were found. Redondo Domínguez et al. [65] examined architecture students' experiences. They formed experimental and control groups. Students in experimental group used AR technology with a mobile application. Their results showed that using AR technology in course provided students to increase their academic performance, motivation, and satisfaction. Montoya et al. [66] developed an AR mobile application and they compared effect of static and dynamic content on students' science learning. Their results revealed that using dynamic content in AR applications helped students to enhance learning perception and performance. Bendicho et al. [67] focused on effect of AR technology on engineering students' academic procrastination. They used marker-based material on paper with mobile application. Results showed that using AR reduced students' academic procrastination.

Salinas et al. [68] developed an AR application in mathematics education. According to their results, students' anxiety towards mathematics could be reduced by understanding mathematical knowledge. In addition, using AR helped to develop students' spatial ability. Carbonell Carrera et al. [69] examined students' spatial orientation when using AR technology. They used marker-based material on paper in engineering field. Their results revealed that using AR enhanced students' spatial orientation skills. Cheng [70] focused on effects of using AR books in culture education on students' cognitive load, motivation, and attitudes. He stated that they had low cognitive load, high motivation, and positive attitude when using AR book. Juan [71] used a mobile AR application for dental education. According to the results, AR application was affective tool to convey knowledge. Participants satisfied this application and perceived it as easy to use. In addition, they wanted to use it in future.

"culture education," and "health education." Mobile applications and marker-based materials on paper have been mostly preferred. Major advantages indicated in the articles are "Learning/Academic Achievement," "Motivation," and "Attitude." After all, it can be said that AR has been gaining importance and will be popular in educational fields by becoming

Augmented Reality Trends in Education between 2016 and 2017 Years

http://dx.doi.org/10.5772/intechopen.74943

91

Address all correspondence to: rabia.kufrevi@gmail.com; rkufrevi@atauni.edu.tr

Department of Computer Education and Instructional Technology, Ataturk University

[1] Azuma RT.A survey of augmented reality. Presence: Teleoperators and Virtual Environments.

[2] Moreno E, MacIntyre B, Bolter JD. Alice's adventure's in new media: An exploration of interactive narratives in augmented reality. Paper presented at CAST'01, Bonn, Germany.

[3] Kirner TG, Reis FMV, Kirner C. Development of an interactive book with Augmented Reality for teaching and learning geometric shapes. Information Systems and Technologies

[4] Wang D, He L, Dou K. StoryCube: Supporting children's storytelling with a tangible tool. The Journal of Supercomputing. 2014;**70.1**:269-283. DOI: 10.1007/s11227-012-0855-x

[5] Wojciechowski R, Walczak K, White M, Cellary W. Building virtual and augmented reality museum exhibitions. In: Proceedings of 9th International Conference on 3D Web

[6] Wojciechowski R, Cellary W. Evaluation of learners' attitude toward learning in ARIES augmented reality environments. Computers and Education. 2013;**68**:570-585

[7] Azuma RT, Baillot Y, Behringer R, Feiner S, Julier S, MacIntyre B. Recent advances in augmented reality. IEEE Computer Graphics and Applications. 2001;**21**:34-37

[8] Alcañiz M, Contero M, Pérez-López DC, Ortega M. Augmented reality technology for education, new achievements in technology education and development. In: Soomro S, editor. InTech. 2010. pp. 247-256. Available form: http://cdn.intechopen.com/pdfs/10538/

(CISTI). 7th Iberian Conference on. IEEE; 2012. pp. 1-6

Technology (Web3D 2004). ACM; 2004. pp. 135-144

InTech-Augmented\_reality\_technology\_for\_education.pdf

widespread use of this technology in daily life.

**Author details**

Rabia M. Yilmaz

Erzurum, Turkey

**References**

1997;**6**(4):355-385

Sep 2001

Ferrer-Torregrosa et al. [72] compared three tools, which were notes with images, videos, and AR (marker-based material on paper) in health education field. They focused on students' time spent, learnings, perception of metacognition, and the prospects of the use of AR. Their results showed that AR was the most effective tool among these tools in all aspects. Harley et al. [73] explored using AR in history education. They organized a historical tour with mobile AR location-based application and examined students' emotions and learning outcomes. They experienced it in and outside of the laboratory. The results revealed that they enjoyed from this experience. In addition, AR application was efficient in both environments. Martin-Gonzalez et al. [74] developed an AR system with Kinect technology for physics and mathematics education. They evaluated its usability and results showed that usability of it was acceptable. In addition, participants had positive attitude towards this system.

#### **1.6. Using augmented reality in other target groups**

When analyzed the articles related to using AR in other target groups, four studies were found. These target groups were primary school teacher, construction workers, special students with intellectual disabilities and autism, and adults. Alkhattabi [75] focused on primary school teachers' acceptance of AR technology. In addition, advantages and barriers of AR were investigated with e-learning tool. Results showed that teachers accepted this technology and would like to use in their lessons. Pejoska et al. [76] used a mobile application for construction workers and analyzed their experiences in informal learning. They suggested some design principles for AR app to support informal learning. McMahon et al. [77] studied on students with intellectual disabilities and autism. They used marker-based material on paper in science education field. The results revealed that these students succeeded in defining new science vocabulary terms with AR technology support. Boletsis et al. [78] tested usability of AR-based cubes gaming system for adults. This game was developed for cognitive field. According to the results, users satisfied to use this game system and gave positive responses.

#### **2. Conclusion**

The aim of this chapter is to review literature regarding the use of AR in education articles published in between 2016 and 2017 years. Fifty-two articles were reviewed; however, 14 of them were not included in the study. As a result, 38 articles were examined. The results showed that AR technology has been mostly carried out in primary and graduate education. "Science education" is the most explored field of education. Reviewed studies have generally focused on "mathematic education," "storytelling," "foreign language education," "culture education," and "health education." Mobile applications and marker-based materials on paper have been mostly preferred. Major advantages indicated in the articles are "Learning/Academic Achievement," "Motivation," and "Attitude." After all, it can be said that AR has been gaining importance and will be popular in educational fields by becoming widespread use of this technology in daily life.

#### **Author details**

using AR books in culture education on students' cognitive load, motivation, and attitudes. He stated that they had low cognitive load, high motivation, and positive attitude when using AR book. Juan [71] used a mobile AR application for dental education. According to the results, AR application was affective tool to convey knowledge. Participants satisfied this application and perceived it as easy to use. In addition, they wanted to use it in future.

Ferrer-Torregrosa et al. [72] compared three tools, which were notes with images, videos, and AR (marker-based material on paper) in health education field. They focused on students' time spent, learnings, perception of metacognition, and the prospects of the use of AR. Their results showed that AR was the most effective tool among these tools in all aspects. Harley et al. [73] explored using AR in history education. They organized a historical tour with mobile AR location-based application and examined students' emotions and learning outcomes. They experienced it in and outside of the laboratory. The results revealed that they enjoyed from this experience. In addition, AR application was efficient in both environments. Martin-Gonzalez et al. [74] developed an AR system with Kinect technology for physics and mathematics education. They evaluated its usability and results showed that usability of it was acceptable. In addition, participants had positive attitude

When analyzed the articles related to using AR in other target groups, four studies were found. These target groups were primary school teacher, construction workers, special students with intellectual disabilities and autism, and adults. Alkhattabi [75] focused on primary school teachers' acceptance of AR technology. In addition, advantages and barriers of AR were investigated with e-learning tool. Results showed that teachers accepted this technology and would like to use in their lessons. Pejoska et al. [76] used a mobile application for construction workers and analyzed their experiences in informal learning. They suggested some design principles for AR app to support informal learning. McMahon et al. [77] studied on students with intellectual disabilities and autism. They used marker-based material on paper in science education field. The results revealed that these students succeeded in defining new science vocabulary terms with AR technology support. Boletsis et al. [78] tested usability of AR-based cubes gaming system for adults. This game was developed for cognitive field. According to the results, users satisfied to use this game system and gave positive responses.

The aim of this chapter is to review literature regarding the use of AR in education articles published in between 2016 and 2017 years. Fifty-two articles were reviewed; however, 14 of them were not included in the study. As a result, 38 articles were examined. The results showed that AR technology has been mostly carried out in primary and graduate education. "Science education" is the most explored field of education. Reviewed studies have generally focused on "mathematic education," "storytelling," "foreign language education,"

towards this system.

**2. Conclusion**

**1.6. Using augmented reality in other target groups**

90 State of the Art Virtual Reality and Augmented Reality Knowhow

Rabia M. Yilmaz

Address all correspondence to: rabia.kufrevi@gmail.com; rkufrevi@atauni.edu.tr

Department of Computer Education and Instructional Technology, Ataturk University Erzurum, Turkey

#### **References**


[9] Alsumait A, Al-Musawi ZS. Creative and innovative e-learning using interactive storytelling. International Journal of Pervasive Computing and Communications. 2013;**9**(3):209-226. DOI: 10.1108/IJPCC-07-2013-0016

[23] Yoon SA, Elinich K, Wang J, Steinmeier C, Tucker S. Using augmented reality and knowledge-building scaffolds to improve learning in a science museum. International Journal

Augmented Reality Trends in Education between 2016 and 2017 Years

http://dx.doi.org/10.5772/intechopen.74943

93

[24] Li Y. Augmented reality for remote education. In: Advanced Computer Theory and Engineering (ICACTE), 2010 3rd International Conference on. IEEE; 2010. pp. V3-187-

[25] Aziz NAA, Aziz KA, Paul A, Yusof AM, Noor NSM. Providing augmented reality based education for students with attention deficit hyperactive disorder via cloud computing: Its advantages. In: Advanced Communication Technology (ICACT), 14th International

[26] Johnson L, Smith R, Willis H, Levine A, Haywood K. The 2011 Horizon Report. Austin,

[27] Cheng K-H, Tsai C-C. Affordances of augmented reality in science learning: Suggestions for future research. Journal of Science Education and Technology. 2012;**22**:449-462. DOI:

[28] Dünser A, Hornecker E. An observational study of children interacting with an augmented story book. In: International Conference on Technologies for E-Learning and

[29] Lester JC, Converse SA, Kahler SE, Barlow ST, Stone BA, Bhogal RS. The persona effect: Affective impact of animated pedagogical agents. In: Proceedings of the ACM SIGCHI Conference on Human factors in computing systems. ACM; 1997. p. 359-366. DOI:

[30] Oh S, Woo W. ARGarden: Augmented edutainment system with a learning companion. In: Transactions on edutainment I. Heidelberg, Berlin: Springer; 2008. pp. 40-50

[31] Zhou Z, Cheok AD, Pan J. 3D story cube: An interactive tangible user interface for storytelling with 3D graphics and audio. Personal Ubiquitous Computing. 2004;**8**:374-376.

[32] Serio ÁD, Ibáñez MB, Kloos CD. Impact of an augmented reality system on students' motivation for a visual art course. Computers and Education. 2013;**68**:586-596

[33] Kerawalla L, Luckin R, Selijefot S, Woolard A. Making it real: Exploring the potential of augmented reality for teaching primary school science. Virtual Reality. 2006;**10**(3-4):

[34] Bujak KR, Radu I, Catrambone R, MacIntyre B, Zheng R, Golubski G. A psychological perspective on augmented reality in the mathematics classroom. Computers and

[35] Kesim M, Ozarslan Y. Augmented reality in education: Current technologies and the potential for education. Procedia – Social and Behavioral Sciences. 2012;**47**:297-302

Digital Entertainment. Heidelberg, Berlin: Springer; 2007. pp. 305-315

of Computer-Supported Collaborative Learning. 2012;**7**(4):519-541

Conference on. IEEE; 2012. pp. 577-581

Texas: The New Media Consortium; 2011

10.1007/s10956-012-9405-9

10.1145/258549.258797

163-174

DOI: 10.1007/s00779-004-0300-0

Education. 2013;**68**:536-544

V3-191


[23] Yoon SA, Elinich K, Wang J, Steinmeier C, Tucker S. Using augmented reality and knowledge-building scaffolds to improve learning in a science museum. International Journal of Computer-Supported Collaborative Learning. 2012;**7**(4):519-541

[9] Alsumait A, Al-Musawi ZS. Creative and innovative e-learning using interactive storytelling. International Journal of Pervasive Computing and Communications.

[10] Nischelwitzer A, Lenz FJ, Searle G, Holzinger A. Some aspects of the development of low-cost augmented reality learning environments as examples for future interfaces in technology enhanced learning. Universal Access in Human-Computer Interaction

Applications and Services Lecture Notes in Computer Science. 2007;**4556**:728-737 [11] Sumadio DD, Rambli DRA. Preliminary evaluation on user acceptance of the augmented reality use for education. In: Computer Engineering and Applications (ICCEA), 2010 Second International Conference on. IEEE; 2010. pp. 461-465. DOI: 10.1109/ICCEA.2010.239

[12] Singhal S, Bagga S, Goyal P, Saxena V.Augmented chemistry: Interactive education system. International Journal of Computer Applications. 2012;**49**(15):1-5. DOI: 10.5120/7700-1041

[13] Azuma RT. Overview of augmented reality. In: ACM SIGGRAPH 2004 Course Notes.

[14] O'Brien HL, Toms EG. Engagement as Process in Computer Mediated Environments.

[15] Squire KD, Jan M Matthews J Wagler M, Martin J Devane B. Wherever You Go, There You Are: Place-Based Augmented Reality Games For Learning. 2008. Available from: http://citeseerx.ist.psu.edu/viewdoc/download; jsessionid=C82FB26748C7B6A2B070 9358AB08AD7B?doi=10.1.1.173.821andrep=rep1andtype=pdf adresinden erişilmiştir

[16] Schrier K. Using augmented reality games to teach 21st century skills. In: ACM SIGGRAPH 2006 Educators program. ACM; 2006. p. 15. DOI: 10.1145/1179295.1179311 [17] Ivanova M, Ivanov G. Enhancement of learning and teaching in computer graphics through marker augmented reality technology. International Journal on New Computer

[18] Yuen S, Yaoyuneyong G, Johnson E. Augmented reality: An overview and five directions for AR in education. Journal of Educational Technology Development and Exchange.

[19] Wu HK, Lee SWY, Chang HY, Liang JC. Current status, opportunities and challenges of

[20] Yilmaz RM. Effects of three dimensional storytelling developed with augmented reality technology on narrative skill and creativity [unpublished doctoral thesis]. Turkey; 2014

[21] Lin KC, Wang SC. Situated learning for computer fabrication based on augmented real-

[22] Arvanitis TN, Petrou A, Knight JF, Savas S, Sotiriou S, Gargalakos M, vd. Human factors and qualitative pedagogical evaluation of a mobile augmented reality system for science education used by learners with physical disabilities. Personal and Ubiquitous

augmented reality in education. Computers and Education. 2013;**62**:41-49

Paper presented at ASISveT, Charlotte, North Carolina. Nov 2005

Architectures and Their Applications. 2011;**1**(1):176-184

ity. Lecture Notes in Information Technology. 2012;**23**:249

2013;**9**(3):209-226. DOI: 10.1108/IJPCC-07-2013-0016

92 State of the Art Virtual Reality and Augmented Reality Knowhow

ACM; 2004. p. 26. DOI: 10.1145/1103900.1103926

[Accessed: Mar 12, 2014]

2011;**4**(1):119-140

Computing. 2007;**13**(3):243-250


[36] Núñez M, Quiros R, Núñez I, Carda JB, Camahort E Collaborative augmented reality for inorganic chemistry education. In: Proceedings of the 5th WSEAS/IASME International Conference on Engineering Education. 2008. pp. 271-277

[49] Hsu TC. Learning English with augmented reality: Do learning styles matter? Computers

Augmented Reality Trends in Education between 2016 and 2017 Years

http://dx.doi.org/10.5772/intechopen.74943

95

[50] Cascales-Martínez A, Martínez-Segura MJ, Pérez-López D, Contero M. Using an augmented reality enhanced tabletop system to promote learning of mathematics: A case study with students with special educational needs. Eurasia Journal of Mathematics

[51] Hwang GJ, Wu PH, Chen CC, Tu NT. Effects of an augmented reality-based educational game on students' learning achievements and attitudes in real-world observations.

[52] Chen CH, Chou YY, Huang CY. An augmented-reality-based concept map to support mobile learning for science. The Asia-Pacific Education Researcher. 2016;**25**(4):567-578

[53] Laine TH, Nygren E, Dirin A, Suk HJ. Science spots AR: A platform for science learning games with augmented reality. Educational Technology Research and Development.

[54] Solak E, Cakır R. Investigating the role of augmented reality Technology in the language

[55] Chang RC, Chung LY, Huang YM. Developing an interactive augmented reality system as a complement to plant education and comparing its effectiveness with video learning.

[56] Cai S, Chiang FK, Sun Y, Lin C, Lee JJ. Applications of augmented reality-based natural interactive learning in magnetic field instruction. Interactive Learning Environments.

[57] Salmi H, Thuneberg H, Vainikainen MP. Making the invisible observable by augmented reality in informal science education context. International Journal of Science Education,

[58] Huang TC, Chen CC, Chou YW.Animating eco-education: To see, feel, and discover in an augmented reality-based experiential learning environment. Computers & Education.

[59] Hsiao HS, Chang CS, Lin CY, Wang YZ. Weather observers: A manipulative augmented reality system for weather simulations at home, in the classroom, and at a museum.

[60] Wang YH. Exploring the Effectiveness of Integrating Augmented Reality-Based Materials

[61] Wang YH. Using augmented reality to support a software editing course for college

[62] Mumtaz K, Iqbal MM, Khalid S, Rafiq T, Owais SM, Al Achhab M. An E-assessment framework for blended learning with augmented reality to enhance the student learning. Eurasia Journal of Mathematics, Science and Technology Education. 2017;**13**(8):4419-4436

to Support Writing Activities. Computers & Education. 2017;**113**:162-176

students. Journal of Computer Assisted Learning. 2017;**33**(5):532-546

classroom. Hrvatski časopis za odgoj i obrazovanje. 2016;**18**(4):1067-1085

& Education. 2017;**106**:137-149

2016;**64**(3):507-531

2017;**25**(6):778-791

2016;**96**:72-82

Part B. 2017;**7**(3):253-268

Science and Technology Education. 2017;**13**(2):355-380

Interactive Learning Environments. 2016;**24**(8):1895-1906

Interactive Learning Environments. 2016;**24**(6):1245-1264

Interactive Learning Environments. 2016;**24**(1):205-223


[49] Hsu TC. Learning English with augmented reality: Do learning styles matter? Computers & Education. 2017;**106**:137-149

[36] Núñez M, Quiros R, Núñez I, Carda JB, Camahort E Collaborative augmented reality for inorganic chemistry education. In: Proceedings of the 5th WSEAS/IASME International

[37] Zhou Z, Cheok AD, Pan J, Li Y. Magic story cube: An interactive tangible interface for storytelling. In: Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technology. ACM; 2004. pp. 364-365. DOI:

[38] Ternier S, Klemke R, Kalz M, Ulzen P, Specht M. ARLearn: Augmented reality meets augmented virtuality. Journal of Universal Computer Science. 2012;**18**(15):2143-2164

[39] Billinghurst M. Augmented Reality in Education. New Horizons for Learning. Dec 2002. Available form: http://www.newhorizons.org/strategies/technology/billinghurst.htm

[40] Klopfer E, Yoon S. Developing games and simulations for today and tomorrow's tech

[41] Yilmaz RM, Kucuk S, Goktas Y. Are augmented reality picture books magic or real for preschool children aged five to six? British Journal of Educational Technology.

[42] Safar AH, Al-Jafar AA, Al-Yousefi ZH. The effectiveness of using augmented reality apps in teaching the english alphabet to kindergarten children: A case study in the State of Kuwait. Eurasia Journal of Mathematics, Science & Technology Education.

[43] Huang Y, Li H, Fong R. Using augmented reality in early art education: A case study in Hong Kong kindergarten. Early Child Development and Care. 2016;**186**(6):879-894

[44] Cheng KH, Tsai CC. The interaction of child–parent shared reading with an augmented reality (AR) picture book and parents' conceptions of AR learning. British Journal of

[45] Joo-Nagata J, Abad FM, Giner JGB, García-Peñalvo FJ. Augmented reality and pedestrian navigation through its implementation in m-learning and e-learning: Evaluation of

[46] Gün ET, Atasoy B. The effects of augmented reality on elementary school students' spa-

[47] Hung YH, Chen CH, Huang SW. Applying augmented reality to enhance learning: A study of different teaching materials. Journal of Computer Assisted Learning.

[48] Nadolny L. Interactive print: The design of cognitive tasks in blended augmented reality and print documents. British Journal of Educational Technology. 2017;**48**(3):

an educational program in Chile. Computers & Education. 2017;**111**:1-17

tial ability and academic achievement. Egitim ve Bilim. 2017;**42**(191)

Conference on Engineering Education. 2008. pp. 271-277

adresinden erişilmiştir [Accessed: Jan 15, 2014]

savvy youth. TechTrends. 2004;**49**(3):41-49

Educational Technology. 2016;**47**(1):203-222

10.1145/1067343.1067404

94 State of the Art Virtual Reality and Augmented Reality Knowhow

2017;**48**(3):824-841

2017;**13**(2):417-440

2017;**33**(3):252-266

814-823


[63] Chang HY, Hsu YS, Wu HK. A comparison study of augmented reality versus interactive simulation technology to support student learning of a socio-scientific issue. Interactive Learning Environments. 2016;**24**(6):1148-1161

[75] Alkhattabi M. Augmented reality as e-learning tool in primary schools' education: Barriers to teachers' adoption. International Journal of Emerging Technologies in

Augmented Reality Trends in Education between 2016 and 2017 Years

http://dx.doi.org/10.5772/intechopen.74943

97

[76] Pejoska J, Bauters M, Purma J, Leinonen T. Social augmented reality: Enhancing contextdependent communication and informal learning at work. British Journal of Educational

[77] McMahon DD, Cihak DF, Wright RE, Bell SM. Augmented reality for teaching science vocabulary to postsecondary education students with intellectual disabilities and

[78] Boletsis C, McCallum S.Augmented reality cubes for cognitive gaming: Preliminary usability and game experience testing. International Journal of Serious Games. 2016;**3**(1):3-18

autism. Journal of Research on Technology in Education. 2016;**48**(1):38-56

Learning. 2017;**12**(2):91-100

Technology. 2016;**47**(3):474-483


[75] Alkhattabi M. Augmented reality as e-learning tool in primary schools' education: Barriers to teachers' adoption. International Journal of Emerging Technologies in Learning. 2017;**12**(2):91-100

[63] Chang HY, Hsu YS, Wu HK. A comparison study of augmented reality versus interactive simulation technology to support student learning of a socio-scientific issue. Interactive

[64] Ibáñez MB, Di-Serio Á, Villarán-Molina D, Delgado-Kloos C.Support for augmented reality simulation systems: The effects of scaffolding on learning outcomes and behavior pat-

[65] Redondo Domínguez E, Fonseca Escudero D, Sánchez Riera A, Navarro Delgado I.Educating urban designers using augmented reality and mobile learning technologies. RIED – Revista

[66] Montoya MH, Díaz CA, Moreno GA. Evaluating the effect on user perception and performance of static and dynamic contents deployed in augmented reality based learning application. Eurasia Journal of Mathematics, Science & Technology Education.

[67] Bendicho PF, Mora CE, Añorbe-Díaz B, Rivero-Rodríguez P. Effect on academic procrastination after introducing augmented reality. Eurasia Journal of Mathematics, Science &

[68] Salinas P, Pulido R.Understanding the conics through augmented reality. Eurasia Journal

[69] Carbonell Carrera C, Bermejo Asensio LA. Landscape interpretation with augmented reality and maps to improve spatial orientation skill. Journal of Geography in Higher

[70] Cheng KH. Reading an augmented reality book: An exploration of learners' cognitive load, motivation, and attitudes. Australasian Journal of Educational Technology.

[71] Juan M. A mobile augmented reality system for the learning of dental morphology.

[72] Ferrer-Torregrosa J, Jiménez-Rodríguez MÁ, Torralba-Estelles J, Garzón-Farinós F, Pérez-Bermejo M, Fernández-Ehrling N. Distance learning ects and flipped classroom in the anatomy learning: Comparative study of the use of augmented reality, video and

[73] Harley JM, Poitras EG, Jarrell A, Duffy MC, Lajoie SP. Comparing virtual and locationbased augmented reality mobile learning: Emotions and learning outcomes. Educational

[74] Martin-Gonzalez A, Chi-Poot A, Uc-Cetina V. Usability evaluation of an augmented reality system for teaching Euclidean vectors. Innovations in Education and Teaching

of Mathematics, Science & Technology Education. 2017;**13**(2):341-354

terns. IEEE Transactions on Learning Technologies. 2016;**9**(1):46-56

Iberoamericana de Educación a Distancia. 2017;**20**(2):141-165

Learning Environments. 2016;**24**(6):1148-1161

96 State of the Art Virtual Reality and Augmented Reality Knowhow

Technology Education. 2017;**13**(2):319-330

Digital Education Review. 2016;**30**:234-247

notes. BMC Medical Education. 2016;**16**(1):230

International. 2016;**53**(6):627-636

Technology Research and Development. 2016;**64**(3):359-388

Education. 2017;**41**(1):119-133

2017;**33**(4):53-69

2017;**13**(2):301-317


**Chapter 7**

Provisional chapter

**Virtual and Augmented Reality: New Frontiers for**

DOI: 10.5772/intechopen.74344

In the last decades, the applied approach for the use of virtual reality (VR) and augmented reality (AR) on clinical and health psychology has grown exponentially. These technologies have been used to treat several mental disorders, for example, phobias, stress-related disorders, depression, eating disorders, and chronic pain. The importance of VR/AR for the mental health field comes from three main concepts: (1) VR/AR as an imaginal technology, people can feel "as if they are" in a reality that does not exist in external world; (2) VR/AR as an embodied technology, the experience to feel user's body inside the virtual environment; and (3) VR/AR as connectivity technology, the "end of geography'. In this chapter, we explore the opportunities provided by VR/AR as technologies to improve people's quality of life and to discuss new frontiers for their application in mental health

Keywords: virtual reality, augmented reality, cybertherapy, clinical psychology,

Clinical psychology is generally perceived as a face-to-face interaction between therapist and patient. However, thanks to technology developments, this picture has been changed. The massive innovation of Information and Communication Technologies (ICTs) has brought a revolution to the view of psychology and also the way how psychotherapists work [1]. Especially, the application of virtual reality (VR) and augmented reality (AR) has given an important contribu-

> © 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and eproduction in any medium, provided the original work is properly cited.

© 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

Virtual and Augmented Reality: New Frontiers for

Sara Ventura, Rosa M. Baños and Cristina Botella

Sara Ventura, Rosa M. Baños and Cristina Botella

Additional information is available at the end of the chapter

Additional information is available at the end of the chapter

and psychological well-being promotion.

advantaged technology

1. Introduction

tion to mental health.

http://dx.doi.org/10.5772/intechopen.74344

**Clinical Psychology**

Clinical Psychology

Abstract

#### **Virtual and Augmented Reality: New Frontiers for Clinical Psychology** Virtual and Augmented Reality: New Frontiers for Clinical Psychology

DOI: 10.5772/intechopen.74344

Sara Ventura, Rosa M. Baños and Cristina Botella Sara Ventura, Rosa M. Baños and Cristina Botella

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.74344

#### Abstract

In the last decades, the applied approach for the use of virtual reality (VR) and augmented reality (AR) on clinical and health psychology has grown exponentially. These technologies have been used to treat several mental disorders, for example, phobias, stress-related disorders, depression, eating disorders, and chronic pain. The importance of VR/AR for the mental health field comes from three main concepts: (1) VR/AR as an imaginal technology, people can feel "as if they are" in a reality that does not exist in external world; (2) VR/AR as an embodied technology, the experience to feel user's body inside the virtual environment; and (3) VR/AR as connectivity technology, the "end of geography'. In this chapter, we explore the opportunities provided by VR/AR as technologies to improve people's quality of life and to discuss new frontiers for their application in mental health and psychological well-being promotion.

Keywords: virtual reality, augmented reality, cybertherapy, clinical psychology, advantaged technology

#### 1. Introduction

Clinical psychology is generally perceived as a face-to-face interaction between therapist and patient. However, thanks to technology developments, this picture has been changed. The massive innovation of Information and Communication Technologies (ICTs) has brought a revolution to the view of psychology and also the way how psychotherapists work [1]. Especially, the application of virtual reality (VR) and augmented reality (AR) has given an important contribution to mental health.

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and eproduction in any medium, provided the original work is properly cited. © 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

In the last decades, a growing number of studies have shown important implications of the use of ICTs for treating several disorders and promotion of well-being. Initially, most of these studies have focused on treating anxiety disorders [2], phobias (e.g., specific phobias, social phobia, agoraphobia) [3], posttraumatic stress disorder (PTSD) [4, 5], eating disorder [6], addiction to nicotine or alcohol [7, 8], among others. Furthermore, VR and AR have been used not only for clinical intervention but also for promoting healthy lifestyles or well-being, for example, the reduction in stress [9], treatment of pain in oncology patients [10], or pain management for variety of known painful medical procedures [11]. In all these studies, the use of these ICTs has supported psychotherapists and researchers to reach the best results for patients. Thanks to the technological advances, it is possible to reproduce virtual environment where people can move as they are in the real world [12], or having some mobile applications which can enlarge the world around us and facing specific phobia [13]. But for professionals, it is not always an easy work because the use of ICTs usually implies that psychologists have to open their mind and co-work with engineers and other professionals who have different backgrounds. Psychologists and engineers have to find a way to cooperate and to integrate their knowledge, a cooperation that till now has changed society exponentially.

Since 1988, Norwegian Telecom Research has initiated and developed several telemedicine applications; the applications were adopted to exchange medical results from clinical chemistry to interactive radiology consultations. All the applications have a common goal to improve efficiency and quality of health care. One of the basic ideas of telemedicine can be expressed by the saying: "Move the information, not the patient" [17]. Indeed, one of the first telemedicine programs was proposed by rural practitioners who required access to certain type of medical

Virtual and Augmented Reality: New Frontiers for Clinical Psychology

http://dx.doi.org/10.5772/intechopen.74344

101

According to Eysenbach, e-health interventions are defined as treatments, typically behaviorally based, that are operationalized and transformed for delivery via the Internet [18]. A branch of e-health is e-mental health, or Internet-based therapy, in which electronic equipment and therapeutic communication converge. E-mental health can be defined as using ICTs to put patients and mental health professionals in contact; to conduct diagnosis or treatment; to disseminate information; or to conduct research studies or any other activity related to mental health care [19]. The online services include email, discussion lists, chats, or audiovisual

The main advance of online therapy is that it can reach people who might not otherwise seek therapy, such as disabled people or those who live in remote areas; it also reduces the contact time between therapist and patient [20]. Today, it is possible to make counseling through Internet, avoiding the face-to-face communication. It does not mean that human interaction disappears, and on the contrary, it faces relevant obstacles such as geographical distance, timetable, and emotive aspects that prevent patients to seek for a psychological therapy [17]. Mohr and colleagues [21] brought the e-health to a forward step. Mohr underlines the "behavioral intervention technologies" where technologies, such as telephone, videoconferencing, and web-based interventions, are integrated with other advanced technologies such as sensors for monitoring, social media, VR, and gaming, promoting e-mental-health interventions. From this perspective, e-mental-health not only provides new delivery media for mental health treatments, but opens the possibility for entirely new interventions. For example, mobile technologies can harness sensors and ubiquitous computing to provide continuous monitoring and/or intervention in the patient's environment. VR creates simulated environments that afford a high degree of control in engineering the provision of therapeutic experiences. Gaming may provide teaching methods that are more engaging. These opportunities may also challenge and expand the limits of the knowledge regarding human behavior processes [18]. After this overview, we can consider cybertherapy as a ramification of e-mental health. It includes all those kinds of treatment done through interactive and immersive technologies such as virtual and augmented reality where people get involved in the "digital" environments. As mentioned, cybertherapy has been used to treat psychological conditions such as anxiety disorders and phobias, eating disorders, autism, substance abuse and addiction, to reduce pain and discomfort perceived during unpleasant medical treatments, to manage stress, to administer exercises for cognitive rehabilitation (e.g., memory and attention disorder), and so on [22]. Evidence has shown that cybertherapy outcomes are comparable with those obtained through therapy protocols that are not supported by technologies [23], with some additional advantages that may make cybertherapy a preferable option. The most obvious

services [17].

conferencing, but also computerized treatments.

In this chapter, we review some of the most important advances in this field and how technology can (or could in the future) support clinical psychology. The aim is to explore the opportunities provided by VR and AR as technologies to improve people's quality of life and to discuss new frontiers for their application in mental health and psychological well-being promotion.

#### 2. What is cybertherapy?

Cybertherapy is the branch of psychology that uses ICTs to induce clinical change [1]. It is also defined as the use of advanced technologies, such as virtual or augmented reality, as an adjunct to traditional form of therapy. Cybertherapy is quickly becoming an accepted and validated method for the treatment of many different health care concerns. It occurs because technology supporting "cybertherapy" provides visual and auditory stimulus that may be otherwise difficult to generate, and it can support and motivate performance, as in rehabilitative exercises [14]. Also, other ICTs are becoming increasingly common in clinical psychology. It is generally agreeing that innovative e-therapy approach is an opportunity for earlier and better care for the most common mental health problems. E-therapy approach allows the patient to engage in treatment without having to accommodate the office appointment, often reducing other limitations in face-to-face treatment [15].

These advances come from the role of telemedicine and e-health. Telemedicine has been defined as the use of telecommunication technologies to provide medical information and services. The defining aspect of telemedicine is the use of electronic signals to transfer information from one site to another. It can be useful for situations in which physical barriers prevent the ready transfer of information between patients and health care providers, and the availability of information is the key to proper medical management [16].

Since 1988, Norwegian Telecom Research has initiated and developed several telemedicine applications; the applications were adopted to exchange medical results from clinical chemistry to interactive radiology consultations. All the applications have a common goal to improve efficiency and quality of health care. One of the basic ideas of telemedicine can be expressed by the saying: "Move the information, not the patient" [17]. Indeed, one of the first telemedicine programs was proposed by rural practitioners who required access to certain type of medical services [17].

In the last decades, a growing number of studies have shown important implications of the use of ICTs for treating several disorders and promotion of well-being. Initially, most of these studies have focused on treating anxiety disorders [2], phobias (e.g., specific phobias, social phobia, agoraphobia) [3], posttraumatic stress disorder (PTSD) [4, 5], eating disorder [6], addiction to nicotine or alcohol [7, 8], among others. Furthermore, VR and AR have been used not only for clinical intervention but also for promoting healthy lifestyles or well-being, for example, the reduction in stress [9], treatment of pain in oncology patients [10], or pain management for variety of known painful medical procedures [11]. In all these studies, the use of these ICTs has supported psychotherapists and researchers to reach the best results for patients. Thanks to the technological advances, it is possible to reproduce virtual environment where people can move as they are in the real world [12], or having some mobile applications which can enlarge the world around us and facing specific phobia [13]. But for professionals, it is not always an easy work because the use of ICTs usually implies that psychologists have to open their mind and co-work with engineers and other professionals who have different backgrounds. Psychologists and engineers have to find a way to cooperate and to integrate

their knowledge, a cooperation that till now has changed society exponentially.

promotion.

2. What is cybertherapy?

100 State of the Art Virtual Reality and Augmented Reality Knowhow

reducing other limitations in face-to-face treatment [15].

the key to proper medical management [16].

In this chapter, we review some of the most important advances in this field and how technology can (or could in the future) support clinical psychology. The aim is to explore the opportunities provided by VR and AR as technologies to improve people's quality of life and to discuss new frontiers for their application in mental health and psychological well-being

Cybertherapy is the branch of psychology that uses ICTs to induce clinical change [1]. It is also defined as the use of advanced technologies, such as virtual or augmented reality, as an adjunct to traditional form of therapy. Cybertherapy is quickly becoming an accepted and validated method for the treatment of many different health care concerns. It occurs because technology supporting "cybertherapy" provides visual and auditory stimulus that may be otherwise difficult to generate, and it can support and motivate performance, as in rehabilitative exercises [14]. Also, other ICTs are becoming increasingly common in clinical psychology. It is generally agreeing that innovative e-therapy approach is an opportunity for earlier and better care for the most common mental health problems. E-therapy approach allows the patient to engage in treatment without having to accommodate the office appointment, often

These advances come from the role of telemedicine and e-health. Telemedicine has been defined as the use of telecommunication technologies to provide medical information and services. The defining aspect of telemedicine is the use of electronic signals to transfer information from one site to another. It can be useful for situations in which physical barriers prevent the ready transfer of information between patients and health care providers, and the availability of information is According to Eysenbach, e-health interventions are defined as treatments, typically behaviorally based, that are operationalized and transformed for delivery via the Internet [18]. A branch of e-health is e-mental health, or Internet-based therapy, in which electronic equipment and therapeutic communication converge. E-mental health can be defined as using ICTs to put patients and mental health professionals in contact; to conduct diagnosis or treatment; to disseminate information; or to conduct research studies or any other activity related to mental health care [19]. The online services include email, discussion lists, chats, or audiovisual conferencing, but also computerized treatments.

The main advance of online therapy is that it can reach people who might not otherwise seek therapy, such as disabled people or those who live in remote areas; it also reduces the contact time between therapist and patient [20]. Today, it is possible to make counseling through Internet, avoiding the face-to-face communication. It does not mean that human interaction disappears, and on the contrary, it faces relevant obstacles such as geographical distance, timetable, and emotive aspects that prevent patients to seek for a psychological therapy [17].

Mohr and colleagues [21] brought the e-health to a forward step. Mohr underlines the "behavioral intervention technologies" where technologies, such as telephone, videoconferencing, and web-based interventions, are integrated with other advanced technologies such as sensors for monitoring, social media, VR, and gaming, promoting e-mental-health interventions. From this perspective, e-mental-health not only provides new delivery media for mental health treatments, but opens the possibility for entirely new interventions. For example, mobile technologies can harness sensors and ubiquitous computing to provide continuous monitoring and/or intervention in the patient's environment. VR creates simulated environments that afford a high degree of control in engineering the provision of therapeutic experiences. Gaming may provide teaching methods that are more engaging. These opportunities may also challenge and expand the limits of the knowledge regarding human behavior processes [18].

After this overview, we can consider cybertherapy as a ramification of e-mental health. It includes all those kinds of treatment done through interactive and immersive technologies such as virtual and augmented reality where people get involved in the "digital" environments. As mentioned, cybertherapy has been used to treat psychological conditions such as anxiety disorders and phobias, eating disorders, autism, substance abuse and addiction, to reduce pain and discomfort perceived during unpleasant medical treatments, to manage stress, to administer exercises for cognitive rehabilitation (e.g., memory and attention disorder), and so on [22]. Evidence has shown that cybertherapy outcomes are comparable with those obtained through therapy protocols that are not supported by technologies [23], with some additional advantages that may make cybertherapy a preferable option. The most obvious advance is that the mediated environment allows patients to experience situations, to display stimuli, or to provide feedback of the patient's action that in vivo would be not controllable (e.g., crowd behavior), not feasible (e.g., scenario variation to improve transfer of skills), or unavailable (e.g., an iced world mitigating pain during medical treatment from severe burning) [22]. In addition, the use of mediated environment minimizes implementation failures because a mediated environment embeds the administration manual: standard task instruction and explanations, organizations of stimuli into subsequence treatment steps, and setting options for personalized treatment. As Botella and colleagues have pointed out [22], the use of ICTs in delivering a psychological therapy allows treatment to reach people in critical conditions, to improve persistency, ubiquity, anonymity, and multimodality of an intervention, as well as the ease with which data can be stored, accessed, and manipulated.

The rationale behind VR use to help exposure technique is simple: in VR, the patient can be intentionally confronted with the feared stimuli while allowing the anxiety to attenuate. What distinguishes VR from other media or communication systems is the sense of presence. What is "presence"? Generally, the sense of presence has been defined as a mental state in which a user feels that he/she is being there, in the computer-mediated environment [28]. These characteristics of VR offer a number of advantages, as we explain below, over in vivo or imaginal

Virtual and Augmented Reality: New Frontiers for Clinical Psychology

http://dx.doi.org/10.5772/intechopen.74344

103

Since the early 1990s, when Hodges and colleagues [26] reported that the use of virtual environment can provide to acrophobic patients the feeling of heights in a safe situation, VR exposure therapy has been proposed as a new medium for exposure therapy [29]. In the past decade, numerous studies have tested the efficacy of VR. Review and meta-analysis [30] studies show how VR therapy works more effectively than imaginal therapy (visualization)

To give a clearer idea about the intervention trough VR, we explain below the intervention for people with flying phobia made by Botella and colleagues [32]. The program includes three virtual scenarios: (1) living room: here, the participant can perform some activities usually associated with the days or hours before the flight: pack, listen the TV news about the weather, and take his/her ticket for the check-in; (2) airport: the time before flight is simulated. The participant can listen and see on the monitor for the announcement of boarding pass, knowing that his flight is near, and listen to other people talking about the flights. It is also possible to see and hear planes landing and take-off. At the end, participant can enter into the virtual airplane scenario; (3) airplane: the participant is sitting on the plane and can experience takeoff, flight, and landing in different conditions (turbulences, storm, etc.) Through the previous virtual environment description, it is possible to figure out how much VR reflects the reality. VR offers several advantages as new options to patients who are unable to utilize imaginal therapy due to difficulties engaging with a situation, or who are resistant to in vivo treatment due to extreme anxiety. It is recognized that there is a large percentage of population (over 80%) that cannot visualize effectively. In addition, many of those suffering from anxiety do not

In addition, VR has an advantage to create safe virtual world where the patient can explore and experience "new realities"; this feeling of safety is essential in therapy, so that the patient can act without feeling threatened. Moreover, in VR, information can be presented gradually, in such a way that the patient can progress from easier tasks to more difficult one. This work in the virtual world helps patients master the strategies need to overcome their fears and limitations in the real world. Furthermore, as VR goes beyond space and time, researchers do not have to wait for specific events to occur. Rather, they can simulate them whenever appropriate

In summary, VR protocols can offer to clinicians and researchers a practical tool to support the clinical tasks (assessment and treatment) in ecologically valid, safe, and controllable

exposure [26].

and as effectively as in vivo exposure therapy [31].

feel that they can approach their feared situation in real life [22].

for the patient and the therapy process [26].

environments [33].

Cybertherapy can adopt different formats: from totally self-guided to more blended, including the presence of the therapist in different graduations, and the protocol can also include other treatments in addition to the virtual one [24].

In technical terms, hardware and software are combined into cybertherapy to achieve the final therapeutic goals. The software content embeds and makes concrete abstract scenarios, imagined situations, feared objects, subjective symbols, and meaning. The hardware shapes the way in which those contests are experienced, whether in isolation from the surrounding or merged with them and whether involving the body in a natural interaction with the environment or mediating the interaction with input devices [25].

As we said before, cybertherapy includes different types of technologies. Now, we focus on two of them: VR and AR.

#### 2.1. Virtual reality in cybertherapy

VR is a collection of technologies that allow people to interact efficiently with 3D computerized database in real time using their natural sense and skills [26]. In terms of behavioral science, VR has been described as an advanced form of human-computer interface that allows the user to interact with and become immersed in a computer-generated environment in a naturalistic fashion [26]. VR has emerged as a potentially effective way to provide general and specialty health care services and appears poised to enter mainstream psychotherapy delivery.

Where does the use of VR in cyberpsychology come from? The pioneering work by Watson demonstrated, contrary to the dominant Freudian theories of psychology, that it was possible to stimulate phobias in a laboratory environment. The little Albert experiment provided empirical evidence of classical conditioning in human. Few years later a study was conducted with the patient named Peter [27]. The therapist treated his rabbit phobia with classical condition model: a pleasant stimulus (food) was presented simultaneously with the rabbit. This case illustrated how fear may be eliminated under laboratory conditions. The study was a pioneer which introduced evidence-based psychological procedures to the field of psychological treatment through the application of "exposure therapy" [22].

The rationale behind VR use to help exposure technique is simple: in VR, the patient can be intentionally confronted with the feared stimuli while allowing the anxiety to attenuate. What distinguishes VR from other media or communication systems is the sense of presence. What is "presence"? Generally, the sense of presence has been defined as a mental state in which a user feels that he/she is being there, in the computer-mediated environment [28]. These characteristics of VR offer a number of advantages, as we explain below, over in vivo or imaginal exposure [26].

advance is that the mediated environment allows patients to experience situations, to display stimuli, or to provide feedback of the patient's action that in vivo would be not controllable (e.g., crowd behavior), not feasible (e.g., scenario variation to improve transfer of skills), or unavailable (e.g., an iced world mitigating pain during medical treatment from severe burning) [22]. In addition, the use of mediated environment minimizes implementation failures because a mediated environment embeds the administration manual: standard task instruction and explanations, organizations of stimuli into subsequence treatment steps, and setting options for personalized treatment. As Botella and colleagues have pointed out [22], the use of ICTs in delivering a psychological therapy allows treatment to reach people in critical conditions, to improve persistency, ubiquity, anonymity, and multimodality of an intervention, as well as the

Cybertherapy can adopt different formats: from totally self-guided to more blended, including the presence of the therapist in different graduations, and the protocol can also include other

In technical terms, hardware and software are combined into cybertherapy to achieve the final therapeutic goals. The software content embeds and makes concrete abstract scenarios, imagined situations, feared objects, subjective symbols, and meaning. The hardware shapes the way in which those contests are experienced, whether in isolation from the surrounding or merged with them and whether involving the body in a natural interaction with the environment or

As we said before, cybertherapy includes different types of technologies. Now, we focus on

VR is a collection of technologies that allow people to interact efficiently with 3D computerized database in real time using their natural sense and skills [26]. In terms of behavioral science, VR has been described as an advanced form of human-computer interface that allows the user to interact with and become immersed in a computer-generated environment in a naturalistic fashion [26]. VR has emerged as a potentially effective way to provide general and specialty

Where does the use of VR in cyberpsychology come from? The pioneering work by Watson demonstrated, contrary to the dominant Freudian theories of psychology, that it was possible to stimulate phobias in a laboratory environment. The little Albert experiment provided empirical evidence of classical conditioning in human. Few years later a study was conducted with the patient named Peter [27]. The therapist treated his rabbit phobia with classical condition model: a pleasant stimulus (food) was presented simultaneously with the rabbit. This case illustrated how fear may be eliminated under laboratory conditions. The study was a pioneer which introduced evidence-based psychological procedures to the field of psycholog-

health care services and appears poised to enter mainstream psychotherapy delivery.

ical treatment through the application of "exposure therapy" [22].

ease with which data can be stored, accessed, and manipulated.

treatments in addition to the virtual one [24].

102 State of the Art Virtual Reality and Augmented Reality Knowhow

mediating the interaction with input devices [25].

two of them: VR and AR.

2.1. Virtual reality in cybertherapy

Since the early 1990s, when Hodges and colleagues [26] reported that the use of virtual environment can provide to acrophobic patients the feeling of heights in a safe situation, VR exposure therapy has been proposed as a new medium for exposure therapy [29]. In the past decade, numerous studies have tested the efficacy of VR. Review and meta-analysis [30] studies show how VR therapy works more effectively than imaginal therapy (visualization) and as effectively as in vivo exposure therapy [31].

To give a clearer idea about the intervention trough VR, we explain below the intervention for people with flying phobia made by Botella and colleagues [32]. The program includes three virtual scenarios: (1) living room: here, the participant can perform some activities usually associated with the days or hours before the flight: pack, listen the TV news about the weather, and take his/her ticket for the check-in; (2) airport: the time before flight is simulated. The participant can listen and see on the monitor for the announcement of boarding pass, knowing that his flight is near, and listen to other people talking about the flights. It is also possible to see and hear planes landing and take-off. At the end, participant can enter into the virtual airplane scenario; (3) airplane: the participant is sitting on the plane and can experience takeoff, flight, and landing in different conditions (turbulences, storm, etc.) Through the previous virtual environment description, it is possible to figure out how much VR reflects the reality.

VR offers several advantages as new options to patients who are unable to utilize imaginal therapy due to difficulties engaging with a situation, or who are resistant to in vivo treatment due to extreme anxiety. It is recognized that there is a large percentage of population (over 80%) that cannot visualize effectively. In addition, many of those suffering from anxiety do not feel that they can approach their feared situation in real life [22].

In addition, VR has an advantage to create safe virtual world where the patient can explore and experience "new realities"; this feeling of safety is essential in therapy, so that the patient can act without feeling threatened. Moreover, in VR, information can be presented gradually, in such a way that the patient can progress from easier tasks to more difficult one. This work in the virtual world helps patients master the strategies need to overcome their fears and limitations in the real world. Furthermore, as VR goes beyond space and time, researchers do not have to wait for specific events to occur. Rather, they can simulate them whenever appropriate for the patient and the therapy process [26].

In summary, VR protocols can offer to clinicians and researchers a practical tool to support the clinical tasks (assessment and treatment) in ecologically valid, safe, and controllable environments [33].

#### 2.2. Augmented reality in cybertherapy

AR is a modification of VR which includes a combination of both real and virtual elements. The most significance aspect of AR is that the virtual elements add relevant and helpful input to physical information available in the real world. User can see images that blend both "realworld elements" and "virtual elements" that have been introduced by the system [13].

As is explained in Table 1, there are differences between AR and VR. The first difference is the immersion of the user inside the system. VR achieves an involved environment for the user and perceptive channels such as vision and sound are controlled by the system. Contrarily, the AR system complements the real world being necessary that the user maintains his/her sense of presence in that world. AR has a mechanism that combines the real and the virtual scenes that is not present in VR settings. In the AR system, the virtual objects generated by computer must be completely fused to the real world, in all of the dimensions.

In few words, while VR immerses fully the user in the entire virtual environment, AR permits the user to see the real world, with the important difference that virtual object merges with actual ones in a composite image [34]. According to Milgram and Kishino [35], AR is a form of mixed reality, that is, a particular subclass of VR-related technologies that, via a single display, expose the user to electronically merged virtual and nonvirtual elements.

AR has been used in various fields such as education and teaching [36], medicine and surgery [37]. However, AR applications for psychological treatment are still scarce and address mailing phobias [33]. Preliminary data show the utility of the system for the treatment of insect phobia [13]. Below is described a study for cockroach phobia to underline how AR system works in therapy (Figure 1) [13].

AR-cockroaches system was developed using a proper engineer software. It uses computer vision techniques in order to obtain the position and orientation of the camera with respect to

> markers. When the camera found a marker in the real world, the program recognized it and activated the feared virtual environment (cockroach). The virtual insect that virtually appears in the hands of participants (cockroach in this case) looks real thanks to a peculiar AR technology. The therapist can watch the virtual stimuli presented to participants during the exposure session in the monitor and can control the application using computer keys (number of cockroaches, movement, size, etc.). All of these combined cockroach's options enable the therapists to apply the treatment progressively [13]. This aspect put focus on the main advantages of AR as it was figured out also in VR: the feeling of safety [22]. In AR, it is possible to modify the virtual

Virtual and Augmented Reality: New Frontiers for Clinical Psychology

http://dx.doi.org/10.5772/intechopen.74344

105

AR offers additional advantages: it can provoke great feeling of presence because the environments and the tools with which the participants interact are real. In AR, the users can see their own body in the environment and interact with the fearing stimuli; in addition, the system allows patients to use real elements and their own hands and body to interact with stimuli.

These pioneer studies show that AR can be a very important alternative treatment for phobia

elements through the participant consent which reduces their rejection for therapy.

and might be useful for other psychological disorders.

Figure 1. Cockroach AR system. Labpsitec, Universidad Jaume I, Spain.


Table 1. Principal difference between VR and AR in clinical psychology.

Figure 1. Cockroach AR system. Labpsitec, Universidad Jaume I, Spain.

2.2. Augmented reality in cybertherapy

104 State of the Art Virtual Reality and Augmented Reality Knowhow

therapy (Figure 1) [13].

Sense of presence

User experience

AR is a modification of VR which includes a combination of both real and virtual elements. The most significance aspect of AR is that the virtual elements add relevant and helpful input to physical information available in the real world. User can see images that blend both "real-

As is explained in Table 1, there are differences between AR and VR. The first difference is the immersion of the user inside the system. VR achieves an involved environment for the user and perceptive channels such as vision and sound are controlled by the system. Contrarily, the AR system complements the real world being necessary that the user maintains his/her sense of presence in that world. AR has a mechanism that combines the real and the virtual scenes that is not present in VR settings. In the AR system, the virtual objects generated by computer

In few words, while VR immerses fully the user in the entire virtual environment, AR permits the user to see the real world, with the important difference that virtual object merges with actual ones in a composite image [34]. According to Milgram and Kishino [35], AR is a form of mixed reality, that is, a particular subclass of VR-related technologies that, via a single display,

AR has been used in various fields such as education and teaching [36], medicine and surgery [37]. However, AR applications for psychological treatment are still scarce and address mailing phobias [33]. Preliminary data show the utility of the system for the treatment of insect phobia [13]. Below is described a study for cockroach phobia to underline how AR system works in

AR-cockroaches system was developed using a proper engineer software. It uses computer vision techniques in order to obtain the position and orientation of the camera with respect to

Feel inside the virtual world Keep feeling inside the "real" world

Generates new experience Enhancing the experience

environment

User can see their own body in context

Uses virtual elements to build upon the existing

world elements" and "virtual elements" that have been introduced by the system [13].

must be completely fused to the real world, in all of the dimensions.

expose the user to electronically merged virtual and nonvirtual elements.

Area Virtual reality Augmented reality

Time study Since the beginning of twentieth century Since the last few years

Point of view Egocentric and allocentric Allocentric

Cost Higher Lower

Table 1. Principal difference between VR and AR in clinical psychology.

Immersion User is completely immersed into the virtual

Environment Substitutes the existing environment with the

environment

virtual one

markers. When the camera found a marker in the real world, the program recognized it and activated the feared virtual environment (cockroach). The virtual insect that virtually appears in the hands of participants (cockroach in this case) looks real thanks to a peculiar AR technology. The therapist can watch the virtual stimuli presented to participants during the exposure session in the monitor and can control the application using computer keys (number of cockroaches, movement, size, etc.). All of these combined cockroach's options enable the therapists to apply the treatment progressively [13]. This aspect put focus on the main advantages of AR as it was figured out also in VR: the feeling of safety [22]. In AR, it is possible to modify the virtual elements through the participant consent which reduces their rejection for therapy.

AR offers additional advantages: it can provoke great feeling of presence because the environments and the tools with which the participants interact are real. In AR, the users can see their own body in the environment and interact with the fearing stimuli; in addition, the system allows patients to use real elements and their own hands and body to interact with stimuli.

These pioneer studies show that AR can be a very important alternative treatment for phobia and might be useful for other psychological disorders.

## 3. The key concepts of virtual and augmented reality for clinical psychology

The artificial environment generates from virtual and augmented reality is closer to daily life people. That is why virtual environment can be considered as an "ecological laboratory" where behaviors, feelings, and human experience can be studied in a controlled and rigorous way [38].

intrusive imagines, flashback, or sensory memories about their traumatic events [42]. In PTSD, the term flashback is used to describe an intense period of dissociation where patient feels that is reliving a traumatic event [42]. Flashback-type mental images have also been identified in other psychological disorders such as social phobia [43], agoraphobia [44], bipolar disorder [45], and also in depression, which is associated with verbal- and imaginative-based process, such as negative rumination. Actually, 90% of depressed patient report distress intrusive

Virtual and Augmented Reality: New Frontiers for Clinical Psychology

http://dx.doi.org/10.5772/intechopen.74344

107

Holmes underlines that mental imagery acts as an amplifier of emotional effects [46]. In fact, imagery has the power to hijack attention (most obviously by flashbacks) further away from the external world, making the internal cognitions more believable and associated emotion more powerful. Amplified anxiety states consequently affect behavior by the avoidance of anxiety-related triggers, for example, the avoidance of crowded places in agoraphobic anxiety [44]. Other important aspects are that mental imageries are capable to induce learning and to promote behaviors. Thanks to mental exposure and its future desensitization, people can face fear situations and learn positive behaviors [44]. All these characteristics including learning, emotional responses, changing behavior, and physiological responses could be translated to VR field. VR can develop simulative environments which are acceptable from the view point of sensory evidence. In fact, VR can be considered as an advanced imaginal system and an experiential form of imagery that is as effective as reality in inducing cognitive, emotional, and behavioral responses [44]. For that reason, VR exposure therapy has been used in contemporary clinical practice with a strong evidence base for treating psychological disorders, such as

There is an increasing evidence that VR exposure is more efficacious than treatment using mental imagery simulations. First of all, the imagination usually decays rapidly. The mental image remains in our mind for approximately 250 ms. [46]. In VR, this risk is not present because the patient is involved into the environment without the need to evoke any imagination. Moreover, imagination of previous events needs a good memory and sometimes is not trustful [22]. For example, during an assessment with PTSD, therapist asks patients to think about a traumatic event. Sometimes patients fall in this task because memories are not clear. In VR, patients are already involved into traumatic event, which gives them the possibility to

Another important role of VR is to induce positive emotions through positive virtual environments [47]. It is common that relaxing imagination is useful to eliminate stress or negative thoughts from our mind. Thanks to virtual environment, it is possible to involve people into a

To summarize, VR gives an experience that is able to reduce the gap between imagination and reality and to go over memory limits increasing the efficacy and effectiveness of psychotherapy.

As we have seen, most VR applications have been used to simulate external reality [48]. In fact, in VR it is possible not only to experience synthetic environment as if it is "our surrounding

specific environment to reduce their level of stress (e.g., virtual island [9]).

memories of past experiences [46].

described previously in this chapter.

relive the traumatic experience and to face it.

3.2. Embodied technology

Virtual and augmented reality can improve some aspects of available treatments [33]. As mentioned, the ecological validity of the assessment could be even better than "in vivo" therapy. First, with AR and VR, the therapist has total control over the virtual situations and elements in the computer program, such as the generation of stimuli, including their order of appearance and their quantity. Second, they can make patients feel more secure during therapy because outcomes that they fear will happen in the real world cannot happen in AR and VR (without consent and planning). For example, the therapist can expose a patient to a virtual elevator and assure him/her that it will not break down, or can expose a patient to a flight with no turbulence. As the patient progresses, the therapist can plan more difficult exposure tasks. Third, AR and VR enable easier access to threatening stimuli. This efficacy is significance because it is not always easy to obtain real elements such as cockroaches or spiders as needed for therapy [22]. In other works, it could be assumed that VR and AR can have numerous applications in the field of psychological treatments. According to Riva [26], these advantages position VR as an "intermediate step between the therapist's office and the real world."

We focus the future perspective of VR and AR on three main aspects which contribute to increase their efficacy and affectivity in clinical psychology. As we described below, VR and AR could be considered as imaginal technology, embodied technology, and connectedness technology.

#### 3.1. Imaginal technology

Mental imagery refers to perceptual experience in the absence of sensory input, commonly described as seeing with the "mind's eyes" or "hearing with the mind's ear" [39]. It is different from perception which occurs when information is directly registered from the senses. Mental imagery is described also as the simulation or recreation of perceptual experience across sensory modalities [40]. Pearson [40] has marked two different routes by which mental imagery can be created within consciousness. First of all, an image can be created directly from immediate perceptual information. For example, someone can look at a picture of a horse, create a mental image of the picture in their mind, and then maintain this mental image as they look away or close their eyes. Second, an image can be created from previously stored information hand in long-term memory. For example, someone can hear the "horse" and then create mental imagery based on their previous experience of what a horse look like.

Imagery has been used frequently in psychotherapy, since the interpretation of dreams by Freud [41]. Today imagination plays a particular role in influencing the key characteristics of mental disorders [41]. This aspect is present especially in patients with PTSD which suffer of intrusive imagines, flashback, or sensory memories about their traumatic events [42]. In PTSD, the term flashback is used to describe an intense period of dissociation where patient feels that is reliving a traumatic event [42]. Flashback-type mental images have also been identified in other psychological disorders such as social phobia [43], agoraphobia [44], bipolar disorder [45], and also in depression, which is associated with verbal- and imaginative-based process, such as negative rumination. Actually, 90% of depressed patient report distress intrusive memories of past experiences [46].

Holmes underlines that mental imagery acts as an amplifier of emotional effects [46]. In fact, imagery has the power to hijack attention (most obviously by flashbacks) further away from the external world, making the internal cognitions more believable and associated emotion more powerful. Amplified anxiety states consequently affect behavior by the avoidance of anxiety-related triggers, for example, the avoidance of crowded places in agoraphobic anxiety [44]. Other important aspects are that mental imageries are capable to induce learning and to promote behaviors. Thanks to mental exposure and its future desensitization, people can face fear situations and learn positive behaviors [44]. All these characteristics including learning, emotional responses, changing behavior, and physiological responses could be translated to VR field. VR can develop simulative environments which are acceptable from the view point of sensory evidence. In fact, VR can be considered as an advanced imaginal system and an experiential form of imagery that is as effective as reality in inducing cognitive, emotional, and behavioral responses [44]. For that reason, VR exposure therapy has been used in contemporary clinical practice with a strong evidence base for treating psychological disorders, such as described previously in this chapter.

There is an increasing evidence that VR exposure is more efficacious than treatment using mental imagery simulations. First of all, the imagination usually decays rapidly. The mental image remains in our mind for approximately 250 ms. [46]. In VR, this risk is not present because the patient is involved into the environment without the need to evoke any imagination. Moreover, imagination of previous events needs a good memory and sometimes is not trustful [22]. For example, during an assessment with PTSD, therapist asks patients to think about a traumatic event. Sometimes patients fall in this task because memories are not clear. In VR, patients are already involved into traumatic event, which gives them the possibility to relive the traumatic experience and to face it.

Another important role of VR is to induce positive emotions through positive virtual environments [47]. It is common that relaxing imagination is useful to eliminate stress or negative thoughts from our mind. Thanks to virtual environment, it is possible to involve people into a specific environment to reduce their level of stress (e.g., virtual island [9]).

To summarize, VR gives an experience that is able to reduce the gap between imagination and reality and to go over memory limits increasing the efficacy and effectiveness of psychotherapy.

#### 3.2. Embodied technology

3. The key concepts of virtual and augmented reality for clinical

The artificial environment generates from virtual and augmented reality is closer to daily life people. That is why virtual environment can be considered as an "ecological laboratory" where behaviors, feelings, and human experience can be studied in a controlled and rigorous

Virtual and augmented reality can improve some aspects of available treatments [33]. As mentioned, the ecological validity of the assessment could be even better than "in vivo" therapy. First, with AR and VR, the therapist has total control over the virtual situations and elements in the computer program, such as the generation of stimuli, including their order of appearance and their quantity. Second, they can make patients feel more secure during therapy because outcomes that they fear will happen in the real world cannot happen in AR and VR (without consent and planning). For example, the therapist can expose a patient to a virtual elevator and assure him/her that it will not break down, or can expose a patient to a flight with no turbulence. As the patient progresses, the therapist can plan more difficult exposure tasks. Third, AR and VR enable easier access to threatening stimuli. This efficacy is significance because it is not always easy to obtain real elements such as cockroaches or spiders as needed for therapy [22]. In other works, it could be assumed that VR and AR can have numerous applications in the field of psychological treatments. According to Riva [26], these advantages position VR as an "intermediate step between the therapist's office and the real world."

We focus the future perspective of VR and AR on three main aspects which contribute to increase their efficacy and affectivity in clinical psychology. As we described below, VR and AR could be considered as imaginal technology, embodied technology, and connectedness

Mental imagery refers to perceptual experience in the absence of sensory input, commonly described as seeing with the "mind's eyes" or "hearing with the mind's ear" [39]. It is different from perception which occurs when information is directly registered from the senses. Mental imagery is described also as the simulation or recreation of perceptual experience across sensory modalities [40]. Pearson [40] has marked two different routes by which mental imagery can be created within consciousness. First of all, an image can be created directly from immediate perceptual information. For example, someone can look at a picture of a horse, create a mental image of the picture in their mind, and then maintain this mental image as they look away or close their eyes. Second, an image can be created from previously stored information hand in long-term memory. For example, someone can hear the "horse" and then

create mental imagery based on their previous experience of what a horse look like.

Imagery has been used frequently in psychotherapy, since the interpretation of dreams by Freud [41]. Today imagination plays a particular role in influencing the key characteristics of mental disorders [41]. This aspect is present especially in patients with PTSD which suffer of

psychology

106 State of the Art Virtual Reality and Augmented Reality Knowhow

way [38].

technology.

3.1. Imaginal technology

As we have seen, most VR applications have been used to simulate external reality [48]. In fact, in VR it is possible not only to experience synthetic environment as if it is "our surrounding world," but also to experience synthetic avatars as if they are "our own body." VR can be defined also as an "embodied technology" for "its ability of modifying the feeling of presence" [49]. People's representations in the virtual world are usually named avatars, and for the first time in human history, they could allow us to watch ourselves being others and doing something they have never done [49].

body continued until the participant decided to stop. Results showed high level of sense of

Virtual and Augmented Reality: New Frontiers for Clinical Psychology

http://dx.doi.org/10.5772/intechopen.74344

109

Other recent studies have also proved how changing the perspective, and living experiences from other bodies, can help us to promote empathy, and even compassion and selfcompassion. For instance, Salter's group has analyzed how the use of virtual bodies can promote compassion and self-compassion. They have analyzed the effects of self-identification with virtual bodies within immersive VR to increase self-compassion in persons with high selfcriticism and depression [53]. The same author and his colleagues investigated also how embodied, in VR, a black avatar decreases the racial prejudice [54]. Moreover, Bailenson's group has studied how embodied an avatar in VR can make us better people. For example, participants embodied a Superman avatar and the results show that, after the experiment, they felt more helpful [55]. In other Bailenson study, participants embodied a sea coral and, after the

Other interesting approaches are based on the development of new immersive technology named: machine to be another. It is an advantage technology to interchange bodies (the real body with the avatar) which offers to the users an immersive experience of seeing themselves in the body of another person. MTBA is a low budget body swapping system, and using a head-mounted display, participant sees the perspective of another person (performer) that

The performer's first-person perspective is captured by a camera controlled by user's head movements, revealing torso, legs and arms of the performer's body. While interacting with the physical space, participants also perceive realistic tactile stimuli. An audio system plays a

experiment, they felt more sensible to ocean pollution [51].

Figure 2. The machine to be another system. Labpsitec, Universidad de Valencia, Spain.

mimics his/her movements (Figure 2).

embodiment [52].

Following the point, VR as an "embodied technology" can be useful to alter the body's bounds and to study the relationship among body (posture, movements, actions) and cognition and emotions. Moreover, virtual avatars can help to amplify the "modeling" learning, and therefore to promote the learning of adaptive behaviors.

For instance, literature points out that [50] the sense of embodiment is principally based on three main aspects: sense of self-location, sense of agency, and sense of body ownership [50]. Self-location is a determinate volume in space where one feels to be located. Normally selflocation and body-space coincide in the sense that one feels self-located inside physical body. This collocation can break down when people have an out-of-body experience in which they perceive themselves outside of their physical body. The sense of agency refers to the sense of having a global motor control, including the subjective experience of action, control, intention, motor selection, and the conscious experience of will. The development of agency depends on the synchronicity of visual motor correlations, for example, the imagination of a tool that touches my hand and the real touch of the hand. The sense of body ownership refers to one's self-attribution of the body. It has been proposed to emerge from a combination of bottom-up and top-down influences. Bottom-up information refers to the afferent sensory information that arrives to put brain from our sensory organs; top-down information consists of the cognitive process that may modulate the processing of sensory stimuli, for example, how much the avatar in VR is similar to real person (this aspect depends most of the time on the graphic computer quality). In fact, the illusion of ownership diminishes when the external object does not resemble or is in a different spatial configuration to the real body [51].

Experimental manipulation of the embodied experience is problematic. However, the use of VR has unique advantages to control the factors associated with the embodied experience. For example, VR makes possible in a relatively easy manner the manipulation of the body representation in terms of structure, morphology, and size, dissociating the egocentric visual perspective from the body, and exploiting the role of multimodal information in spatiotemporal term for body perception [52].

For instance, Slater's study embodied participants alternately in two virtual bodies such that they could have an extended conversation with themselves. In the study, one body represented themselves and the other represented Dr. Sigmund Freud with whom they would discuss a personal problem. While embodying their own body (lookalike) representation, they described the problem. They then transfer to the counselor body and, from that perspective, saw and heard their lookalike body describing the problem, and then gave some insight into how the problem might be solved. They would then transfer back to their own lookalike body and look at and listen to the counselor body giving them the advice, and then respond to it. If they chose to, they could then once again see and hear this response from the perspective of the counselor body and again respond to it. This process of switching between the lookalike and counselor body continued until the participant decided to stop. Results showed high level of sense of embodiment [52].

world," but also to experience synthetic avatars as if they are "our own body." VR can be defined also as an "embodied technology" for "its ability of modifying the feeling of presence" [49]. People's representations in the virtual world are usually named avatars, and for the first time in human history, they could allow us to watch ourselves being others and doing some-

Following the point, VR as an "embodied technology" can be useful to alter the body's bounds and to study the relationship among body (posture, movements, actions) and cognition and emotions. Moreover, virtual avatars can help to amplify the "modeling" learning, and there-

For instance, literature points out that [50] the sense of embodiment is principally based on three main aspects: sense of self-location, sense of agency, and sense of body ownership [50]. Self-location is a determinate volume in space where one feels to be located. Normally selflocation and body-space coincide in the sense that one feels self-located inside physical body. This collocation can break down when people have an out-of-body experience in which they perceive themselves outside of their physical body. The sense of agency refers to the sense of having a global motor control, including the subjective experience of action, control, intention, motor selection, and the conscious experience of will. The development of agency depends on the synchronicity of visual motor correlations, for example, the imagination of a tool that touches my hand and the real touch of the hand. The sense of body ownership refers to one's self-attribution of the body. It has been proposed to emerge from a combination of bottom-up and top-down influences. Bottom-up information refers to the afferent sensory information that arrives to put brain from our sensory organs; top-down information consists of the cognitive process that may modulate the processing of sensory stimuli, for example, how much the avatar in VR is similar to real person (this aspect depends most of the time on the graphic computer quality). In fact, the illusion of ownership diminishes when the external

object does not resemble or is in a different spatial configuration to the real body [51].

Experimental manipulation of the embodied experience is problematic. However, the use of VR has unique advantages to control the factors associated with the embodied experience. For example, VR makes possible in a relatively easy manner the manipulation of the body representation in terms of structure, morphology, and size, dissociating the egocentric visual perspective from the body, and exploiting the role of multimodal information in spatiotemporal

For instance, Slater's study embodied participants alternately in two virtual bodies such that they could have an extended conversation with themselves. In the study, one body represented themselves and the other represented Dr. Sigmund Freud with whom they would discuss a personal problem. While embodying their own body (lookalike) representation, they described the problem. They then transfer to the counselor body and, from that perspective, saw and heard their lookalike body describing the problem, and then gave some insight into how the problem might be solved. They would then transfer back to their own lookalike body and look at and listen to the counselor body giving them the advice, and then respond to it. If they chose to, they could then once again see and hear this response from the perspective of the counselor body and again respond to it. This process of switching between the lookalike and counselor

thing they have never done [49].

term for body perception [52].

fore to promote the learning of adaptive behaviors.

108 State of the Art Virtual Reality and Augmented Reality Knowhow

Other recent studies have also proved how changing the perspective, and living experiences from other bodies, can help us to promote empathy, and even compassion and selfcompassion. For instance, Salter's group has analyzed how the use of virtual bodies can promote compassion and self-compassion. They have analyzed the effects of self-identification with virtual bodies within immersive VR to increase self-compassion in persons with high selfcriticism and depression [53]. The same author and his colleagues investigated also how embodied, in VR, a black avatar decreases the racial prejudice [54]. Moreover, Bailenson's group has studied how embodied an avatar in VR can make us better people. For example, participants embodied a Superman avatar and the results show that, after the experiment, they felt more helpful [55]. In other Bailenson study, participants embodied a sea coral and, after the experiment, they felt more sensible to ocean pollution [51].

Other interesting approaches are based on the development of new immersive technology named: machine to be another. It is an advantage technology to interchange bodies (the real body with the avatar) which offers to the users an immersive experience of seeing themselves in the body of another person. MTBA is a low budget body swapping system, and using a head-mounted display, participant sees the perspective of another person (performer) that mimics his/her movements (Figure 2).

The performer's first-person perspective is captured by a camera controlled by user's head movements, revealing torso, legs and arms of the performer's body. While interacting with the physical space, participants also perceive realistic tactile stimuli. An audio system plays a

Figure 2. The machine to be another system. Labpsitec, Universidad de Valencia, Spain.

personal narrative recorded by the performer. The goal of the MTBA is to induce empathy and compassion to participants as the study of Falconer [53].

SimCoach was created to face the urgent need of reducing the stigma of seeking mental health treatment for veteran population who have a strong PTSD and cutting geographical barriers. SimCoach allows users to initiate and engage in a dialog about their health care concerns with an interactive AI therapist. Generally, these intelligent graphical characters were designed to use speech, gestures, and emotions to introduce the capabilities of the system, solicit basic anonymous background information about the user's history and clinical-psychosocial concerns, provide advice and support, direct the user to relevant online content, and facilitate the process of seeking appropriate care with a live clinical provider. The SimCoach project is not conceived to deliver a diagnosis or treatment or serve as a replacement for human provides and experts, but to support users who are determined to be in need, to make the decision to take the first step toward initiating psychological or

Virtual and Augmented Reality: New Frontiers for Clinical Psychology

http://dx.doi.org/10.5772/intechopen.74344

111

The connectedness potential of technology seems to be limitless. Thanks to its feature, it was possible to reach people who were in need and without physical or social resource for seeking

As we discussed, the three features of VR and AR, imagery, embodiment, and connectedness are fundamental for the efficacy and effectiveness of cybertherapy. Clinical psychology reached an excellent level in the society, and it contributes to the well-being of community. These goals were reached easily, thanks to the supply of new technologies, and in few years,

The future of VR and AR therapy includes treatment of a wide variety of disorders. Internet dissemination allows therapists easy access to new virtual environments and provides them with a broader selection of options for use in therapy. The possibility of offering VR services to patients, under therapist supervision, at home has already become a reality, but this practice

Adding to the previous methods, exciting new methods are allowing for the introduction of real digitized images into the virtual world [22]. This technique was published at the last Facebook congress by Mark Zuckerberg. He introduced, in the social virtual world, the possibility to show some personal pictures or the own smartphone pictures to increase the sense of presence [59]. These techniques have the potential to help disorders as attention, social phobia, public speaking, anxiety disorder, PTSD, and so on. For instance, including photograph of child's actual classroom, the therapist can work with them to improve concentration skills during tests and assignments. In addition, working with the photographs of classmates, coworkers, or family members can help people to improve their social skills in a safety virtual world and then can actualize their abilities in the real one [22]. For example, after practicing with the therapist in a closed system, the client can visit a virtual world populated by other avatar, initiate conversation, and obtain feedback from other avatars in real time audio.

medical care with a real psychotherapist, who is the best [58].

a therapy assessment.

we assist to their potential growth.

4. The future of cybertherapy

has yet to become widespread.

To summarize, the potentialities of VR as an "embodied technology" open its use wider than the only reproduction of real worlds. Thanks to this advanced technology, it is possible to study behavioral, cognitive, and emotional aspects that were hard to realize so far. And by designing meaningful embodied activities, VR can enhance therapy and promote significant changes.

#### 3.3. Connectedness technology

Recently, software developers and ICT industries have emphasized that next interesting and compelling work for VR and AR focuses on helping people to connect with others through shared experiences. The new developments of VR are changing not only the way people interact with computers, but also the way individuals interact each other. One of the evident signs of this change has been the creation of totally new interactive communication environments like computer-mediated communication (CMC). CMC created a new social space, called cyberspace [56]. Interaction and connection are the key features of cyberspace, from which a sense of self can be built. VR, more than other technologies, carries the detachment of interaction from physical interlocutor co-presence to its logical extreme and challenges the very concept of interlocutor identity. The concept of cyberspace clearly shows that VR is, in fact, a parallel universe created and maintained by the networks in which people interact.

The CMC brought new challenges to clinical psychology. There are objective barriers that stop people the seeking for a therapy program (e.g., geographical distance and lack of local skilled clinicians). To go over all these barriers, new technologies have found some solutions. For instance, videoconferencing means simulating face-to-face psychotherapy using a clinician at a site distant from the client [57]. Some clinicians are not agreeing to use videoconference technology for therapy because they argue that the therapeutic alliance will be impaired [57]. However, other studies have shown that therapeutic alliance is established equally well in videoconferencing and face-to-face therapy and, in some case, participants prefer the online therapy (e.g., younger participants, people living in rural and remote areas). Cybertherapy can go even further over this problem, thanks to its power to cut geographical limit and to reach people anywhere and anytime [57]. The growing of new technology gives also the opportunities to interact with people inside the virtual environment. Thanks to this forward step, it is now possible not just to call a psychotherapist by Skype, but also to interact with the artificial therapist.

Since several years, virtual environment was populated by avatar, a graphical image that represents a person, and it can interact with real people, thanks to its artificial intelligence (AI). This innovative tool addressed numerous issues in clinical research, assessment, and intervention. The first AI tool was "Eliza" that was designed to imitate a Rogerian therapist. The system allowed a computer user to interact with a virtual therapist by typing simple sentence responses to the computerized therapist's questions. "Eliza" disappeared soon due its inability to handle complexity questions from the users [58]. A new version of AI therapist was created by the Institute of Creative Technology, Los Angeles (USA). The program SimCoach was created to face the urgent need of reducing the stigma of seeking mental health treatment for veteran population who have a strong PTSD and cutting geographical barriers. SimCoach allows users to initiate and engage in a dialog about their health care concerns with an interactive AI therapist. Generally, these intelligent graphical characters were designed to use speech, gestures, and emotions to introduce the capabilities of the system, solicit basic anonymous background information about the user's history and clinical-psychosocial concerns, provide advice and support, direct the user to relevant online content, and facilitate the process of seeking appropriate care with a live clinical provider. The SimCoach project is not conceived to deliver a diagnosis or treatment or serve as a replacement for human provides and experts, but to support users who are determined to be in need, to make the decision to take the first step toward initiating psychological or medical care with a real psychotherapist, who is the best [58].

The connectedness potential of technology seems to be limitless. Thanks to its feature, it was possible to reach people who were in need and without physical or social resource for seeking a therapy assessment.

As we discussed, the three features of VR and AR, imagery, embodiment, and connectedness are fundamental for the efficacy and effectiveness of cybertherapy. Clinical psychology reached an excellent level in the society, and it contributes to the well-being of community. These goals were reached easily, thanks to the supply of new technologies, and in few years, we assist to their potential growth.

### 4. The future of cybertherapy

personal narrative recorded by the performer. The goal of the MTBA is to induce empathy and

To summarize, the potentialities of VR as an "embodied technology" open its use wider than the only reproduction of real worlds. Thanks to this advanced technology, it is possible to study behavioral, cognitive, and emotional aspects that were hard to realize so far. And by designing meaningful embodied activities, VR can enhance therapy and promote significant changes.

Recently, software developers and ICT industries have emphasized that next interesting and compelling work for VR and AR focuses on helping people to connect with others through shared experiences. The new developments of VR are changing not only the way people interact with computers, but also the way individuals interact each other. One of the evident signs of this change has been the creation of totally new interactive communication environments like computer-mediated communication (CMC). CMC created a new social space, called cyberspace [56]. Interaction and connection are the key features of cyberspace, from which a sense of self can be built. VR, more than other technologies, carries the detachment of interaction from physical interlocutor co-presence to its logical extreme and challenges the very concept of interlocutor identity. The concept of cyberspace clearly shows that VR is, in fact, a

parallel universe created and maintained by the networks in which people interact.

The CMC brought new challenges to clinical psychology. There are objective barriers that stop people the seeking for a therapy program (e.g., geographical distance and lack of local skilled clinicians). To go over all these barriers, new technologies have found some solutions. For instance, videoconferencing means simulating face-to-face psychotherapy using a clinician at a site distant from the client [57]. Some clinicians are not agreeing to use videoconference technology for therapy because they argue that the therapeutic alliance will be impaired [57]. However, other studies have shown that therapeutic alliance is established equally well in videoconferencing and face-to-face therapy and, in some case, participants prefer the online therapy (e.g., younger participants, people living in rural and remote areas). Cybertherapy can go even further over this problem, thanks to its power to cut geographical limit and to reach people anywhere and anytime [57]. The growing of new technology gives also the opportunities to interact with people inside the virtual environment. Thanks to this forward step, it is now possible not just to call a psychotherapist by Skype, but also to interact with the artificial

Since several years, virtual environment was populated by avatar, a graphical image that represents a person, and it can interact with real people, thanks to its artificial intelligence (AI). This innovative tool addressed numerous issues in clinical research, assessment, and intervention. The first AI tool was "Eliza" that was designed to imitate a Rogerian therapist. The system allowed a computer user to interact with a virtual therapist by typing simple sentence responses to the computerized therapist's questions. "Eliza" disappeared soon due its inability to handle complexity questions from the users [58]. A new version of AI therapist was created by the Institute of Creative Technology, Los Angeles (USA). The program

compassion to participants as the study of Falconer [53].

110 State of the Art Virtual Reality and Augmented Reality Knowhow

3.3. Connectedness technology

therapist.

The future of VR and AR therapy includes treatment of a wide variety of disorders. Internet dissemination allows therapists easy access to new virtual environments and provides them with a broader selection of options for use in therapy. The possibility of offering VR services to patients, under therapist supervision, at home has already become a reality, but this practice has yet to become widespread.

Adding to the previous methods, exciting new methods are allowing for the introduction of real digitized images into the virtual world [22]. This technique was published at the last Facebook congress by Mark Zuckerberg. He introduced, in the social virtual world, the possibility to show some personal pictures or the own smartphone pictures to increase the sense of presence [59]. These techniques have the potential to help disorders as attention, social phobia, public speaking, anxiety disorder, PTSD, and so on. For instance, including photograph of child's actual classroom, the therapist can work with them to improve concentration skills during tests and assignments. In addition, working with the photographs of classmates, coworkers, or family members can help people to improve their social skills in a safety virtual world and then can actualize their abilities in the real one [22]. For example, after practicing with the therapist in a closed system, the client can visit a virtual world populated by other avatar, initiate conversation, and obtain feedback from other avatars in real time audio.

Another future perspective for the cybertherapy is the use of the "data glove." It allows for tactile interaction in the virtual environment, giving to the users the ability to grasp and manipulate virtual objects. This technology can be used to increase the sense of presence in the virtual environment, or to help desensitize patients to disturbing tactile stimuli, or to distract them for a painful surgery [22].

promotes the experience of feeling user's body inside the virtual environment (embodied technology). And now, VR/AR could help users to connect with others and to share experiences (connectivity technology). These characteristics have enormous potential for clinical psychology and to improve psychological treatments. However, several barriers still remain. The first obstacle refers to the expense, especially of designing and creating virtual environments [26]. However, the development of technologies has reduced the cost dramatically. Now is easy to find economic equipment and there are several free online programs to develop virtual environments. Second, VR software and clinical protocols still lack standardization, and most VR systems available are not interoperable. These issues force most researchers to spend a lot of time and money designing and developing "one-off" VR creations. Third, from therapists' side, the operation of PC-based VR programs requires basic computer skills. In addition, there is a different therapist-patient dynamic in VR therapy that also must be taken into account. Fourth refers to the patient side. The insufficient sense of presence level felt by patients in the virtual environment could negatively affect the therapy. Other barrier from patients is cybersickness, and symptoms can include motion sickness, oculomotor problems, and migraines. One more barrier regards cultural adaptation. VR applications, as other psychological tools, need to be culturally adapted in order to make it compatible with the patients'

Virtual and Augmented Reality: New Frontiers for Clinical Psychology

http://dx.doi.org/10.5772/intechopen.74344

113

Overall, the future of VR and AR seems to be limitless. Technology progress is growing exponentially, and it can generate great and significant changes for clinical psychology. Till now, cybertherapy made huge steps which contributed to the spread of well-being in the society, but it is just on the starting point. Clinical researchers are working hard to keep the

This chapter was supported by "PROMOSAM" (PSI2014-56303-REDT) and "BODYTA" (Plan Nacional I + D + I 2013-2016 PSI2014-51928-R), and Santiago Grisolía program—Generalitat

relationship between clinical psychology and ICTs in privileged position.

Valenciana 2017, CIBEROBN is an initiative of the ISCIII (Spain).

There is no financial or personal interest to report.

ICT information and communication technology

experience and with the general therapeutic goal [12].

Acknowledgements

Conflict of interest

Abbreviations

VR virtual reality

AR augmented reality

As technology advances and more disorders are being treated through it, research continues into ways in which the boundaries of cybertherapy can be expanded. Technology is growing more and more, and its contribution to clinical psychology will be a crucial issue. Furthermore, in developing new virtual and augmented reality tools, it is important to keep several concepts in mind as therapeutic concept and ethic, the practice and the costs of equipment, and the multidisciplinary teams of experts in particular psychologists and engineers.

### 5. Conclusion

A technological revolution in mental health care is approaching. As Freeman and colleagues have stated, at the forefront may be virtual and augmented reality, the powerful tools for individuals to make new learning for the benefit of their psychological well-being [60]. Cybertherapy is quickly becoming an accepted and validated method for the treatment of many different mental disorders. In this chapter, we have described the potential of VR and AR to improve the role of clinic psychology. In Table 2, the advantages and limits of VR and AR in clinical psychology are summarized.

VR and AR have some differences: in VR people are immersed in virtual environments and interact with it. The perception to interact with the environment is possible, thanks to the sense of presence (the feeling to be inside the environment); AR is an extension of the real world, mixing real and artificial reality where people keeping the opportunity to see their own body, which is not possible with VR, and to interact with the artificial objects. In the last decades, many studies have shown that VR and AR are successful tools for therapy with a high level of efficacy, satisfaction from participants, and decrease in participant's symptoms.

Imagery, embodiment, and connectedness play the keys roles for understanding VR and AR and their potentialities in clinical psychology. In virtual environments, people can feel "as if they are" in a reality that does not exist in external world (imagery technology). VR/AR also


Table 2. Advantages and limits of VR and AR in clinical psychology.

promotes the experience of feeling user's body inside the virtual environment (embodied technology). And now, VR/AR could help users to connect with others and to share experiences (connectivity technology). These characteristics have enormous potential for clinical psychology and to improve psychological treatments. However, several barriers still remain. The first obstacle refers to the expense, especially of designing and creating virtual environments [26]. However, the development of technologies has reduced the cost dramatically. Now is easy to find economic equipment and there are several free online programs to develop virtual environments. Second, VR software and clinical protocols still lack standardization, and most VR systems available are not interoperable. These issues force most researchers to spend a lot of time and money designing and developing "one-off" VR creations. Third, from therapists' side, the operation of PC-based VR programs requires basic computer skills. In addition, there is a different therapist-patient dynamic in VR therapy that also must be taken into account. Fourth refers to the patient side. The insufficient sense of presence level felt by patients in the virtual environment could negatively affect the therapy. Other barrier from patients is cybersickness, and symptoms can include motion sickness, oculomotor problems, and migraines. One more barrier regards cultural adaptation. VR applications, as other psychological tools, need to be culturally adapted in order to make it compatible with the patients' experience and with the general therapeutic goal [12].

Overall, the future of VR and AR seems to be limitless. Technology progress is growing exponentially, and it can generate great and significant changes for clinical psychology. Till now, cybertherapy made huge steps which contributed to the spread of well-being in the society, but it is just on the starting point. Clinical researchers are working hard to keep the relationship between clinical psychology and ICTs in privileged position.

#### Acknowledgements

Another future perspective for the cybertherapy is the use of the "data glove." It allows for tactile interaction in the virtual environment, giving to the users the ability to grasp and manipulate virtual objects. This technology can be used to increase the sense of presence in the virtual environment, or to help desensitize patients to disturbing tactile stimuli, or to

As technology advances and more disorders are being treated through it, research continues into ways in which the boundaries of cybertherapy can be expanded. Technology is growing more and more, and its contribution to clinical psychology will be a crucial issue. Furthermore, in developing new virtual and augmented reality tools, it is important to keep several concepts in mind as therapeutic concept and ethic, the practice and the costs of equipment, and the

A technological revolution in mental health care is approaching. As Freeman and colleagues have stated, at the forefront may be virtual and augmented reality, the powerful tools for individuals to make new learning for the benefit of their psychological well-being [60]. Cybertherapy is quickly becoming an accepted and validated method for the treatment of many different mental disorders. In this chapter, we have described the potential of VR and AR to improve the role of clinic psychology. In Table 2, the advantages and limits of VR and

VR and AR have some differences: in VR people are immersed in virtual environments and interact with it. The perception to interact with the environment is possible, thanks to the sense of presence (the feeling to be inside the environment); AR is an extension of the real world, mixing real and artificial reality where people keeping the opportunity to see their own body, which is not possible with VR, and to interact with the artificial objects. In the last decades, many studies have shown that VR and AR are successful tools for therapy with a high level of

Imagery, embodiment, and connectedness play the keys roles for understanding VR and AR and their potentialities in clinical psychology. In virtual environments, people can feel "as if they are" in a reality that does not exist in external world (imagery technology). VR/AR also

Source of information on patient's performance achievements Basic technological capabilities for therapist and patients Ecological validity Deficits in sense of presence can influence therapy success

efficacy, satisfaction from participants, and decrease in participant's symptoms.

Control of stimulus presentation and response measurement Lack of interoperability of VR applications Safe virtual situations Different therapist-patient dynamic

multidisciplinary teams of experts in particular psychologists and engineers.

distract them for a painful surgery [22].

112 State of the Art Virtual Reality and Augmented Reality Knowhow

AR in clinical psychology are summarized.

Advantages Limits More naturalistic or "real-life" environments Cost required

Increased standardization of rehabilitation protocols Cybersickness

Table 2. Advantages and limits of VR and AR in clinical psychology.

5. Conclusion

This chapter was supported by "PROMOSAM" (PSI2014-56303-REDT) and "BODYTA" (Plan Nacional I + D + I 2013-2016 PSI2014-51928-R), and Santiago Grisolía program—Generalitat Valenciana 2017, CIBEROBN is an initiative of the ISCIII (Spain).

### Conflict of interest

There is no financial or personal interest to report.

#### Abbreviations




#### Author details

Sara Ventura<sup>1</sup> \*, Rosa M. Baños1,2 and Cristina Botella2,3

\*Address all correspondence to: saven2@alumni.uv.es


[7] Girard B, Turcotte V, Bouchard S, Girard B. Crushing virtual cigarettes reduces tobacco addiction and treatment discontinuation. CyberPsychology & Behavior. Oct 1, 2009;12(5):

Virtual and Augmented Reality: New Frontiers for Clinical Psychology

http://dx.doi.org/10.5772/intechopen.74344

115

[8] Bordnick PS, Traylor A, Copp HL, Graap KM, Carter B, Ferrer M, et al. Assessing reactivity to virtual reality alcohol based cues. Addictive Behaviors. Jun 1, 2008;33(6):

[9] Serino S, Cipresso P, Gaggioli A, Pallavicini F, Cipresso S, Campanaro D, et al. Smartphone for self-management of psychological stress: a preliminary evaluation of positive technology app. Revista de Psicopatología y Psicología Clínica. Dec 1, 2014;19(3):253-260.

[10] Baños RM, Espinoza M, García-Palacios A, Cervera JM, Esquerdo G, Barrajón E, et al. A positive psychological intervention using virtual reality for patients with advanced cancer in a hospital setting: A pilot study to assess feasibility. Supportive Care in Cancer. Jan

[11] Li A, Montaño Z, Chen VJ, Gold JI. Virtual reality and pain management: Current trends and future directions. Pain Management. Mar 1, 2011;1(2):147-157. DOI: 10.2217/pmt.10.15

[12] Steuer J. Defining virtual reality: Dimensions determining telepresence. Journal of Com-

[13] Botella C, Bretón-López J, Quero S, Baños R, García-Palacios A. Treating cockroach phobia with augmented reality. Behavior Therapy. Sep 1, 2010;41(3):401-413. DOI: 10.10

[14] Alcañiz M, Botella C, Baños RM, Zaragoza I, Guixeres J. The intelligent e-therapy system: A new paradigm for telepsychology and cybertherapy. British Journal of Guidance & Counsel-

[15] Andersson G. Internet-delivered psychological treatments. Annual Review of Clinical Psychology. 2016;12(1):157-179. DOI: 10.1146/annurev-clinpsy-021815-093006

[16] Perednia DA, Allen A. Telemedicine technology and clinical applications. Journal of the American Medical Association. Feb 8, 1995;273(6):483-488. DOI: 10.1001/jama.1995.0352

[17] Brown FW. Rural telepsychiatry. Psychiatric Services. Jul 1, 1998;49(7):963-964. DOI: 10.1176/

[18] Eysenbach G and CONSORT-EHEALTH Group. CONSORT-EHEALTH: Improving and standardizing evaluation reports of web-based and mobile health interventions. Journal of Medical Internet Research. Dec 31, 2011;13(4):e126. DOI: http://doi.org/10.2196/jmir.1923

[19] Alleman JR. Online counseling: The Internet and mental health treatment. Psychotherapy: Theory, Research, Practice, Training. 2002;39(2):199-209. DOI: http://doi.org/10.2196/jmir.1923

[20] Andersson G, Titov N. Advantages and limitations of Internet-based interventions for common mental disorders. World Psychiatry. 2014;13:4-11. DOI: 10.1002/wps.20083

munication. 1992;42:73-93. DOI: 10.1111/j.1460-2466.1992.tb00812.x

ling. Aug 1, 2009;37(3):287-296. DOI: 10.1080/03069880902957015

477-483. DOI: 10.1089/cpb.2009.0118

743-756. DOI: 10.1016/j.addbeh.2007.12.010

DOI: 10.5944/rppc.vol.19.num.3.2014.13906

16/j.beth.2009.07.002

0300057037

ps.49.7.963

1, 2013;21(1):263-270. DOI: 10.1007/s00520-012-1520-x

3 Universitat Jaume I, Castellón, Spain

#### References


[7] Girard B, Turcotte V, Bouchard S, Girard B. Crushing virtual cigarettes reduces tobacco addiction and treatment discontinuation. CyberPsychology & Behavior. Oct 1, 2009;12(5): 477-483. DOI: 10.1089/cpb.2009.0118

PTSD posttraumatic stress disorder

CMC computer-mediated communication

114 State of the Art Virtual Reality and Augmented Reality Knowhow

\*, Rosa M. Baños1,2 and Cristina Botella2,3

2 CIBER Fisiopatología Obesidad y Nutrición (CIBEROBN), Instituto Carlos III, Spain

[1] Riva G, Calvo R, Lisetti C. Cyberpsychology and affective computing. In: The Oxford Handbook of Affective Computing. 2015. pp. 547-558. DOI: 10.1093/oxfordhb/9780199942237.

[2] Wiederhold MD. Virtual Reality Therapy for Anxiety Disorders: Advances in Evaluation and Treatment. In: Wiederhold BK, editor. American Journal of Psychiatry. 2005;162(9):

[3] Botella C, García-Palacios A, Villa H, Baños RM, Quero S, Alcañiz M, et al. Virtual reality exposure in the treatment of panic disorder and agoraphobia: A controlled study. Clinical

[4] Baños RM, Guillen V, Quero S, García-Palacios A, Alcaniz M, Botella C. A virtual reality system for the treatment of stress-related disorders: A preliminary analysis of efficacy compared to a standard cognitive behavioral program. International Journal of Human-Computer Studies. 2011;69(9):602-613. DOI: https://doi.org/10.1016/j.ijhcs.2011.06.002 [5] Botella C, Serrano B, Baños RM, Garcia-Palacios A. Virtual reality exposure-based therapy for the treatment of post-traumatic stress disorder: a review of its efficacy, the adequacy of the treatment protocol, and its acceptability. Neuropsychiatric Disease and

[6] Perpiñá C, Botella C, Baños R, Marco H, Alcañiz M, Quero S. Body image and virtual reality in eating disorders: Is exposure to virtual reality more effective than the classical body image treatment? CyberPsychology & Behavior. Apr 1, 1999;2(2):149-155. DOI:

Psychology & Psychotherapy. May 1, 2007;14(3):164-175. DOI: 10.1002/cpp.524

Treatment. Oct 3, 2015;11:2533-2545. DOI: 10.2147/NDT.S89542

\*Address all correspondence to: saven2@alumni.uv.es

1772. DOI: http://dx.doi.org/10.1037/10858-000

1 Universitat de València, Valencia, Spain

3 Universitat Jaume I, Castellón, Spain

MTBA machine to be another

AI artificial intelligence

Author details

Sara Ventura<sup>1</sup>

References

013.017

10.1089/cpb.1999.2.149


[21] Mohr DC, Duffecy J, Jin L, Ludman EJ, Lewis A, Begale M, McCarthy Jr M. Multimodal emental health treatment for depression: A feasibility trial. Journal of Medical Internet Research. Dec 19, 2010;12(5):e48. DOI: http://doi.org/10.2196/jmir.1370

[35] Milgram P, Kishino F. A taxonomy of mixed reality visual displays. IEICE Transactions

Virtual and Augmented Reality: New Frontiers for Clinical Psychology

http://dx.doi.org/10.5772/intechopen.74344

117

[36] Hughes CE, Stapleton CB, Hughes DE, Smith EM. Mixed reality in education, entertainment, and training. IEEE Computer Graphics and Applications. Nov 2005;25(6):24-30.

[37] Shuhaiber JH. Augmented reality in surgery. Archives of Surgery. Feb 1, 2004;139(2):170-

[38] Botella C, Perpiñá C, Baños RM, García-Palacios A. Virtual reality: A new clinical setting lab. In: Riva G, Wiederhold BK, Molinari E, editors. Studies in Health Technology and Informatics, Virtual Environments in Clinical Psychology and Neuroscience: Methods and Techniques in Advanced Patient–Therapist Interaction. Vol. 58. Amsterdam: IOS

[39] Ji JL, Heyes SB, MacLeod C, Holmes EA. Emotional mental imagery as simulation of reality: Fear and beyond—A tribute to Peter Lang. Behavior Therapy. Sep 1, 2016;47(5):

[40] Pearson DG, Deeprose C, Wallace-Hadrill SMA, Heyes SB, Holmes EA. Assessing mental imagery in clinical psychology: A review of imagery measures and a guiding framework.

[41] Holmes EA, Bourne C. Inducing and modulating intrusive emotional memories: A review of the trauma film paradigm. Acta Psychologica. Mar 1, 2008;127(3):553-566.

[42] Clark IA, James EL, Iyadurai L, Holmes EA. Mental imagery in psychopathology: From the lab to the clinic. In: Watson LA, Berntsen D, editors. Clinical Perspectives on Autobiographical Memory [Internet]. Cambridge (UK): Cambridge University Press; 2015

[43] Hackmann A, Clark DM, McManus F. Recurrent images and early memories in social phobia. Behaviour Research and Therapy. Jun 1, 2000;38(6):601-610. DOI: 10.1016/S0005-

[44] Day S, Holmes E, Hackmann A. Occurrence of imagery and its link with early memories in agoraphobia. Memory. Jul 1, 2004;12(4):416-427. DOI: 10.1080/09658210444000034 [45] Holmes EA, Geddes JR, Colom F, Goodwin GM. Mental imagery as an emotional amplifier: Application to bipolar disorder. Behaviour Research and Therapy. Dec 1, 2008;46(12):

[46] Holmes EA, Mathews A. Mental imagery and emotion: A special relationship? Emotion.

[47] Baños RM, Etchemendy E, Castilla D, García-Palacios A, Quero S, Botella C. Positive mood induction procedures for virtual environments designed for elderly people. Interacting with Computers. May 1, 2012;24(3):131-138. DOI: 10.1016/j.intcom.2012.

Clinical Psychology Review. Feb 1, 2013;33(1):1-23. DOI: 10.1016/j.cpr.2012.09.001

on Information and Systems. Dec 25, 1994;77(12):1321-1329

DOI: 10.1109/MCG.2005.139

Press; 1998. pp. 73-81

174. DOI: 10.1001/archsurg.139.2.170

702-719. DOI: 10.1016/j.beth.2015.11.004

DOI: 10.1016/j.actpsy.2007.11.002

1251-1258. DOI: 10.1016/j.brat.2008.09.005

2005;5(4):489-497. DOI: 10.1037/1528-3542.5.4.489

7967(99)00161-8

04.002


[35] Milgram P, Kishino F. A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems. Dec 25, 1994;77(12):1321-1329

[21] Mohr DC, Duffecy J, Jin L, Ludman EJ, Lewis A, Begale M, McCarthy Jr M. Multimodal emental health treatment for depression: A feasibility trial. Journal of Medical Internet

[22] Botella C, Garcia-Palacios A, Baños RM, Quero S. Cybertherapy: Advantages, limitations,

[23] Powers MB, Emmelkamp PMG. Virtual reality exposure therapy for anxiety disorders: A meta-analysis. Journal of Anxiety Disorders. 2008 Apr 1;22(3):561-569. DOI: 10.1016/j.

[24] Schultheis MT, Rizzo AA. The application of virtual reality technology in rehabilitation. Rehabilitation Psychology. 2001;46(3):296-311. DOI: 10.1037/0090-5550.46.3.296

[25] Pegden CD. The evolution of simulation languages. In: Advances in Modeling and Simulation [Internet]. Cham: Springer; 2017. pp. 81-96. (Simulation Foundations, Methods

[26] Riva G. Virtual reality in psychotherapy: Review. CyberPsychology & Behavior. Jun 1,

[27] Jones MC. A laboratory study of fear: The case of peter. The Journal of Genetic Psychol-

[28] Sanchez-Vives MV, Slater M. From presence to consciousness through virtual reality.

[29] Hodges LF, Kooper R, Meyer TC, Rothbaum BO, Opdyke D, de GJJ, et al. Virtual environments for treating the fear of heights. Computer. Jul 1995;28(7):27-34. DOI: 10.1109/

[30] Opriş D, Pintea S, García-Palacios A, Botella C, Szamosközi Ş, David D. Virtual reality exposure therapy in anxiety disorders: a quantitative meta-analysis. Depress Anxiety.

[31] Botella C, Fernández-Álvarez J, Guillén V, García-Palacios A, Baños R. Recent progress in virtual reality exposure therapy for phobias: A systematic review. Current Psychiatry

[32] Botella C, Osma J, Garcia-Palacios A, Quero S, Baños RM. Treatment of flying phobia using virtual reality: data from a 1-year follow-up using a multiple baseline design.

[33] Botella C, Baños RM, García-Palacios A, Quero S. Chapter 22 – Virtual Reality and Other Realities A2 – Hofmann, Stefan G. In: Asmundson GJG, editor. The Science of Cognitive Behavioral Therapy [Internet]. San Diego: Academic Press; 2017. pp. 551-590. DOI: 10.1016/

[34] Emmelkamp PM, Krijn M, Hulsbosch A, de Vries S, Schuemie M, van der Mast CAP. Virtual reality treatment versus exposure in vivo: a comparative evaluation in acrophobia. Behaviour Research and Therapy. May 1, 2002;40(5):509-516. DOI: 10.1016/S0005-7967(01)00023-7

Clinical Psychology & Psychotherapy. 2004;11:311-323. DOI: 10.1002/cpp.404

ogy. Dec 1, 1991;152(4):462-469. DOI: 10.1080/00221325.1991.9914707

Reports. Jul 1, 2017;19(7):42. DOI: 10.1007/s11920-017-0788-4

Nature Reviews Neuroscience. Apr 2005;6(4):332. DOI: 10.1038/nrn1651

Research. Dec 19, 2010;12(5):e48. DOI: http://doi.org/10.2196/jmir.1370

and ethical issues. PsychNology Journal. 2009;7(1):77-100

116 State of the Art Virtual Reality and Augmented Reality Knowhow

and Applications). DOI: 10.1007/978-3-319-64182-9\_6

2005;8(3):220-230. DOI: 10.1089/cpb.2005.8.220

2012;29:85-93. DOI: 10.1002/da.20910

B978-0-12-803457-6.00022-2

janxdis.2007.04.006

2.391038


[48] Riva G, Baños RM, Botella C, Mantovani F, Gaggioli A. Transforming experience: The potential of augmented reality and virtual reality for enhancing personal and clinical change. Front Psychiatry. Sep 30, 2016;7:164. DOI: 10.3389/fpsyt.2016.00164

**Chapter 8**

**Provisional chapter**

**How to Create Suitable Augmented Reality Application**

**Application to Teach Social Skills for Children with ASD**

Autism spectrum disorders (ASDs) are characterized by a reduced ability to appropriately express social greetings. Studies have indicated that individuals with ASD might not recognize the crucial nonverbal cues that usually aid social interaction. This study applied augmented reality (AR) with tabletop role-playing game (AR-RPG) to focus on the standard nonverbal social cues to teach children with ASD, how to appropriately reciprocate when they socially interact with others. The results showed that intervention system provides an AR combined with physical manipulatives and presents corresponding specific elements in an AR 3D animation with dialogue; thus, it can be used to help them increase their social interaction skills and drive their attention toward the meaning and social value of greeting behavior in specific social situations. We conclude that AR-RPG of social situations helped children with ASD recognize and better understand these situations and moderately effective in teaching the target greeting responses. **Keywords:** autism spectrum disorders, augmented reality, educational technology,

Autism spectrum disorders (ASD) are characterized by social interaction difficulties, communication challenges, and tendencies to engage in repetitive behaviors [1]. In particular, social reciprocity defects are one of the core deficits in social interaction for people with ASD [2]. Social reciprocity depends on the ability to empathize with others, to be aware of emotional and interpersonal cues, and to respond appropriately [3]. In addition, ASD is characterized

greeting behavior, physical manipulatives, tabletop role-playing game

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

© 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

DOI: 10.5772/intechopen.76476

**to Teach Social Skills for Children with ASD**

**How to Create Suitable Augmented Reality** 

I-Jui Lee, Ling-Yi Lin, Chien-Hsu Chen and

I-Jui Lee, Ling-Yi Lin, Chien-Hsu Chen and

Additional information is available at the end of the chapter

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.76476

Chi-Hsuan Chung

Chi-Hsuan Chung

**Abstract**

**1. Introduction**


#### **How to Create Suitable Augmented Reality Application to Teach Social Skills for Children with ASD How to Create Suitable Augmented Reality Application to Teach Social Skills for Children with ASD**

DOI: 10.5772/intechopen.76476

I-Jui Lee, Ling-Yi Lin, Chien-Hsu Chen and Chi-Hsuan Chung I-Jui Lee, Ling-Yi Lin, Chien-Hsu Chen and Chi-Hsuan Chung

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.76476

#### **Abstract**

[48] Riva G, Baños RM, Botella C, Mantovani F, Gaggioli A. Transforming experience: The potential of augmented reality and virtual reality for enhancing personal and clinical

[49] Longo MR, Schüür F, Kammers MPM, Tsakiris M, Haggard P. What is embodiment? A psychometric approach. Cognition. Jun 1, 2008;107(3):978-998. DOI: 10.1016/j.cogni-

[50] Kilteni K, Groten R, Slater M. The sense of embodiment in virtual reality. Presence: Teleoperators and Virtual Environments. Nov 1, 2012;21(4):373-387. DOI: 10.1162/PRES\_

[51] Fox J, Bailenson JN, Tricase L. The embodiment of sexualized virtual selves: The Proteus effect and experiences of self-objectification via avatars. Computers in Human Behavior.

[52] Osimo SA, Pizarro R, Spanlang B, Slater M. Conversations between self and self as Sigmund Freud—A virtual body ownership paradigm for self-counselling. Scientific

[53] Falconer CJ, Slater M, Rovira A, King JA, Gilbert P, Antley A, et al. Embodying compassion: A virtual reality paradigm for overcoming excessive self-criticism. Plos One. Nov 12,

[54] Peck TC, Seinfeld S, Aglioti SM, Slater M. Putting yourself in the skin of a black avatar reduces implicit racial bias. Consciousness and Cognition. Sep 1, 2013;22(3):779-787. DOI:

[55] Rosenberg RS, Baughman SL, Bailenson JN. Virtual superheroes: Using superpowers in virtual reality to encourage prosocial behavior. Plos One. Jan 30, 2013;8(1):e55003. DOI:

[56] Riva G, Galimberti C. The psychology of cyberspace: A socio-cognitive framework to computer-mediated communication. New Ideas in Psychology. Aug 1, 1997;15(2):141-

[57] Dunstan DA, Tooth SM. Treatment via videoconferencing: A pilot study of delivery by clinical psychology trainees. Australian Journal of Rural Health. Apr 1, 2012;20(2):88-94.

[58] Rizzo A, Lange B, Buckwalter JG, et al. Simcoach: An intelligent virtual human system for providing healthcare information and support. International Journal on Disability and

[59] Available from: https://www.wired.com/2016/10/oculus-facebook-social-vr/ [Accessed:

[60] Freeman D, Reeve S, Robinson A, Ehlers A, Clark D, Spanlang B, et al. Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychological

Human Development. 2011;10(4):277-281. DOI: 10.1515/IJDHD.2011.046

Medicine. Oct 2017;47(14):2393-2400. DOI: 10.1017/S003329171700040X

May 1, 2013;29(3):930-938. DOI: 10.1016/j.chb.2012.12.027

Reports. Sep 10, 2015;5:13899. DOI: 10.1038/srep13899

2014;9(11):e111933. DOI: 10.1371/journal.pone.0111933

10.1016/j.concog.2013.04.016

10.1371/journal.pone.0055003

158. DOI: 10.1016/S0732-118X(97)00015-9

DOI: 10.1111/j.1440-1584.2012.01260.x

Jul 10, 2016]

change. Front Psychiatry. Sep 30, 2016;7:164. DOI: 10.3389/fpsyt.2016.00164

tion.2007.12.004

118 State of the Art Virtual Reality and Augmented Reality Knowhow

a\_00124

Autism spectrum disorders (ASDs) are characterized by a reduced ability to appropriately express social greetings. Studies have indicated that individuals with ASD might not recognize the crucial nonverbal cues that usually aid social interaction. This study applied augmented reality (AR) with tabletop role-playing game (AR-RPG) to focus on the standard nonverbal social cues to teach children with ASD, how to appropriately reciprocate when they socially interact with others. The results showed that intervention system provides an AR combined with physical manipulatives and presents corresponding specific elements in an AR 3D animation with dialogue; thus, it can be used to help them increase their social interaction skills and drive their attention toward the meaning and social value of greeting behavior in specific social situations. We conclude that AR-RPG of social situations helped children with ASD recognize and better understand these situations and moderately effective in teaching the target greeting responses.

**Keywords:** autism spectrum disorders, augmented reality, educational technology, greeting behavior, physical manipulatives, tabletop role-playing game

#### **1. Introduction**

Autism spectrum disorders (ASD) are characterized by social interaction difficulties, communication challenges, and tendencies to engage in repetitive behaviors [1]. In particular, social reciprocity defects are one of the core deficits in social interaction for people with ASD [2]. Social reciprocity depends on the ability to empathize with others, to be aware of emotional and interpersonal cues, and to respond appropriately [3]. In addition, ASD is characterized

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

by an impaired ability to engage in social relationships and can result in serious deficits in the ability to make friends or interact with others [4]. Typical deficits include an absence of appropriate greeting behaviors and a failure to acknowledge the presence of a familiar person [5]. Such deficits in social greeting appear to be common among individuals with ASD [6]. This impairment has far-reaching consequences for social interaction, communication, and imagination [7].

can effectively enhance the child with ASD to recognize and understand emotions in themselves and to generalize them to other situations. Social Stories™ can increase awareness and understanding of social situations; simultaneously, VM can also provide the video for children with ASD to mimic and pretend character's gesture [28]. In addition, VM has been therapeutically effective for teaching functional, social, and behavioral skills to children with ASD [22, 29, 30]. That is why many experiments adopt this strategy to teach their students with ASD.

How to Create Suitable Augmented Reality Application to Teach Social Skills for Children…

http://dx.doi.org/10.5772/intechopen.76476

121

However, most studies in Social Stories™ combined with VM have only a unidirectional way to teach children with ASD to mimic behaviors that they see, but they cannot directly trigger spontaneous social events because children only watch videos and do not directly interact with others. Moreover, most VM presents all flat displays, and children with ASD can only imitate actions presented on single-perspective 2D views of scenarios to understand everything that is going on in the scene; nor do they understand three-dimensional (3D) facial expressions and body movements or how to reciprocate [31]. In addition, although VM is advantageous for promoting the motivation of children with ASD to learn, children still have difficulty dynamically adjusting the size of their attentional focus and switching the locus of their attention [32–34], especially in patterns that include dynamic, repetitive, or social stimuli [8]; however, that decreased multiple object tracking (MOT) performance is not due to deficits in dynamic attention but to a diminished capacity to select and maintain attention on multiple targets. Therefore, several problems that may occur, such as a lack of progress, could be due to a lack of reinforcement of sustained attention, poor video content, or a lack of prerequisites [35], while children with ASD may engage in pretense when instructed to do so, and they still

Fortunately, there are other assistant tools to help children with ASD promote their social skills and also help focus children's attention on specific target social cues. In some specific cases, AR technology has been proven effective for teaching social interaction skills. For example, AR was effective for helping people select and maintain their focus on the 3D model and helping them better judge the correct mental task [37]. In addition, AR can effectively reduce cognitive load and increase participants' interest in training [38]. Other studies also indicate that physical object manipulatives can support children with ASD to collaborate and communicate in new ways [39]. Additionally, AR with physical manipulatives provides physical behaviors and aids children's active learning [40]. Foundational evidence from LeGoff [41] suggests that interaction with physical manipulatives supported children with ASD to collaborate for extended periods of time by helping channel children's attention and providing a common context for sharing objects and ideas [42]. Physical manipulatives might be particularly well suited to children with ASD because they take advantage of children as active learners, whose experience is grounded in the body and improved through sensory awareness. The kinesthetic learning experience might be ideal for the development of social skills because tangible interface offers expressive activity, programmability, and the construction

**2.2. Primary problem with current training**

find it difficult to develop creative extensions [36].

**2.3. Benefits of AR combined with physical manipulatives**

In contrast, typically developing (TD) children are socially interacting with people effectively, including maintaining eye contact, initiating interactions, responding to the initiations of others, sharing enjoyment, reading body language or nonverbal cues of others, and adopting the perspective of another person. However, most children with ASD seem to find it difficult learning how to engage in the give-and-take process of everyday human social interaction. Even in the first few months of life, many children with ASD do not interact, and they avoid eye contact [8]. They seem indifferent to other people and often seem to prefer being alone [9].

### **2. Related works**

#### **2.1. Current training methods for social interaction skills**

#### *2.1.1. Social stories™*

Social Stories™ (http://carolgraysocialstories.com/social-stories/) is a promising strategy that has been used to teach social interaction skills for individuals with ASD [10, 11]. A Social Stories™ intervention involves creating brief stories that describe social situations, what others are thinking or feeling, and how to behave in the specific situation. As part of describing how to behave, a good social story would also highlight what social cues the person should look for and how to respond to others [12]. Specifically, teaching through the use of a Social Stories™ strategy with a storybook is widely used [13–15]. Social Stories™ provides a catalyst for change, providing children with other perspectives and options for thoughts, feelings, and behaviors [16]. Previous studies examining the effectiveness of Social Stories™ interventions have yielded varying results, but overall Social Stories™ appears to have a positive effect [11, 17]. Social Stories™ interventions have targeted a range of skills: initiating verbal greetings [10], initiating and responding to interactions [18], and maintaining appropriate social engagement [19, 20].

#### *2.1.2. Video modeling*

In addition, another strategy that has been used to teach social skills to people with ASD is video modeling (VM) [21]. VM entails showing the participant a video segment that demonstrates how to perform a task or behavior. The participant is expected to learn by observing the instructional video segment and repeating the modeled behavior [22, 23]. VM has been effective for teaching a variety of social skills: social initiation [24], social language in play situations [25], social engagement [22], and expressive behaviors, such as intonation and facial expressions [26]. Therefore, some studies combine Social Stories™ and VM [27]. This strategy can effectively enhance the child with ASD to recognize and understand emotions in themselves and to generalize them to other situations. Social Stories™ can increase awareness and understanding of social situations; simultaneously, VM can also provide the video for children with ASD to mimic and pretend character's gesture [28]. In addition, VM has been therapeutically effective for teaching functional, social, and behavioral skills to children with ASD [22, 29, 30]. That is why many experiments adopt this strategy to teach their students with ASD.

#### **2.2. Primary problem with current training**

by an impaired ability to engage in social relationships and can result in serious deficits in the ability to make friends or interact with others [4]. Typical deficits include an absence of appropriate greeting behaviors and a failure to acknowledge the presence of a familiar person [5]. Such deficits in social greeting appear to be common among individuals with ASD [6]. This impairment has far-reaching consequences for social interaction, communication, and

In contrast, typically developing (TD) children are socially interacting with people effectively, including maintaining eye contact, initiating interactions, responding to the initiations of others, sharing enjoyment, reading body language or nonverbal cues of others, and adopting the perspective of another person. However, most children with ASD seem to find it difficult learning how to engage in the give-and-take process of everyday human social interaction. Even in the first few months of life, many children with ASD do not interact, and they avoid eye contact [8]. They seem indifferent to other people and often seem to prefer being alone [9].

Social Stories™ (http://carolgraysocialstories.com/social-stories/) is a promising strategy that has been used to teach social interaction skills for individuals with ASD [10, 11]. A Social Stories™ intervention involves creating brief stories that describe social situations, what others are thinking or feeling, and how to behave in the specific situation. As part of describing how to behave, a good social story would also highlight what social cues the person should look for and how to respond to others [12]. Specifically, teaching through the use of a Social Stories™ strategy with a storybook is widely used [13–15]. Social Stories™ provides a catalyst for change, providing children with other perspectives and options for thoughts, feelings, and behaviors [16]. Previous studies examining the effectiveness of Social Stories™ interventions have yielded varying results, but overall Social Stories™ appears to have a positive effect [11, 17]. Social Stories™ interventions have targeted a range of skills: initiating verbal greetings [10], initiating and responding to interactions [18], and maintaining appropriate social

In addition, another strategy that has been used to teach social skills to people with ASD is video modeling (VM) [21]. VM entails showing the participant a video segment that demonstrates how to perform a task or behavior. The participant is expected to learn by observing the instructional video segment and repeating the modeled behavior [22, 23]. VM has been effective for teaching a variety of social skills: social initiation [24], social language in play situations [25], social engagement [22], and expressive behaviors, such as intonation and facial expressions [26]. Therefore, some studies combine Social Stories™ and VM [27]. This strategy

imagination [7].

**2. Related works**

*2.1.1. Social stories™*

engagement [19, 20].

*2.1.2. Video modeling*

**2.1. Current training methods for social interaction skills**

120 State of the Art Virtual Reality and Augmented Reality Knowhow

However, most studies in Social Stories™ combined with VM have only a unidirectional way to teach children with ASD to mimic behaviors that they see, but they cannot directly trigger spontaneous social events because children only watch videos and do not directly interact with others. Moreover, most VM presents all flat displays, and children with ASD can only imitate actions presented on single-perspective 2D views of scenarios to understand everything that is going on in the scene; nor do they understand three-dimensional (3D) facial expressions and body movements or how to reciprocate [31]. In addition, although VM is advantageous for promoting the motivation of children with ASD to learn, children still have difficulty dynamically adjusting the size of their attentional focus and switching the locus of their attention [32–34], especially in patterns that include dynamic, repetitive, or social stimuli [8]; however, that decreased multiple object tracking (MOT) performance is not due to deficits in dynamic attention but to a diminished capacity to select and maintain attention on multiple targets. Therefore, several problems that may occur, such as a lack of progress, could be due to a lack of reinforcement of sustained attention, poor video content, or a lack of prerequisites [35], while children with ASD may engage in pretense when instructed to do so, and they still find it difficult to develop creative extensions [36].

#### **2.3. Benefits of AR combined with physical manipulatives**

Fortunately, there are other assistant tools to help children with ASD promote their social skills and also help focus children's attention on specific target social cues. In some specific cases, AR technology has been proven effective for teaching social interaction skills. For example, AR was effective for helping people select and maintain their focus on the 3D model and helping them better judge the correct mental task [37]. In addition, AR can effectively reduce cognitive load and increase participants' interest in training [38]. Other studies also indicate that physical object manipulatives can support children with ASD to collaborate and communicate in new ways [39]. Additionally, AR with physical manipulatives provides physical behaviors and aids children's active learning [40]. Foundational evidence from LeGoff [41] suggests that interaction with physical manipulatives supported children with ASD to collaborate for extended periods of time by helping channel children's attention and providing a common context for sharing objects and ideas [42]. Physical manipulatives might be particularly well suited to children with ASD because they take advantage of children as active learners, whose experience is grounded in the body and improved through sensory awareness. The kinesthetic learning experience might be ideal for the development of social skills because tangible interface offers expressive activity, programmability, and the construction of moving objects with structural integrity [43]. Hence, most AR combined with physical manipulatives can bring beneficial opportunities that children can simply imitate and as they pretend to do the modeled behaviors, without actually facing the conflict activity in a real situation. For example, Chen et al. [31] used AR technology with tangible facial masks as physical manipulatives to enable three adolescents with ASD to become aware of facial expressions observed in situations in a simulated school setting. They provided 3D animations of six basic facial expressions overlaid on participant faces to facilitate practicing and to pretend emotional status and social skills. They indicated that AR technology can allow children with ASD to pretend their own facial expression correspond to occurring events and can also let the children with ASD pay attention focused on this specific facial part of social cues. Moreover, a physical manipulative interface also enables people with ASD to control the system in an intuitive way and provide immediate haptic feedback during social interaction, and this helps develop social and language skills [44]. In addition, AR environments can emphasize learners' participation in different roles and can improve the sense of presence, immediacy, and immersion [45].

physical space by overlaying real space with flexible virtual objects. AR-RPG system is like a "miniature theater" in which children with ASD pretend, using role-play, to participate in specific social events. This role-playing game platform uses a small 3D space where virtual and real spaces are tightly linked together and can accommodate single or multiple users (see **Figure 1**). We used several settings, a living room, a classroom, and a community (see **Figure 2**), all of which are familiar places to our participants. We also included virtual furniture or some

How to Create Suitable Augmented Reality Application to Teach Social Skills for Children…

The AR-RPG takes a series of scenario stories created as training materials that portray the participants' everyday life activities to focus on training their greeting behavior as it might happen in school, home, and community. For example, we got some information from their

**Table 1.** Comparison of conventional video modeling (VM) and augmented reality (AR) tabletop role-playing game

**Figure 1.** The therapist taught the children how to use the AR-RPG system and how to manipulate the cardboard avatar

1. Realistic full 3D virtual objects with tangible

2. Content is not fixed but can change in real time 3. Multiple-user based; makes learning interesting 4. Rich entity operability can increase users'

interact with setting

them to pretend play.

learning incentive and motivation

help users recognize the setting

manipulation controller; user can directly manipulate and

http://dx.doi.org/10.5772/intechopen.76476

123

understanding of the relationship between roles and help

5. AR with Social Stories™ can establish their social network concept map and also enhance children's

6. Multiple viewing angles are allowed; users can see gestures and facial expressions from different angles 7. Gives extra real spatial information as a reference to

other objects to make the space more realistic.

1. 2D flat image and video; user can watch content only on

2. Content is fixed and cannot change in real time 3. Primarily single-user based to watch; no interaction 4. Lack of real manipulation can help facilitate mental

5. Most video has single development context; lacks concept of visualization network; and cannot enhance

6. Provides only single view angle from the video; multiple

7. User cannot obtain additional spatial information from their surrounding environment to help users recognize the

to observe and pretend (role play) the avatar's situation.

learning experience incentive or motivation

monitor; few direct interactions

skills such as pretend play skills

viewing angles are not allowed

setting

(AR-RPG) system.

**Conventional VM AR-RPG system**

#### **2.4. The gaps in these studies**

In this study, we used AR technology combined with the tabletop role-playing game (AR-RPG) as training platform that focused on social greetings. This system encourages imaginative social activities and allows children with ASD to interact directly with physical role props, as they do in natural circumstance. An AR system can guide children with ASD to express what they understand and feel in an easy and interesting way. We summarized some differences between our research and that of previous studies. First, we focused on social reciprocity behavior training (i.e., greeting behavior), not on work skills or basic daily life skills (e.g., washing dishes, eating food, getting dressed) [46]. Second, our training materials were created using the Social Stories™ strategy and related to each child's daily life situations at home, in school, and in the community, whereas previous studies created their materials using textbooks or treatment manuals without Social Stories™ strategy. Third, our research focused on children with ASD, not on TD children; most previous studies [47] used AR for children without ASD. Fourth, our AR system was based on markerless 3D tracking technology that uses Qualcomm's Vuforia to increase its realism. This allowed children with ASD to directly sense which marker is their role-play photo image, which reduces the confusion that many different role-play markers produce. Fifth, our AR system had more advantages than traditional VM and conventional AR systems do (see **Table 1**). Sixth, our study focused on greeting gestures and related social interaction, not only on facial expression pretend training [31].

#### **2.5. Developing the AR-RPG system**

AR is a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, or 2D and 3D graphics; thus, AR enhances one's current perception of reality [48]. It is for developing instructional content that needs to present augmented spatial information to reference and other parts of its features. AR creates mediated space that exists between the mind and physical space by overlaying real space with flexible virtual objects. AR-RPG system is like a "miniature theater" in which children with ASD pretend, using role-play, to participate in specific social events. This role-playing game platform uses a small 3D space where virtual and real spaces are tightly linked together and can accommodate single or multiple users (see **Figure 1**). We used several settings, a living room, a classroom, and a community (see **Figure 2**), all of which are familiar places to our participants. We also included virtual furniture or some other objects to make the space more realistic.

of moving objects with structural integrity [43]. Hence, most AR combined with physical manipulatives can bring beneficial opportunities that children can simply imitate and as they pretend to do the modeled behaviors, without actually facing the conflict activity in a real situation. For example, Chen et al. [31] used AR technology with tangible facial masks as physical manipulatives to enable three adolescents with ASD to become aware of facial expressions observed in situations in a simulated school setting. They provided 3D animations of six basic facial expressions overlaid on participant faces to facilitate practicing and to pretend emotional status and social skills. They indicated that AR technology can allow children with ASD to pretend their own facial expression correspond to occurring events and can also let the children with ASD pay attention focused on this specific facial part of social cues. Moreover, a physical manipulative interface also enables people with ASD to control the system in an intuitive way and provide immediate haptic feedback during social interaction, and this helps develop social and language skills [44]. In addition, AR environments can emphasize learners' participation in different roles and can improve the sense of presence,

In this study, we used AR technology combined with the tabletop role-playing game (AR-RPG) as training platform that focused on social greetings. This system encourages imaginative social activities and allows children with ASD to interact directly with physical role props, as they do in natural circumstance. An AR system can guide children with ASD to express what they understand and feel in an easy and interesting way. We summarized some differences between our research and that of previous studies. First, we focused on social reciprocity behavior training (i.e., greeting behavior), not on work skills or basic daily life skills (e.g., washing dishes, eating food, getting dressed) [46]. Second, our training materials were created using the Social Stories™ strategy and related to each child's daily life situations at home, in school, and in the community, whereas previous studies created their materials using textbooks or treatment manuals without Social Stories™ strategy. Third, our research focused on children with ASD, not on TD children; most previous studies [47] used AR for children without ASD. Fourth, our AR system was based on markerless 3D tracking technology that uses Qualcomm's Vuforia to increase its realism. This allowed children with ASD to directly sense which marker is their role-play photo image, which reduces the confusion that many different role-play markers produce. Fifth, our AR system had more advantages than traditional VM and conventional AR systems do (see **Table 1**). Sixth, our study focused on greeting gestures

and related social interaction, not only on facial expression pretend training [31].

AR is a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, or 2D and 3D graphics; thus, AR enhances one's current perception of reality [48]. It is for developing instructional content that needs to present augmented spatial information to reference and other parts of its features. AR creates mediated space that exists between the mind and

immediacy, and immersion [45].

122 State of the Art Virtual Reality and Augmented Reality Knowhow

**2.4. The gaps in these studies**

**2.5. Developing the AR-RPG system**

The AR-RPG takes a series of scenario stories created as training materials that portray the participants' everyday life activities to focus on training their greeting behavior as it might happen in school, home, and community. For example, we got some information from their


**Table 1.** Comparison of conventional video modeling (VM) and augmented reality (AR) tabletop role-playing game (AR-RPG) system.

**Figure 1.** The therapist taught the children how to use the AR-RPG system and how to manipulate the cardboard avatar to observe and pretend (role play) the avatar's situation.

**Figure 2.** Different scenes with corresponding scenario events (classroom and playground in the community).

daily experience: (e.g., greeting parents, greeting friends, and greeting strangers). Those scenes are useful for children with ASD to be trained while pretending to make appropriate responses to greetings. In addition, we focus on teaching the six types of greeting behaviors of social etiquette, including (1) nodded head and tiny smile, (2) handshakes, (3) waved hand and said hello, (4) kisses their head or face, (5) hugs, and (6) shrug your shoulders; this greeting is often seen in Taiwan's daily life and also useful to train children with ASD social interaction skill. Beyond the greeting, which may involve a verbal acknowledgment and sometimes a facial expression, gestures, body language, and eye contact can all signal what type of greeting is expected. Therefore, we also add the social story and dialog contents with text description within our greeting scenario, to let the specific greeting behavior situation present more closely to the real status.

you?", wave at the neighbor, smile, etc. By providing a wide range of scenarios that reflect their everyday life and analyzing stories suitable for them, therapists might better understand what actually happens in the lives of their child clients and how the children feel about it.

**Figure 3.** Operational scenario and facilities (different cards represent different pattern IDs, e.g., the role itself and others

How to Create Suitable Augmented Reality Application to Teach Social Skills for Children…

http://dx.doi.org/10.5772/intechopen.76476

125

people's cards, decision about which greeting behavior card, and scenario card to choose).

We recruited three participants. The three children with ASD (two boys and one girl (Oliver, Peter, and Jane), all pseudonyms to guarantee anonymity) (mean age = 7.73 years old; age range 7–9 years; intelligence quotient (IQ) scores: (a) full scale IQ (FIQ) = 100.67±3.79, (b) verbal IQ (VIQ) = 100.33±4.04, and (c) performance IQ (PIQ) = 101.33±2.08) were enrolled in the mainstream school with other TD students (see **Table 2**). The inclusion criteria for this study were (1) a clinical diagnosis of ASD based on DSM-IV-TR criteria, (2) no other specific disabilities, (3) not taking medications, (4) no physician-diagnosed comorbidities, (5) not undergoing any other interventions at the time of the testing, and (6) no intellectual disability (a full scale IQ > 70). Oliver, Peter, and Jane had additional assistance from a teacher's aide in general lessons. Once a week, an occupational therapist provides educational services to their class. The teachers reported that the three students with ASD did not greet or acknowledge the teaching staff or other adults at the school. The parents also said that their children did not greet or acknowledge others across different contexts. Their teacher and therapists believed that it

**3. Method**

**3.1. Participants**

#### **2.6. Operational settings and facilities**

The researcher created 20 scenario stories after discussing suitable scenes and scripts with each participant's parents and special education teacher. We designed those stories based on the child who would do the role-play, the greeting response to be learned, the conversation content, the role's intention, and its emotional status. We let them read the script or watch the scene, and we ask participants to mimic and role-play with therapist, who would help them comprehend. We used AR to construct our tabletop role-play game platform because it allowed us to quickly and easily change scenes. The AR-RPG system has two functions: (a) it is a tangible scene simulator with a physical manipulative function, and (b) it is a vehicle for providing virtual content. In addition, we created each child's pattern printed on tangible cardboard as avatars to represent each one's identification (ID) (see **Figure 3**). Children with ASD can directly handle their own cardboard avatar when they interact with others (e.g., peers, parents, or other characters) or when different characters contact each other at the different situations (e.g., home, school, or park). Our system will trigger the event's related AR 3D greeting animation based on the background scene (see **Figure 2**). For example, when a role-playing child sees a neighbor appeared on the street corner, he will say "Hi! How are How to Create Suitable Augmented Reality Application to Teach Social Skills for Children… http://dx.doi.org/10.5772/intechopen.76476 125

**Figure 3.** Operational scenario and facilities (different cards represent different pattern IDs, e.g., the role itself and others people's cards, decision about which greeting behavior card, and scenario card to choose).

you?", wave at the neighbor, smile, etc. By providing a wide range of scenarios that reflect their everyday life and analyzing stories suitable for them, therapists might better understand what actually happens in the lives of their child clients and how the children feel about it.

#### **3. Method**

daily experience: (e.g., greeting parents, greeting friends, and greeting strangers). Those scenes are useful for children with ASD to be trained while pretending to make appropriate responses to greetings. In addition, we focus on teaching the six types of greeting behaviors of social etiquette, including (1) nodded head and tiny smile, (2) handshakes, (3) waved hand and said hello, (4) kisses their head or face, (5) hugs, and (6) shrug your shoulders; this greeting is often seen in Taiwan's daily life and also useful to train children with ASD social interaction skill. Beyond the greeting, which may involve a verbal acknowledgment and sometimes a facial expression, gestures, body language, and eye contact can all signal what type of greeting is expected. Therefore, we also add the social story and dialog contents with text description within our greeting scenario, to let the specific greeting behavior situa-

**Figure 2.** Different scenes with corresponding scenario events (classroom and playground in the community).

The researcher created 20 scenario stories after discussing suitable scenes and scripts with each participant's parents and special education teacher. We designed those stories based on the child who would do the role-play, the greeting response to be learned, the conversation content, the role's intention, and its emotional status. We let them read the script or watch the scene, and we ask participants to mimic and role-play with therapist, who would help them comprehend. We used AR to construct our tabletop role-play game platform because it allowed us to quickly and easily change scenes. The AR-RPG system has two functions: (a) it is a tangible scene simulator with a physical manipulative function, and (b) it is a vehicle for providing virtual content. In addition, we created each child's pattern printed on tangible cardboard as avatars to represent each one's identification (ID) (see **Figure 3**). Children with ASD can directly handle their own cardboard avatar when they interact with others (e.g., peers, parents, or other characters) or when different characters contact each other at the different situations (e.g., home, school, or park). Our system will trigger the event's related AR 3D greeting animation based on the background scene (see **Figure 2**). For example, when a role-playing child sees a neighbor appeared on the street corner, he will say "Hi! How are

tion present more closely to the real status.

124 State of the Art Virtual Reality and Augmented Reality Knowhow

**2.6. Operational settings and facilities**

#### **3.1. Participants**

We recruited three participants. The three children with ASD (two boys and one girl (Oliver, Peter, and Jane), all pseudonyms to guarantee anonymity) (mean age = 7.73 years old; age range 7–9 years; intelligence quotient (IQ) scores: (a) full scale IQ (FIQ) = 100.67±3.79, (b) verbal IQ (VIQ) = 100.33±4.04, and (c) performance IQ (PIQ) = 101.33±2.08) were enrolled in the mainstream school with other TD students (see **Table 2**). The inclusion criteria for this study were (1) a clinical diagnosis of ASD based on DSM-IV-TR criteria, (2) no other specific disabilities, (3) not taking medications, (4) no physician-diagnosed comorbidities, (5) not undergoing any other interventions at the time of the testing, and (6) no intellectual disability (a full scale IQ > 70). Oliver, Peter, and Jane had additional assistance from a teacher's aide in general lessons. Once a week, an occupational therapist provides educational services to their class. The teachers reported that the three students with ASD did not greet or acknowledge the teaching staff or other adults at the school. The parents also said that their children did not greet or acknowledge others across different contexts. Their teacher and therapists believed that it


people behave, but it also highlights what social cues the person should look for and how to respond to others [12]. Those SSTs were created by the school's ASD therapist and reviewed by two other experts with experience in implementing Social Stories™ interventions. Social Stories™ provides a catalyst for change and provides children with other perspectives and options for thoughts, feelings, and behaviors ([16]). Therefore, we had all our participants take the standard SSTs in the different phases according to multiple baselines across participants' design. We arranged 20 SSTs to occur at random in our SSTs to reduce boredom. Each question was different, but they were all at the same level of difficulty to reduce the

How to Create Suitable Augmented Reality Application to Teach Social Skills for Children…

http://dx.doi.org/10.5772/intechopen.76476

127

The SSTs used for the intervention phase were different from those used for the baseline and maintenance phases; they were counterbalanced to reduce bias. One test question was asked per short scenario script, and there was no prompting for answers in any session. We provide the six basic fundamental greeting behavior answer options that most happen in Taiwan's daily life for them to choose, including the (1) nodded head and tiny smile, (2) handshakes, (3) waved hand and said hello, (4) kisses their head or face, (5) hugs, and (6) shrug your shoulders; those behaviors presented in AR animation also add with the dialog and extra detail information (e.g., facial expressions, communication, and dialog's tone tone); yet, in the channel choice, we use the physical paper card (it also used as the AR marker for camera to detect their choice) with text description for them to choose. In each session of the experimental, 20 SSTs were given in three phases. After the test, we recorded the answer to determine the correct response rate. For example, if a participant got 10 correct answers in 20 SSTs, their correct response rate was 50%. The answers were checked by therapists and researchers who tested

All of the participants role-played the greeting behavior after each SST question in each session with the therapist and their special education teacher. The therapist evaluated their greeting behavior feedback to evaluate their learning performance. The therapist evaluated performance on a 5-point Likert scale (5—Strongly Agree, 4—Agree, 3—Undecided, 2— Disagree, 1—Strongly Disagree) to evaluate the children's role-playing performance. Performance included (1) gestures, (2) emotional responses, (3) interactive methods, (4) intention emotion, and (5) conversation. These separate channels were evaluated by the therapist and two social behavior experts' score for expert assessment to ensure that each evaluation of

In this study, we focus on teaching children with ASD how to appropriately reciprocate when they socially interact with others. Therefore, we arrange the different scenarios for them to

test–retest effect.

*3.2.2. Test evaluation*

for normative answers.

*3.2.3. Role-play evaluation*

role-play is accurate.

*3.3.1. Procedure*

**3.3. Intervention program**

**Table 2.** Summarized demographic information of the participants.

would be appropriate and desirable for them to learn how to greet familiar people at school in order to promote greater socialization and improve their communication skills. The participants' sensory abilities were within the normal range; however, their parents also reported that they usually engaged in unusual responses to greetings: hand-wringing, slapping their own face, hand tapping, or rubbing, especially when their parents force them to greet others. For example, they do not understand why some of their relatives greet them by hugging and kissing them. Those behaviors make them feel awkward and confused.

#### **3.2. Instruments**

Data on the participants' intelligence, sensory abilities, and social and communication skills are based on multiple information sources: parental interviews, teachers' reports, verbal IQ scores (Wechsler Intelligence Scale for Children), and levels of functional language and social adaptation levels (based on clinical observations or behavior and adaptation scales). All participants had a disability identification card issued by a medical institution in Taiwan and had been counseled in special education schools and institutes in Taiwan. The relevant university ethics committee approved the study, and parental consent was obtained for the students' participation. We used Social Stories™ ([49, 50]) to create the Social Story trial (SSTs) tests to evaluate the AR-RPG training effect: pre- and post-intervention scores were compared using Kolmogorov–Smirnov test (KS test) to determine whether a particular intervention method improved their social greeting skills. Each scenario was associated with a different event, but each was consistent in length and at a similar level of difficulty. The entire scenario stories pretend that the process that was used followed the same rules for content creation and discussion with a special education expert and their teacher. The SST used for the intervention phase was different from those used for the baseline and maintenance phases; they were counterbalanced to reduce bias.

#### *3.2.1. SST tests*

Social Stories™ provides a standard strategy that has been used to teach social interaction skills to people with ASD [11, 51]. A Social Stories™ intervention involves creating brief stories that describe social situations and what others are thinking or feeling and how to behave in a specific situation. We created the SSTs using the rules dictated by Baker [49, 50]) for different phases (baseline and maintenance phases). A good scenario not only describes how people behave, but it also highlights what social cues the person should look for and how to respond to others [12]. Those SSTs were created by the school's ASD therapist and reviewed by two other experts with experience in implementing Social Stories™ interventions. Social Stories™ provides a catalyst for change and provides children with other perspectives and options for thoughts, feelings, and behaviors ([16]). Therefore, we had all our participants take the standard SSTs in the different phases according to multiple baselines across participants' design. We arranged 20 SSTs to occur at random in our SSTs to reduce boredom. Each question was different, but they were all at the same level of difficulty to reduce the test–retest effect.

#### *3.2.2. Test evaluation*

would be appropriate and desirable for them to learn how to greet familiar people at school in order to promote greater socialization and improve their communication skills. The participants' sensory abilities were within the normal range; however, their parents also reported that they usually engaged in unusual responses to greetings: hand-wringing, slapping their own face, hand tapping, or rubbing, especially when their parents force them to greet others. For example, they do not understand why some of their relatives greet them by hugging and

**Participants Age FIQ VIQ PIQ Diagnosis** Oliver 7.2 98 104 99 ASD Peter 7.6 105 96 102 ASD Jane 8.4 99 101 103 ASD

Data on the participants' intelligence, sensory abilities, and social and communication skills are based on multiple information sources: parental interviews, teachers' reports, verbal IQ scores (Wechsler Intelligence Scale for Children), and levels of functional language and social adaptation levels (based on clinical observations or behavior and adaptation scales). All participants had a disability identification card issued by a medical institution in Taiwan and had been counseled in special education schools and institutes in Taiwan. The relevant university ethics committee approved the study, and parental consent was obtained for the students' participation. We used Social Stories™ ([49, 50]) to create the Social Story trial (SSTs) tests to evaluate the AR-RPG training effect: pre- and post-intervention scores were compared using Kolmogorov–Smirnov test (KS test) to determine whether a particular intervention method improved their social greeting skills. Each scenario was associated with a different event, but each was consistent in length and at a similar level of difficulty. The entire scenario stories pretend that the process that was used followed the same rules for content creation and discussion with a special education expert and their teacher. The SST used for the intervention phase was different from those used for the baseline and maintenance phases; they were

Social Stories™ provides a standard strategy that has been used to teach social interaction skills to people with ASD [11, 51]. A Social Stories™ intervention involves creating brief stories that describe social situations and what others are thinking or feeling and how to behave in a specific situation. We created the SSTs using the rules dictated by Baker [49, 50]) for different phases (baseline and maintenance phases). A good scenario not only describes how

kissing them. Those behaviors make them feel awkward and confused.

Mean 7.73 100.67 100.33 101.33

**Information**

126 State of the Art Virtual Reality and Augmented Reality Knowhow

**Table 2.** Summarized demographic information of the participants.

**3.2. Instruments**

counterbalanced to reduce bias.

*3.2.1. SST tests*

The SSTs used for the intervention phase were different from those used for the baseline and maintenance phases; they were counterbalanced to reduce bias. One test question was asked per short scenario script, and there was no prompting for answers in any session. We provide the six basic fundamental greeting behavior answer options that most happen in Taiwan's daily life for them to choose, including the (1) nodded head and tiny smile, (2) handshakes, (3) waved hand and said hello, (4) kisses their head or face, (5) hugs, and (6) shrug your shoulders; those behaviors presented in AR animation also add with the dialog and extra detail information (e.g., facial expressions, communication, and dialog's tone tone); yet, in the channel choice, we use the physical paper card (it also used as the AR marker for camera to detect their choice) with text description for them to choose. In each session of the experimental, 20 SSTs were given in three phases. After the test, we recorded the answer to determine the correct response rate. For example, if a participant got 10 correct answers in 20 SSTs, their correct response rate was 50%. The answers were checked by therapists and researchers who tested for normative answers.

#### *3.2.3. Role-play evaluation*

All of the participants role-played the greeting behavior after each SST question in each session with the therapist and their special education teacher. The therapist evaluated their greeting behavior feedback to evaluate their learning performance. The therapist evaluated performance on a 5-point Likert scale (5—Strongly Agree, 4—Agree, 3—Undecided, 2— Disagree, 1—Strongly Disagree) to evaluate the children's role-playing performance. Performance included (1) gestures, (2) emotional responses, (3) interactive methods, (4) intention emotion, and (5) conversation. These separate channels were evaluated by the therapist and two social behavior experts' score for expert assessment to ensure that each evaluation of role-play is accurate.

#### **3.3. Intervention program**

#### *3.3.1. Procedure*

In this study, we focus on teaching children with ASD how to appropriately reciprocate when they socially interact with others. Therefore, we arrange the different scenarios for them to judge and ask them to consider appropriate reciprocate in each scenario, and they need to imagine role's situation and did the appropriate reciprocate judgment (choice one of the six greeting behaviors with facial expression and dialog). And, after the judgment, they watch the 3D animation, and then they need to role-play and mimic the character's status; our therapist will evaluate their learning performance.

*3.3.2.2. Intervention phase*

individual child.

acquired.

*3.3.2.3. Maintenance phase*

**3.4. Experimental setting**

this experiment.

In the intervention phase, the children were required to use the AR-RPG system to activate their understanding of the social reciprocity contexts and judge the questions about greeting behavior. (a) In the first session of the intervention phase, the children were instructed by the therapist on how to operate the AR-RPG system and how to perceive cues, to ensure that they felt comfortable using the AR technology. The instruction time was 40–45 min. (b) The children began the experimental sessions by following the therapist's instructions to use the AR-RPG system. (c) In each scenario, the therapist shows the SST questions to ask the children with ASD. (d) After the SST questions, the children had to choose the appropriate greeting behavior card. (e) When the choice was correct, the AR-RPG system showed the avatar's 3D greeting behavior animation and triggered the "Correct!" sound signal. When the choice was incorrect, the system showed the error signal and triggered the "Incorrect!" sound. (f) Then the therapist asked them to role-play the scenario with appropriate gestures, conversation, intention, and facial expressions of emotion. (g) After the children had completed this part of the training, the therapist taught them the correct response and reconstructed their greeting behavior map. In the intervention process, each child was given the same training by the same therapist, but each was separately trained for different durations, as required by the

How to Create Suitable Augmented Reality Application to Teach Social Skills for Children…

http://dx.doi.org/10.5772/intechopen.76476

129

Between the intervention and maintenance phases, there was a 6-week hiatus to reduce recall interference in order to determine, using the baseline phase procedure, but not the AR-RPG system intervention materials, whether the children had maintained the skills that they had

All sessions were conducted in a quiet 3.5 m × 6 m room of the day treatment center at the school. The room contained a table and chairs, an Intel Core i7 laptop computer, a web camera, several tangible role-play markers with avatars, and a 37-inch LCD display set up in front of the children. The therapist sat on the right of the participant and guided the training process; the researcher operated the computer, set up the environment, and helped the children use the AR-RPG system (see **Figure 2**). To begin the intervention test, the therapist showed the test sample to the children, taught them how to use the AR-RPG system, and ensured that they felt comfortable using the AR technology. The children started controlling each step of the test after the first trial task was presented. During the test, they used their hands to control the cardboard avatars to make interact with other avatars that appeared in the scene. After they had chosen the appropriate greeting, the next task trial was presented. The system ran the AR application in the background and showed the image on the LCD screen. The children were able to practice pretend play and mimic the greetings with their therapist, which allowed them to feel comfortable doing

#### *3.3.2. Experimental conditions*

One certified occupational therapist with more than 7 years of experience working with children with ASD conducted all the sessions and taught all the children how to use the AR-RPG system. The experiment consisted of three phases: (a) the baseline phase, in which baseline information on the children was collected; (b) the intervention phase, in which the AR-RPG system was used to obtain the performance data for assessment; and (c) the maintenance phase, done 6 weeks after the intervention had been completed, in which the performance of the children was assessed. During the intervention phase, the AR-RPG system was used twice a week for 1–1.5 months to train all of the children in social greeting skills. The three children in our study group had a congenital condition that manifested differently in each; therefore, we used a single subject in a multiple baseline across subject design [52] to confirm whether the intervention was effective in single participants, despite their being ostensibly members of a group of similar participants. This is regarded as a standard and evidence-based method in many computer-based treatments used in special education. It is a fundamental experimental method for research in the field, and, in actual practice, it does not require control groups or many participants. The multiple baseline design is a style of research that involves carefully measuring multiple persons, traits, or settings, both before and after a treatment. This design is used in medical, psychological, and biological research and is especially relevant for ASD studies [53–55]. Because the manifestations of ASD are different in each individual, the purposes of the research were to see whether the intervention was effective by documenting whether and how each individual had improved.

#### *3.3.2.1. Baseline phase*

In the baseline phase, the therapist (a) first explained to the children the meanings of the greeting behavior that they would be asked about. (b) The ASD therapist and the children's special education teacher created the series of greeting behavior scenarios into the SSTs, which were reviewed by two other experts with experience in implementing Social Stories™ interventions. (c) The scenarios were presented on the treatment room's computer. (d) The therapist then asked the children some questions about the scenarios. (e) After the children had completed the scenarios, the therapist showed them a picture of six real greeting situations and asked them to determine the proper response to each. (f) The children chose one of the six greeting behavior pictures from the target pictures that they thought best reflected the correct response in this scenario. (g) Then the therapist asked them to role-play the scenario with appropriate gestures, conversation, intention, and facial expressions of emotion. Correct answers prompted the therapist to guide them to the next scenario. Correct and incorrect answers were identified and recorded, and the rate of correct answers was determined.

#### *3.3.2.2. Intervention phase*

judge and ask them to consider appropriate reciprocate in each scenario, and they need to imagine role's situation and did the appropriate reciprocate judgment (choice one of the six greeting behaviors with facial expression and dialog). And, after the judgment, they watch the 3D animation, and then they need to role-play and mimic the character's status; our therapist

One certified occupational therapist with more than 7 years of experience working with children with ASD conducted all the sessions and taught all the children how to use the AR-RPG system. The experiment consisted of three phases: (a) the baseline phase, in which baseline information on the children was collected; (b) the intervention phase, in which the AR-RPG system was used to obtain the performance data for assessment; and (c) the maintenance phase, done 6 weeks after the intervention had been completed, in which the performance of the children was assessed. During the intervention phase, the AR-RPG system was used twice a week for 1–1.5 months to train all of the children in social greeting skills. The three children in our study group had a congenital condition that manifested differently in each; therefore, we used a single subject in a multiple baseline across subject design [52] to confirm whether the intervention was effective in single participants, despite their being ostensibly members of a group of similar participants. This is regarded as a standard and evidence-based method in many computer-based treatments used in special education. It is a fundamental experimental method for research in the field, and, in actual practice, it does not require control groups or many participants. The multiple baseline design is a style of research that involves carefully measuring multiple persons, traits, or settings, both before and after a treatment. This design is used in medical, psychological, and biological research and is especially relevant for ASD studies [53–55]. Because the manifestations of ASD are different in each individual, the purposes of the research were to see whether the intervention was effective by documenting

In the baseline phase, the therapist (a) first explained to the children the meanings of the greeting behavior that they would be asked about. (b) The ASD therapist and the children's special education teacher created the series of greeting behavior scenarios into the SSTs, which were reviewed by two other experts with experience in implementing Social Stories™ interventions. (c) The scenarios were presented on the treatment room's computer. (d) The therapist then asked the children some questions about the scenarios. (e) After the children had completed the scenarios, the therapist showed them a picture of six real greeting situations and asked them to determine the proper response to each. (f) The children chose one of the six greeting behavior pictures from the target pictures that they thought best reflected the correct response in this scenario. (g) Then the therapist asked them to role-play the scenario with appropriate gestures, conversation, intention, and facial expressions of emotion. Correct answers prompted the therapist to guide them to the next scenario. Correct and incorrect answers were identified and recorded, and the rate of correct answers was determined.

will evaluate their learning performance.

128 State of the Art Virtual Reality and Augmented Reality Knowhow

whether and how each individual had improved.

*3.3.2.1. Baseline phase*

*3.3.2. Experimental conditions*

In the intervention phase, the children were required to use the AR-RPG system to activate their understanding of the social reciprocity contexts and judge the questions about greeting behavior. (a) In the first session of the intervention phase, the children were instructed by the therapist on how to operate the AR-RPG system and how to perceive cues, to ensure that they felt comfortable using the AR technology. The instruction time was 40–45 min. (b) The children began the experimental sessions by following the therapist's instructions to use the AR-RPG system. (c) In each scenario, the therapist shows the SST questions to ask the children with ASD. (d) After the SST questions, the children had to choose the appropriate greeting behavior card. (e) When the choice was correct, the AR-RPG system showed the avatar's 3D greeting behavior animation and triggered the "Correct!" sound signal. When the choice was incorrect, the system showed the error signal and triggered the "Incorrect!" sound. (f) Then the therapist asked them to role-play the scenario with appropriate gestures, conversation, intention, and facial expressions of emotion. (g) After the children had completed this part of the training, the therapist taught them the correct response and reconstructed their greeting behavior map. In the intervention process, each child was given the same training by the same therapist, but each was separately trained for different durations, as required by the individual child.

#### *3.3.2.3. Maintenance phase*

Between the intervention and maintenance phases, there was a 6-week hiatus to reduce recall interference in order to determine, using the baseline phase procedure, but not the AR-RPG system intervention materials, whether the children had maintained the skills that they had acquired.

#### **3.4. Experimental setting**

All sessions were conducted in a quiet 3.5 m × 6 m room of the day treatment center at the school. The room contained a table and chairs, an Intel Core i7 laptop computer, a web camera, several tangible role-play markers with avatars, and a 37-inch LCD display set up in front of the children. The therapist sat on the right of the participant and guided the training process; the researcher operated the computer, set up the environment, and helped the children use the AR-RPG system (see **Figure 2**). To begin the intervention test, the therapist showed the test sample to the children, taught them how to use the AR-RPG system, and ensured that they felt comfortable using the AR technology. The children started controlling each step of the test after the first trial task was presented. During the test, they used their hands to control the cardboard avatars to make interact with other avatars that appeared in the scene. After they had chosen the appropriate greeting, the next task trial was presented. The system ran the AR application in the background and showed the image on the LCD screen. The children were able to practice pretend play and mimic the greetings with their therapist, which allowed them to feel comfortable doing this experiment.

#### **4. Results and discussion**

Oliver, Peter, and Jane could perform tasks when supervised. Results from the Vineland II Adaptive Behavior Scales [56] indicated a moderately medium adaptive level. They all could engage in typical conversations about daily activities and things they liked. However, they had few friends, and they were reported to interact appropriately only with their parents or relatives. The researcher who examined the procedural reliability of this study was the same certified occupational therapist who conducted all of the tests. We followed the related experimental methods used in other studies [57] to train and test the participants' ability to identify the correct greeting behavior response. Those separate channels were evaluated by occupational therapists and special educational experts. We used a checklist in the test procedure to follow standard operating procedures for a therapist to ensure consistency in the processes and related controls. We also used the same AR role-playing strategy and context design to control the consistency of each story event to ensure that there were no unclear or emotionally confusing parts.

single subjects. The baseline phase consisted of four sessions for Oliver, six for Peter, and eight for Jane. The intervention phase consisted of eight sessions for each. The maintenance phase consisted of eight sessions for Oliver, six for Peter, and four for Jane (see **Figure 4**). All three children started with low scores (error rate range, 66.88–70%) during the baseline phase (see **Table 3**). All three scores rose significantly (*p* < .05) and dramatically (error rate range, 26.88–29.38%) during the intervention phase and remained significantly higher than at baseline (error rate range, 18.75–25.63%). The most dramatic error reduce rate was for Jane,

How to Create Suitable Augmented Reality Application to Teach Social Skills for Children…

http://dx.doi.org/10.5772/intechopen.76476

131

The three curves in **Figure 4** indicate that the correct assessment rates of all the children significantly improved (*p* < .05) (Kolmogorov–Smirnov test) after training and that the children retained in the maintenance phase the social expression and social skills that they had learned in the intervention phase. In addition, the mean difference in the performance level between

The training effect was also evaluated by the therapist using a 5-point Likert scale (see **Table 4**). The means of each phase of the score were recorded. All three children started with low scores (mean range, 1.5–1.75) during the baseline phase. All three scores rose significantly (*p* < .05), dramatically (mean range, 3.43–3.73) during the intervention phase, and remained signifi-

After the training, the curves indicate that the overall role-playing score was significantly

who started with 66.88% and ended with 18.75% (*p* < .05).

the baseline and maintenance phases was significant (*p* < .05).

**4.2. Their overall expression performances of role-playing**

cantly higher than at baseline (mean range, 3.57–4.18).

Data collection and test reliability.

**Table 3.** Summarized results for the participants.

**Table 4.** Summarized results for the participants.

(p < 0.05) higher in the maintenance phase than in the baseline phase.

**Role-playing performance**

Oliver 1.75 3.73 4.18 Peter 1.53 3.43 3.57 Jane 1.50 3.48 3.70 Mean 1.59 3.55 3.82

**Participants Baseline Intervention Maintenance**

**Training effects of AR-RPG system (error rate) Participants Baseline Intervention Maintenance** Oliver 70.00% 29.38% 25.63% Peter 68.33% 26.88% 20.00% Jane 66.88% 27.50% 18.75% Mean 66.40% 27.92% 21.46%

#### **4.1. Training effects of AR-RPG system**

The purpose of this experiment was to examine the differences in answers and greeting behavior responses between baseline and maintenance. We used a multiple baseline design across

**Figure 4.** The error rates of the participants during three phases.

single subjects. The baseline phase consisted of four sessions for Oliver, six for Peter, and eight for Jane. The intervention phase consisted of eight sessions for each. The maintenance phase consisted of eight sessions for Oliver, six for Peter, and four for Jane (see **Figure 4**). All three children started with low scores (error rate range, 66.88–70%) during the baseline phase (see **Table 3**). All three scores rose significantly (*p* < .05) and dramatically (error rate range, 26.88–29.38%) during the intervention phase and remained significantly higher than at baseline (error rate range, 18.75–25.63%). The most dramatic error reduce rate was for Jane, who started with 66.88% and ended with 18.75% (*p* < .05).

The three curves in **Figure 4** indicate that the correct assessment rates of all the children significantly improved (*p* < .05) (Kolmogorov–Smirnov test) after training and that the children retained in the maintenance phase the social expression and social skills that they had learned in the intervention phase. In addition, the mean difference in the performance level between the baseline and maintenance phases was significant (*p* < .05).

#### **4.2. Their overall expression performances of role-playing**

The training effect was also evaluated by the therapist using a 5-point Likert scale (see **Table 4**). The means of each phase of the score were recorded. All three children started with low scores (mean range, 1.5–1.75) during the baseline phase. All three scores rose significantly (*p* < .05), dramatically (mean range, 3.43–3.73) during the intervention phase, and remained significantly higher than at baseline (mean range, 3.57–4.18).

After the training, the curves indicate that the overall role-playing score was significantly (p < 0.05) higher in the maintenance phase than in the baseline phase.


Data collection and test reliability.

**Table 3.** Summarized results for the participants.


**Table 4.** Summarized results for the participants.

**Figure 4.** The error rates of the participants during three phases.

**4. Results and discussion**

130 State of the Art Virtual Reality and Augmented Reality Knowhow

**4.1. Training effects of AR-RPG system**

Oliver, Peter, and Jane could perform tasks when supervised. Results from the Vineland II Adaptive Behavior Scales [56] indicated a moderately medium adaptive level. They all could engage in typical conversations about daily activities and things they liked. However, they had few friends, and they were reported to interact appropriately only with their parents or relatives. The researcher who examined the procedural reliability of this study was the same certified occupational therapist who conducted all of the tests. We followed the related experimental methods used in other studies [57] to train and test the participants' ability to identify the correct greeting behavior response. Those separate channels were evaluated by occupational therapists and special educational experts. We used a checklist in the test procedure to follow standard operating procedures for a therapist to ensure consistency in the processes and related controls. We also used the same AR role-playing strategy and context design to control the consistency of

each story event to ensure that there were no unclear or emotionally confusing parts.

The purpose of this experiment was to examine the differences in answers and greeting behavior responses between baseline and maintenance. We used a multiple baseline design across

#### **5. Conclusions**

The present study's findings provide new insights into the innovations and technologies that promote social greeting behaviors for children with ASD. Children with ASD who participated in the study improved significantly. We found that, when closely monitored by an experienced therapist, our AR-RPG system was useful for teaching children with ASD how to recognize and understand the social greetings of others and how to appropriately respond. AR was important for helping the children learn how to properly greet people.

AR-RPG system, they seemed quite excited that the content was a 3D animation that brought the scenario "to life," and they began, without prompting, to ask the therapist a series of questions about the characters' facial expressions, gestures, and related social greeting activities. In addition, when students in their class were noisy, the teacher put a hand in front of his mouth to ask them to be quiet. Or, when classmates shrug their shoulders, it might have meant "I have no idea," "I can't help you," or "I don't care"; their teacher found those participants actively trying to observe those people's behavior differently by noticing target character's gesture, dialog's tone, and background situation to understand their body language. Although we cannot say that children with ASD easily learn to understand social situations, sometimes they still make wrong judgment. However, we can say that they learned to imitate

How to Create Suitable Augmented Reality Application to Teach Social Skills for Children…

http://dx.doi.org/10.5772/intechopen.76476

133

The children's parents also said this on their questionnaires and in interview feedback reports. They said that when their children met their teacher or other people at school, and when they met their neighbors or other adults in the community, they would more frequently respond appropriately. Sometimes, they would ask why some people did not wave or say goodbye or why some would cry and cover their face with their hand. When unexpected situations occurred, the children were unable to respond in real time to response, but they were trying to notice the social clues that they had learned. Furthermore, therapist also found that our AR-RPG system gives them a chance to not only pretend different role situations but also cause their pretend play habit frequency to increase, and the therapist said that their students with ASD always want to pretend they are different roles (e.g., pretend they are teacher) and sometimes will disturb their peer or other people. For example, they will pretend some situation that they learn some specific situation from our system and repeat to do this with their teacher or classmates (e.g., hugs to their teacher); they feel that different responses from those people are fun and ridiculous. However, this behavior might bore others, but, in a positive perspective, AR-RPG system takes advantage of children as active learners to social interac-

This study has some limitations. First, because this is a fairly new intervention strategy for children with ASD, it was difficult to recruit participants to join the study; moreover, the participants had limited time for the tests because many had their routine school's homework or family gatherings to take part in. Thus, it would be advantageous to recruit and enroll larger samples and extend the experiment period to provide stronger evidence. Second, it was difficult to determine whether the social skills of our participants had actually improved because it included many complex reciprocal social behaviors (e.g., effective modulation of eye contact, sharing affect, nonverbal reciprocity), not easily to separately measure each part of whole social behavior. Our positive findings indicate that children with ASD might change their behavior when they are aware of being observed; however, this will require a great deal of prospective observation and a long-term study. Third, we focused on the improvement of the greeting behavior as a whole. Future research might also take into account individual differences, e.g.,

eye contact and facial expressions, and how each individual uses the AR technology.

social etiquette that they had been taught.

tion with others.

**5.2. Limitation**

There are a few possible reasons for this. First, AR uses 3D virtual animation to show social behavior and real-time interactive scenarios. AR holds unique and promising potential to bridge between real-world activities and digital experiences, allowing users to engage their imagination and boost their creativity. Traditional VM cannot do this. Second, AR combined the cardboard and 3D avatars to help the children understand the scenarios, and the multisensory methods can help integrate the senses, such as focused attention, understanding situation boundaries, kinesthetic learning experience, and watch change. Third, using the physical manipulative avatar increased the children's learning motivation and comprehension because they could directly manipulate and observe the scenario's interactions. Fourth, AR and the related environment setup can help children with ASD quickly understand the situation in a specific place that is happening in a living room, classroom, park, or supermarket. This process can foster the development of more direct communication by employing techniques such as shared role-play situations than only reading a text description or watching a video.

In addition, many studies [58–60] support our perspectives that 3D animation with real spatial information in scenarios helps facilitate children's mental skills because they can directly see the characters' gestures and imitate them their feeling and action, and it does not require the user to image the scenario from 2D flat image [31]. What is most important, however, is that AR reduces a learner's cognitive load and increases their interest in being trained because they can directly see the scenario [38, 61, 62]. AR has other advantages. For example, AR interfaces can directly provide real-time 3D visual support and promote spatial visualization, which is related to cognitive ability [63]. We designed an AR platform that combined real avatars that used their real facial expressions. It allows users to manipulate a real tangible object to trigger different events in the setting. It is easy to change the scene and add some real tangible object such as a table, chair, etc., inside the AR system to enhance the user's immersion experience and motivation.

#### **5.1. Feedback from participants**

We also found clear changes in the social behavior of the three children before and after the training. Our therapist reported that, when children with ASD encounter a new situation and an unexpected greeting scenario, they first shrink back and look around for help or they focus on noncritical clues like clothing decoration patterns and other things that they are interested in. After they had been taught and trained using the AR-RPG system and 3D animation, they began to observe gestures and facial expressions. For example, when they used the AR-RPG system, they seemed quite excited that the content was a 3D animation that brought the scenario "to life," and they began, without prompting, to ask the therapist a series of questions about the characters' facial expressions, gestures, and related social greeting activities. In addition, when students in their class were noisy, the teacher put a hand in front of his mouth to ask them to be quiet. Or, when classmates shrug their shoulders, it might have meant "I have no idea," "I can't help you," or "I don't care"; their teacher found those participants actively trying to observe those people's behavior differently by noticing target character's gesture, dialog's tone, and background situation to understand their body language. Although we cannot say that children with ASD easily learn to understand social situations, sometimes they still make wrong judgment. However, we can say that they learned to imitate social etiquette that they had been taught.

The children's parents also said this on their questionnaires and in interview feedback reports. They said that when their children met their teacher or other people at school, and when they met their neighbors or other adults in the community, they would more frequently respond appropriately. Sometimes, they would ask why some people did not wave or say goodbye or why some would cry and cover their face with their hand. When unexpected situations occurred, the children were unable to respond in real time to response, but they were trying to notice the social clues that they had learned. Furthermore, therapist also found that our AR-RPG system gives them a chance to not only pretend different role situations but also cause their pretend play habit frequency to increase, and the therapist said that their students with ASD always want to pretend they are different roles (e.g., pretend they are teacher) and sometimes will disturb their peer or other people. For example, they will pretend some situation that they learn some specific situation from our system and repeat to do this with their teacher or classmates (e.g., hugs to their teacher); they feel that different responses from those people are fun and ridiculous. However, this behavior might bore others, but, in a positive perspective, AR-RPG system takes advantage of children as active learners to social interaction with others.

#### **5.2. Limitation**

**5. Conclusions**

132 State of the Art Virtual Reality and Augmented Reality Knowhow

sion experience and motivation.

**5.1. Feedback from participants**

The present study's findings provide new insights into the innovations and technologies that promote social greeting behaviors for children with ASD. Children with ASD who participated in the study improved significantly. We found that, when closely monitored by an experienced therapist, our AR-RPG system was useful for teaching children with ASD how to recognize and understand the social greetings of others and how to appropriately respond.

There are a few possible reasons for this. First, AR uses 3D virtual animation to show social behavior and real-time interactive scenarios. AR holds unique and promising potential to bridge between real-world activities and digital experiences, allowing users to engage their imagination and boost their creativity. Traditional VM cannot do this. Second, AR combined the cardboard and 3D avatars to help the children understand the scenarios, and the multisensory methods can help integrate the senses, such as focused attention, understanding situation boundaries, kinesthetic learning experience, and watch change. Third, using the physical manipulative avatar increased the children's learning motivation and comprehension because they could directly manipulate and observe the scenario's interactions. Fourth, AR and the related environment setup can help children with ASD quickly understand the situation in a specific place that is happening in a living room, classroom, park, or supermarket. This process can foster the development of more direct communication by employing techniques such as shared role-play situations than only reading a text description or watching a video. In addition, many studies [58–60] support our perspectives that 3D animation with real spatial information in scenarios helps facilitate children's mental skills because they can directly see the characters' gestures and imitate them their feeling and action, and it does not require the user to image the scenario from 2D flat image [31]. What is most important, however, is that AR reduces a learner's cognitive load and increases their interest in being trained because they can directly see the scenario [38, 61, 62]. AR has other advantages. For example, AR interfaces can directly provide real-time 3D visual support and promote spatial visualization, which is related to cognitive ability [63]. We designed an AR platform that combined real avatars that used their real facial expressions. It allows users to manipulate a real tangible object to trigger different events in the setting. It is easy to change the scene and add some real tangible object such as a table, chair, etc., inside the AR system to enhance the user's immer-

We also found clear changes in the social behavior of the three children before and after the training. Our therapist reported that, when children with ASD encounter a new situation and an unexpected greeting scenario, they first shrink back and look around for help or they focus on noncritical clues like clothing decoration patterns and other things that they are interested in. After they had been taught and trained using the AR-RPG system and 3D animation, they began to observe gestures and facial expressions. For example, when they used the

AR was important for helping the children learn how to properly greet people.

This study has some limitations. First, because this is a fairly new intervention strategy for children with ASD, it was difficult to recruit participants to join the study; moreover, the participants had limited time for the tests because many had their routine school's homework or family gatherings to take part in. Thus, it would be advantageous to recruit and enroll larger samples and extend the experiment period to provide stronger evidence. Second, it was difficult to determine whether the social skills of our participants had actually improved because it included many complex reciprocal social behaviors (e.g., effective modulation of eye contact, sharing affect, nonverbal reciprocity), not easily to separately measure each part of whole social behavior. Our positive findings indicate that children with ASD might change their behavior when they are aware of being observed; however, this will require a great deal of prospective observation and a long-term study. Third, we focused on the improvement of the greeting behavior as a whole. Future research might also take into account individual differences, e.g., eye contact and facial expressions, and how each individual uses the AR technology.

#### **5.3. Future work**

AR-RPG system made the way we children with ASD interesting and fun for them and, therefore, more efficacious than traditional methods. Our intervention system was effective for helping the three children with ASD maintain their focus on greeting behavior clues. It triggered the children's learning incentive, encouraged them to observe nonverbal social signals, and improved their social interaction skills. In future studies, experiments involving more participants of all ages with ASD and AR technology should be included to spur research in this area. In addition, training materials for ASD need to be more complete and more reflective of real life. Finally, we hope that our findings will encourage new research projects on how to reinvent visual media to increase in adolescents and others with ASD the recognition of nonverbal social reciprocity cues in social situations.

[2] American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders

How to Create Suitable Augmented Reality Application to Teach Social Skills for Children…

http://dx.doi.org/10.5772/intechopen.76476

135

[3] Sucksmith E, Allison C, Baron-Cohen S, Chakrabarti B, Hoekstra RA. Empathy and emotion recognition in people with autism, first-degree relatives, and controls.

[4] Fodstad JC, Matson JL, Hess J, Neal D. Social and communication behaviours in infants and toddlers with autism and pervasive developmental disorder-not otherwise speci-

[5] Hobson RP, Lee A. Hello and goodbye: A study of social engagement in autism. Journal

[6] Attwood T. The Complete Guide to Asperger's Syndrome. Jessica Kingsley Publishers; 2006 [7] Wing L, Gould J. Severe impairments of social interaction and associated abnormalities in children: Epidemiology and classification. Journal of Autism and Developmental

[8] Koldewyn K, Weigelt S, Kanwisher N, Jiang Y. Multiple object tracking in autism spectrum disorders. Journal of Autism and Developmental Disorders. 2013;**43**(6):1394-1405 [9] Geschwind DH, Levitt P. Autism spectrum disorders: Developmental disconnection

[10] Reichow B, Sabornie EJ. Brief report: Increasing verbal greeting initiations for a student with autism via a social story™ intervention. Journal of Autism and Developmental

[11] Reynhout G, Carter M. Social stories™ for children with disabilities. Journal of Autism

[12] Gray CA. Social stories and comic strip conversations with students with Asperger syndrome and high-functioning autism. In Asperger Syndrome or High-Functioning

[13] Cartledge G, Kiarie MW. Learning social skills through literature for children and ado-

[15] Kehret P. Encouraging empathy an author makes a case for teaching interpersonal skills.

[16] Heath MA, Sheen D, Leavy D, Young E, Money K. Bibliotherapy: A resource to facilitate emotional healing and growth. School Psychology International. 2005;**26**(5):563-580 [17] Sansosti FJ, Powell-Smith KA, Kincaid D. A research synthesis of social story interventions for children with autism spectrum disorders. Focus on Autism and Other

[18] Scattone D, Tingstrom DH, Wilczynski SM. Increasing appropriate social interactions of children with autism spectrum disorders using social stories™. Focus on Autism and

[14] Cress SW, Holm DT. Developing Empathy through Children's Literature. 1998

fied. Developmental Neurorehabilitation. 2009;**12**(3):152-157

of Autism and Developmental Disorders. 1998;**28**(2):117-127

syndromes. Current Opinion in Neurobiology. 2007;**17**(1):103-111

and Developmental Disorders. 2006;**36**(4):445-469

lescents. Teaching Exceptional Children. 2001;**34**(2):40

Autism? Boston, MA: Springer; 1998:167-198

School Library Journal. 2001;**47**(8):44-45

Developmental Disabilities. 2004;**19**(4):194-204

Other Developmental Disabilities. 2006;**21**(4):211-222

(DSM-5®). American Psychiatric Pub; 2013

Neuropsychologia. 2013;**51**(1):98e105

Disorders. 1979;**9**(1):11-29

Disorders. 2009;**39**(12):1740

#### **Conflict of interest**

The authors declare that they have no conflicts of interest.

#### **Informed consent**

Informed consent was obtained from all individual participants included in the study.

#### **Author details**

I-Jui Lee1 \*, Ling-Yi Lin2 , Chien-Hsu Chen3,4 and Chi-Hsuan Chung<sup>3</sup>

\*Address all correspondence to: ericlee@ntut.edu.tw

1 Ergonomics and Interaction Design Lab, Department of Industrial Design, National Taipei University of Technology, Taipei, Taiwan

2 Department of Occupational Therapy, College of Medicine, National Cheng Kung University, Tainan, Taiwan

3 Ergonomics and Interaction Design Lab, Department of Industrial Design, National Cheng Kung University, Tainan, Taiwan

4 Hierarchical Green-Energy Materials (Hi-GEM) Research Center, National Cheng Kung University, Tainan, Taiwan

#### **References**

[1] Boelte S, Hallmayer J. Autism spectrum conditions: FAQs on autism, asperger syndrome, and atypical autism answered by international experts. Ashland, OH: Hogrefe Publishing; 2013

[2] American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (DSM-5®). American Psychiatric Pub; 2013

**5.3. Future work**

**Conflict of interest**

**Informed consent**

**Author details**

\*, Ling-Yi Lin2

University, Tainan, Taiwan

University, Tainan, Taiwan

Publishing; 2013

**References**

Kung University, Tainan, Taiwan

University of Technology, Taipei, Taiwan

I-Jui Lee1

AR-RPG system made the way we children with ASD interesting and fun for them and, therefore, more efficacious than traditional methods. Our intervention system was effective for helping the three children with ASD maintain their focus on greeting behavior clues. It triggered the children's learning incentive, encouraged them to observe nonverbal social signals, and improved their social interaction skills. In future studies, experiments involving more participants of all ages with ASD and AR technology should be included to spur research in this area. In addition, training materials for ASD need to be more complete and more reflective of real life. Finally, we hope that our findings will encourage new research projects on how to reinvent visual media to increase in adolescents and others with ASD the recognition

Informed consent was obtained from all individual participants included in the study.

, Chien-Hsu Chen3,4 and Chi-Hsuan Chung<sup>3</sup>

1 Ergonomics and Interaction Design Lab, Department of Industrial Design, National Taipei

3 Ergonomics and Interaction Design Lab, Department of Industrial Design, National Cheng

[1] Boelte S, Hallmayer J. Autism spectrum conditions: FAQs on autism, asperger syndrome, and atypical autism answered by international experts. Ashland, OH: Hogrefe

4 Hierarchical Green-Energy Materials (Hi-GEM) Research Center, National Cheng Kung

2 Department of Occupational Therapy, College of Medicine, National Cheng Kung

of nonverbal social reciprocity cues in social situations.

134 State of the Art Virtual Reality and Augmented Reality Knowhow

The authors declare that they have no conflicts of interest.

\*Address all correspondence to: ericlee@ntut.edu.tw


[19] Delano M, Snell ME. The effects of social stories on the social engagement of children with autism. Journal of Positive Behavior Interventions. 2006;**8**(1):29-42

[33] Facoetti A, Ruffino M, Peru A, Paganoni P, Chelazzi L. Sluggish engagement and disengagement of non-spatial attention in dyslexic children. Cortex. 2008;**44**(9):1221-1233 [34] Kikuchi Y, Senju A, Akechi H, Tojo Y, Osanai H, Hasegawa T. Atypical disengagement from faces and its modulation by the control of eye fixation in children with autism spectrum disorder. Journal of Autism and Developmental Disorders. 2011;**41**(5):629-645

How to Create Suitable Augmented Reality Application to Teach Social Skills for Children…

http://dx.doi.org/10.5772/intechopen.76476

137

[35] Sigafoos J, O'Reilly M, De La Cruz B. How to Use Video Modeling and Video Prompting; 2007 [36] Jarrold C, Boucher J, Smith PK. Generativity deficits in pretend play in autism. British

[37] Lee IJ, Chen CH, Chang KP. Augmented reality technology combined with threedimensional holography to train the mental rotation ability of older adults. Computers

[38] Klatzky RL, Wu B, Shelton D, Stetten G. Effectiveness of augmented-reality visualization versus cognitive mediation for learning actions in near space. ACM Transactions on

[39] Trawick-Smith J. The Physical Play and Motor Development of Young Children: A Review of Literature and Implications for Practice. Vol. 13. Viitattu: The center for aerly

[40] Shelton BE. How augmented reality helps students learn dynamic spatial relationships

[41] LeGoff DB. Use of LEGO© as a therapeutic medium for improving social competence.

[42] Hornecker E, Buur J. Getting a grip on tangible interaction: A framework on physical space and social interaction: In Proceedings of the SIGCHI conference on Human

[43] Marshall P. Do tangible interfaces enhance learning?. In: Proceedings of the 1st international conference on Tangible and embedded interaction. pp. 163-170; February 2007. ACM

[44] Farr W, Yuill N, Hinske S. An augmented toy and social interaction in children with

[45] Wu HK, Lee SWY, Chang HY, Liang JC. Current status, opportunities and challenges of

[46] Gardner S, Wolfe P. Use of video modeling and video prompting interventions for teaching daily living skills to individuals with autism spectrum disorders: A review. Research

[47] Chen CH, Chou YY, Huang CY. An augmented-reality-based concept map to support mobile learning for science. The Asia-Pacific Education Researcher. 2016;**25**(4):567-578

[48] Graham M, Zook M, Boulton A. Augmented reality in urban places: Contested content and the duplicity of code. Transactions of the Institute of British Geographers.

autism. International Journal of Arts and Technology. 2012;**5**(2-4):104-125

augmented reality in education. Computers & Education. 2013;**62**:41-49

and Practice for Persons with Severe Disabilities. 2013;**38**(2):73-87

Journal of Developmental Psychology. 1996;**14**(3):275-300

[Doctoral dissertation]. University of Washington; 2003

Journal of Autism and Developmental Disorders. 2004;**34**(5):557-571

Factors in computing systems. pp. 437-446; 2006, April. ACM

in Human Behavior. 2016;**65**:488-500

Applied Perception (TAP). 2008;**5**(1):1

childhood education; 2014. p. 2016

2013;**38**(3):464-479


[33] Facoetti A, Ruffino M, Peru A, Paganoni P, Chelazzi L. Sluggish engagement and disengagement of non-spatial attention in dyslexic children. Cortex. 2008;**44**(9):1221-1233

[19] Delano M, Snell ME. The effects of social stories on the social engagement of children

[20] Sansosti FJ, Powell-Smith KA. Using social stories to improve the social behavior of children with Asperger syndrome. Journal of Positive Behavior Interventions. 2006;**8**(1):43-57

[21] Shukla-Mehta S, Miller T, Callahan KJ. Evaluating the effectiveness of video instruction on social and communication skills training for children with autism spectrum disorders: A review of the literature. Focus on Autism and Other Developmental Disabilities.

[22] Bellini S, Akullian J. A meta-analysis of video modeling and video self-modeling interventions for children and adolescents with autism spectrum disorders. Exceptional

[23] Lang R, Shogren K, Machalicek W, Rispoli M, O'Reilly MF, Regester A, Baker S. Use of video self-modeling to teach classroom rules to two children with Asperger's syndrome.

[24] Nikopoulos CK, Keenan M. Promoting social initiation in children with autism using

[25] Maione L, Mirenda P. Effects of video modeling and video feedback on peer-directed social language skills of a child with autism. Journal of Positive Behavior Interventions.

[26] Charlop MH, Dennis B, Carpenter MH, Greenberg AL. Teaching socially expressive behaviors to children with autism through video modeling. Education and Treatment of

[27] Kagohara DM, Achmadi D, van der Meer L, Lancioni GE, O'Reilly MF, Lang R, Sigafoos J. Teaching two students with Asperger syndrome to greet adults using social stories™ and video modeling. Journal of Developmental and Physical Disabilities.

[28] Bernad-Ripoll S. Using a self-as-model video combined with social stories™ to help a child with Asperger syndrome understand emotions. Focus on Autism and Other

[29] Ayres KM, Langone J. Intervention and instruction with video for students with autism: A review of the literature. Education and Training in Developmental Disabilities.

[30] Corbett BA, Abdullah M. Video modeling: Why does it work for children with autism?

[31] Chen CH, Lee IJ, Lin LY. Augmented reality-based self-facial modeling to promote the emotional expression and social skills of adolescents with autism spectrum disorders.

[32] Elsabbagh M, Volein A, Holmboe K, Tucker L, Csibra G, Baron-Cohen S, et al. Visual orienting in the early broader autism phenotype: Disengagement and facilitation. Journal

Journal of Early and Intensive Behavior Intervention. 2005;**2**(1):2

Research in Developmental Disabilities. 2015;**36**:396-403

of Child Psychology and Psychiatry. 2009;**50**(5):637-642

Research in Autism Spectrum Disorders. 2009;**3**:483-488

video modeling. Behavioral Interventions. 2003;**18**(2):87-108

with autism. Journal of Positive Behavior Interventions. 2006;**8**(1):29-42

2010;**25**(1):23-36

2006;**8**(2):106-118

2013;**25**(2):241-251

2005:183-196

Children. 2010;**33**(3):371-393

Developmental Disabilities. 2007;**22**(2):100-106

Children. 2007;**73**(3):264-287

136 State of the Art Virtual Reality and Augmented Reality Knowhow


[49] Baker JE. Social Skills Picture Book: Teaching Play, Emotion, and Communication to

[50] Baker JE. Social Skills Training. Overland Park, KS: Autism Asperger Publishing; 2003b [51] Matson JL. Applied Behavior Analysis for Children with Autism Spectrum Disorders.

[52] Plavnick JB, Ferreri SJ. Single-case experimental designs in educational research: A methodology for causal analyses in teaching and learning. Educational Psychology Review.

[53] Apple AL, Billingsley F, Schwartz IS, Carr EG. Effects of video modeling alone and with self-management on compliment-giving behaviors of children with high-functioning

[54] Cardon TA, Azuma T. Deciphering single-subject research design and autism spectrum

[55] Ingersoll B, Dvortcsak A, Whalen C, Sikora D. The effects of a developmental, social— Pragmatic language intervention on rate of expressive language production in young children with autistic spectrum disorders. Focus on Autism and Other Developmental

[56] Sparrow SS. Vineland adaptive behavior scales. In: Encyclopedia of Clinical Neuropsy-

[57] Castelli F. Understanding emotions from standardized facial expressions in autism and

[58] Hoyek N, Di Rienzo F, Collet C, Creveaux T, Guillot A. Hand mental rotation is not systematically altered by actual body position: Laterality judgment versus same–different

[59] Wraga M, Thompson WL, Alpert NM, Kosslyn SM. Implicit transfer of motor strategies

[60] Wexler M, Kosslyn SM, Berthoz A. Motor processes in mental rotation. Cognition. 1998;

[61] Hedley NR. Empirical evidence for advanced geographic visualization interface use. In:

[62] Shelton BE, Hedley NR. Exploring a cognitive basis for learning spatial relationships with augmented reality. Technology, Instruction, Cognition and Learning. 2004;**1**(4):323

[63] Kaufmann H, Schmalstieg D. Mathematics and geometry education with collaborative

International cartographic congress; Durban, South Africa; August, 2003

comparison tasks. Attention, Perception, & Psychophysics. 2014;**76**(2):519-526

Children with Autism. Arlington, TX: Future Horizons; 2003a

ASD. Journal of Positive Behavior Interventions. 2005;**7**(1):33-46

disorders. The ASHA Leader. 2011;**16**(7) online-only

chology. New York: Springer; 2011. pp. 2618-2621

normal development. Autism. 2005;**9**(4):428-449

in mental rotation. Brain and Cognition. 2003;**52**(2):135-143

augmented reality. Computers & Graphics. 2003;**27**(3):339-345

New York: Springer; 2009

138 State of the Art Virtual Reality and Augmented Reality Knowhow

Disabilities. 2005;**20**(4):213-222

**68**(1):77-94

2013;**25**(4):549-569
