**NeuAR – A Review of the VR/AR Applications in the Neuroscience Domain**

Pedro Gamito1,2,3, Jorge Oliveira1,2, Diogo Morais1,2, Pedro Rosa1,2,4 and Tomaz Saraiva1 *1Universidade Lusófona de Humanidades e Tecnologias, 2Centro de Estudos em Psicologia Cognitiva e da Aprendizagem, 3Clínica S. João de Deus, 4ISCTE-IUL/CIS Portugal* 

## **1. Introduction**

130 Augmented Reality – Some Emerging Application Areas

Lucas B.D. and Kanade T. (1981). An iterative image registration technique with an

Milgram P. and Kishino F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems, Vol E77-D, No 12, December 1994 Pasman W. and Woodward C. (2003). Implementation of an augmented reality system on a

*Augmented Reality (ISMAR 2003)*, Tokyo, Japan, October 2003, pp. 276-277. Reitmayr G. and Drummond T. (2006). Going out: robust, model based tracking for outdoor

Schall G., Mendez E. and Schmalstieg D. (2008). Virtual redlining for civil engineering in

*Reality (ISMAR 2008)*, Orlando, Florida, USA, October 2008, pp. 153-162. Senate (2007). Senate BIM Guidelines. http://www.senaatti.fi/document.asp?siteID

Shi J. and Tomasi C. (1994). Good features to track. *Proc. 1994 IEEE Conference on Computer* 

Shin D.H. and Dunston P.S. (2008). Identification of application areas for augmented reality

Takacs G., Chandrasekhar V., Tsai S.S., Chen D.M., Grzeszczuk R. and Girod B. (2010).

Wither J., DiVerdi S. and Höllerer T. (2006). Using aerial photographs for improved mobile

*Virtual Reality (CONVR2010)*, Sendai, Japan, Nov 4-5, 2010, pp. 35-44.

*(CVPR 2010)*, San Francisco, CA, USA, June 13-18, 2010, pp. 934-941. TNO (2010). TNO IFC Engine. http://www.ifcbrowser.com/ifcenginedll.html. VTT (2010a). Video on the Forchem pilot. http://youtu.be/xpZDXih9zLg. VTT (2010b). Video on the Skanska pilot. http://youtu.be/rVt86NGXQv4.

*European Conference on Computer Vision*, May 2006, pp. 430-443.

*Vision and Pattern Recognition (CVPR'94)*, pp. 593-600.

*Intelligence (IJCAI '81)*, April 1981, pp. 674-679.

=2&docID=588.

*Construction* 17, pp. 882-894.

VTT (2011). ALVAR. www.vtt.fi/multimedia/alvar.html.

application to stereo vision. *Proc. of the 7th International Joint Conference on Artificial* 

PDA. *Proc. The Second IEEE and ACM International Symposium on Mixed and* 

augmented reality. *Proc. The Fifth IEEE and ACM International Symposium on Mixed and Augmented Reality (2006)*, Santa Barbara, USA, 22-25 October 2006, pp.109-118. Rosten E. and Drummond T. (2006). Machine learning for high-speed corner detection.

real environments. *Proc. The 7th IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2008)*, Cambridge, UK, September 2008, pp. 95-98. Schall G., Wagner D., Reitmayr G., Taichmann E., Wieser M., Schmalstieg D. and Hoffmann-

Wellenhof B. (2008). Global pose estimation using multi-sensor fusion for outdoors augmented reality. *Proc. 8th IEEE International Symposium on Mixed and Augmented* 

in industrial construction based on technological suitability. *Automation in* 

Unified real-time tracking and recognition with rotation-invariant fast features. *Proc. The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition,* 

AR annotation. *Proc. The Fifth IEEE and ACM International Symposium on Mixed and Augmented Reality (2006)*, Santa Barbara, USA, 22-25 October 2006, pp.159-162. Woodward C., Hakkarainen M., Korkalo O., Kantonen T. Aittala M., Rainio K. and

Kähkönen K. (2010). Mixed reality for mobile construction site visualization and communication. *Proc. 10th International Conference on Construction Applications of*  Since the 1980's, computational applications based on virtual reality (VR) aimed at treating mental disorders and rehabilitating individuals with cognitive or motor disabilities have been around. They started off by focusing on simple phobias like acrophobia (Emmelkamp et al., 2002) and agoraphobia (Botella et al., 2004), fear of flying (Rothbaum, Hodges, Smith, Lee & Price, 2000), and evolved to fear of driving (Saraiva et al., 2007) or posttraumatic stress disorder (PTSD) (Gamito et al., 2010), schizophrenia (Costa & Carvalho, 2004) or traumatic brain injuries (Gamito et al., 2011a), among many others (Gamito et al., 2011b).

VR holds two chief properties that enable patients to experience the synthetic environment as being real: immersion and interaction. The first relates to the sensation of being physical present and perceptually included in the VR world. The second stands for the ability to change the world properties, i.e. the environment and its constituents react according to participants actions. Along with imagination, interaction and immersion concur to create the so called "sense of being there" or presence.

This characteristic of VR settings has been acknowledged by the psychotherapists as a media to expose patients with anxiety disorders (AD) to anxiogenic cues within an ecologically sound and controlled environment. VR designed for therapeutic purposes can replicate any of the ansiogenic situations, enabling a better approximation to the ansiogenic world and inducing higher levels of engagement when compared to traditional imagination exposure (Riva et al., 2002). Hyperrealistic threatening stimuli provided by VR lead to higher attention, and subsequent encapsulation, which means, once the fear system is activated the participant perceives the synthetic world as being real (Hamm & Weike, 2005). Also, VR reduces the decalage between reality and imagination, by diminishing potential distraction or cognitive avoidance to the threatening stimuli (Vincelli & Riva, 2000). These and other studies revealed that VR exposure therapy (VRET) may be an alternative to *in vivo* and imagination exposure.

NeuAR – A Review of the VR/AR Applications in the Neuroscience Domain 133

with the exercise. Bryanton and colleagues (2006) found that when compared to conventional exercise, children with cerebral palsy had more fun and tended to repeat more

VR seems, during hospitalization, to promote a more intensive and program supportive approach to the execution of the exercise, providing appropriate feedback to the patient. Also, exercises may be displayed with an adapting degree of difficulty, making possible the use of non-invasive forms of physiological monitoring. VR, in addition, gives therapist the ability to individualize treatment needs, while providing the opportunity for repeated learning trials and offer the capacity to gradually increase the complexity tasks while decreasing therapist support and feedback (Weiss & Katz, 2004). VR is a promising response

Studies on VR rehabilitation are usually focused on motor rehabilitation following brain damage and on training people with intellectual disabilities (Attree et al., 2005). However, VR has been also applied to rehabilitate patients that had suffered traumatic brain injuries (TBI). Slobounov and colleagues (2006) found VR to be useful as a tool to assess brain concussion. A VR system was developed to inspect the temporal restoration of the effect of visual field motion on TBI's subjects with short term and long term balance anomalies. The study of memory and attentional problems is important for many patients with a history of TBI, even when they are not a primary problem. Wilson and colleagues (2006) stressed that the automaticity of basic movement skills is often learned in controlled environments. Once the patient is required to apply skills in real-world settings, demands on attention and on working memory often exceed their processing and response capabilities. Also, skills' compliance in the previous stages of rehabilitation is inhibited by disruptions to attention and working memory processes. Patients with acquired brain injuries may find it tricky to train both a primary task (e.g. walking) and a simultaneous secondary task (e.g. signal detection). During the skill learning phase, the function of attention and memory can be supported by visual and verbal cues that can signal attention to obstacles and forthcoming

In-between "real" reality and virtual reality rests, according to Milgram's continuum (Milgram & Kishino, 1994), another form of interaction with the real world coined by Caudell and Mizell (1992) as augmented reality (AR). This technique, as mentioned in previous chapters, consists (through at least three different approaches, video see-through, optical see through or projection) on superimposing a computer generated object onto a real

As a result, it is expected participant's perception to be tricked so that the virtual object should be perceived as being part of the real world. But this perquisite is not sufficed. Also, real time interaction and 3D registration are required (Azuma et al., 2001). Likewise in VR, interaction is a key feature. In order to guarantee that the user recognizes the synthetic object as being part of the real world, it is paramount that he or she may interact with it as if it is a real entity. Interaction is perceived by many authors (Witmer & Singer, 1998; Riva et al., 2002) as the cornerstone of any virtual or close to virtual experience as it promotes the immersion on the synthetic world. Also, the precise alignment between the real world and the plan where the 3D image is placed is essential so that the illusion of non-real and real coexistence may take place. AR properties will be fully discussed in the

often at home ankle dorsiflexion and long-sitting VR exercises.

to shorter hospitalization and foster homecare (Giorgino et al., 2008).

events.

world setting.

next section.

In fact, Parsons and Rizzo (2008), on a meta-analysis found an average size effect of 0.96, reflecting a large effect for VRET on the decrease of negative affective symptoms of the six affective domains studied (PTSD, Social Phobia, Arachnophobia, Acrophobia, Panic Disorder with Agoraphobia and Aerophobia). Also, Powers and Emmelkamp (2008), on another metaanalysis, observed a large mean effect size for VRET, when compared with *in vivo* and control conditions (waiting list, attention control, etc.).

In this way, VRET as a form of a psycotherapeutic approach presents some advantages when compared with the two traditional exposure techniques: *in vivo* and imagination exposure. Concerning the first, VRET is able to replace real traumatic events, such as, war scenarios or motor vehicles accidents, eliciting traumatic emotions as if the patient were really there. The interactive simulation that VR encloses ensures a rich full sensorial experience similar to an *in vivo* occurrence.

Regarding the last, in VRET the therapeutic environment is controlled by the therapist, something that does not occur when a patient is asked to imagine the anxiogenic situation as occurs during imagination exposure. Both typology and intensity of cues can be managed by the therapist. For instance, Rizzo and colleagues (2006), in a VR world devised to treat veterans with PTSD from Iraqi war, have included a console in which scenario's assets such as alternation between day and night, night vision, fog, helicopter coming in, spawn of enemies, among others functions, can be placed, as requested, in the VR world.

Concerning rehabilitation, its three core pillars: repetition, feedback and motivation may gain from the use of VR (Holden, 2003). In rehabilitation, one of the most common procedures is the repeated and systematic training of the impaired functions (Allred et al., 2005). In agreement to the review from Sveistrup (2004), VR can provide training environments where visual and auditory feedback can be systematically manipulated according to individual differences. Furthermore, the use of 3D (3 dimensions) virtual environments offers the possibility of real-time feedback of subject's position and progression (Sveistrup, 2004). For Levin and colleagues (2005) the use of VR applications in rehabilitation can be effective because of the 3D spatial correspondence between movements in the real world and movements in the virtual worlds which, in turn, may facilitate realtime performance feedback. Cirstea and Levin (2007) referred that performance feedback can provide information regarding impaired motor movements. For example, Feintuch and colleagues (2006) developed a haptic-tactile feedback system that, when integrated on a video-capture-based VR environment, enables patients to feel a vibration on their fingers whenever they "touch" a ball on the VR world. Viau and colleagues (2004) analyzed movements performed by participants with hemiparesis with virtual objects in VR and real objects in real environments. These authors found no differences between performances in VR and real environments and suggested that this VR technique can be an effective training for rehabilitation.

The repetitive practice is also an important aspect in motor and cognitive training as it improves performance in disabled patients (Chen et al., 2004). These authors used VR environments in children with cerebral palsy and observed that the repetitive practice of a particular motor aspect enables the coordination of a specific muscular system.

And because VR is usually presented on a multimodal platform with several sorts of immersive cues, such as images and sounds, patients are more willing to engage and pursue 132 Augmented Reality – Some Emerging Application Areas

In fact, Parsons and Rizzo (2008), on a meta-analysis found an average size effect of 0.96, reflecting a large effect for VRET on the decrease of negative affective symptoms of the six affective domains studied (PTSD, Social Phobia, Arachnophobia, Acrophobia, Panic Disorder with Agoraphobia and Aerophobia). Also, Powers and Emmelkamp (2008), on another metaanalysis, observed a large mean effect size for VRET, when compared with *in vivo* and control

In this way, VRET as a form of a psycotherapeutic approach presents some advantages when compared with the two traditional exposure techniques: *in vivo* and imagination exposure. Concerning the first, VRET is able to replace real traumatic events, such as, war scenarios or motor vehicles accidents, eliciting traumatic emotions as if the patient were really there. The interactive simulation that VR encloses ensures a rich full sensorial

Regarding the last, in VRET the therapeutic environment is controlled by the therapist, something that does not occur when a patient is asked to imagine the anxiogenic situation as occurs during imagination exposure. Both typology and intensity of cues can be managed by the therapist. For instance, Rizzo and colleagues (2006), in a VR world devised to treat veterans with PTSD from Iraqi war, have included a console in which scenario's assets such as alternation between day and night, night vision, fog, helicopter coming in, spawn of

Concerning rehabilitation, its three core pillars: repetition, feedback and motivation may gain from the use of VR (Holden, 2003). In rehabilitation, one of the most common procedures is the repeated and systematic training of the impaired functions (Allred et al., 2005). In agreement to the review from Sveistrup (2004), VR can provide training environments where visual and auditory feedback can be systematically manipulated according to individual differences. Furthermore, the use of 3D (3 dimensions) virtual environments offers the possibility of real-time feedback of subject's position and progression (Sveistrup, 2004). For Levin and colleagues (2005) the use of VR applications in rehabilitation can be effective because of the 3D spatial correspondence between movements in the real world and movements in the virtual worlds which, in turn, may facilitate realtime performance feedback. Cirstea and Levin (2007) referred that performance feedback can provide information regarding impaired motor movements. For example, Feintuch and colleagues (2006) developed a haptic-tactile feedback system that, when integrated on a video-capture-based VR environment, enables patients to feel a vibration on their fingers whenever they "touch" a ball on the VR world. Viau and colleagues (2004) analyzed movements performed by participants with hemiparesis with virtual objects in VR and real objects in real environments. These authors found no differences between performances in VR and real environments and suggested that this VR technique can be an effective training

The repetitive practice is also an important aspect in motor and cognitive training as it improves performance in disabled patients (Chen et al., 2004). These authors used VR environments in children with cerebral palsy and observed that the repetitive practice of a

And because VR is usually presented on a multimodal platform with several sorts of immersive cues, such as images and sounds, patients are more willing to engage and pursue

particular motor aspect enables the coordination of a specific muscular system.

enemies, among others functions, can be placed, as requested, in the VR world.

conditions (waiting list, attention control, etc.).

experience similar to an *in vivo* occurrence.

for rehabilitation.

with the exercise. Bryanton and colleagues (2006) found that when compared to conventional exercise, children with cerebral palsy had more fun and tended to repeat more often at home ankle dorsiflexion and long-sitting VR exercises.

VR seems, during hospitalization, to promote a more intensive and program supportive approach to the execution of the exercise, providing appropriate feedback to the patient. Also, exercises may be displayed with an adapting degree of difficulty, making possible the use of non-invasive forms of physiological monitoring. VR, in addition, gives therapist the ability to individualize treatment needs, while providing the opportunity for repeated learning trials and offer the capacity to gradually increase the complexity tasks while decreasing therapist support and feedback (Weiss & Katz, 2004). VR is a promising response to shorter hospitalization and foster homecare (Giorgino et al., 2008).

Studies on VR rehabilitation are usually focused on motor rehabilitation following brain damage and on training people with intellectual disabilities (Attree et al., 2005). However, VR has been also applied to rehabilitate patients that had suffered traumatic brain injuries (TBI). Slobounov and colleagues (2006) found VR to be useful as a tool to assess brain concussion. A VR system was developed to inspect the temporal restoration of the effect of visual field motion on TBI's subjects with short term and long term balance anomalies. The study of memory and attentional problems is important for many patients with a history of TBI, even when they are not a primary problem. Wilson and colleagues (2006) stressed that the automaticity of basic movement skills is often learned in controlled environments. Once the patient is required to apply skills in real-world settings, demands on attention and on working memory often exceed their processing and response capabilities. Also, skills' compliance in the previous stages of rehabilitation is inhibited by disruptions to attention and working memory processes. Patients with acquired brain injuries may find it tricky to train both a primary task (e.g. walking) and a simultaneous secondary task (e.g. signal detection). During the skill learning phase, the function of attention and memory can be supported by visual and verbal cues that can signal attention to obstacles and forthcoming events.

In-between "real" reality and virtual reality rests, according to Milgram's continuum (Milgram & Kishino, 1994), another form of interaction with the real world coined by Caudell and Mizell (1992) as augmented reality (AR). This technique, as mentioned in previous chapters, consists (through at least three different approaches, video see-through, optical see through or projection) on superimposing a computer generated object onto a real world setting.

As a result, it is expected participant's perception to be tricked so that the virtual object should be perceived as being part of the real world. But this perquisite is not sufficed. Also, real time interaction and 3D registration are required (Azuma et al., 2001). Likewise in VR, interaction is a key feature. In order to guarantee that the user recognizes the synthetic object as being part of the real world, it is paramount that he or she may interact with it as if it is a real entity. Interaction is perceived by many authors (Witmer & Singer, 1998; Riva et al., 2002) as the cornerstone of any virtual or close to virtual experience as it promotes the immersion on the synthetic world. Also, the precise alignment between the real world and the plan where the 3D image is placed is essential so that the illusion of non-real and real coexistence may take place. AR properties will be fully discussed in the next section.

NeuAR – A Review of the VR/AR Applications in the Neuroscience Domain 135

world is provided by the cameras which are not perfectly aligned with the eyes' position. This incongruity may lead to disorientation resulting from the eye offset (Biocca & Rolland, 1998). However, the VST display is able to create more compelling experiences than its optical counterpart (Azuma, 1997). Despite being a replacement of the real view, the video

In order to provide a realistic AR experience it is imperative an optimal level of registration. Virtual objects must appear perfectly aligned with the real ones in order create a believable experience. An obstacle to this registration is the delay in time, between the moment when the position of the object is measured and the moment when the digitized imaged is presented. Using an OST display, the user has an immediate view of the real world but a slightly delayed view of the virtual overlay. This gives the impression that virtual objects are not fixed in the environment, something referred as a swimming effect (Azuma & Bishop, 1994). Solutions have been developed, from using predictive tracking to optimizing the system for low latency (Azuma et al., 1994). While this is a serious limitation in the optical method, with the VST technology it is possible to take advantage of having two video feeds, the real and the virtual. As demonstrated by Bajura and Neumann (1995) it is possible to enforce registration, matching both views by delaying the original video in order to equal the presentation of the synthetic objects. This way, both feeds are matched and the objects

One of the most important advantages of the VST technology is related to how it deals with occlusion. Depth cues are extremely important when creating realistic environments and occlusion is one of the strongest. Occlusion depth cue consists in how an object is hidden by another one which is closer and in front of it relatively to the users' line of sight. The optical method is limited when dealing with occlusion (Azuma, 1997). The optical combiners receive light simultaneously from real and virtual world, which makes it impossible to obscure completely the real objects with the virtual ones. In this way, the virtual objects appear as semi-transparent, affecting the sense of occlusion and therefore the overall realism of the experience. On the other hand the VST technology can deal perfectly with occlusion. While using a digitized version of the reality it is possible to obscure completely the real, the virtual or to blend both using a pixel-by-pixel comparison. In the same way this method allows a better matching between real and virtual brightness and contrast, which is not

The VST technology also benefits from additional tracking methods that enhance the alignment of real and virtual objects. Using a video feed of the real scene it is possible to employ additional registration methods based on image processing techniques. The same methods are unavailable on OST displays, which can only rely on tracking information from

One last issue regarding these technologies is the Field-of-View (FOV). Both systems present limited FOVs. OST displays support a 20º to 60º overlay FOVs but provide a close match to our eyes' natural real-world FOV, since the peripheral vision is available to look around the device. On the other hand, the VST displays may support overlay FOVs similar to the viewing optics but the peripheral FOV is occluded, resulting in a smaller real-world FOV. This limitation can affect applications where situation awareness is necessary since the users need to perform larger head movements when scanning the environment (Rolland et al.,

live feed introduces several advantages on the augmentation process.

appear perfectly aligned (Rolland et al., 1994).

possible on optical methods.

the users' body movement.

1995).

These characteristics of AR were adopted by practitioners and researchers of the area of neurosciences. Similarly to VR, also AR applications were developed to treat anxiety disorders and to rehabilitate individuals with cognitive and motor impairments, following the same VR principles of application referred above. The upcoming sections of the chapter will focus on the work of several research groups that employ AR as a media to treat mental disorders and to rehabilitate patients with acquired central nervous injuries. But first, a fly over the "techy" bits and bites of AR.

## **2. Art – Augmented reality technology**

Despite the several approaches that can be used to achieve the augmentation effect, the AR systems' architecture relies essentially on the combination of two components: the visual display and the tracking system. The visual display is decisive on the immersive ability of the system, while an efficient tracking is required to reach an optimal alignment of virtual and real objects, also known as registration (Zlatanova, 2002).

## **2.1 Visual display**

The visual display is an image-forming system responsible for how the virtual content is combined with the real one and presented in the user's line of sight. The type of display is a product of the combination of the technology with the positioning of the display relatively to the user (Van Krevelen & Poelman, 2010). Currently there are four different available technologies: optical see-through (OST); video see-through (VST); virtual retinal display (VRD) and projective display.

#### **2.1.1 Visual display technology**

#### **2.1.1.1 Optical See-through (OST) vs. Video See-through (VST)**

With the OST technology the user sees the real world through optical combiners, usually half-silvered mirrors or transparent LCD displays. These allow an unobstructed view of the real world while supporting the superimposition of virtual content (Azuma, 1997).

The VST technology consists in overlaying the virtual content on a live video feed of a real environment. The real world is captured in real-time by one or two video cameras and the virtual content is digitally blended into the original recording (Van Krevelen et al., 2010) using video keying techniques or pixel-by-pixel depth comparison (Azuma, 1997). The main difference between these systems consists in the nearly intact real world view provided by the OST technology as opposed to the live video feed replacement created by the VST.

Preserving the real world view allows a higher resolution on OST in comparison to VST displays. While on the optical method the user looks through a thin lens that leaves the realworld resolution intact, the video method clamps it to the maximum resolution supported by the display or video source (Rolland et al., 1994). The OST method is also safer than VST. In power failure incidents, the OST only loses the virtual overlay, while in the video replacement such an incident would leave the user completely blind, which could be critical in medical or military applications (Azuma, 1997). The OST is also a parallax free method. In the OST displays the view of the world corresponds exactly to the viewpoint of the eye, while in the VST there is a mismatch in the viewpoint information. The users' view of the 134 Augmented Reality – Some Emerging Application Areas

These characteristics of AR were adopted by practitioners and researchers of the area of neurosciences. Similarly to VR, also AR applications were developed to treat anxiety disorders and to rehabilitate individuals with cognitive and motor impairments, following the same VR principles of application referred above. The upcoming sections of the chapter will focus on the work of several research groups that employ AR as a media to treat mental disorders and to rehabilitate patients with acquired central nervous injuries. But first, a fly

Despite the several approaches that can be used to achieve the augmentation effect, the AR systems' architecture relies essentially on the combination of two components: the visual display and the tracking system. The visual display is decisive on the immersive ability of the system, while an efficient tracking is required to reach an optimal alignment of virtual

The visual display is an image-forming system responsible for how the virtual content is combined with the real one and presented in the user's line of sight. The type of display is a product of the combination of the technology with the positioning of the display relatively to the user (Van Krevelen & Poelman, 2010). Currently there are four different available technologies: optical see-through (OST); video see-through (VST); virtual retinal display

With the OST technology the user sees the real world through optical combiners, usually half-silvered mirrors or transparent LCD displays. These allow an unobstructed view of the

The VST technology consists in overlaying the virtual content on a live video feed of a real environment. The real world is captured in real-time by one or two video cameras and the virtual content is digitally blended into the original recording (Van Krevelen et al., 2010) using video keying techniques or pixel-by-pixel depth comparison (Azuma, 1997). The main difference between these systems consists in the nearly intact real world view provided by the OST technology as opposed to the live video feed replacement created by the VST.

Preserving the real world view allows a higher resolution on OST in comparison to VST displays. While on the optical method the user looks through a thin lens that leaves the realworld resolution intact, the video method clamps it to the maximum resolution supported by the display or video source (Rolland et al., 1994). The OST method is also safer than VST. In power failure incidents, the OST only loses the virtual overlay, while in the video replacement such an incident would leave the user completely blind, which could be critical in medical or military applications (Azuma, 1997). The OST is also a parallax free method. In the OST displays the view of the world corresponds exactly to the viewpoint of the eye, while in the VST there is a mismatch in the viewpoint information. The users' view of the

real world while supporting the superimposition of virtual content (Azuma, 1997).

over the "techy" bits and bites of AR.

**2.1 Visual display** 

(VRD) and projective display.

**2.1.1 Visual display technology** 

**2. Art – Augmented reality technology** 

and real objects, also known as registration (Zlatanova, 2002).

**2.1.1.1 Optical See-through (OST) vs. Video See-through (VST)** 

world is provided by the cameras which are not perfectly aligned with the eyes' position. This incongruity may lead to disorientation resulting from the eye offset (Biocca & Rolland, 1998). However, the VST display is able to create more compelling experiences than its optical counterpart (Azuma, 1997). Despite being a replacement of the real view, the video live feed introduces several advantages on the augmentation process.

In order to provide a realistic AR experience it is imperative an optimal level of registration. Virtual objects must appear perfectly aligned with the real ones in order create a believable experience. An obstacle to this registration is the delay in time, between the moment when the position of the object is measured and the moment when the digitized imaged is presented. Using an OST display, the user has an immediate view of the real world but a slightly delayed view of the virtual overlay. This gives the impression that virtual objects are not fixed in the environment, something referred as a swimming effect (Azuma & Bishop, 1994). Solutions have been developed, from using predictive tracking to optimizing the system for low latency (Azuma et al., 1994). While this is a serious limitation in the optical method, with the VST technology it is possible to take advantage of having two video feeds, the real and the virtual. As demonstrated by Bajura and Neumann (1995) it is possible to enforce registration, matching both views by delaying the original video in order to equal the presentation of the synthetic objects. This way, both feeds are matched and the objects appear perfectly aligned (Rolland et al., 1994).

One of the most important advantages of the VST technology is related to how it deals with occlusion. Depth cues are extremely important when creating realistic environments and occlusion is one of the strongest. Occlusion depth cue consists in how an object is hidden by another one which is closer and in front of it relatively to the users' line of sight. The optical method is limited when dealing with occlusion (Azuma, 1997). The optical combiners receive light simultaneously from real and virtual world, which makes it impossible to obscure completely the real objects with the virtual ones. In this way, the virtual objects appear as semi-transparent, affecting the sense of occlusion and therefore the overall realism of the experience. On the other hand the VST technology can deal perfectly with occlusion. While using a digitized version of the reality it is possible to obscure completely the real, the virtual or to blend both using a pixel-by-pixel comparison. In the same way this method allows a better matching between real and virtual brightness and contrast, which is not possible on optical methods.

The VST technology also benefits from additional tracking methods that enhance the alignment of real and virtual objects. Using a video feed of the real scene it is possible to employ additional registration methods based on image processing techniques. The same methods are unavailable on OST displays, which can only rely on tracking information from the users' body movement.

One last issue regarding these technologies is the Field-of-View (FOV). Both systems present limited FOVs. OST displays support a 20º to 60º overlay FOVs but provide a close match to our eyes' natural real-world FOV, since the peripheral vision is available to look around the device. On the other hand, the VST displays may support overlay FOVs similar to the viewing optics but the peripheral FOV is occluded, resulting in a smaller real-world FOV. This limitation can affect applications where situation awareness is necessary since the users need to perform larger head movements when scanning the environment (Rolland et al., 1995).

NeuAR – A Review of the VR/AR Applications in the Neuroscience Domain 137

Besides the regular HMDs, there are also available head-mounted projective displays (HMPD) and projective HMD. The HMPD use a mirror beam-splitter to project the synthetic images onto retro-reflective surfaces (Hua et al., 2005), while the projective HMD beam the virtual overlay onto the ceiling and then integrate those images onto the users' visual field

In comparison to the regular HMDs, these provide a wider FOV and prevent disorientation resulting from viewpoint information mismatch. However, they present specific limitations regarding the synthetic objects' brightness. The HMPD require special display surfaces in order to display the objects with an adequate level of brightness, while on projective HMD it

The handheld displays are the best solution for mobile applications since they integrate on a single device the graphics, display and interaction technology which support unrestrained handling (Bimber et al., 2006). It supports video/optical and projective technologies. Although, the video method is preferred, there are also optical devices, such as the real time tomographic reflection of Stetten and colleagues (2001) or projection handheld solutions as

Since handheld devices use common technologies such as Tablet PCs, PDAs and mobile phones, its main advantages are related with the mass diffusion of AR technology, low production costs and ease of use applications (Van Krevelen et al., 2010). The use of such ordinary technologies has also its shortcomings (Bimber et al., 2006). Low-end devices cannot provide enough processing power for AR applications resulting in system delay and very low frame rates in addition to limited image quality resultant from their integrated cameras. The screen-size may also limit the FOV. However this effect may be counteracted by the occurrence of a perception effect known as Parks Effect. In a nutshell, when moving a display over a stationary scene, the virtual display actual size becomes larger than its physical size because of the persistence of the image in the retina (Parks, 1965). In comparison to HMD, these types of devices don't allow a complete hands-free experience.

The spatial displays are positioned on fixed places in the environment and therefore are completely detached from the user. It supports optical, video and projective technologies respectively, screen-based video see-through display, spatial optical see-through display and projection-based spatial displays. The screen-based video see-through displays are the most cost-effective AR technology. They are similar to the video see-through HMDs but instead of presenting the images on a head-attached device, it uses a regular computer monitor. This simple setup has several limitations. It provides a small FOV, because it depends on the screen size and a low resolution of the real environment since it needs to be adapted to the system specifications. It does not support direct interaction with the environment being more a system for remote viewing than a proper see-through technology

According to the same authors, the spatial optical see-through displays use a diverse range of optical combiners (planar or curves mirror beam splitters, transparent screens and optical holograms) in order to generate images aligned within the real environment. Besides the

using two half-silvered mirrors (Kijima & Ojika, 1997).

demonstrated by Bimber and colleagues (2000).

**2.1.2.2 Handheld displays** 

**2.1.2.3 Spatial displays** 

(Bimber et al., 2006).

depends on the environmental light conditions (Bimber et al., 2006).

#### **2.1.1.2 Virtual retinal display**

In addition to OST and VST technologies, more recent methods are being developed. One method is called Virtual Retinal Display (VRT) (Pryor et al., 1998; Kollin, 1993; Lewis, 2004). Although being analogous to the OST display, since it preserves the real world view, it is a screen free method. The virtual overlay is drawn directly on the retina using low-power lasers discarding the need of a screen (Kollin, 1993).

Although being still in development, the VRT shows promising advantages. According to Kollin (1993), these are low-profile portable displays that allow also wider FOVs. Since its technology its independent of pixel size, it allows a much brighter and higher resolution virtual overlays (Van Krevelen et al., 2010). However, it still presents some drawbacks. Most systems are monocular and monochromatic (red), do not support stereoscopic vision and provide fixed focal length (Bimber & Raskar, 2006). However, Schowengerdt and colleagues (2004) are developing a full colour, low-cost, light-weight binocular version with dynamic refocus.

#### **2.1.1.3 Projective display**

There is also the possibility to use projective technology, which consists in using virtual overlays being directly projected onto real objects instead of being presented on a plane or surface on the users' line of sight (Van Krevelen et al., 2010).

The main advantages of this technology are related with the absence of special eye-wear in order to see the virtual overlay. Projecting the synthetic information directly on the real environment decreases the incongruity of accommodation and convergence usually present on the other methods and also allows for a wider FOV (Bimber et al., 2006). However, it lacks on providing a reasonable occlusion effect and it is restricted to indoor use, since the projected images have low brightness and contrast (Van Krevelen et al., 2010).

## **2.1.2 Visual display positioning**

In addition to the display technology it is fundamental to decide about the display positioning: head-mounted displays (HMD); handheld displays; spatial displays. Each type of positioning has specific advantages and limitations and should be chosen in regard to the application requirements and the technologies supported.

#### **2.1.2.1 Head-Mounted Displays (HMDs)**

The HMDs require the user to wear the display connected to his/her head. This type of display supports the optical and video see-through technologies, the virtual retinal display (VRD) and the projection method.

Relatively to the optical/video see-through HMD and VRD, technology limitations aside, this type of display positioning also lacks in mobility since it requires to be connected to a laptop which battery life is rather limited (Van Krevelen et al., 2010). It is also quite difficult to find a balance between display quality and ergonomics, since most systems vary from high quality cumbersome displays to low quality ergonomic ones (Bimber et al., 2006). Another issue related to HMD is the incidence of simulator sickness during fast head movements (Patrick et al., 2000).

Besides the regular HMDs, there are also available head-mounted projective displays (HMPD) and projective HMD. The HMPD use a mirror beam-splitter to project the synthetic images onto retro-reflective surfaces (Hua et al., 2005), while the projective HMD beam the virtual overlay onto the ceiling and then integrate those images onto the users' visual field using two half-silvered mirrors (Kijima & Ojika, 1997).

In comparison to the regular HMDs, these provide a wider FOV and prevent disorientation resulting from viewpoint information mismatch. However, they present specific limitations regarding the synthetic objects' brightness. The HMPD require special display surfaces in order to display the objects with an adequate level of brightness, while on projective HMD it depends on the environmental light conditions (Bimber et al., 2006).

#### **2.1.2.2 Handheld displays**

136 Augmented Reality – Some Emerging Application Areas

In addition to OST and VST technologies, more recent methods are being developed. One method is called Virtual Retinal Display (VRT) (Pryor et al., 1998; Kollin, 1993; Lewis, 2004). Although being analogous to the OST display, since it preserves the real world view, it is a screen free method. The virtual overlay is drawn directly on the retina using low-power

Although being still in development, the VRT shows promising advantages. According to Kollin (1993), these are low-profile portable displays that allow also wider FOVs. Since its technology its independent of pixel size, it allows a much brighter and higher resolution virtual overlays (Van Krevelen et al., 2010). However, it still presents some drawbacks. Most systems are monocular and monochromatic (red), do not support stereoscopic vision and provide fixed focal length (Bimber & Raskar, 2006). However, Schowengerdt and colleagues (2004) are developing a full colour, low-cost, light-weight binocular version with dynamic

There is also the possibility to use projective technology, which consists in using virtual overlays being directly projected onto real objects instead of being presented on a plane or

The main advantages of this technology are related with the absence of special eye-wear in order to see the virtual overlay. Projecting the synthetic information directly on the real environment decreases the incongruity of accommodation and convergence usually present on the other methods and also allows for a wider FOV (Bimber et al., 2006). However, it lacks on providing a reasonable occlusion effect and it is restricted to indoor use, since the projected images have low brightness and contrast (Van Krevelen et al.,

In addition to the display technology it is fundamental to decide about the display positioning: head-mounted displays (HMD); handheld displays; spatial displays. Each type of positioning has specific advantages and limitations and should be chosen in regard to the

The HMDs require the user to wear the display connected to his/her head. This type of display supports the optical and video see-through technologies, the virtual retinal display

Relatively to the optical/video see-through HMD and VRD, technology limitations aside, this type of display positioning also lacks in mobility since it requires to be connected to a laptop which battery life is rather limited (Van Krevelen et al., 2010). It is also quite difficult to find a balance between display quality and ergonomics, since most systems vary from high quality cumbersome displays to low quality ergonomic ones (Bimber et al., 2006). Another issue related to HMD is the incidence of simulator sickness during fast head

**2.1.1.2 Virtual retinal display** 

refocus.

2010).

**2.1.1.3 Projective display** 

**2.1.2 Visual display positioning** 

(VRD) and the projection method.

movements (Patrick et al., 2000).

**2.1.2.1 Head-Mounted Displays (HMDs)** 

lasers discarding the need of a screen (Kollin, 1993).

surface on the users' line of sight (Van Krevelen et al., 2010).

application requirements and the technologies supported.

The handheld displays are the best solution for mobile applications since they integrate on a single device the graphics, display and interaction technology which support unrestrained handling (Bimber et al., 2006). It supports video/optical and projective technologies. Although, the video method is preferred, there are also optical devices, such as the real time tomographic reflection of Stetten and colleagues (2001) or projection handheld solutions as demonstrated by Bimber and colleagues (2000).

Since handheld devices use common technologies such as Tablet PCs, PDAs and mobile phones, its main advantages are related with the mass diffusion of AR technology, low production costs and ease of use applications (Van Krevelen et al., 2010). The use of such ordinary technologies has also its shortcomings (Bimber et al., 2006). Low-end devices cannot provide enough processing power for AR applications resulting in system delay and very low frame rates in addition to limited image quality resultant from their integrated cameras. The screen-size may also limit the FOV. However this effect may be counteracted by the occurrence of a perception effect known as Parks Effect. In a nutshell, when moving a display over a stationary scene, the virtual display actual size becomes larger than its physical size because of the persistence of the image in the retina (Parks, 1965). In comparison to HMD, these types of devices don't allow a complete hands-free experience.

#### **2.1.2.3 Spatial displays**

The spatial displays are positioned on fixed places in the environment and therefore are completely detached from the user. It supports optical, video and projective technologies respectively, screen-based video see-through display, spatial optical see-through display and projection-based spatial displays. The screen-based video see-through displays are the most cost-effective AR technology. They are similar to the video see-through HMDs but instead of presenting the images on a head-attached device, it uses a regular computer monitor. This simple setup has several limitations. It provides a small FOV, because it depends on the screen size and a low resolution of the real environment since it needs to be adapted to the system specifications. It does not support direct interaction with the environment being more a system for remote viewing than a proper see-through technology (Bimber et al., 2006).

According to the same authors, the spatial optical see-through displays use a diverse range of optical combiners (planar or curves mirror beam splitters, transparent screens and optical holograms) in order to generate images aligned within the real environment. Besides the

NeuAR – A Review of the VR/AR Applications in the Neuroscience Domain 139

based on the variation of the angles in mechanical linkages. This type of tracking provides a good accuracy and precision, update rate and lag. However, in order to achieve such

The magnetic sensors measure orientation and position using magnetic fields to obtain the distance between emitters and receivers. These type of tracking is not sensitive to physical obstacles, provides a great update rate, low lag, is inexpensive and small. However, it works on small areas and is quite sensitive to electromagnetic noise and metallic objects. Most of these tracking systems are dominant in virtual reality environments. However, in AR, studies using only sensor-based tracking techniques are rare (Zhou et al., 2008). In order to achieve the necessary tracking precision on AR environments, most setups tend to combine

In comparison to sensor-based tracking, the vision-based methods are more accurate since provide dynamic correction of tracking errors (Bajura et al., 1995) and the possibility for a pixel-perfect registration of virtual objects (Van Krevelen et al., 2010). Vision-based tracking techniques may be divided into an earlier fiducial-based approach and more recent modelbased and feature-based techniques (Pressigout & Marchand, 2006). The fiducial-based approach places recognizable artificial markers or LEDs (light emitting diodes) on the scene in order to compute in real time the position and orientation of the camera. These fiducial markers are placed in known locations and have certain properties (shape or colour) that allow the camera to easily recognize and extract them from a video frame. By identifying exactly the position of the markers, it is also possible to perfectly align the virtual and real

The fiducials have the advantage of being cheap, customizable for greater efficiency (identification and extraction by the camera) and can be place arbitrarily on the scene (Park, You & Neumman, 1998). Even though this approach is quite efficient in small and prepared environments, it is not useful when considering large environments or even multiple

The feature-based approach is based on identifying 2D natural features (points, lines, edges or textures) in the environment in order to provide a robust and markerless vision-based tracking (Pressigout et al., 2006). The system is able to detect natural features on each image frame and to achieve the correspondence through images using a feature tracking algorithm. Therefore the image coordinates and their estimated 3D positions may be used to track the camera position in space (Park et al., 1998). This approach is quite sensitive to

The model-based approach instead of using 2D natural features of the environment is based on a model constructed of the features of the tracked object. This can be a CAD (computer assisted design) model or a 2D template built from the indistinguishable features of the target object (Zhou et al., 2008). This method is considered more robust then the featurebased. This approach may adopt an edge-based or a texture-based method to construct the feature model. The edge-based method is more commonly used since edges are easier to identify and quite robust to light changes. The texture-based method is usually applied as a complementary method in order to reinforce the tracking accuracy (Zhou et al., 2008).

instances of the same setting. In this way, markerless tracking approaches emerged.

measurements there is a great downside in terms of movement freedom.

the use of sensors with the vision-based tracking techniques (Pinz et al., 2002).

**2.2.2 Vision-based techniques** 

objects on the scene enhancing the level of registration.

changes in illumination (Pressigout et al., 2006).

optical technology limitations this kind of systems are not appropriated for mobile applications and the applied optics restrict the number of simultaneous users.

For Bimber and colleagues (2006), projection-based spatial displays use front projection to display the virtual overlay directly on the physical objects. This technique presents several limitations. On the one hand the front projection method limits the interaction since the interacting users and other physical objects may cast shadows on the display. There are also restrictions on the display area. Since the synthetic images are projected directly onto the physical objects, their surfaces become the display and therefore it is constrained to their size, shape and colour.

#### **2.2 Tracking system**

The other fundamental component, maybe even more important than the visual display, is the tracking system. Without tracking the system cannot know what, when or where to display the virtual overlay. In order to correctly present the synthetic information, the system must acknowledge the position and relative movement of the users' viewpoint in the real world, so that virtual objects may appear exactly where they should be. There is a vital relationship between the tracking systems and the level of registration (Van Krevelen et al., 2010).

However, to this day, a perfect single solution is still missing and so several approaches and possible combinations are still in study. Currently, two main categories of tracking may be defined: sensor-based and vision-based tracking techniques (Zhou et al., 2008).

#### **2.2.1 Sensor-based techniques**

Sensor-based techniques acquire tracking information from a diverse range of sensors such as ultrasonic, optical, inertial, mechanical and magnetic. Each of these sensors present advantages and limitations as demonstrated in Rolland and colleagues (2001).

The ultrasonic sensors measure movement and orientation through acoustic pulse propagation. Essentially, these sensors are able to acquire the distance between emitter/receivers attached to reference positions and a moving target, by measuring the time of propagation of pulsed signals between those features. As upsides, these sensors are small, light and with no distortion. As downsides they provide low update rate, are sensitive to environment conditions (temperature, pressure, humidity) and to physical obstacles. Optical sensors are able to track position and orientation using cameras to acquire the shape of the target features. In contrast to the ultrasonic sensors, these provide a good update rate, but are sensitive to optical noise, spurious light, ambiguity of surface and physical obstacles.

The principle of the inertial sensors is based on the inertia principle. Any physical object tends to resist to a possible change in its state of motion or rest. Measuring the variation between an initial and final position/rotation it is possible to determine the movement of the target feature. Gyroscopes are used to measure orientation and accelerometers to measure position. The main advantage of these methods is the absence of a reference point while its main limitation is related with an increase in error with time due to relative measurements (measurements are relative to the previous ones). Mechanical sensors are based on the variation of the angles in mechanical linkages. This type of tracking provides a good accuracy and precision, update rate and lag. However, in order to achieve such measurements there is a great downside in terms of movement freedom.

The magnetic sensors measure orientation and position using magnetic fields to obtain the distance between emitters and receivers. These type of tracking is not sensitive to physical obstacles, provides a great update rate, low lag, is inexpensive and small. However, it works on small areas and is quite sensitive to electromagnetic noise and metallic objects. Most of these tracking systems are dominant in virtual reality environments. However, in AR, studies using only sensor-based tracking techniques are rare (Zhou et al., 2008). In order to achieve the necessary tracking precision on AR environments, most setups tend to combine the use of sensors with the vision-based tracking techniques (Pinz et al., 2002).

## **2.2.2 Vision-based techniques**

138 Augmented Reality – Some Emerging Application Areas

optical technology limitations this kind of systems are not appropriated for mobile

For Bimber and colleagues (2006), projection-based spatial displays use front projection to display the virtual overlay directly on the physical objects. This technique presents several limitations. On the one hand the front projection method limits the interaction since the interacting users and other physical objects may cast shadows on the display. There are also restrictions on the display area. Since the synthetic images are projected directly onto the physical objects, their surfaces become the display and therefore it is constrained to their

The other fundamental component, maybe even more important than the visual display, is the tracking system. Without tracking the system cannot know what, when or where to display the virtual overlay. In order to correctly present the synthetic information, the system must acknowledge the position and relative movement of the users' viewpoint in the real world, so that virtual objects may appear exactly where they should be. There is a vital relationship between the tracking systems and the level of registration (Van Krevelen et al.,

However, to this day, a perfect single solution is still missing and so several approaches and possible combinations are still in study. Currently, two main categories of tracking may be

Sensor-based techniques acquire tracking information from a diverse range of sensors such as ultrasonic, optical, inertial, mechanical and magnetic. Each of these sensors present

The ultrasonic sensors measure movement and orientation through acoustic pulse propagation. Essentially, these sensors are able to acquire the distance between emitter/receivers attached to reference positions and a moving target, by measuring the time of propagation of pulsed signals between those features. As upsides, these sensors are small, light and with no distortion. As downsides they provide low update rate, are sensitive to environment conditions (temperature, pressure, humidity) and to physical obstacles. Optical sensors are able to track position and orientation using cameras to acquire the shape of the target features. In contrast to the ultrasonic sensors, these provide a good update rate, but are sensitive to optical noise, spurious light, ambiguity of surface and

The principle of the inertial sensors is based on the inertia principle. Any physical object tends to resist to a possible change in its state of motion or rest. Measuring the variation between an initial and final position/rotation it is possible to determine the movement of the target feature. Gyroscopes are used to measure orientation and accelerometers to measure position. The main advantage of these methods is the absence of a reference point while its main limitation is related with an increase in error with time due to relative measurements (measurements are relative to the previous ones). Mechanical sensors are

defined: sensor-based and vision-based tracking techniques (Zhou et al., 2008).

advantages and limitations as demonstrated in Rolland and colleagues (2001).

applications and the applied optics restrict the number of simultaneous users.

size, shape and colour.

**2.2 Tracking system** 

**2.2.1 Sensor-based techniques** 

physical obstacles.

2010).

In comparison to sensor-based tracking, the vision-based methods are more accurate since provide dynamic correction of tracking errors (Bajura et al., 1995) and the possibility for a pixel-perfect registration of virtual objects (Van Krevelen et al., 2010). Vision-based tracking techniques may be divided into an earlier fiducial-based approach and more recent modelbased and feature-based techniques (Pressigout & Marchand, 2006). The fiducial-based approach places recognizable artificial markers or LEDs (light emitting diodes) on the scene in order to compute in real time the position and orientation of the camera. These fiducial markers are placed in known locations and have certain properties (shape or colour) that allow the camera to easily recognize and extract them from a video frame. By identifying exactly the position of the markers, it is also possible to perfectly align the virtual and real objects on the scene enhancing the level of registration.

The fiducials have the advantage of being cheap, customizable for greater efficiency (identification and extraction by the camera) and can be place arbitrarily on the scene (Park, You & Neumman, 1998). Even though this approach is quite efficient in small and prepared environments, it is not useful when considering large environments or even multiple instances of the same setting. In this way, markerless tracking approaches emerged.

The feature-based approach is based on identifying 2D natural features (points, lines, edges or textures) in the environment in order to provide a robust and markerless vision-based tracking (Pressigout et al., 2006). The system is able to detect natural features on each image frame and to achieve the correspondence through images using a feature tracking algorithm. Therefore the image coordinates and their estimated 3D positions may be used to track the camera position in space (Park et al., 1998). This approach is quite sensitive to changes in illumination (Pressigout et al., 2006).

The model-based approach instead of using 2D natural features of the environment is based on a model constructed of the features of the tracked object. This can be a CAD (computer assisted design) model or a 2D template built from the indistinguishable features of the target object (Zhou et al., 2008). This method is considered more robust then the featurebased. This approach may adopt an edge-based or a texture-based method to construct the feature model. The edge-based method is more commonly used since edges are easier to identify and quite robust to light changes. The texture-based method is usually applied as a complementary method in order to reinforce the tracking accuracy (Zhou et al., 2008).

NeuAR – A Review of the VR/AR Applications in the Neuroscience Domain 141

relevant and helpful information to the real scene. Although VR and AR share and present some advantages over traditional approaches (e.g. improving acceptance and therapy duration), AR in some cases also presents additional advantages over VR. (Botella et al., 2004). First, in VRET is expensive to create different areas of high level of detail (LOD). Second in VRET one can include for instance avatars that simulate patients' bodies; however patients cannot see their own body (arms, hands, etc.) as can be seen in augmented reality exposure therapy (ARET). Third, animated avatars with close-to-real artificial intelligence are difficult to find. On the other hand, with ARET a delicate issue arouses. The integration of real and virtual elements should fit perfectly and remain during the entire length of exposure. Otherwise when an error is perceived, patients will not get the sense that the two

In ARET patient sees the real world ''augmented" by virtual elements, which means that, AR attempts to improve the reality and not to replace it (Azuma, 1997). The basis of ARET is that the virtual elements add information to the physical details of the real world. For instance, a therapist can present certain information by imposing virtual stimuli (such as personalized threatening snake) over real objects and environments. In ARET, the patient can see images that are merged in both real and virtual elements. Whereas in VRET the patient is in a totally artificial environment, in AR patients are *de factum* in a real world, with the essential difference that virtual elements are fused with real ones in a composite image

AR applications are already available in the areas of education (Arvanitis et al., 2007; Kerawalla et al., 2006; Squire & Klopfer, 2007; Squire & Mingfong, 2007) and medicine (Wörn et al., 2005). In the domain of psychotherapy, however, there are not many studies around. The ones that were conducted confirmed the benefits of ARET in the treatment of specific animal phobias, namely cockroaches (Botella et al., 2005; Botella et al., 2010) and

In cockroach phobia, Botella and colleagues (2005) conducted a one-session ARET, following the guidelines developed by Öst (1989). In a more recent study, ARET was applied in the short and long term (three-, six- and twelve-month follow-up) using a multiple baseline design across individuals (Botella et al., 2010). In both studies ARET was capable of inducing fear and all the participants showed an improvement on the outcome measures in the post-treatment assessment (less fear and less avoidance). In addition, the results were maintained at follow-up periods. In the study of acrophobia (Juan et al., 2005), ARET was conducted using immersive pictures (180º view) with encouraging results. In these studies, ARET induced high sense of presence probably due to a hyperrealist merged context, leading to a higher attention and subsequent fear encapsulation (Hamm & Weike, 2005).

Given that ARET may lead to high sense of presence the emotional processing of the phobiarelated information is facilitated and the access to the patient's fear memory structure promoted (Foa & Kozak, 1986). Under this view, the higher level of presence, better the

Both the three studies demonstrate how effective ARET is and can be a motivating factor to develop applications not only on specific phobias, but other mental illnesses as well. ARET which is in its infancy when it comes to psychotherapeutic applications may spark a change of paradigm, not only in the way how ET is conducted, but also in the therapeutic project

spiders and in the treatment of acrophobia (Juan et al, 2005).

itself, being a new challenge for future clinical applications.

worlds blend into one, decreasing the sense of being there (Milgram et al., 1994).

(Milgram et al., 1994).

therapeutic results are.

According to Van Krevelen and colleagues (2010), despite being necessary more robustness and lower computational costs on these methods, they demonstrate very promising results.

## **3. AR applications on psychotherapy**

#### **3.1 Traditional therapeutic approaches**

Surprisingly or not, many people still think that mental illnesses are not treatable, being the main reason for not seeking mental health treatment (Sussman et al., 1987). A large *palette* of efficacious treatments is available to ameliorate symptoms. In fact, for most mental disorders, there is generally not just one but a range of treatments of proven efficacy. Most treatments fall under two general categories, non-pharmacological and pharmacological (Gazzaniga & Heatherton, 2006).

Among non-pharmacological treatments, cognitive–behavioural therapy (CTB) is the gold standard. CBT seeks to change faulty biased cognitions and replace them with thoughts and self-statements that promote adaptive behavior (Beck et al., 1976). For instance, CBT tries to replace self-defeatist expectations ("I can't do anything right") with positive expectations ("I can do this right"). CBT has gained such ascendancy as a means of integrating cognitive and behavioral views of human functioning, being empirically validated and a common approach in anxiety disorders (Hofmann & Smits, 2008), mood disorders (Gloaguen et al., 1998) and schizophrenia (Wykes et al., 2007).

Under CBT approach, exposure therapy (ET) is the most common psychotherapeutic technique for the treatment of anxiety disorders (Foa et al., 2000). Particularly for phobias and PTSD, ET is an effective therapeutic technique which involves the exposure to the feared stimulus or context without any danger while the psychotherapist helps patients relieving their anxiety (e.g. Cooper et al., 2008; Rothbaum & Schwartz, 2002).

#### **3.2 Psychotherapy powered by technology: AR as a 3T (Therapeutic Technological Tool)**

As seen on the introductory section the advance of technology brought about new approaches and new computational applications. One of those is virtual reality exposure therapy (VRET). This human–computer interaction system is a medium in which patients can be immersed within a virtual anxiogenic environment where the fear structure is effectively elicited and the emotional processing of fears fired-up (Rothbaum et al., 1995). In VRET patients are immersed within a computer-generated simulation or virtual environment, bypassing, as previously mentioned, some limitations of imagination and *in vivo* exposure (e.g., the risks of distressing patients). VRET is a better-quality technique to control potential distracters and cognitive avoidance to threatening stimuli when imagination exposure or *in vivo* exposure is compromised (Vincelli & Riva, 2002). In VRET, cues of events which are not replicable in real-life situations can be reproduced *ad infinitum* in the therapist' room (Gamito et al., 2010, 2011a). When a patient is immersed in a synthetic world, he/she can be systematically exposed to specific feared stimuli integrated in a relevant context.

In AR, patients see an image made up of a real image and virtual elements that are superimposed over it. The most relevant aspect in AR is that the virtual elements add 140 Augmented Reality – Some Emerging Application Areas

According to Van Krevelen and colleagues (2010), despite being necessary more robustness and lower computational costs on these methods, they demonstrate very promising results.

Surprisingly or not, many people still think that mental illnesses are not treatable, being the main reason for not seeking mental health treatment (Sussman et al., 1987). A large *palette* of efficacious treatments is available to ameliorate symptoms. In fact, for most mental disorders, there is generally not just one but a range of treatments of proven efficacy. Most treatments fall under two general categories, non-pharmacological and pharmacological

Among non-pharmacological treatments, cognitive–behavioural therapy (CTB) is the gold standard. CBT seeks to change faulty biased cognitions and replace them with thoughts and self-statements that promote adaptive behavior (Beck et al., 1976). For instance, CBT tries to replace self-defeatist expectations ("I can't do anything right") with positive expectations ("I can do this right"). CBT has gained such ascendancy as a means of integrating cognitive and behavioral views of human functioning, being empirically validated and a common approach in anxiety disorders (Hofmann & Smits, 2008), mood disorders (Gloaguen et al.,

Under CBT approach, exposure therapy (ET) is the most common psychotherapeutic technique for the treatment of anxiety disorders (Foa et al., 2000). Particularly for phobias and PTSD, ET is an effective therapeutic technique which involves the exposure to the feared stimulus or context without any danger while the psychotherapist helps patients

**3.2 Psychotherapy powered by technology: AR as a 3T (Therapeutic Technological** 

As seen on the introductory section the advance of technology brought about new approaches and new computational applications. One of those is virtual reality exposure therapy (VRET). This human–computer interaction system is a medium in which patients can be immersed within a virtual anxiogenic environment where the fear structure is effectively elicited and the emotional processing of fears fired-up (Rothbaum et al., 1995). In VRET patients are immersed within a computer-generated simulation or virtual environment, bypassing, as previously mentioned, some limitations of imagination and *in vivo* exposure (e.g., the risks of distressing patients). VRET is a better-quality technique to control potential distracters and cognitive avoidance to threatening stimuli when imagination exposure or *in vivo* exposure is compromised (Vincelli & Riva, 2002). In VRET, cues of events which are not replicable in real-life situations can be reproduced *ad infinitum* in the therapist' room (Gamito et al., 2010, 2011a). When a patient is immersed in a synthetic world, he/she can be systematically exposed to specific feared stimuli integrated in a

In AR, patients see an image made up of a real image and virtual elements that are superimposed over it. The most relevant aspect in AR is that the virtual elements add

relieving their anxiety (e.g. Cooper et al., 2008; Rothbaum & Schwartz, 2002).

**3. AR applications on psychotherapy 3.1 Traditional therapeutic approaches** 

1998) and schizophrenia (Wykes et al., 2007).

(Gazzaniga & Heatherton, 2006).

**Tool)** 

relevant context.

relevant and helpful information to the real scene. Although VR and AR share and present some advantages over traditional approaches (e.g. improving acceptance and therapy duration), AR in some cases also presents additional advantages over VR. (Botella et al., 2004). First, in VRET is expensive to create different areas of high level of detail (LOD). Second in VRET one can include for instance avatars that simulate patients' bodies; however patients cannot see their own body (arms, hands, etc.) as can be seen in augmented reality exposure therapy (ARET). Third, animated avatars with close-to-real artificial intelligence are difficult to find. On the other hand, with ARET a delicate issue arouses. The integration of real and virtual elements should fit perfectly and remain during the entire length of exposure. Otherwise when an error is perceived, patients will not get the sense that the two worlds blend into one, decreasing the sense of being there (Milgram et al., 1994).

In ARET patient sees the real world ''augmented" by virtual elements, which means that, AR attempts to improve the reality and not to replace it (Azuma, 1997). The basis of ARET is that the virtual elements add information to the physical details of the real world. For instance, a therapist can present certain information by imposing virtual stimuli (such as personalized threatening snake) over real objects and environments. In ARET, the patient can see images that are merged in both real and virtual elements. Whereas in VRET the patient is in a totally artificial environment, in AR patients are *de factum* in a real world, with the essential difference that virtual elements are fused with real ones in a composite image (Milgram et al., 1994).

AR applications are already available in the areas of education (Arvanitis et al., 2007; Kerawalla et al., 2006; Squire & Klopfer, 2007; Squire & Mingfong, 2007) and medicine (Wörn et al., 2005). In the domain of psychotherapy, however, there are not many studies around. The ones that were conducted confirmed the benefits of ARET in the treatment of specific animal phobias, namely cockroaches (Botella et al., 2005; Botella et al., 2010) and spiders and in the treatment of acrophobia (Juan et al, 2005).

In cockroach phobia, Botella and colleagues (2005) conducted a one-session ARET, following the guidelines developed by Öst (1989). In a more recent study, ARET was applied in the short and long term (three-, six- and twelve-month follow-up) using a multiple baseline design across individuals (Botella et al., 2010). In both studies ARET was capable of inducing fear and all the participants showed an improvement on the outcome measures in the post-treatment assessment (less fear and less avoidance). In addition, the results were maintained at follow-up periods. In the study of acrophobia (Juan et al., 2005), ARET was conducted using immersive pictures (180º view) with encouraging results. In these studies, ARET induced high sense of presence probably due to a hyperrealist merged context, leading to a higher attention and subsequent fear encapsulation (Hamm & Weike, 2005).

Given that ARET may lead to high sense of presence the emotional processing of the phobiarelated information is facilitated and the access to the patient's fear memory structure promoted (Foa & Kozak, 1986). Under this view, the higher level of presence, better the therapeutic results are.

Both the three studies demonstrate how effective ARET is and can be a motivating factor to develop applications not only on specific phobias, but other mental illnesses as well. ARET which is in its infancy when it comes to psychotherapeutic applications may spark a change of paradigm, not only in the way how ET is conducted, but also in the therapeutic project itself, being a new challenge for future clinical applications.

NeuAR – A Review of the VR/AR Applications in the Neuroscience Domain 143

Survivors of acquired brain injuries live with minor to severe functional impairments (Merians et al., 2002). These deficits, such as loss of range motion in upper or lower limbs along with lack of organization and motor planning are associated to decreased autonomy and independence on activities of daily living. Occupational Therapy can be applied to patients with upper or lower limb disabilities in order to promote their functional ability. Traditional occupational sessions are carried out in rehabilitation centers where the patients are instructed on how to manage basic motor skills (Alamri et al., 2010). Repetitive practice is considered to be helpful for effective therapy, even after discharge from the rehabilitation's hospital. However, the vast majority of the patients with brain injury are not able to travel to rehabilitation centers located essentially in urban areas for maintenance sessions. The contribution of the new information technologies by means of using VR settings for neuro-rehabilitation could offer opportunities for neuropsychological rehabilitation. On one hand, the use of on-line virtual environments as a form of telerehabilitation may increase the accessibility to training environments, enabling home training for patients that are far from the rehabilitation centre (Gamito et al., 2011b). On the other hand, Correa and colleagues (2006), suggest that novel VR applications in a form of an augmented reality (AR) system could offer new possibilities for motor and cognitive training of patients with acquired brain injury. For Leitener and colleagues (2007) the use of AR in rehabilitation allows patients to touch and move the objects in a natural way and without the use of electronic input devices (e.g., mouse, keyboard or gamepads), which may improve interaction and the sense of presence when performing the predetermined tasks. Luo and colleagues (2005), consider that one advantage of AR over VR is that disabled patients following stroke are less disoriented when performing the exercises in AR than in

There is increasing interest in the use of AR/VR technology in motor and cognitive rehabilitation (Riva, 2005). The use of interactive AR/VR environments may also help the transfer of the learned skills during training. Although, the transfer process of skills from virtual to real worlds are poorly understood, rehabilitation paradigms using AR/VR techniques should be based on previous assumptions of neuroplasticity, that effective rehabilitation is achieved mainly through repetition, rewarding or reinforcing adaptive skill acquisition. The AR/VR environments for rehabilitation offer the opportunity to include naturalistic challenges that are important for adjustment in real-world activities (Rizzo et al., 2004). The use of this technology in rehabilitation has the advantage of simulating the learning of real tasks in a controlled reality, where training repetition and intensity can be gradually increased in function of patients' achievements. In addition, the visual correspondence between motor or cognitive exercises in AR training allow real time feedback of performance, providing well suited and personalised applications for function

For Edmans and colleagues (2006), an important question is whether the difficulties that affect a task in the real world are similar to those in the virtual world or whether the errors committed in the virtual world are the same of an analogous task in real world.

**4.2 Neuro-rehabilitation using AR technology** 

immersive VR environments.

based training.

**4.2.1 Motor rehabilitation** 

## **4. AR applications on neuro-rehabilitation**

## **4.1 Principles of neuro-rehabilitation**

The consequences of acquired brain injury (ABI) can be very severe and depending on the etiology and distribution, the effects are seen immediately after brain injury or at long term as a result of metabolic disturbances of the primary neural damage (Sohlberg & Mateer, 2001). The etiology of brain injury varies from infectious (e.g., encephalitis) and degenerative diseases (e.g., Alzheimer's) to brain tumors, stroke or traumatic brain injury. The nature of the neurological disease determines specific patterns of disability, being associated with different syndromes of impaired physical, cognitive, behavioral and emotional domains.

In agreement with Wilson (2003), neuropsychological rehabilitation can be defined as a set of techniques to restore and/or compensate for acquired physical or intellectual disability. The techniques used for physical and functional rehabilitation are aimed to assist and promote the natural recovery process, decreasing the development of maladaptive patterns (e.g. disrupting behaviors) and implementing physical, pharmacological, cognitive and behavioral interventions to facilitate the functional recovery of these patients.

ABI may result in motor and/or cognitive impairment. In this context, neuropsychological rehabilitation can be classified into two broad categories, motor and cognitive rehabilitation. Motor rehabilitation plans rely on the assumption that flexion and extension exercises are important to enhance muscle functioning, while cognitive rehabilitation approaches consider that training basic, instrumental or complex tasks of daily living will improve overall adjustment.

The scientific literature is more extensive regarding the neuropsychological interventions for stroke or traumatic brain injuries (TBI). Previous work from Sohlberg and Mateer (1989) suggest that early interventions after severe brain injury are directed essentially to environmental management to control the level of stimulation provided to these patients. During spontaneous recovery, the first signs of change are shown by involuntary responses to environmental stimuli, where cognitive skills such as self-orientation and memory are being partially recovered. According to these authors, this phase is the focus of rehabilitation, with emphasis on training in self-care activities, usually involving motor training to work muscle tone and postural control. Cognitive training is also applied during this stage, aimed at improving communication, attention and memory deficits (Sohlberg et al., 2001).

The conventional rehabilitation plans for motor and cognitive rehabilitation consider that repeated and massive practice of a predetermined function can actually affect neural reorganization, allowing for synaptic reconnectivity and neural reorganization (Butefisch et al., 1995). Previous work from Taub and colleagues (1999) suggests that motor recovery may be possible when training is used to stimulate a specific motor activity. These authors claim that even after damage to the central or peripheral nervous system, implicit information regarding motor schemes may persist in the central nervous system (CNS). In this way, the stimulation of the impaired motor functions benefit the functional reorganization of the CNS, in which, the intact neural systems may reorganize to achieve a given motor act.

#### **4.2 Neuro-rehabilitation using AR technology**

142 Augmented Reality – Some Emerging Application Areas

The consequences of acquired brain injury (ABI) can be very severe and depending on the etiology and distribution, the effects are seen immediately after brain injury or at long term as a result of metabolic disturbances of the primary neural damage (Sohlberg & Mateer, 2001). The etiology of brain injury varies from infectious (e.g., encephalitis) and degenerative diseases (e.g., Alzheimer's) to brain tumors, stroke or traumatic brain injury. The nature of the neurological disease determines specific patterns of disability, being associated with different syndromes of impaired physical, cognitive, behavioral and

In agreement with Wilson (2003), neuropsychological rehabilitation can be defined as a set of techniques to restore and/or compensate for acquired physical or intellectual disability. The techniques used for physical and functional rehabilitation are aimed to assist and promote the natural recovery process, decreasing the development of maladaptive patterns (e.g. disrupting behaviors) and implementing physical, pharmacological, cognitive and

ABI may result in motor and/or cognitive impairment. In this context, neuropsychological rehabilitation can be classified into two broad categories, motor and cognitive rehabilitation. Motor rehabilitation plans rely on the assumption that flexion and extension exercises are important to enhance muscle functioning, while cognitive rehabilitation approaches consider that training basic, instrumental or complex tasks of daily living will improve

The scientific literature is more extensive regarding the neuropsychological interventions for stroke or traumatic brain injuries (TBI). Previous work from Sohlberg and Mateer (1989) suggest that early interventions after severe brain injury are directed essentially to environmental management to control the level of stimulation provided to these patients. During spontaneous recovery, the first signs of change are shown by involuntary responses to environmental stimuli, where cognitive skills such as self-orientation and memory are being partially recovered. According to these authors, this phase is the focus of rehabilitation, with emphasis on training in self-care activities, usually involving motor training to work muscle tone and postural control. Cognitive training is also applied during this stage, aimed at improving communication, attention and memory deficits (Sohlberg et

The conventional rehabilitation plans for motor and cognitive rehabilitation consider that repeated and massive practice of a predetermined function can actually affect neural reorganization, allowing for synaptic reconnectivity and neural reorganization (Butefisch et al., 1995). Previous work from Taub and colleagues (1999) suggests that motor recovery may be possible when training is used to stimulate a specific motor activity. These authors claim that even after damage to the central or peripheral nervous system, implicit information regarding motor schemes may persist in the central nervous system (CNS). In this way, the stimulation of the impaired motor functions benefit the functional reorganization of the CNS, in which, the intact neural systems may reorganize to achieve

behavioral interventions to facilitate the functional recovery of these patients.

**4. AR applications on neuro-rehabilitation** 

**4.1 Principles of neuro-rehabilitation** 

emotional domains.

overall adjustment.

al., 2001).

a given motor act.

Survivors of acquired brain injuries live with minor to severe functional impairments (Merians et al., 2002). These deficits, such as loss of range motion in upper or lower limbs along with lack of organization and motor planning are associated to decreased autonomy and independence on activities of daily living. Occupational Therapy can be applied to patients with upper or lower limb disabilities in order to promote their functional ability. Traditional occupational sessions are carried out in rehabilitation centers where the patients are instructed on how to manage basic motor skills (Alamri et al., 2010). Repetitive practice is considered to be helpful for effective therapy, even after discharge from the rehabilitation's hospital. However, the vast majority of the patients with brain injury are not able to travel to rehabilitation centers located essentially in urban areas for maintenance sessions. The contribution of the new information technologies by means of using VR settings for neuro-rehabilitation could offer opportunities for neuropsychological rehabilitation. On one hand, the use of on-line virtual environments as a form of telerehabilitation may increase the accessibility to training environments, enabling home training for patients that are far from the rehabilitation centre (Gamito et al., 2011b). On the other hand, Correa and colleagues (2006), suggest that novel VR applications in a form of an augmented reality (AR) system could offer new possibilities for motor and cognitive training of patients with acquired brain injury. For Leitener and colleagues (2007) the use of AR in rehabilitation allows patients to touch and move the objects in a natural way and without the use of electronic input devices (e.g., mouse, keyboard or gamepads), which may improve interaction and the sense of presence when performing the predetermined tasks. Luo and colleagues (2005), consider that one advantage of AR over VR is that disabled patients following stroke are less disoriented when performing the exercises in AR than in immersive VR environments.

There is increasing interest in the use of AR/VR technology in motor and cognitive rehabilitation (Riva, 2005). The use of interactive AR/VR environments may also help the transfer of the learned skills during training. Although, the transfer process of skills from virtual to real worlds are poorly understood, rehabilitation paradigms using AR/VR techniques should be based on previous assumptions of neuroplasticity, that effective rehabilitation is achieved mainly through repetition, rewarding or reinforcing adaptive skill acquisition. The AR/VR environments for rehabilitation offer the opportunity to include naturalistic challenges that are important for adjustment in real-world activities (Rizzo et al., 2004). The use of this technology in rehabilitation has the advantage of simulating the learning of real tasks in a controlled reality, where training repetition and intensity can be gradually increased in function of patients' achievements. In addition, the visual correspondence between motor or cognitive exercises in AR training allow real time feedback of performance, providing well suited and personalised applications for function based training.

#### **4.2.1 Motor rehabilitation**

For Edmans and colleagues (2006), an important question is whether the difficulties that affect a task in the real world are similar to those in the virtual world or whether the errors committed in the virtual world are the same of an analogous task in real world.

NeuAR – A Review of the VR/AR Applications in the Neuroscience Domain 145

rehabilitation since these mechanisms require 3 axis based movements, similar to those

A separate literature defines two general approaches in cognitive rehabilitation, an intervention specifically targeted for rehabilitation and compensation of acquired cognitive deficits, and more recently the focus on a more global and holistic approach with a growing interest in other individual variables at personal, emotional and social levels and their relations to cognitive functioning (Sohlberg et al., 2001). Actually, the literature on this topic is not consistent on whether neuropsychological rehabilitation should focus on the process of cognitive training or the overall adjustment of ABI patients (Clare & Woods, 2004). Several forms of cognitive interventions are described in the literature, however the distinction between cognitive stimulation and training in some cases is unclear and rather confusing. Clare and colleagues (2004) describe cognitive stimulation as a form of cognitive intervention to maintain an adequate level of cognitive functioning when deficits are related to diffuse and progressive brain injury such as in degenerative brain diseases (e.g. Alzheimer). Cognitive stimulation aims at improving the patient's everyday living activities and seeks the management of its cognitive deficits, rather than the recovery of brain

When cognitive interventions refer to restoration or recovery of a specific function, neurorehabilitation should be considered as training since these are based in a cognitive retraining rationale as suggested earlier. The most frequent cases of cognitive training are related to memory, attention and executive functioning. These deficits are in most cases associated

There are other interesting systems in development as alternatives to cognitive recovery. An example is provided by Sandor and Klinker (2006) that are developing an AR system, termed as Mixed Reality Kitchen, to train organization and planning functions in activities of daily living (e.g., making breakfast). The authors studied a stroke patient in their own environment when performing specific routines. After a five-session training, a decreased time spent on task and decreased location errors were reported when transferring the

Nevertheless, one of the most interesting AR systems in cognitive training is the ARVe (Augmented Reality to Vegetal field) – (Richard et al., 2007). The ARVe is an educational environment to assist cognitive disabled children in decision making process. This system consists of a book with several sorts of virtual markers, each of them representing a type of vegetable (leaves, flowers, fruits and seeds). The main goal of this application is to match vegetable entities according to their functions shown on a reference page. The authors found that cognitive disabled children in the AR condition were more motivated to complete the

In sum, the studies reported here for motor and cognitive rehabilitation are promising and may encourage the use of AR/VR applications for function based training. In fact, as suggested before, the use of this technology has the key advantage of being an ecological valid application, where generalization or transfer of learned skills from virtual to real world may be

improved, augmenting functionality and overall adjustment in disabled patients.

with focal brain injury after a traumatic brain injury or acute stroke episodes.

performed in real world situations.

**4.2.2 Cognitive rehabilitation** 

function.

exercises to the real world.

exercises than other children in the control condition.

Furthermore, several experiments undertaken by Kosslyn and collaborators (2001) have suggested the mental imagery of movement may activate cortical regions involved in planning and execution of movements. These findings may also encourage the use of AR/VR systems in rehabilitation, specifically the use of AR environments that promote mental practice of a desired motor movement may stimulate the activation of wider neural networks.

Alamri and colleagues (2010) are developing an AR based Rehabilitation (AR-REHAB) system to provide motor training in activities of daily living. These authors used several virtual objects in a real kitchen setup, where the patients were able to interact with virtual and real objects. Preliminary results from fifteen healthy male participants are promising and support the use of AR applications in neuropsychological rehabilitation.

According to Kahn and colleagues (2001), motor training can be facilitated through the use of mechanical devices in an AR setup, such as haptic gloves or even body accelerometers. Luo and colleagues (2005), highlight that the use of AR/VR may be beneficial when combined with assistive devices for kinematics. For these authors, the combination of these different technologies in rehabilitation training provides new possibilities that are not possible in conventional rehabilitation programmes. They have developed a training environment that combines AR and assistive devices. This system comprised an assistive device to provide assistance for finger extension. The preliminary results of one case study showed user acceptance along with an improvement in finger extension of the impaired hand after 6 weeks of training.

Riess (1995) developed an AR system to decrease the maladaptive patterns of movement (akinesia) in Parkinson disease, by superimposing virtual images to the real world. This system was designed to compensate for paradoxical motion or kinesis paradoxa, which describes the incapacity for walk without the presence of visual cues. The author suggest that the use AR with visual cues can help the patients to start walking by themselves, however the results are unclear and should be more fully explored.

Baram and colleagues (2002) describe a similar system that combines an AR portable technology with body accelerometers, allowing the generation of a virtual tiled floor to provide a greater sense of reality. These authors tested this system in a sample of fourteen Parkinson patients and found that walking speed and stride length can be effectively manipulated through the use of virtual visual cues.

The AR/VR systems also incorporate game elements that increase motivation to participate in training plans. Commercial video games like Nintendo Wii or Sony PlayStation EyeToy are being used for motor recovery. Yavuzer and colleagues (2008) studied a small sample of disabled patients with the use of conventional therapy sessions combined with AR with PlayStation Eye-Toy. The results showed a significant enhancement of range motion of movements and satisfaction with training in the experimental group using AR in comparison to a control group without AR. Similar results were observed by Deutsch and colleagues (2009) using Wii technology.

In agreement with Kirner and colleagues (2007) the AR games allow an enhanced and wider environment that stimulates perception and spatial orientation. In addition, the new interaction systems developed by the video game industry can also be beneficial for rehabilitation since these mechanisms require 3 axis based movements, similar to those performed in real world situations.

#### **4.2.2 Cognitive rehabilitation**

144 Augmented Reality – Some Emerging Application Areas

Furthermore, several experiments undertaken by Kosslyn and collaborators (2001) have suggested the mental imagery of movement may activate cortical regions involved in planning and execution of movements. These findings may also encourage the use of AR/VR systems in rehabilitation, specifically the use of AR environments that promote mental practice of a desired motor movement may stimulate the activation of wider neural

Alamri and colleagues (2010) are developing an AR based Rehabilitation (AR-REHAB) system to provide motor training in activities of daily living. These authors used several virtual objects in a real kitchen setup, where the patients were able to interact with virtual and real objects. Preliminary results from fifteen healthy male participants are promising

According to Kahn and colleagues (2001), motor training can be facilitated through the use of mechanical devices in an AR setup, such as haptic gloves or even body accelerometers. Luo and colleagues (2005), highlight that the use of AR/VR may be beneficial when combined with assistive devices for kinematics. For these authors, the combination of these different technologies in rehabilitation training provides new possibilities that are not possible in conventional rehabilitation programmes. They have developed a training environment that combines AR and assistive devices. This system comprised an assistive device to provide assistance for finger extension. The preliminary results of one case study showed user acceptance along with an improvement in finger extension of the impaired

Riess (1995) developed an AR system to decrease the maladaptive patterns of movement (akinesia) in Parkinson disease, by superimposing virtual images to the real world. This system was designed to compensate for paradoxical motion or kinesis paradoxa, which describes the incapacity for walk without the presence of visual cues. The author suggest that the use AR with visual cues can help the patients to start walking by themselves,

Baram and colleagues (2002) describe a similar system that combines an AR portable technology with body accelerometers, allowing the generation of a virtual tiled floor to provide a greater sense of reality. These authors tested this system in a sample of fourteen Parkinson patients and found that walking speed and stride length can be effectively

The AR/VR systems also incorporate game elements that increase motivation to participate in training plans. Commercial video games like Nintendo Wii or Sony PlayStation EyeToy are being used for motor recovery. Yavuzer and colleagues (2008) studied a small sample of disabled patients with the use of conventional therapy sessions combined with AR with PlayStation Eye-Toy. The results showed a significant enhancement of range motion of movements and satisfaction with training in the experimental group using AR in comparison to a control group without AR. Similar results were observed by Deutsch and

In agreement with Kirner and colleagues (2007) the AR games allow an enhanced and wider environment that stimulates perception and spatial orientation. In addition, the new interaction systems developed by the video game industry can also be beneficial for

and support the use of AR applications in neuropsychological rehabilitation.

however the results are unclear and should be more fully explored.

manipulated through the use of virtual visual cues.

colleagues (2009) using Wii technology.

networks.

hand after 6 weeks of training.

A separate literature defines two general approaches in cognitive rehabilitation, an intervention specifically targeted for rehabilitation and compensation of acquired cognitive deficits, and more recently the focus on a more global and holistic approach with a growing interest in other individual variables at personal, emotional and social levels and their relations to cognitive functioning (Sohlberg et al., 2001). Actually, the literature on this topic is not consistent on whether neuropsychological rehabilitation should focus on the process of cognitive training or the overall adjustment of ABI patients (Clare & Woods, 2004). Several forms of cognitive interventions are described in the literature, however the distinction between cognitive stimulation and training in some cases is unclear and rather confusing. Clare and colleagues (2004) describe cognitive stimulation as a form of cognitive intervention to maintain an adequate level of cognitive functioning when deficits are related to diffuse and progressive brain injury such as in degenerative brain diseases (e.g. Alzheimer). Cognitive stimulation aims at improving the patient's everyday living activities and seeks the management of its cognitive deficits, rather than the recovery of brain function.

When cognitive interventions refer to restoration or recovery of a specific function, neurorehabilitation should be considered as training since these are based in a cognitive retraining rationale as suggested earlier. The most frequent cases of cognitive training are related to memory, attention and executive functioning. These deficits are in most cases associated with focal brain injury after a traumatic brain injury or acute stroke episodes.

There are other interesting systems in development as alternatives to cognitive recovery. An example is provided by Sandor and Klinker (2006) that are developing an AR system, termed as Mixed Reality Kitchen, to train organization and planning functions in activities of daily living (e.g., making breakfast). The authors studied a stroke patient in their own environment when performing specific routines. After a five-session training, a decreased time spent on task and decreased location errors were reported when transferring the exercises to the real world.

Nevertheless, one of the most interesting AR systems in cognitive training is the ARVe (Augmented Reality to Vegetal field) – (Richard et al., 2007). The ARVe is an educational environment to assist cognitive disabled children in decision making process. This system consists of a book with several sorts of virtual markers, each of them representing a type of vegetable (leaves, flowers, fruits and seeds). The main goal of this application is to match vegetable entities according to their functions shown on a reference page. The authors found that cognitive disabled children in the AR condition were more motivated to complete the exercises than other children in the control condition.

In sum, the studies reported here for motor and cognitive rehabilitation are promising and may encourage the use of AR/VR applications for function based training. In fact, as suggested before, the use of this technology has the key advantage of being an ecological valid application, where generalization or transfer of learned skills from virtual to real world may be improved, augmenting functionality and overall adjustment in disabled patients.

NeuAR – A Review of the VR/AR Applications in the Neuroscience Domain 147

interaction while AR is in use (Zhou et al., 2008). Current research shows, however, that there are still not enough rehab applications using such combination, even though it has

**5.2 Augmented Reality Exposure Therapy (ARET) contribution to psychotherapy and** 

Bearing in mind that some applications mentioned in this chapter concerns mental illness, ARET is only a slight improvement on more spread forms of VRET use. If a closer look is taken at phobia treatment using CBT with VRET, ARET's major contribution can be attributed to its ability to insert virtual objects in the real worlds which is an excellent substitute for *in vivo* exposure, since it also addresses ecological validity issues. Maybe it is wiser to insert virtual elements in the real world than to draw people entirely into the VR scenario. Moreover, ARET simplifies the need for world/scenario construction since it uses real places with superimposed objects, making it less time-consuming. Therefore, ARET seems to be able to be used as a coadjutant or as a substitute to VRET in CBT's approach

This is also true for another important field of application: rehabilitation. Since one of the key aspects of rehabilitation is repetition, VR/AR solutions may represent a clear path to swifter recoveries. AR also provides a more realistic environment where the individual can practice while being motivated by the insertion of virtual elements. Additionally, solutions where videogame consoles are being used can also serve as an added bonus since prices of off-the-shelf products are becoming more accessible. On the other hand, videogames bring in the fun of a game. Videogame based solutions also guarantee top notch CGI (computer graphic imaging) and, more importantly, the products that are market leaders all have motion detection hardware that can easily be used to rehabilitation. But some fences are still needed to be crossed as the available applications were designed for entertainment and

One aspect that is shared by both rehabilitation and mental illness fields of research when addressing AR is that this technology ensures a smoother transition from therapy to real life as it removes most mediated elements, ensuring a more proximal experience to real objects and situations. This is probably one of the most essential aspects of AR versus VR and constitutes a major opportunity for AR based applications. Knowledge transfer between mediated environments and real-life situations is still an issue and more research is needed

A collateral issue to the full and unrestricted use in both rehabilitation and mental health is the need to gain the support of more clinical practitioners. There is still some reserve from many mental health professionals about the use and the advantages that these technologybased solutions have to offer. For that reason, a bigger effort on education and results

For all the above mentioned reasons, AR solutions have a bright future ahead in deploying exciting and fruitful solutions for some serious issues. And with the continuous development of exciting new technological solutions, AR based solutions may be available

to dissipate all doubts about the efficacy of VR/AR in competence development.

dissemination is the way to get more support for these applications development.

been proved to be a cost-efficient response.

using exposure therapy.

must be adapted for these specific ends.

in every home in just a few years.

**neuro-rehabilitation: How to explore its full potential?** 

## **5. How much more can we augment reality? Future applications and present limitations**

#### **5.1 Augmented hardware – Future prospects and issues concerning psychotherapy and neuro-rehabilitation**

As has been discussed in this chapter, AR has a part to play in the neurosciences area, namely on psychotherapy and neuro-rehabilitation. This role derived from the technological evolution in the last decade. One of the clearest signs of this evolution rests upon two pillars that are usually related to any new technology that strolled from research labs into households' living-room: price and availability.

If it is true that prices have significantly decreased in most of the equipments needed to deploy AR applications to clinicians, therapists and other caregivers, it is also clear that some of these technologies will never have a large audience since their technical specs are just too obscure for daily-life personal use. Be as it may, technology prices tend to decrease overtime as production increases and become optimized. And, of course, as the market demands for better and easy-to-use applications new and superior products will emerge. In fact, AR is already in our living-room. Examples can be found in some game console applications (Playstation 3 has some videogames, like EyePet and Eye of Judgment, that can take AR into our homes for roughly 30€) and more is on the way if we consider the technological development surrounding portable videogame devices and tablets PCs.

The development of these technologies is, as usually, associated with the investment made in the military, entertainment and medical research since these are industries that attract considerable amounts of investment and have the spending power to promote major advances in a very short time (Gamito et al., 2011b). These advances have helped surpassing most of the constraints associated with the use of AR in therapy and rehabilitation. For example, HMDs, which still are the most common devices to display VR and AR worlds, saw some of their limitations such as weight and ergonomic characteristics resolved. But further work is still needed on addressing technical matters, such as resolution, FOV, registration and occlusion. As discussed before, OST displays have some advantages as they rely mainly on the optical apparatus that constitutes the human eye. On the other hand, VST displays have some features that are appealing when compared to OST. Most recent technologies like projective displays and VRT show great prospect in solving most of the shortcoming concerning displays in AR technology. Nevertheless, and to our best knowledge, even these two state-of-the-art solutions have some limitations. The fact is the ideal solution will probably come from some new technology that it is able to combine the most features present in all of these types of displays, or a significant evolution in one of these, even though it seems at the moment that projective technology is the way to move forward.

Beyond the discussion about displays, it is also important to understand current issues in positioning. As has been shown in this chapter, some positioning solutions have received more attention to specific ends like rehabilitation. HMDs are still used frequently even with all the obvious constraints they pose. However, handheld devices are becoming increasingly popular and are receiving closer attention from both scholars and solution providers in rehabilitation technologies. The future will probably bring forth a solution where handheld devices can work with projection technologies to ensure that hands are available for 146 Augmented Reality – Some Emerging Application Areas

**5. How much more can we augment reality? Future applications and present** 

**5.1 Augmented hardware – Future prospects and issues concerning psychotherapy** 

As has been discussed in this chapter, AR has a part to play in the neurosciences area, namely on psychotherapy and neuro-rehabilitation. This role derived from the technological evolution in the last decade. One of the clearest signs of this evolution rests upon two pillars that are usually related to any new technology that strolled from research labs into

If it is true that prices have significantly decreased in most of the equipments needed to deploy AR applications to clinicians, therapists and other caregivers, it is also clear that some of these technologies will never have a large audience since their technical specs are just too obscure for daily-life personal use. Be as it may, technology prices tend to decrease overtime as production increases and become optimized. And, of course, as the market demands for better and easy-to-use applications new and superior products will emerge. In fact, AR is already in our living-room. Examples can be found in some game console applications (Playstation 3 has some videogames, like EyePet and Eye of Judgment, that can take AR into our homes for roughly 30€) and more is on the way if we consider the technological development surrounding portable videogame devices and tablets PCs.

The development of these technologies is, as usually, associated with the investment made in the military, entertainment and medical research since these are industries that attract considerable amounts of investment and have the spending power to promote major advances in a very short time (Gamito et al., 2011b). These advances have helped surpassing most of the constraints associated with the use of AR in therapy and rehabilitation. For example, HMDs, which still are the most common devices to display VR and AR worlds, saw some of their limitations such as weight and ergonomic characteristics resolved. But further work is still needed on addressing technical matters, such as resolution, FOV, registration and occlusion. As discussed before, OST displays have some advantages as they rely mainly on the optical apparatus that constitutes the human eye. On the other hand, VST displays have some features that are appealing when compared to OST. Most recent technologies like projective displays and VRT show great prospect in solving most of the shortcoming concerning displays in AR technology. Nevertheless, and to our best knowledge, even these two state-of-the-art solutions have some limitations. The fact is the ideal solution will probably come from some new technology that it is able to combine the most features present in all of these types of displays, or a significant evolution in one of these, even though it seems at the moment that projective technology is the way to move

Beyond the discussion about displays, it is also important to understand current issues in positioning. As has been shown in this chapter, some positioning solutions have received more attention to specific ends like rehabilitation. HMDs are still used frequently even with all the obvious constraints they pose. However, handheld devices are becoming increasingly popular and are receiving closer attention from both scholars and solution providers in rehabilitation technologies. The future will probably bring forth a solution where handheld devices can work with projection technologies to ensure that hands are available for

**limitations** 

forward.

**and neuro-rehabilitation** 

households' living-room: price and availability.

interaction while AR is in use (Zhou et al., 2008). Current research shows, however, that there are still not enough rehab applications using such combination, even though it has been proved to be a cost-efficient response.

#### **5.2 Augmented Reality Exposure Therapy (ARET) contribution to psychotherapy and neuro-rehabilitation: How to explore its full potential?**

Bearing in mind that some applications mentioned in this chapter concerns mental illness, ARET is only a slight improvement on more spread forms of VRET use. If a closer look is taken at phobia treatment using CBT with VRET, ARET's major contribution can be attributed to its ability to insert virtual objects in the real worlds which is an excellent substitute for *in vivo* exposure, since it also addresses ecological validity issues. Maybe it is wiser to insert virtual elements in the real world than to draw people entirely into the VR scenario. Moreover, ARET simplifies the need for world/scenario construction since it uses real places with superimposed objects, making it less time-consuming. Therefore, ARET seems to be able to be used as a coadjutant or as a substitute to VRET in CBT's approach using exposure therapy.

This is also true for another important field of application: rehabilitation. Since one of the key aspects of rehabilitation is repetition, VR/AR solutions may represent a clear path to swifter recoveries. AR also provides a more realistic environment where the individual can practice while being motivated by the insertion of virtual elements. Additionally, solutions where videogame consoles are being used can also serve as an added bonus since prices of off-the-shelf products are becoming more accessible. On the other hand, videogames bring in the fun of a game. Videogame based solutions also guarantee top notch CGI (computer graphic imaging) and, more importantly, the products that are market leaders all have motion detection hardware that can easily be used to rehabilitation. But some fences are still needed to be crossed as the available applications were designed for entertainment and must be adapted for these specific ends.

One aspect that is shared by both rehabilitation and mental illness fields of research when addressing AR is that this technology ensures a smoother transition from therapy to real life as it removes most mediated elements, ensuring a more proximal experience to real objects and situations. This is probably one of the most essential aspects of AR versus VR and constitutes a major opportunity for AR based applications. Knowledge transfer between mediated environments and real-life situations is still an issue and more research is needed to dissipate all doubts about the efficacy of VR/AR in competence development.

A collateral issue to the full and unrestricted use in both rehabilitation and mental health is the need to gain the support of more clinical practitioners. There is still some reserve from many mental health professionals about the use and the advantages that these technologybased solutions have to offer. For that reason, a bigger effort on education and results dissemination is the way to get more support for these applications development.

For all the above mentioned reasons, AR solutions have a bright future ahead in deploying exciting and fruitful solutions for some serious issues. And with the continuous development of exciting new technological solutions, AR based solutions may be available in every home in just a few years.

NeuAR – A Review of the VR/AR Applications in the Neuroscience Domain 149

Bryanton, C., Bossé, J., Brien, M., Mclean, J., McCormick, A., Sveistrup, H. (2006). Feasibility,

Butefisch, C., Hummelsheim, H., Denzler, P. & Mauritz, K.H. (1995). Repetitive training of

Caudell, T. & Mizell, D. (1992) Augmented reality: An application of heads-up display

Chen, Y., Kang, L.J., Chuang, T.Y., Doong, J.L., Lee, S.J. & Tsai, M.W. (2004). Use of Virtual

Cirstea, M.C. & Levin, M.F. (2007). Improvement in arm movement patterns and endpoint

Clare, L. & Woods, R. T. (2004). Cognitive training and cognitive rehabilitation for people

Cooper, R.A., Dicianno, B.E., Brewer, B., LoPresti, E., Ding, D., Simpson, R., Grindle, G. &

Correa, A. G., Klein, A. N., Lopes, R. D. (2009). Augmented Reality Musical System for

Costa, R. & Carvalho, L. (2004). The acceptance of virtual reality devices for cognitive

Deutsch J.E., Robbins D., Morrison J. & Bowlby P. G. (2009). Wii-based compared to

Emmelkamp, P.M., Krijn, M., Hulsbosch, A.M., de Vries, S., Schuemie, M.J., & van der Mast,

evaluation in acrophobia. *Behavior Research and Therapy*, 40, 5, pp. (509-516). Feintuch, U., Raz, L., Hwang, J., Yongseok, J., Josman, N., Katz, N., Kizony, R., Rand , D.,

Foa, E. & Kozak. M.(1986). Emotional processing of fear: exposure to corrective information.

Foa, E., Keane, T, & Friedman M. (2000). *Effective treatments for PTSD: practice guidelines from the International Society for Traumatic Stress Studies*. Guilford Press, Guilford Press.

*Virtual Rehabilitation International Conference*. Haifa, Israel. pp. (11-129). Edmans, J.A. Gladman, J.R., Cobb, S., Sunderland, A., Pridemore, T., Hilton, D. & Walker,

paretic hand. *Journal of the Neurological Sciences*, 130, pp. (59–68).

Single-Subject Design. *Physical Therapy*, 87, 11, pp. (1441-1457).

rehabilitation. *Medical Engineering & Physics*, 30, pp. (1387–1398).

(Eds.) *Rehabilitation Engineering*. InTech, Publishing, pp. (2–36).

*Conference on Systems Sciences*, pp. (659–669).

*Neurorehabilitation & Neural Repair*, 21, pp. (1-14).

*Programs in Biomedicine*, 73, 3, pp. (173-182).

*Psychological Bulletin,* 99, 1, pp. (20-35).

*Behavior*, 9, 2, pp. 1(23-128).

4, pp. (385-401).

pp. (2770-5).

(129-135).

Motivation, and Selective Motor Control: Virtual Reality Compared to Conventional Home Exercise in Children with Cerebral Palsy. *CyberPsychology &* 

isolated movements improves the outcome of motor rehabilitation of the centrally

technology to manual manufacturing processes. *Proceedings Hawaii International* 

Reality to Improve Upper-Extremity Control in Children With Cerebral Palsy: A

control depends on type of feedback during practice in stroke survivors.

with early-stage Alzheimer's disease: A review. *Neuropsychological Rehabilitation*, 14,

Wang, H. (2008). A perspective on intelligent devices and environments in medical

Rehabilitation of Patients with Duchenne Muscular Dystrophy. In Kheng T.Y.

rehabilitation: a report of positive results with schizophrenia. *Computer Methods and* 

standard of care balance and mobility rehabilitation for two individuals pot-stroke.

M. (2006). Validity of a virtual environment for stroke rehabilitation. *Stroke*, 37, 11,

C.A. (2002). Virtual reality treatment versus exposure in vivo: A comparative

Rizzo, A.A., Shahar, M. & Weiss, P.L. (2006). Integrating haptic-tactle feedback into a video capture based VE for rehabilitation. *CyberPsychology & Behavior*, 9, 2, pp.

#### **6. References**


148 Augmented Reality – Some Emerging Application Areas

Alamri, A., Cha, J., & Saddik, A. (2010). AR-REHAB: An Augmented Reality Framework for

Allred, R.P., Maldonado, M.A., Hsu, J.E. & Jones, T.A. (2005). Training the "lessaffected"

Attree, E.A., Brooks, B.M., & Rose, F.D. (2005). Virtual environments in rehabilitation and training: international perspectives. *Cyberpsychology & Behavior*, 8, 3, pp. (187–188).

Azuma, R. & G. Bishop (1994). Improving static and dynamic registration in an optical see-

Azuma, R. T. (1997). A survey of augmented reality. Presence: Teleoperators and Virtual

Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S. & MacIntyre, B. (2001). Recent

Bajura, M. & Neumann, U. (1995). Dynamic Registration Correction in Video-Based

Baram, Y., Aharon-Peretz, J., Simionovici, Y. & Ron, L. (2002). Walking on virtual tiles.

Beck, A.T. (1976). *Cognitive therapy and emotional disorders*. New York: International

Bimber, O. & Raskar, R. (2007). Modern approaches to augmented reality . In *ACM* 

Bimber, O., Encarnação, L.M., & Schmalstieg, D. (2000). Augmented Reality with Back-

Biocca, F.A. & Rolland J.P., (1998). Virtual eyes can rearrange your body-adaptation to visual displacement in see-through, head-mounted displays. *Presence*, 7, 3, pp. (262–277). Botella C., Villa, H., García-Palacios, A., Baños, R., Perpiñá, C., & Alcañiz. M. (2004).

Botella, C., Baños, R., Quero, S., Perpiñá, C., & Fabregat, S. (2004). Telepsychology and self-

Botella, C., Bretón-López, J., Quero, S., Baños, R. M., & García-Palacios, A. (2010). Treating cockroach phobia with augmented reality. *Behavior Therapy*, 41, 3, pp. (401-413). Botella, C., Juan, M. C., Baños, R., Alcañiz, M., Guillén, V., & Rey, B. (2005). Mixing realities?

and Agoraphobia. *CyberPsychology & Behavior,* 7, 5, pp. (527-535).

Advances in Augmented Reality. *IEEE Computers Graphics & Applications*, pp. (1-

Augmented Reality Systems. *IEEE Computer Graphics and Applications*, 15, 5, pp. (52-

*SIGGRAPH 2007 Courses* (San Diego, California, August 05 - 09, 2007). SIGGRAPH

Projection Systems using Transflective Surfaces. *Proceedings of EUROGRAPHICS* 

Clinically Significant Virtual Environments for the Treatment of Panic Disorder

help: The treatment of phobias using the internet. *CyberPsychology&Behavior,* 7, pp.

An application of augmented reality for the treatment of cockroach phobia.

through HMD. *Proceedings SIGGRAPH '94*, pp. (197-204).

impaired forelimb in rats. *Restor Neuroly Neuroscience*, 23, pp. (297-302). Arvanitis, T. N., Petrou, A., Knight, J. F., Savas, S., Sotiriou, S., Gargalakos, M., et al. (2007).

Poststroke-Patient Rehabilitation. *IEEE Transactions on Instrumentation and* 

forelimb after unilateral cortical infarcts interferes with functional recovery of the

Human factors and qualities pedagogical evaluation of a mobile augmented reality system for science education used by learners with physical disabilities. *Personal* 

**6. References** 

15).

60).

Universities Press.

(272–273).

'07. ACM, New York, NY.

*2000 - EG'2000*, 19, 3, pp. (161-168).

*CyberPsychology & Behavior*, 8, pp. (162–171).

*Measurment,* pp. (1-10).

*and Ubiquitous Computing,* 11, pp. (1–8).

Environments, 6, 4, pp. (355-385).

*Neural Processing Letters*, 16, pp. (227-33).


NeuAR – A Review of the VR/AR Applications in the Neuroscience Domain 151

Kosslyn, S.M., Thompson, W.L., Wraga, M. & Alpert, N.M. (2001). Imagining rotation by

Leitener, M., Tomitsch, T., Koltringer, T., Kappel, K. & Grechenig, T. (2007). Designing

Levin, M. F., Musampa, N. K., Henderson, A. K. & Knaut, L. A. (2005). New approaches to

Luo, X., Kline, T., Fischer, H., Stubblefield, K., Kenyon, R. & Kamper, D. (2005). Integration

Merians, A. S., Jack, D., Boian, R., Tremaine, M., Burdea, G. C., Adamovich, S. V., Recce, M.

Milgram, P., Kishino, F. (1994). A Taxonomy of Mixed Reality Visual Displays. *IEICE Trans.* 

Öst, L. G. (1989). One-session treatment for specific phobias. *Behaviour Research & Therapy*,

Öst, L. G. (1997). Rapid treatment of specific phobias. In G. C. L. Davey (Ed.), *Phobias: A handbook of theory, research, and Treatment.* pp. (227–247). New York: Wiley. Park, J., You, S. and Neumann, U. (1998). Natural feature tracking for extendible robust

Parks, T.E. (1965). Post Retinal Visual Storage. *American Journal of Psychology*, 78, pp. (145-

Parsons & Rizzo (2008). Affective Outcomes of Virtual Reality Exposure Therapy for

Patrick, E., Cosgrove, D., Slavkovic, A., Rode, J.A., Verratti, T., & Chiselko, G. (2000). Using

Pinz, M. Brandner, H. Ganster, A. Kusej, P. Lang, P. & Ribo, M. (2002). Hybrid tracking for

Powers, M. & Emmelkamp, P. (2008). Virtual reality exposure therapy for anxiety disorders:

Pressigout, M. & Marchand, E. (2006). Hybrid tracking algorithms for planar and non-planar

Pryor, Homer L., Furness, Thomas A. & Viirre, E. (1998). The Virtual Retinal Display: A

Richard, E., Billaudeau, V., Richard, P. & Gaudin, G. (2007). Augmented Reality for

New Display Technology Using scanned Laser Light. *Proceedings of Human Factors* 

rehabilitation of cognitive disabled children: A preliminary study. *Virtual* 

A meta-analysis. Journal of Anxiety Disorders, 22, 3, pp. (561-569).

structures subject to illumination changes. *ISMAR '06,* pp. (52-55).

*and Ergonomics Society in the 42nd Annual Meeting*, pp. (1570-1574).

Anxiety and Specific Phobias: A Meta-Analysis. *Journal of Behavior Therapy &* 

a Large Projection Screen as an Alternative to Head-Mounted Displays for Virtual

*Assistive Technology for All Ages*. M. Hersh (Eds.), Spain. pp. (1-7).

Lewis, J.R. (2004). In the eye of the beholder. *IEEE Spectrum*, pp. (16-20).

Following Stroke. *Physical Therapy,* 82, 9, pp. (898-915).

strategies. *NeuroReport*, 12, pp. (2519–25).

*Physiotherapy Journal*, 23, pp. (2-5).

*Annual Conference in Shanghai*, pp. (1-4).

*Information Systems*. 77, 12, pp. (1321-1329).

*Experimental Psychiatry*, 39, 3, pp. (250-261).

Environments. *Proceedings of CHI' 2000*, 2, 1, pp. (479-485).

augmented reality. *ÖGAI Journal*, 21, 1, pp. (17-24).

augmented realities. *In IWAR '98*.

*Rehabilitation*, pp. (100-106).

27, pp. (1–7).

147).

endogenous and exogenous forces: distinct neural mechanisms for different

Tangible Tabletop Interfaces for Patients in Rehabilitation. *Proceedings of Conference & Workshop on Assistive Technologies for People with Vision and Hearing Impairments:* 

enhance motor function of the upper limb in patients with hemiparesis. *Hong Kong* 

of Augmented Reality and Assistive Devices for Post-Stroke Hand Opening Rehabilitation. *Proceedings of the IEEE Engineering in Medicine and Biology 27th* 

& Poizner, H. (2002). Virtual Reality–Augmented Rehabilitation for Patients


150 Augmented Reality – Some Emerging Application Areas

Gamito P., Oliveira J, Morais D., Rosa P. & Saraiva T. (2011b). Serious Games for Serious

Gamito, P., Oliveira, J., Pacheco, J., Morais, D., Saraiva, T., Lacerda, R., Baptista, A., Santos,

Gamito, P., Oliveira, J., Rosa, P., Morais, D., Duarte, N., Oliveira, S. & Saraiva, T., (2010).

Gazzaniga, M.S. & Heatherton, T.F. (2006). *The Psychological Science: Mind, Brain, and Behavior*

Giorgino, T., Tormene, P., Maggioni, G., Cattani, B., Pistarini, C. & Quaglini, S. (2008). My

Gloaguen, V., Cottraux, J., Cucherat, M., Blackburn, I. (1998). A meta-analysis of the effects

Hamm A, & Weike A. (2005). The neuropsychology of fear-learning and fear regulation.

Hofmann, S. & Smits, J. (2008). Cognitive-behavioral therapy for adult anxiety disorders: a

Holden, M. (2003). Virtual Environments for Motor Rehabilitation: Review. *Cyberpsychology* 

Hua, H., Gao, C., Brown, L., Ahuja, N. & Rolland, J.P. (2005). Using a head-mounted

Juan, M. C., Alcañiz, M., Monserrat, C., Botella, C., Baños, R. & Guerrero, B. (2005). Using

Kahn, L. E., Averbuch, M., Rimer, W. Z. & Reinkensmeyer, D. J. (2001). Comparison of

Kerawalla, L., Luckin, R., Seljeflot, S. & Woolard, A. (2006). Making it real: exploring the

Kijima, R. and Ojika, T. (1997). Transition between virtual environment & workstation

Kirner, C., Gonçalves, T. & Zorzal, E.R. (2007). Collaborative Augmented Reality

*Conference on Enterprise Information Systems*. Lisboa: INSTICC, pp. (257-262). Kollin, J.A. (1993). Retinal Display For Virtual-Environment Applications. In *Proceedings of* 

*Reality Annual International Symposium*, pp. (130- 137).

*SIDInternational Symposium Digest of Technical Papers*.

*ACM International Symposium on Augmented Reality*, pp. (217–223).

Publishing, pp. (527 – 548).

72).

*69*, 4, pp. (621-32).

*& Behaviour*, 8, 3.

(31–37).

pp. (39–44).

10, pp. (163–174).

*Human Development*, 10, 3, pp. (309-312).

*Behavior, and Social Networking*, 13, 1, pp. (43-48).

(2nd edition). New York: W.W. Norton & Company

Treatment. *Cybertherapy & Rehabilitation*, 1, pp. (17-19).

*International Journal of Psychophysiology*, 57, pp. (5–14).

problems: from Ludicus to Therapeuticus. In J. J. Kim (Ed.) *Virtual Reality*. InTech,

N., Soares, F., Gamito, L & Rosa, P. (2011a). Traumatic Brain Injury memory training: a Virtual Reality online solution. *International Journal on Disability and* 

PTSD Elderly War Veterans: A Clinical Controlled Pilot Study. *Cyberpsychology,* 

Hearth's Neurological Rehabilitation Concept – Technologies for Post-Stroke

of cognitive therapy in depressed patients. *Journal of Affective Disorders,* 49, pp. (59-

meta-analysis of randomized placebo-controlled trials. *Journal of Clinical Psychiatry,* 

projective display in interactive augmented environments. In *Proceedings IEEE and* 

augmented reality to treat phobias. *IEEE Computer Graphics and Applications,* 25, pp.

robot-assisted reaching to free reaching in promoting recovery from chronic stroke. In *International of Assisted Technology in the Information Age*, IOS Press, Amsterdam:

potential of augmented reality for teaching primary school science. *Virtual Reality,*

environment with projective head-mounted display. *Proceedings of IEEE Virtual* 

Environment for Educational Applications. *Proceedings of the 9th International* 


NeuAR – A Review of the VR/AR Applications in the Neuroscience Domain 153

Slobounov, S., Slobounov, E. & Newell, K. (2006). Application of Virtual Reality Graphics in Assessment of Concussion. *Cyberpsychology & Behavior*, 9, 2, pp. (188–191). Sohlberg, M. M. & Mateer, C. A. (1989). *Introduction to cognitive rehabilitation : theory and* 

Sohlberg, M. M. & Mateer, C. A. (2001). *Cognitive rehabilitation: An integrative* 

Squire, K., & Klopfer, E. (2007). Augmented reality simulations onhandheld computers.

Squire, K., & Mingfong, J. (2007). Mad city mystery: Developing scientific argumentation

Stetten, G., Chib, V., Hildebrand, D., & Bursee, J. (2001). Real Time Tomographic Reflection:

Sussman, L. K., Robins, L. N., & Earls, F. (1987). Treatment-seeking for depression by black

Sveistrup, H. (2004). Motor rehabilitation using virtual reality. *Journal of NeuroEngineering* 

Taub, E., Uswatte, G. & Pidikiti, R. (1999). Constraint-induced movement therapy: A new

review. *Journal of Rehabilitation Research and Development*, 36, pp. (237–251). Van Krevelen, D.W.F. & Poelman, R. (2010). A Survey of Augmented RealityTechnologies,

Viau, A., Levin, M.F., McFadyen, B.J. & Feldman, A.G. (2004). Reaching in reality and in

Vincelli F, Riva G (2000). Virtual reality as a new imaginative tool in psychotherapy. *Studies* 

Vincelli, F. & Riva, G. (2002). Virtual Reality: a new tool for panic disorder therapy. *Expert* 

Weissman, M., Markowitz, J. & Klerman G. (2000). *Comprehensive guide to interpersonal* 

Weiss, P.L. & Katz, N. (2004). Guest Editorial: The Potential of Virtual Reality for Rehabilitation. *Journal of Rehabilitation Research & Development*, 41, 5, pp. (7-10). Wilson, B. (2003). *Neuropsychological rehabilitation : theory and practice*. Swets & Zeitlinger

Wilson, P., Thomas, P., Sum, D., Duckworth, J., Gugliemetti, M., Rudolph, H, Mumford, N.

Witmer, G. & Singer M. (1998). Measuring Presence in Virtual Environments: A Presence Questionnaire. *Presence: Teleoperators and Virtual Environments*, 7, 3. pp. (225-240). Wörn, H., Aschke, M., & Kahrs, L. A. (2005). New augmented reality and robotic based

& Eldridge, R. (2006). A multilevel model for movement rehabilitation in Traumatic Brain Injury (TBI) using Virtual Environments. *IEEE, 5th International Workshop of* 

methods for head-surgery. *International Journal of Medical Robotics and Computer* 

*International Society of Electrophysiology and Kinesiology Congress*, 51.

*in Health Technologies and Informatics*, 70, pp. (356-358).

*Review of Neurotherapeutics*, 2, 3, pp. (89-95).

*psychotherapy*. New York: Basic Books.

*Virtual Rehabilitation* (IWVR 2006).

*Assisted Surgery,* 1, pp. (49–56).

and white Americans. *Social Science and Medicine*, 24, pp. (187–196).

skills with a placebased augmented reality game on handheld computers. *Journal of* 

Phantoms for Calibration and Biopsy, In *Proceedings of IEEE/ACM International* 

family of techniques with broad application to physical rehabilitation—a clinical

Applications and Limitations. *The International Journal of Virtual Reality*, 9, 2, pp. (1-

virtual reality: a comparison of movement kinematics. In *Proceedings of the 15th* 

*practice*. New York: Gildford Press.

*and Rehabilitation*, pp. (1-10).

20).

Publishers.

*neuropsychological approach*. New York: Gildford Press.

*Symposium on Augmented Reality (ISMAR'01)*, pp. (11-19).

*Journal of the Learning Sciences,* 16, pp. (371–413).

*Science Education and Technology,* 16, pp. (5–29).


152 Augmented Reality – Some Emerging Application Areas

Riess T. (1995). Augmented reality in the treatment of Parkinson's disease. In K. Morgan, M.

Riva G. (2005). Virtual reality in psychotherapy: review. *CyberPsychology & Behavior*, 8, pp.

Riva, G., Molinari, E. & Vincelli, F. (2002). Interaction and presence in the clinical

Rizzo, A., Schultheis, M. Kerns, K.A. & Meteer, C. (2004). Analysis of assets for virtual

Rizzo, A., Pair J., Graap K., Manson, B., McNerney, P., Wiederhold, B., Wiederhold M. &

Rolland, J. P., Davis L. & Baillot. Y. (2001). A survey of tracking technology for virtual

Rolland, J.P., Biocca, F., Barlow, T. & Kancherla, A. (1995). Quantification of Perceptual

Rolland, J.P., Holloway, R.L. & Fuchs, H. (1994a). A comparison of optical and video see-

Rothbaum, B. & Schwartz, A. (2002). Exposure therapy for posttraumatic stress disorder.

Rothbaum, B., Hodges, L., Smith, S., Lee, J. & Price, L. (2000) A controlled study of virtual

Sandor, C. & Klinker, G. (2006). Lessons Learned in Designing Ubiquitous Augmented

Saraiva,T., Gamito, P., Oliveira, J., Morais, D., Pombal, M. Gamito, L. & Anastácio, M. (2007).

Schowengerdt, B.T, Seibel, E.J. Silverman, N.L. & Furness, T.A. (2004). Stereoscopic retinal

*the new paradigm of healthcare*. Amsterdam: IOS Press: pp. (298-302).

(220 –230).

205).

(207-239).

pp. (293-307).

pp. (218-235).

pp. (366–376).

Washington D.C., pp. (235-250).

*Augmented Reality*. Mahwah, NJ, pp. (67-112).

*American Journal of Psycotherapy,* 56, pp. (59-75).

*Cybertherapy & Telemedicine*, 5, pp. (199-206).

*Proceedings IEEE VRAIS'95,* pp. (56-66).

*Psychology*, 68, 6, pp. (1020-1026).

Satava, HB Sieburg, R Mattheus & JP Christensen (Eds.). *Interactive technology and* 

relationship: virtual reality (VR) as communicative medium between patient and therapist. *IEEE Transactions on Information Technology in Biomedicine*, 6, 3, pp. (198-

reality applications in neuropsychology. *Neuropsychological Rehabilitation,* 14, 2,

Spira J. (2006). A Virtual Reality Exposure Therapy Application for Iraq War Military Personnel with Post Traumatic Stress Disorder: From Training to Toy to Treatment. In: M. Roy. (Eds.). *NATO Advanced Research Workshop on Novel Approaches to the Diagnosis and Treatment of Posttraumatic Stress Disorder*. IOS Press,

environments. In Barfield & Caudell (Eds.). *Fundamentals of Wearable Computers and* 

Adaptation to Visual Displacement in See-Through Head-Mounted Displays.

through head mounted displays. *SPIE Telemanipulator and Telepresence Technologies*,

reality exposure therapy for the fear of flying. *Journal of Consulting and Clinical* 

Reality User Interfaces. In M. Haller, M. BillingHurst, B. Thomas (Eds.). *Emerging Technologies of Augmented Reality: Interfaces and Design*, Idea Group Publishing, USA,

VR exposure: reducing acute stress disorder derived from motor vehicle accidents. In Wiederhold et al. (Eds.). Interactive Media Institute. *Annual Review of* 

scanning laser display with integrated focus cues for ocular accommodation. A.J. Woods, J.O. Merritt, S.A. Benton, & M.T. Bolas (Eds.). Stereoscopic Displays and Virtual Reality Systems XI. *Proceedings of SPIE-IS&T Electronic Imaging*, SPIE 5291,


**8** 

**Augmented Reality Assisted Upper** 

Kimberlee Jordan and Marcus King

*Industrial Research Ltd.* 

*New Zealand* 

**Limb Rehabilitation Following Stroke** 

Stroke is a rapidly developing loss of brain function(s) resulting from lack of blood flow caused by either a blockage, or a hemorrhage (Sims & Muyderman, 2009). Common motor impairments after stroke include muscle weakness, reduced reaction time, loss of joint range of motion and disordered movement organization: hemiparesis or weakness on one side of the body is common. It is widely regarded as a leading cause of disability in the developed world (Adamson et al., 2004). For survivors of stroke, these impairments can severely limit daily activities and participation in social and family environments (Merians et al., 2002; Werner & Kessler, 1996). The upper limb (UL) in particular remains problematic post stroke. Whereas up to 83% of stroke survivors learn to walk again (Skilbeck et al., 1983), it is estimated that only 5 to 20% of stroke survivors attain complete functional recovery of their affected UL (Kwakkel et al., 2003). This may be in part because the high costs of standard therapy mean that treatment frequently ceases once patients are released from acute care and long term outcomes are often poor (Feys et al., 1998; Carter et al., 2006). Another possibility is that it is possible for the patient to learn to use their unaffected arm almost exclusively for most activities of daily living, whereas the patient has no choice but to use both legs for walking (Feys et al., 1998; Kwakkel et al., 1999). Additionally, significant loss of hand function may cause some people to abandon any residual function in that hand

Best practice rehabilitation requires multiple trained personnel up to 30 hours of therapy a week (anon, 2007), but this is seldom achieved and outcomes of conventional therapy are poor (Feys et al., 1998). Evidence suggests that repetitive training of functional UL motor tasks (Van Peppen et al., 2004; Kwakkel et al., 1999; van der Lee, 2001); intensity of practice, and functional relevance of the motor tasks (French et al., 2009; Butefisch et al., 1995) are the critical components of successful UL rehabilitation. Neuroplasticity is the process by which neural circuits in the brain are modified by experience, learning and/or injury (e.g. Nudo, 2003) that allows for motor-relearning and recovery after stroke. A large number of repetitions of the same movement pattern forms the physiological basis of motor learning and is thus an essential component of motor-relearning (Butefisch et al., 1995). Animal studies indicate that as many as 400 to 600 repetitions per day may be required to induce changes in neuroplasticity following stroke (Nudo et al., 1996). But repeated motor activity on its own is not sufficient however to promote recovery: the specificity and functionality of

**1. Introduction** 

through learned non-use (Sterr et al., 2002).


Zlatanova, S. (2002). Augmented Reality Technology. *GISt Report*, 17, pp. (1-76).
