**Augmented Reality Assisted Upper Limb Rehabilitation Following Stroke**

Kimberlee Jordan and Marcus King

*Industrial Research Ltd. New Zealand* 

#### **1. Introduction**

154 Augmented Reality – Some Emerging Application Areas

Wykes, T., Reeder, C., Landau, S. , Everitt, B., Knapp, M., Patel A. & Romeo, R. (2007).

Yavuzer G., Senel A., Atay M.B. & Stam H. J. (2008). Playstation eyetoy games' improve

Zhou, F., Duh, H.B.L. & Billinghurst, M. (2008) Trends in Augmented Reality Tracking,

Zlatanova, S. (2002). Augmented Reality Technology. *GISt Report*, 17, pp. (1-76).

*British Journal of Psychiatry,* 190, pp. (421-427).

pp. (237-44).

(ISMAR 2008), pp. (15-18).

Cognitive remediation therapy in schizophrenia. Randomised controlled trial.

upper extremity-related motor functioning in subacute stroke: a randomized controlled clinical trial. *European Journal of Physical and Rehabilitation Medicine*, 44,

Interaction and Display: A Review of Ten Years of ISMAR. Cambridge, UK: 7th IEEE and ACM International Symposium on Mixed and Augmented Reality

> Stroke is a rapidly developing loss of brain function(s) resulting from lack of blood flow caused by either a blockage, or a hemorrhage (Sims & Muyderman, 2009). Common motor impairments after stroke include muscle weakness, reduced reaction time, loss of joint range of motion and disordered movement organization: hemiparesis or weakness on one side of the body is common. It is widely regarded as a leading cause of disability in the developed world (Adamson et al., 2004). For survivors of stroke, these impairments can severely limit daily activities and participation in social and family environments (Merians et al., 2002; Werner & Kessler, 1996). The upper limb (UL) in particular remains problematic post stroke. Whereas up to 83% of stroke survivors learn to walk again (Skilbeck et al., 1983), it is estimated that only 5 to 20% of stroke survivors attain complete functional recovery of their affected UL (Kwakkel et al., 2003). This may be in part because the high costs of standard therapy mean that treatment frequently ceases once patients are released from acute care and long term outcomes are often poor (Feys et al., 1998; Carter et al., 2006). Another possibility is that it is possible for the patient to learn to use their unaffected arm almost exclusively for most activities of daily living, whereas the patient has no choice but to use both legs for walking (Feys et al., 1998; Kwakkel et al., 1999). Additionally, significant loss of hand function may cause some people to abandon any residual function in that hand through learned non-use (Sterr et al., 2002).

> Best practice rehabilitation requires multiple trained personnel up to 30 hours of therapy a week (anon, 2007), but this is seldom achieved and outcomes of conventional therapy are poor (Feys et al., 1998). Evidence suggests that repetitive training of functional UL motor tasks (Van Peppen et al., 2004; Kwakkel et al., 1999; van der Lee, 2001); intensity of practice, and functional relevance of the motor tasks (French et al., 2009; Butefisch et al., 1995) are the critical components of successful UL rehabilitation. Neuroplasticity is the process by which neural circuits in the brain are modified by experience, learning and/or injury (e.g. Nudo, 2003) that allows for motor-relearning and recovery after stroke. A large number of repetitions of the same movement pattern forms the physiological basis of motor learning and is thus an essential component of motor-relearning (Butefisch et al., 1995). Animal studies indicate that as many as 400 to 600 repetitions per day may be required to induce changes in neuroplasticity following stroke (Nudo et al., 1996). But repeated motor activity on its own is not sufficient however to promote recovery: the specificity and functionality of

Augmented Reality Assisted Upper Limb Rehabilitation Following Stroke 157

Fig. 1. Robotic armskate with PC controller for UL rehabilitation.

Rizzo, 2001).

2008).

deficits.

**3.1 The Able-X** 

user with a disability a stimulating experience, engaging and motivating to the practice of UL movements as the user manipulates the interface device (Sviestrup, 2004). VR allows the creation and control of dynamic 3-dimensional, ecologically valid stimulus environments within which behavioral response can be recorded and measured, therby offering clinical assessment and rehabilitation options not available with traditional methods (Schultheis &

Studies have reported increased participant motivation, enjoyment or perceived improvement in physical ability following the inclusion of VR into stroke rehabilitation (Broeren et al., 2008; Housman et al., 2009; Yavuzer et al., 2008). Importantly, computer games can improve compliance with prescribed rehabilitation exercises (Kwakkel et al.,

Assistive devices may be combined with virtual reality in order to provide a valuable rehabilitation experience for stroke survivors. For example Pyk et al. (2008) describe a combination of data gloves and virtual reality for UL rehabilitation in children with motor

Industrial Research Ltd (IRL) developed the Able-X for UL rehabilitation of stroke survivors who are able to move their arm against gravity, but have minimal strength or control over their actions (Hijmans et al., 2011). It is a light-weight handlebar (Fig 2) which contains a

the repeated task is also an important consideration associated with functional recovery (Butefisch et al., 1995; Nudo et al., 1996; Plautz et al., 2000).

Subjective well-being is reduced one year post stroke and this is mainly attributed to poor arm function (Wyller et al., 1997). However, improvements in outcomes can be seen up to 3 years post stroke (Stinear et al 2007), well past the point at which standard rehabilitation therapy stops (Stein et al., 2009). There is, therefore, a clear need for effective UL rehabilitation tools that provides large repetition of functional movements and can be used in a home setting without the constant supervision of a therapist.

## **2. Robot assisted rehabilitation therapy**

From the stand point of the therapist, the large number of repetitions necessary for effective rehabilitation is very labour intensive. Robot-assisted physiotherapy (e.g. robotic or automated exercise machines) can help in alleviating therapist work load by providing autonomous training where patients can engage in repeated and intense practice of goaldirected tasks (Prange et al., 2006). For example, Krebs et al (2008) describe a 2 DOF robot, the MIT Manus, that trained shoulder and elbow movements in a horizontal plane, although robots can be used for "teaching a trajectory" or "enabling a movement through minimal assistance", the amount of guidance for maximising recovery is not known (Adamovich, 2009). Robotic-assisted therapy devices can also provide an objective and reliable means of monitoring patient progress, and can be used in conjunction with computer-based augmented or virtual reality environments, leading to improvements in motor function (Fasoli et al., 2003; Krebs et al., 2002; Kwakkel et al., 2008).

Reports indicate that robotic therapy can improve motor control in stroke survivors to a greater extent than conventional therapy and results of clinical trials using these systems to rehabilitate the UL are positive (Prange et al., 2006). However, this is likely to be due primarily to the greater intensity of practice that can be achieved using these devices (Kwakkel et al., 2008).

There is a crucial need to improve the cost-to benefit ratio of robot-assisted therapy and their effectiveness in rehabilitation of the impaired arm (Johnson et al., 2007). The rehabilitation interventions described in Kwakkel et al. (2008) require patient time on the devices of an average of 48 min per day for 8 weeks. A 6-axis robot can be expected to cost about US\$60,000 and deployment (e.g., training, programming, adding tools etc.) may cost another US\$200,000. Such a capital item, with a depreciation rate of say 15%, operating 40 h per week, would need to be charged at US\$19/h, meaning that robotic arm treatment of one patient's rehabilitation programme will cost US\$800 in capital depreciation alone, regardless of maintenance, running or therapist costs (King et al., 2010). Peattie et al. (2009) describe a reaching training robot, based on an armskate (Fig 1), using a personal computer as the controller. If a low-cost (under US\$1000) peripheral device for a personal computer can supply a beneficial therapy, the cost-effectiveness of such therapies would be significantly improved and clinical uptake will be expedited.

#### **3. Virtual reality rehabilitation**

Virtual reality (VR) allows the user to interact with a simulated "real" environment via dedicated computer hardware and software (Holden & Dyer, 2002). It potentially provides a 156 Augmented Reality – Some Emerging Application Areas

the repeated task is also an important consideration associated with functional recovery

Subjective well-being is reduced one year post stroke and this is mainly attributed to poor arm function (Wyller et al., 1997). However, improvements in outcomes can be seen up to 3 years post stroke (Stinear et al 2007), well past the point at which standard rehabilitation therapy stops (Stein et al., 2009). There is, therefore, a clear need for effective UL rehabilitation tools that provides large repetition of functional movements and can be used

From the stand point of the therapist, the large number of repetitions necessary for effective rehabilitation is very labour intensive. Robot-assisted physiotherapy (e.g. robotic or automated exercise machines) can help in alleviating therapist work load by providing autonomous training where patients can engage in repeated and intense practice of goaldirected tasks (Prange et al., 2006). For example, Krebs et al (2008) describe a 2 DOF robot, the MIT Manus, that trained shoulder and elbow movements in a horizontal plane, although robots can be used for "teaching a trajectory" or "enabling a movement through minimal assistance", the amount of guidance for maximising recovery is not known (Adamovich, 2009). Robotic-assisted therapy devices can also provide an objective and reliable means of monitoring patient progress, and can be used in conjunction with computer-based augmented or virtual reality environments, leading to improvements in motor function

Reports indicate that robotic therapy can improve motor control in stroke survivors to a greater extent than conventional therapy and results of clinical trials using these systems to rehabilitate the UL are positive (Prange et al., 2006). However, this is likely to be due primarily to the greater intensity of practice that can be achieved using these devices

There is a crucial need to improve the cost-to benefit ratio of robot-assisted therapy and their effectiveness in rehabilitation of the impaired arm (Johnson et al., 2007). The rehabilitation interventions described in Kwakkel et al. (2008) require patient time on the devices of an average of 48 min per day for 8 weeks. A 6-axis robot can be expected to cost about US\$60,000 and deployment (e.g., training, programming, adding tools etc.) may cost another US\$200,000. Such a capital item, with a depreciation rate of say 15%, operating 40 h per week, would need to be charged at US\$19/h, meaning that robotic arm treatment of one patient's rehabilitation programme will cost US\$800 in capital depreciation alone, regardless of maintenance, running or therapist costs (King et al., 2010). Peattie et al. (2009) describe a reaching training robot, based on an armskate (Fig 1), using a personal computer as the controller. If a low-cost (under US\$1000) peripheral device for a personal computer can supply a beneficial therapy, the cost-effectiveness of such therapies would be significantly

Virtual reality (VR) allows the user to interact with a simulated "real" environment via dedicated computer hardware and software (Holden & Dyer, 2002). It potentially provides a

(Butefisch et al., 1995; Nudo et al., 1996; Plautz et al., 2000).

in a home setting without the constant supervision of a therapist.

(Fasoli et al., 2003; Krebs et al., 2002; Kwakkel et al., 2008).

improved and clinical uptake will be expedited.

**3. Virtual reality rehabilitation** 

(Kwakkel et al., 2008).

**2. Robot assisted rehabilitation therapy** 

Fig. 1. Robotic armskate with PC controller for UL rehabilitation.

user with a disability a stimulating experience, engaging and motivating to the practice of UL movements as the user manipulates the interface device (Sviestrup, 2004). VR allows the creation and control of dynamic 3-dimensional, ecologically valid stimulus environments within which behavioral response can be recorded and measured, therby offering clinical assessment and rehabilitation options not available with traditional methods (Schultheis & Rizzo, 2001).

Studies have reported increased participant motivation, enjoyment or perceived improvement in physical ability following the inclusion of VR into stroke rehabilitation (Broeren et al., 2008; Housman et al., 2009; Yavuzer et al., 2008). Importantly, computer games can improve compliance with prescribed rehabilitation exercises (Kwakkel et al., 2008).

Assistive devices may be combined with virtual reality in order to provide a valuable rehabilitation experience for stroke survivors. For example Pyk et al. (2008) describe a combination of data gloves and virtual reality for UL rehabilitation in children with motor deficits.

#### **3.1 The Able-X**

Industrial Research Ltd (IRL) developed the Able-X for UL rehabilitation of stroke survivors who are able to move their arm against gravity, but have minimal strength or control over their actions (Hijmans et al., 2011). It is a light-weight handlebar (Fig 2) which contains a

Augmented Reality Assisted Upper Limb Rehabilitation Following Stroke 159

intervention was provided, after which participants were again re-assessed (T2). The 2.5 week intervention was then performed and consisted of playing computer games with the Able-X for eight to ten therapist supervised sessions lasting 45 to 60 minutes. Game difficulties were adjusted to mach the ability of the participant and progressed as participant ability improved. Finally the participants were reassessed (T3). The Fugl Meyer Upper Limb assessment (FMA-UL) (Fugl-Meyer, 1975) was the primary outcome measure. Additional outcome measures were the DASH (Disabilities of the Arm, Shoulder and Hand), a questionnaire for self reported symptoms and abilities in certain activities (Beaton et al., 2001) and the WOLF test (Wolf Motor

The Able-X intervention resulted in a significant improvement in UL motor performance as measured by the FMA-UL (Table 1). The mean improvement post intervention (T3) ranged from 4.2 (compared to T2) to 5.2 (compared to T1). These results are especially encouraging in light of the fact that the group comprised adults with chronic stroke and the cohort was heterogeneous (FMA-UL ranged from 14 to 65 at inclusion). Post-intervention focus groups showed that participants enjoyed the experience and reported perceived gains in UL

FMA-UL (n=13) 44.2±17.9 44.0±17.2 45.0±16.2 49.2±16.6 10.41 <0.001 WMFT (n=13) 32.7±50.4 32.4±50.6 31.5±50.9 30.5±51.2 2.74 0.06 DASH (n=13) 51.8±21.5 54.5±23.4 55.0±24.4 55.6±23.2 0.66 0.58 Table 1. Means ± standard deviation of the outcome measures pre-control (T0), post-control/ pre-washout (T1), post-washout / pre-intervention (T2), and post-intervention (T3).

To test use of the system in an unsupervised home situation, three participants from the above trial were asked to use the system at home for 8 weeks. Participants were assessed prior to the intervention (T1); the systems were left with them for up to 61 days; after which they were reassessed (T2). As patient self-reports are suggested as the best method of evaluating adherence to home-based physiotherapy (Bassett, 2003), participants diaries of usage and the Intrinsic Motivation Inventory (McAuley, 1987) were the outcome measures.

Diary: number of sessions in intervention 44 / 55 days 46 / 58 days 49 / 61 days

The diaries (Table 2) indicated that all of participants continued therapy at a rate of from 4.5 to 5.5 sessions per week and maintained that throughout the 8 week period. The IMI results

Participant J D W IMI: average 70% 70.5% 49%

Diary: Average session duration (minutes) 46 35 38 Diary: Average sessions per week 5.5 5.5 4.5 Table 2. IMI and diary results for unsupervised homecare therapy using Able-X.

T0 T1 T2 T3 F p

Function Test for functional movement (Wolf & Catlin, 2001)).

**3.2.3 Able-X results** 

movement, concentration and balance.

**3.3 Able-X home care pilot study** 

motion sensitive game controller (CyWee Z, Taiwan) that interfaces with a PC. The Able-X couples the arms so that the unaffected arm can assist movement of the hemiparetic arm.

A suite of computer games provided a graduated series of physical challenges, from target hitting games, to faster sports games, as well as some casual and puzzle games.

Fig. 2. Against-gravity bilateral exercise device that interacts with computer games.

## **3.2 Able-X pilot study**

#### **3.2.1 Able-X participants**

To trial the system fourteen participants with post-stroke UL hemiparesis were recruited: their mean age was 71 years (range 47-85); nine were male, five female; time since stroke one to six years; eight had a stroke in their left hemisphere and six in their right and in nine participants the dominant hand was affected while in five it was the non-dominant.

#### **3.2.2 Able-X assessment, outcome measures and Intervention**

Assessments were conducted after enrolment in the study (T0). A 2.5 week sham-intervention was provided which consisted of playing mouse based computer games on a personal computer using their unaffected arm for eight to ten sessions, each lasting at least 45 minutes. Following this, participants were re-assessed (T1). There was a 2 ½ week period where no intervention was provided, after which participants were again re-assessed (T2). The 2.5 week intervention was then performed and consisted of playing computer games with the Able-X for eight to ten therapist supervised sessions lasting 45 to 60 minutes. Game difficulties were adjusted to mach the ability of the participant and progressed as participant ability improved. Finally the participants were reassessed (T3). The Fugl Meyer Upper Limb assessment (FMA-UL) (Fugl-Meyer, 1975) was the primary outcome measure. Additional outcome measures were the DASH (Disabilities of the Arm, Shoulder and Hand), a questionnaire for self reported symptoms and abilities in certain activities (Beaton et al., 2001) and the WOLF test (Wolf Motor Function Test for functional movement (Wolf & Catlin, 2001)).

## **3.2.3 Able-X results**

158 Augmented Reality – Some Emerging Application Areas

motion sensitive game controller (CyWee Z, Taiwan) that interfaces with a PC. The Able-X couples the arms so that the unaffected arm can assist movement of the hemiparetic arm.

A suite of computer games provided a graduated series of physical challenges, from target

hitting games, to faster sports games, as well as some casual and puzzle games.

Fig. 2. Against-gravity bilateral exercise device that interacts with computer games.

participants the dominant hand was affected while in five it was the non-dominant.

**3.2.2 Able-X assessment, outcome measures and Intervention** 

To trial the system fourteen participants with post-stroke UL hemiparesis were recruited: their mean age was 71 years (range 47-85); nine were male, five female; time since stroke one to six years; eight had a stroke in their left hemisphere and six in their right and in nine

Assessments were conducted after enrolment in the study (T0). A 2.5 week sham-intervention was provided which consisted of playing mouse based computer games on a personal computer using their unaffected arm for eight to ten sessions, each lasting at least 45 minutes. Following this, participants were re-assessed (T1). There was a 2 ½ week period where no

**3.2 Able-X pilot study 3.2.1 Able-X participants**  The Able-X intervention resulted in a significant improvement in UL motor performance as measured by the FMA-UL (Table 1). The mean improvement post intervention (T3) ranged from 4.2 (compared to T2) to 5.2 (compared to T1). These results are especially encouraging in light of the fact that the group comprised adults with chronic stroke and the cohort was heterogeneous (FMA-UL ranged from 14 to 65 at inclusion). Post-intervention focus groups showed that participants enjoyed the experience and reported perceived gains in UL movement, concentration and balance.


Table 1. Means ± standard deviation of the outcome measures pre-control (T0), post-control/ pre-washout (T1), post-washout / pre-intervention (T2), and post-intervention (T3).

#### **3.3 Able-X home care pilot study**

To test use of the system in an unsupervised home situation, three participants from the above trial were asked to use the system at home for 8 weeks. Participants were assessed prior to the intervention (T1); the systems were left with them for up to 61 days; after which they were reassessed (T2). As patient self-reports are suggested as the best method of evaluating adherence to home-based physiotherapy (Bassett, 2003), participants diaries of usage and the Intrinsic Motivation Inventory (McAuley, 1987) were the outcome measures.


Table 2. IMI and diary results for unsupervised homecare therapy using Able-X.

The diaries (Table 2) indicated that all of participants continued therapy at a rate of from 4.5 to 5.5 sessions per week and maintained that throughout the 8 week period. The IMI results

Augmented Reality Assisted Upper Limb Rehabilitation Following Stroke 161

presentation, type and amount of feedback provided (Holden, 2005). Particular elements can be enriched or downplayed as required to exploit principles of motor learning and plasticity (Rose et al., 2005; Levin 2011). Particular elements of a task can be shown selectively to highlight the relevant information. For example, the specifications for reaching movements are determined from observation of the end point trajectory of a moving limb. This can therefore be highlighted to focus the attention of the patient onto the relevant information. Effectively this reduces the complexity of the environment in terms of clutter and

Feedback is a critical component to motor learning which is easily manipulated in an augmented environment and can be delivered in a variety of different ways. Holden (2005) suggests creation of a virtual teacher for demonstration of movements. The virtual teacher has the advantage that they can perform an unlimited number of repetitions (unlike a human therapist). Many studies have investigated the VR environment for stroke rehabilitation, but few have shown effective transfer of training to the real world. Pridmore et al. (2004) discussed how moving VR environments closer to the real world will ease the transfer of rehabilitation activities into daily life. This means moving to the right of the continuum shown in fig 3, i.e. making use of AR. Rand et al. (2005) found that older adults prefer a head-mounted display to seeing an image projected on a screen, which is an example of elderly users requiring the environment to be closer to reality than younger

A significant advantage of AR is that it provides for activation of mirror neurons, thought to be the basis of imitative learning. A mirror neuron is one that fires when someone either acts or observes a goal directed action performed by another (Rizzolatti & Craighero, 2004). However, they do not fire when a simple movement is executed or observed, so a task must be associated with the movement (Gallese et al., 1996). Mirror neurons have a direct input to the motor cortex and a role for these neurons in recovery from stroke has been suggested by several authors (Eng et al., 2007; Ramachandran & Altschuler, 2009; Sathian et al., 2000). Human fMRI and TMS studies have shown increased mirror neuron activity and formation of motor memories during learning of new motor tasks by imitation or action-observation

Observation of another individual performing a motor training task (action observation) results in increased cortical excitability of the primary motor cortex and can enhance the beneficial effects of motor training on motor memory formation in patients with chronic stroke (Celnik et al., 2008). The mirror system plays the role of 'movement organizer' and is sensitive therefore to observation and thereon action. Preliminary data indicate that the

unnecessary information that may distract and/or frustrate the patient.

Fig. 3. Technologies across the virtual divide (Pridmore et al., 2004).

(Buccino et al., 2004; Stefan et al., 2005).

people do.

suggest that the therapy successfully motivated the participants. Patient motivation and compliance with prescribed exercise/practice is a key component of successful home-based therapy or telerehabilitation. We have shown this can be provided with a computer game therapy system that did not rely on regular therapist prompting to exercise.

## **4. Augmented reality rehabilitation**

In contrast to VR, Augmented Reality (AR) superimposes a computer-generated image on a user's view of the real world, thus providing a composite view, which has significant advantages over VR for neurorehabilitation. Neurological injuries, such as stroke, cause considerable environmental impoverishment due to the stroke survivor's either limited ability or inability to interact physically with their environment (Stein et al., 1995). There is substantial literature on the negative impact of impoverished environments and their effects on the cortex in rats and primates (Renner & Rosenzweig, 1987). Similarly there is evidence to suggest that enriching an environment has a beneficial effect on both behavioural and cognitive function in brain damaged animals. Although more research on humans is necessary, there is general agreement that this holds for humans too (Rose et al., 1998). There is widespread agreement amongst clinicians that the reduction of interaction with the physical environment resulting from stroke is detrimental to recovery of function (Rose et al., 2005). Thus a key factor in successful stroke rehabilitation is whether the patient's level of interaction with their environment can be dramatically increased.

Augmented environments offer a unique rehabilitation solution for stroke survivors in that the information within the environment and the way in which patients interact with this information can be controlled very specifically (Levin, 2011; Rose et al., 1998). Many of the physical issues giving rise to a stroke-induced impoverished environment can be solved or circumvented using augmented environments. For example, a given sensory aspect or modality can be emphasised or augmented to make up for other sensory loss. If a patient is unable to walk through their physical environment, VR or AR can be combined concurrently with a robotic walking aid, e.g. Hocoma Locomat, therapy to provide visual flow and thus the overall sensation of moving through an environment. In fact the computer technology underlying AR is ideally suited to delivering an enriched environment to humans with disabilities as it is generally possible to cater for whatever motor capability an individual may have. Similarly the particular sensory aspects of an environment may be augmented to offset partial sensory loss.

Because of the above, augmented environments have the potential to be highly engaging for patients undergoing rehabilitation, leading to greater levels of compliance. Approximately 65% of patients are likely to be non-adherent to physical therapy rehabilitation programmes, to varying degrees (Bassett, 2003). Additionally, Tinson (1989) found that stroke patients within a rehabilitation unit spent 30 - 40% of their time in "disengaged and inactive tasks" and only 30-60 minutes per day in formal therapy. It is crucial that a person engaged in rehabilitation following neurological injury is engaged in their therapy and an augmented reality rehabilitation has the potential to fill some of the patient "down-time" with valuable therapeutic activity.

AR environments and the skills practiced or learned within them can be tailored according to a patient's impairment(s) in terms of task complexity, required response(s), sensory 160 Augmented Reality – Some Emerging Application Areas

suggest that the therapy successfully motivated the participants. Patient motivation and compliance with prescribed exercise/practice is a key component of successful home-based therapy or telerehabilitation. We have shown this can be provided with a computer game

In contrast to VR, Augmented Reality (AR) superimposes a computer-generated image on a user's view of the real world, thus providing a composite view, which has significant advantages over VR for neurorehabilitation. Neurological injuries, such as stroke, cause considerable environmental impoverishment due to the stroke survivor's either limited ability or inability to interact physically with their environment (Stein et al., 1995). There is substantial literature on the negative impact of impoverished environments and their effects on the cortex in rats and primates (Renner & Rosenzweig, 1987). Similarly there is evidence to suggest that enriching an environment has a beneficial effect on both behavioural and cognitive function in brain damaged animals. Although more research on humans is necessary, there is general agreement that this holds for humans too (Rose et al., 1998). There is widespread agreement amongst clinicians that the reduction of interaction with the physical environment resulting from stroke is detrimental to recovery of function (Rose et al., 2005). Thus a key factor in successful stroke rehabilitation is whether the patient's level

Augmented environments offer a unique rehabilitation solution for stroke survivors in that the information within the environment and the way in which patients interact with this information can be controlled very specifically (Levin, 2011; Rose et al., 1998). Many of the physical issues giving rise to a stroke-induced impoverished environment can be solved or circumvented using augmented environments. For example, a given sensory aspect or modality can be emphasised or augmented to make up for other sensory loss. If a patient is unable to walk through their physical environment, VR or AR can be combined concurrently with a robotic walking aid, e.g. Hocoma Locomat, therapy to provide visual flow and thus the overall sensation of moving through an environment. In fact the computer technology underlying AR is ideally suited to delivering an enriched environment to humans with disabilities as it is generally possible to cater for whatever motor capability an individual may have. Similarly the particular sensory aspects of an environment may be

Because of the above, augmented environments have the potential to be highly engaging for patients undergoing rehabilitation, leading to greater levels of compliance. Approximately 65% of patients are likely to be non-adherent to physical therapy rehabilitation programmes, to varying degrees (Bassett, 2003). Additionally, Tinson (1989) found that stroke patients within a rehabilitation unit spent 30 - 40% of their time in "disengaged and inactive tasks" and only 30-60 minutes per day in formal therapy. It is crucial that a person engaged in rehabilitation following neurological injury is engaged in their therapy and an augmented reality rehabilitation has the potential to fill some of the patient "down-time" with valuable

AR environments and the skills practiced or learned within them can be tailored according to a patient's impairment(s) in terms of task complexity, required response(s), sensory

therapy system that did not rely on regular therapist prompting to exercise.

of interaction with their environment can be dramatically increased.

**4. Augmented reality rehabilitation** 

augmented to offset partial sensory loss.

therapeutic activity.

presentation, type and amount of feedback provided (Holden, 2005). Particular elements can be enriched or downplayed as required to exploit principles of motor learning and plasticity (Rose et al., 2005; Levin 2011). Particular elements of a task can be shown selectively to highlight the relevant information. For example, the specifications for reaching movements are determined from observation of the end point trajectory of a moving limb. This can therefore be highlighted to focus the attention of the patient onto the relevant information. Effectively this reduces the complexity of the environment in terms of clutter and unnecessary information that may distract and/or frustrate the patient.

Feedback is a critical component to motor learning which is easily manipulated in an augmented environment and can be delivered in a variety of different ways. Holden (2005) suggests creation of a virtual teacher for demonstration of movements. The virtual teacher has the advantage that they can perform an unlimited number of repetitions (unlike a human therapist). Many studies have investigated the VR environment for stroke rehabilitation, but few have shown effective transfer of training to the real world. Pridmore et al. (2004) discussed how moving VR environments closer to the real world will ease the transfer of rehabilitation activities into daily life. This means moving to the right of the continuum shown in fig 3, i.e. making use of AR. Rand et al. (2005) found that older adults prefer a head-mounted display to seeing an image projected on a screen, which is an example of elderly users requiring the environment to be closer to reality than younger people do.

Fig. 3. Technologies across the virtual divide (Pridmore et al., 2004).

A significant advantage of AR is that it provides for activation of mirror neurons, thought to be the basis of imitative learning. A mirror neuron is one that fires when someone either acts or observes a goal directed action performed by another (Rizzolatti & Craighero, 2004). However, they do not fire when a simple movement is executed or observed, so a task must be associated with the movement (Gallese et al., 1996). Mirror neurons have a direct input to the motor cortex and a role for these neurons in recovery from stroke has been suggested by several authors (Eng et al., 2007; Ramachandran & Altschuler, 2009; Sathian et al., 2000). Human fMRI and TMS studies have shown increased mirror neuron activity and formation of motor memories during learning of new motor tasks by imitation or action-observation (Buccino et al., 2004; Stefan et al., 2005).

Observation of another individual performing a motor training task (action observation) results in increased cortical excitability of the primary motor cortex and can enhance the beneficial effects of motor training on motor memory formation in patients with chronic stroke (Celnik et al., 2008). The mirror system plays the role of 'movement organizer' and is sensitive therefore to observation and thereon action. Preliminary data indicate that the

Augmented Reality Assisted Upper Limb Rehabilitation Following Stroke 163

Fig. 4. Able-B to allow an individual to exercise their hemiparetic right arm, using AR to

attached to the hand acts as a "computer mouse" to play specially designed games.

A web camera is used to display the hemiparetic arm (Fig 3) on the monitor such that it appears to interact with the computer games. Using color-tracking software, a colored patch

Fig. 5. The series of computer images of the four AR games used for arm reaching exercises.

show their hemiparetic arm interacting with a computer game.

action / observation approach may produce significant clinical results (Ertelt et al., 2007). The action/observation/execution mechanism requires the stroke survivor to view their hemiparetic arm carrying out the task, even though it may be too weak to function independently and this can stimulate the cortical mirror neurone system.

A related treatment, mirror box therapy has been shown to be effective in UL stroke therapy (Sütbeyaz et al., 2007; Yavuzer et al., 2008). Mirror box therapy, was originally developed to reduce phantom pain in amputees by Ramachandran et al. (1995). The patient places their stump into a box which has mirrors arranged so that the image of the good limb appears to be superimposed onto the stump of the amputated or phantom limb. When the patient moves their good limb, it appears as if their lost limb is also moving and through the use of this artificial visual feedback it becomes possible for the patient to "move" the phantom limb, and to unclench it from potentially painful positions. This demonstrates a real-world attempt to create an AR environment for rehabilitation.

#### **4.1 The Able-B: an AR mirror box for stroke rehabilitation**

AR therapy may be utilised to stimulate a similar experience to mirror box therapy and we have developed the Able-B (Figure 3) based on these principles as well as bilateral UL exercise therapy (Caraugh et al., 2005; McCombe-Waller et al., 2008). This device is designed to facilitate re-learning of reaching movements for patients who are either unable or have great difficulty moving their hemiparetic arm. The patient can use the strength of their unaffected arm to assist the hemiparetic arm as required while observing the mirrored movement of the impaired arm through the AR system. The patient sits with both arms in the moulded forearm supports, held in place using padded velcro straps. The hands rest on either a palmar mound or a joy-stick style handle, depending on user preference. The Able-B provides a platform for high repetition, intense bilateral exercise within a relatively low-cost package that decreases therapist supervision requirements and allows persons with more severe UL hemiparesis to exercise independently (Sampson et al., 2011).

#### **4.2 The software**

It is necessary to create a specific set of games for rehabilitation of stroke survivors, who are generally over 65 years old. Stroke often results in reduced attention span, short term memory and problem solving skills (anon, 2009) and most computer games are very fast and provide negative feedback when losing (IJsselsteijn, 2007). We have developed a suite of computer games to provide a graduated series of physical challenges, from stationary target hitting games, to strategic target hitting games, to moving target hitting games. These games always provide positive feedback for actions performed and do not have significant memory or problem solving challenges. The graphics are large sized, in high contrast colours and the movements do not require fast reactions. Use of sound is maximised, so that every time an object appears and every time an interaction occurs, a sound is provided.

All games required large cursor movements in both horizontal and vertical directions. The games are presented using a simple AR environment using a web camera and conventional computer monitor as display. The computer game characters were superimposed over the image of the stroke-affected arm and hand (fig 5) to create an action/observation/ execution scenario within an affordable and easy to implement AR environment.

162 Augmented Reality – Some Emerging Application Areas

action / observation approach may produce significant clinical results (Ertelt et al., 2007). The action/observation/execution mechanism requires the stroke survivor to view their hemiparetic arm carrying out the task, even though it may be too weak to function

A related treatment, mirror box therapy has been shown to be effective in UL stroke therapy (Sütbeyaz et al., 2007; Yavuzer et al., 2008). Mirror box therapy, was originally developed to reduce phantom pain in amputees by Ramachandran et al. (1995). The patient places their stump into a box which has mirrors arranged so that the image of the good limb appears to be superimposed onto the stump of the amputated or phantom limb. When the patient moves their good limb, it appears as if their lost limb is also moving and through the use of this artificial visual feedback it becomes possible for the patient to "move" the phantom limb, and to unclench it from potentially painful positions. This demonstrates a real-world

AR therapy may be utilised to stimulate a similar experience to mirror box therapy and we have developed the Able-B (Figure 3) based on these principles as well as bilateral UL exercise therapy (Caraugh et al., 2005; McCombe-Waller et al., 2008). This device is designed to facilitate re-learning of reaching movements for patients who are either unable or have great difficulty moving their hemiparetic arm. The patient can use the strength of their unaffected arm to assist the hemiparetic arm as required while observing the mirrored movement of the impaired arm through the AR system. The patient sits with both arms in the moulded forearm supports, held in place using padded velcro straps. The hands rest on either a palmar mound or a joy-stick style handle, depending on user preference. The Able-B provides a platform for high repetition, intense bilateral exercise within a relatively low-cost package that decreases therapist supervision requirements and allows persons with more

It is necessary to create a specific set of games for rehabilitation of stroke survivors, who are generally over 65 years old. Stroke often results in reduced attention span, short term memory and problem solving skills (anon, 2009) and most computer games are very fast and provide negative feedback when losing (IJsselsteijn, 2007). We have developed a suite of computer games to provide a graduated series of physical challenges, from stationary target hitting games, to strategic target hitting games, to moving target hitting games. These games always provide positive feedback for actions performed and do not have significant memory or problem solving challenges. The graphics are large sized, in high contrast colours and the movements do not require fast reactions. Use of sound is maximised, so that every time an object appears and every time an interaction occurs, a sound is provided.

All games required large cursor movements in both horizontal and vertical directions. The games are presented using a simple AR environment using a web camera and conventional computer monitor as display. The computer game characters were superimposed over the image of the stroke-affected arm and hand (fig 5) to create an action/observation/ execution

independently and this can stimulate the cortical mirror neurone system.

attempt to create an AR environment for rehabilitation.

**4.1 The Able-B: an AR mirror box for stroke rehabilitation** 

severe UL hemiparesis to exercise independently (Sampson et al., 2011).

scenario within an affordable and easy to implement AR environment.

**4.2 The software** 

Fig. 4. Able-B to allow an individual to exercise their hemiparetic right arm, using AR to show their hemiparetic arm interacting with a computer game.

A web camera is used to display the hemiparetic arm (Fig 3) on the monitor such that it appears to interact with the computer games. Using color-tracking software, a colored patch attached to the hand acts as a "computer mouse" to play specially designed games.

Fig. 5. The series of computer images of the four AR games used for arm reaching exercises.

Augmented Reality Assisted Upper Limb Rehabilitation Following Stroke 165

Fig. 6. AR arm rehabilitation using polished board, showing the virtual butterfly net

To test the concept of integrating AR with a simple reaching exercise, four participants with UL hemiparesis as a result of stroke were recruited. The age ranged from 55 to 85; time since stroke ranged from 2 to 12 years; three were male and one female; two were left side hemiparetic, and all were right hand dominant; all participants were free of other

The FMA-UL was the primary outcome measure, but additional outcome measures were the WOLF and the DASH. Assessors conducted pre- (T0) and post-intervention (T1) assessments. Participants completed 4 weeks of AR and polished board intervention consisting of 8 supervised sessions which lasted 45 minutes each. An overhead mounted web camera using the computer vision algorithm ARToolkit (HIT Lab, University of Canterbury, NZ), tracked the position of a fiducial marker attached to the user's hand. A computer game showed a butterfly net superimposed over the user's hand. The user placed the marker on a cross marked in the centre of the screen to start the game. Virtual butterflies "flew" from eight

**4.5.2 AR and polished board: assessment, outcome measures and intervention** 

superimposed on the image of the hand.

**4.5 AR and polished board case study 4.5.1 AR and polished board: participants** 

neurological deficits.

#### **4.3 Able-B case study**

## **4.3.1 Able-B participants**

To test the efficacy of the Able-B, five post-stroke participants with UL hemiparesis were recruited: their mean age was 61.8 years (range 45 – 76 years); 3 were male, 2 were female; all were right dominant with 3 participants having right hemiparesis and 2 having left hemiparesis. The mean time post stroke that the intervention was started was 45.2 weeks (range 9 – 64 weeks) and participants were free of other neurological deficits.

## **4.3.2 Able-B assessment, Outcome measures and Intervention**

Assessors conducted blinded assessments of the trial participants, in the week prior to the start of intervention (T0) and in the week following the cessation of interventions (T1). The FMA-UL was the primary outcome measure and participant motivation was assessed using the Intrinsic Motivation Inventory (IMI) after the intervention.

Participants completed 6 weeks of Able-B + AR intervention consisting of therapist supervised, 45 minute sessions, 4x/week. Game difficulties were adjusted to match the ability of the participant and progressed as participant ability improved.

#### **4.3.3 Able-B results**

Overall the results indicate a positive effect from intervention using the Able-B system (Table 3), with the FMA-UL score increase ranging from 1 to 5 points after 6 weeks of therapy.


Table 3. FMA-UE scores before and after Able-B therapy.

The results of the IMI ranged from 80% for participant 4 to 95% for participant 2, indicating that all participants were highly motivated to comply with the Able-B therapy.

#### **4.4 AR rehabilitation using a polished board**

Polished board or armskate exercising is designed for patients who have some ability to move their hemiparetic arm, provided they are supported against gravity e.g. by sliding their arm across a polished table top. King et al. (2010) describes an affordable, computerised table-based exercise system using AR in conjunction with such an exercise regime (fig 6).

164 Augmented Reality – Some Emerging Application Areas

To test the efficacy of the Able-B, five post-stroke participants with UL hemiparesis were recruited: their mean age was 61.8 years (range 45 – 76 years); 3 were male, 2 were female; all were right dominant with 3 participants having right hemiparesis and 2 having left hemiparesis. The mean time post stroke that the intervention was started was 45.2 weeks

Assessors conducted blinded assessments of the trial participants, in the week prior to the start of intervention (T0) and in the week following the cessation of interventions (T1). The FMA-UL was the primary outcome measure and participant motivation was assessed using

Participants completed 6 weeks of Able-B + AR intervention consisting of therapist supervised, 45 minute sessions, 4x/week. Game difficulties were adjusted to match the

Overall the results indicate a positive effect from intervention using the Able-B system (Table 3), with the FMA-UL score increase ranging from 1 to 5 points after 6 weeks of

The results of the IMI ranged from 80% for participant 4 to 95% for participant 2, indicating

Polished board or armskate exercising is designed for patients who have some ability to move their hemiparetic arm, provided they are supported against gravity e.g. by sliding their arm across a polished table top. King et al. (2010) describes an affordable, computerised table-based exercise system using AR in conjunction with such an exercise

Participant T0 T1 Change

1 16 21 5 2 15 18 3 3 7 11 4 4 19 20 1 5 7 9 2

that all participants were highly motivated to comply with the Able-B therapy.

(range 9 – 64 weeks) and participants were free of other neurological deficits.

**4.3.2 Able-B assessment, Outcome measures and Intervention** 

the Intrinsic Motivation Inventory (IMI) after the intervention.

ability of the participant and progressed as participant ability improved.

Fugl-Meyer Upper extremity scores (/66)

Table 3. FMA-UE scores before and after Able-B therapy.

**4.4 AR rehabilitation using a polished board** 

(Wk 0) (Wk 7)

**4.3 Able-B case study 4.3.1 Able-B participants** 

**4.3.3 Able-B results** 

therapy.

regime (fig 6).

Fig. 6. AR arm rehabilitation using polished board, showing the virtual butterfly net superimposed on the image of the hand.

## **4.5 AR and polished board case study**

#### **4.5.1 AR and polished board: participants**

To test the concept of integrating AR with a simple reaching exercise, four participants with UL hemiparesis as a result of stroke were recruited. The age ranged from 55 to 85; time since stroke ranged from 2 to 12 years; three were male and one female; two were left side hemiparetic, and all were right hand dominant; all participants were free of other neurological deficits.

#### **4.5.2 AR and polished board: assessment, outcome measures and intervention**

The FMA-UL was the primary outcome measure, but additional outcome measures were the WOLF and the DASH. Assessors conducted pre- (T0) and post-intervention (T1) assessments.

Participants completed 4 weeks of AR and polished board intervention consisting of 8 supervised sessions which lasted 45 minutes each. An overhead mounted web camera using the computer vision algorithm ARToolkit (HIT Lab, University of Canterbury, NZ), tracked the position of a fiducial marker attached to the user's hand. A computer game showed a butterfly net superimposed over the user's hand. The user placed the marker on a cross marked in the centre of the screen to start the game. Virtual butterflies "flew" from eight

Augmented Reality Assisted Upper Limb Rehabilitation Following Stroke 167

Fig. 7. Able-M, a gravity supported arm exercise device that interacts with computer games.

impairment, or weakness presented by the patient. Fig 8 shows the relationship of the components of the system to the patient's strength as measured by the Oxford scale of

There is significant amount of evidence to show that learning in a virtual environment can transfer to the real environment. However, there is also evidence that transfer of rehabilitation training into real world activities of daily living is easier if the training is

muscle strength (Parkinson, 2000).

Fig. 8. IRL devices aligned to level of strength.

**6. Discussion** 

evenly spaced different directions, starting at the outer edge of the screen and traveling towards the centre at a fixed speed. The user's task was to move their hand to the outside of the screen to "catch" the butterfly in the net (Figure 6). After the butterfly was caught, the user moved back to the centre cross and the next butterfly appeared. After all 8 butterflies had been caught, the time taken, and therefore the distance moved in each direction was shown with a graph.

## **4.5.3 AR and polished board case study results**

This study indicated a positive effect of the therapy (Table 4) with two participants increasing their FMA-UL scores and Wolf times significantly. Interestingly their selfreported standard of life as shown by the DASH, showed a decrease for two participants, possibly as a result of the focus of attention on their arm impairment.


Table 4. Results of the outcome measures pre and post ARS intervention for the 4 participants.

## **5. A computer assisted upper limb rehabilitation system for stroke**

We have developed a computer assisted UL rehabilitation system which combines the advantages of both AR and VR with three assistive devices that provide a lower cost alternative to rehabilitation robotics for training of shoulder and elbow reaching. The Able-X and Able-B described above are suitable for people with severe and mild arm movement impairments. A third device, the Able-M has been designed to enable the delivery of the AR with polished board exercises in a rehabilitation setting.

#### **5.1 The able-M**

The Able-M is designed for patients who have some ability to move their hemiparetic arm, provided they are supported against gravity. It is based on a traditional arm-skate or sliding board to provide gravity support while practising reaching exercises across a tabletop. The Able-M (Fig 7) interacts with the computer in the same way as a wireless computer mouse, controlling the cursor of the PC. A button, which acts as the mouse clicker switch, is flexibly mounted to position it around the dorsal side of the hand or fingers, hence requiring finger or wrist extension exercises to operate.

#### **5.2 The suite of devices**

The assistive devices allow people with hemiparesis to interact with a computer during rehabilitation exercises, providing a system which can are aligned to the degree of 166 Augmented Reality – Some Emerging Application Areas

evenly spaced different directions, starting at the outer edge of the screen and traveling towards the centre at a fixed speed. The user's task was to move their hand to the outside of the screen to "catch" the butterfly in the net (Figure 6). After the butterfly was caught, the user moved back to the centre cross and the next butterfly appeared. After all 8 butterflies had been caught, the time taken, and therefore the distance moved in each direction was

This study indicated a positive effect of the therapy (Table 4) with two participants increasing their FMA-UL scores and Wolf times significantly. Interestingly their selfreported standard of life as shown by the DASH, showed a decrease for two participants,

No T0 T1 Change T0 T1 Change T0 T1 Change 1 33 33 0 34.3 24.5 -9.8 33.3 34.2 0.9 2 10 17 7 84.0 77.0 -7.0 67.5 71.7 3.2 3 31 31 0 43.4 44.4 1.0 - - - 4 36 42 6 4.2 4.1 -0.1 21.7 40.0 18.3

We have developed a computer assisted UL rehabilitation system which combines the advantages of both AR and VR with three assistive devices that provide a lower cost alternative to rehabilitation robotics for training of shoulder and elbow reaching. The Able-X and Able-B described above are suitable for people with severe and mild arm movement impairments. A third device, the Able-M has been designed to enable the delivery of the AR

The Able-M is designed for patients who have some ability to move their hemiparetic arm, provided they are supported against gravity. It is based on a traditional arm-skate or sliding board to provide gravity support while practising reaching exercises across a tabletop. The Able-M (Fig 7) interacts with the computer in the same way as a wireless computer mouse, controlling the cursor of the PC. A button, which acts as the mouse clicker switch, is flexibly mounted to position it around the dorsal side of the hand or fingers, hence requiring finger

The assistive devices allow people with hemiparesis to interact with a computer during rehabilitation exercises, providing a system which can are aligned to the degree of

shown with a graph.

participants.

**5.1 The able-M** 

**4.5.3 AR and polished board case study results** 

possibly as a result of the focus of attention on their arm impairment.

Fugl-Meyer (/44) Wolf, average time /task (sec) DASH (/100)

Table 4. Results of the outcome measures pre and post ARS intervention for the 4

**5. A computer assisted upper limb rehabilitation system for stroke** 

with polished board exercises in a rehabilitation setting.

or wrist extension exercises to operate.

**5.2 The suite of devices** 

Fig. 7. Able-M, a gravity supported arm exercise device that interacts with computer games.

impairment, or weakness presented by the patient. Fig 8 shows the relationship of the components of the system to the patient's strength as measured by the Oxford scale of muscle strength (Parkinson, 2000).

Fig. 8. IRL devices aligned to level of strength.

## **6. Discussion**

There is significant amount of evidence to show that learning in a virtual environment can transfer to the real environment. However, there is also evidence that transfer of rehabilitation training into real world activities of daily living is easier if the training is

Augmented Reality Assisted Upper Limb Rehabilitation Following Stroke 169

A comparison should be made between the use of VR with avatars, versus AR in a stroke affected population. Adamovich et al. (2009) suggested that VR avatars can create an observation condition and hence it may be possible to use avatars, rather than real time images of a person's limbs, to provide positive rehabilitaion outcomes. The authors have recorded anecdotal statements from study participants that they do not like viewing their

The acute phase of stroke recovery is the optimal period for maximum recovery following stroke and so research must move into that area. Currently most researchers work with participants who are in the chronic stage of their stroke recovery, due to their stable baseline

AR rehabilitation has a large potential for benefit to stroke survivors and the community as it can be delivered in a home environment. But it is necessary to integrate these systems with homecare, using telerehabilitation services, to ensure adherence to the programme and tailoring the service to the individual's requirements. Such systems must be robust for home installation by non-specialised technicians (e.g. physiotherapists) and daily use by the

It is not yet clear what mix of cognitive and physical rehabilitation challenges should be provided for optimal recovery following stroke. For example Schimmelpfennig (2000) found that, following orthopaedic or phlebological procedures, patients who underwent mental activation training, showed a greater motivation in rehabilitation and recovered their movement coordination quicker. This area is an ideal research opportunity for AR

Environmental enrichment, in addition to being beneficial in its own right, can optimise other treatments designed to restore function following brain damage. There is evidence of enrichment induced neuroplastic change in primates and it would be interesting to see how

Neurological injury leads to environmental impoverishment and AR provides the ability to enrich the environment and increase a stroke survivor's interaction / experiences, leading to

Improvements in UL function and isometric strength were observed during studies of augmented physical therapy utilising a combination of assistive devices and AR that allow people with UL disabilities that range from very severe hemiparesis to almost able-

Engaging in AR therapy using simple assistive devices seems to be a cost effective alternative to other forms of therapy such as robotic therapy, catering for repetitive selfsupported UL exercises. It also can integrate easily with conventional physiotherapy.

Overall these studies suggest that further research on a larger sample size and range of stroke survivors with UL hemiparesis is warranted to provide a greater level of clinical

and the lack of interference with medical interventions by research protocols.

individual who may have a cognitive disability as a result of the injury.

this translates to humans, especially in relation to an AR environment.

beneficial effects in terms of engagement and rehabilitation outcomes.

evidence as to the effect of the system for people with UL hemiparesis.

stroke-affected limb, due to it's "old look".

therapies.

**8. Conclusions** 

bodied.

carried out in as close to the real environment as possible. Therefore, the AR environment has potential for better rehabilitation outcomes than a VR environment.

Mirror neurons were first described in 1992 so the use of mirror therapy is relatively new. AR seems to be an ideal method for bringing mirror therapy to people with a variety of disabilities or injuries. The development of a suite of devices that can bring this technology into a homecare situation, for people to rehabilitate themselves, with minimal supervision from clinicians, has potential to allow delivery of rehabilitation to a large numbers in a cost effective manner. The operating range of the three IRL devices is wide enough to provide smooth transition and overlap from one to the next, thereby allowing a person to exercise at their own rate until they are able to "graduate" from one to the next. The suite demonstrates the application of a systems approach to the rehabilitation of the UL.

Although the quantity of UL repetitions per session were not recorded, they were estimated (conservatively based on the game parameters set during sessions) to range from 600 to 1000 per 45 min session for each of the 3 devices, depending upon which games were used. Lang et al. (2009) found that in typical stroke rehabilitation units, 32 UL repetitions per session was typical, so this is greater than many conventional UL therapy sessions and would compare favourably with other non-robotic platforms with or without AR or VR. The Able-X results are comparable to more intensive robotic-aided therapies reported previously, with improvements on the FMA-UL of between 3 and 6 (Prange et al., 2006).

Computer game play has been shown to be very motivating for stroke survivors to undertake as part of a rehabilitation programme, even without supervision. However, it is important that the game play actually constitutes task-based movements, because the mirror neuron system is only activated when task are undertaken, rather than when passive movements are made. Further research is required to ensure that AR gameplay activates these neural regions, leading to learning and recovery.

### **7. Future directions / considerations**

Our research has been carried out with small numbers in pilot trial situations. Larger randomised, controlled trials across multiple centres are necessary for clinical validation of these devices. Future clinical research should be aimed at developing protocols for rehabilitation using the range of devices and software or games that may be used with the devices. Additionally, for delivery of a rehabilitation service to stroke survivors, products must meet medical devices standards.

Brain imaging (e.g. fMRI, EEG) should be incorporated into future research programme. This will allow clinicians to determine which patients will benefit most from specific treatments (e.g. bilateral therapy vs. unilateral therapy or AR versus VR environments). The importance of including information on lesion size and location in the planning of stroke treatment trials has also been highlighted. Incorporation of brain imaging can assist with the planning and prescription of rehabilitation (Hamzei et al., 2006). The development of functional imaging promises major advances within neuroscience and there is significant potential for combining these methods with virtual environments. The combination of neuroimaging and activity in virtual environments could be used to investigate the extent to which interaction with virtual and real environments is cognitively equivalent.

A comparison should be made between the use of VR with avatars, versus AR in a stroke affected population. Adamovich et al. (2009) suggested that VR avatars can create an observation condition and hence it may be possible to use avatars, rather than real time images of a person's limbs, to provide positive rehabilitaion outcomes. The authors have recorded anecdotal statements from study participants that they do not like viewing their stroke-affected limb, due to it's "old look".

The acute phase of stroke recovery is the optimal period for maximum recovery following stroke and so research must move into that area. Currently most researchers work with participants who are in the chronic stage of their stroke recovery, due to their stable baseline and the lack of interference with medical interventions by research protocols.

AR rehabilitation has a large potential for benefit to stroke survivors and the community as it can be delivered in a home environment. But it is necessary to integrate these systems with homecare, using telerehabilitation services, to ensure adherence to the programme and tailoring the service to the individual's requirements. Such systems must be robust for home installation by non-specialised technicians (e.g. physiotherapists) and daily use by the individual who may have a cognitive disability as a result of the injury.

It is not yet clear what mix of cognitive and physical rehabilitation challenges should be provided for optimal recovery following stroke. For example Schimmelpfennig (2000) found that, following orthopaedic or phlebological procedures, patients who underwent mental activation training, showed a greater motivation in rehabilitation and recovered their movement coordination quicker. This area is an ideal research opportunity for AR therapies.

Environmental enrichment, in addition to being beneficial in its own right, can optimise other treatments designed to restore function following brain damage. There is evidence of enrichment induced neuroplastic change in primates and it would be interesting to see how this translates to humans, especially in relation to an AR environment.

## **8. Conclusions**

168 Augmented Reality – Some Emerging Application Areas

carried out in as close to the real environment as possible. Therefore, the AR environment

Mirror neurons were first described in 1992 so the use of mirror therapy is relatively new. AR seems to be an ideal method for bringing mirror therapy to people with a variety of disabilities or injuries. The development of a suite of devices that can bring this technology into a homecare situation, for people to rehabilitate themselves, with minimal supervision from clinicians, has potential to allow delivery of rehabilitation to a large numbers in a cost effective manner. The operating range of the three IRL devices is wide enough to provide smooth transition and overlap from one to the next, thereby allowing a person to exercise at their own rate until they are able to "graduate" from one to the next. The suite demonstrates

Although the quantity of UL repetitions per session were not recorded, they were estimated (conservatively based on the game parameters set during sessions) to range from 600 to 1000 per 45 min session for each of the 3 devices, depending upon which games were used. Lang et al. (2009) found that in typical stroke rehabilitation units, 32 UL repetitions per session was typical, so this is greater than many conventional UL therapy sessions and would compare favourably with other non-robotic platforms with or without AR or VR. The Able-X results are comparable to more intensive robotic-aided therapies reported previously,

Computer game play has been shown to be very motivating for stroke survivors to undertake as part of a rehabilitation programme, even without supervision. However, it is important that the game play actually constitutes task-based movements, because the mirror neuron system is only activated when task are undertaken, rather than when passive movements are made. Further research is required to ensure that AR gameplay activates

Our research has been carried out with small numbers in pilot trial situations. Larger randomised, controlled trials across multiple centres are necessary for clinical validation of these devices. Future clinical research should be aimed at developing protocols for rehabilitation using the range of devices and software or games that may be used with the devices. Additionally, for delivery of a rehabilitation service to stroke survivors, products

Brain imaging (e.g. fMRI, EEG) should be incorporated into future research programme. This will allow clinicians to determine which patients will benefit most from specific treatments (e.g. bilateral therapy vs. unilateral therapy or AR versus VR environments). The importance of including information on lesion size and location in the planning of stroke treatment trials has also been highlighted. Incorporation of brain imaging can assist with the planning and prescription of rehabilitation (Hamzei et al., 2006). The development of functional imaging promises major advances within neuroscience and there is significant potential for combining these methods with virtual environments. The combination of neuroimaging and activity in virtual environments could be used to investigate the extent to

which interaction with virtual and real environments is cognitively equivalent.

has potential for better rehabilitation outcomes than a VR environment.

the application of a systems approach to the rehabilitation of the UL.

with improvements on the FMA-UL of between 3 and 6 (Prange et al., 2006).

these neural regions, leading to learning and recovery.

**7. Future directions / considerations** 

must meet medical devices standards.

Neurological injury leads to environmental impoverishment and AR provides the ability to enrich the environment and increase a stroke survivor's interaction / experiences, leading to beneficial effects in terms of engagement and rehabilitation outcomes.

Improvements in UL function and isometric strength were observed during studies of augmented physical therapy utilising a combination of assistive devices and AR that allow people with UL disabilities that range from very severe hemiparesis to almost ablebodied.

Engaging in AR therapy using simple assistive devices seems to be a cost effective alternative to other forms of therapy such as robotic therapy, catering for repetitive selfsupported UL exercises. It also can integrate easily with conventional physiotherapy.

Overall these studies suggest that further research on a larger sample size and range of stroke survivors with UL hemiparesis is warranted to provide a greater level of clinical evidence as to the effect of the system for people with UL hemiparesis.

Augmented Reality Assisted Upper Limb Rehabilitation Following Stroke 171

Carter, K.; Anderson, C.; Hacket, M.; Feigin, V.; Barber, P.; Broad, J. & Bonita, R. (2006).

Cauraugh, J. & Summers J. (2005). Neural Plasticity and bilateral movements: A

Celnik, P.; Webster, B.; Glasser, D. & Cohen, L. (2008). Effects of action observation on physical training after stroke. *Stroke*, Vol.39, No.6, pp. 1814-1820, ISSN 00392499 Eng, K.; Siekierka, E.; Pyk, P.; Chevrier, E.; Hauser, Y.; Cameirao, M.; Holper, L.; Hägni, K.;

Ertelt, D.; Small, S.; Solodkin, A.; Dettmers, C.; McNamara, A.; Binkofski, F. & Buccino, G.

Fasoli, S.; Krebs, H.; Stein, J.; Frontera, W. & Hogan, N. (2003). Effects of robotic therapy on

Feys, H.; De Weerdt, W.; Selz, B.; Cox Steck, G.; Spichiger, R.; Vereeck, L.; Putman, K. et al.

French, B.; Thomas, L.; Leathley, M.; Sutton, C.; McAdam, J.; Forster, A.; Langhorne, P. et al.

Fugl-Meyer, A.; Jääskö, L.; Leyman, I.; Olsson, S. & Steglind, S. (1975). The post-stroke

Gallese, V.; Fadiga, L.; Fogassi, L. & Rizzolatti, G. (1996). Action recognition in the premotor

Hamzei, F., Liepert, J., Dettmers, C., Weiller, C., & Rijntjes, M. (2006). Two different

Feys, H.; De Weerdt, W.; Selz, B.; Cox Steck, G.; Spichiger, R.; Vereeck, L.; Putman, K. & Van

*Society*, Vol.8, No.3, pp. 187-211; discussion pp. 212-219, ISSN: 1094-9313

fMRI and TMS. *NeuroImage*, Vol.31, No.2, pp. 710-720, ISSN: 1053-8119 Hijmans, J.; Hale, L.; Satherley, J.; McMillan, N. & King, M. (2011). Bilateral upper limb

publication in Journal of Rehabilitation Research and Development

after stroke. *NeuroImage,* Vol.36, No.2, pp. 164-73, ISSN: 1053-8119

*Rehabilitation*, Vol.84, No.4, pp. 477-482, ISSN 0003-9993

*Rehabilitation Medicine*, Vol.7, pp. 13-31, ISSN 0036-5505

cortex. *Brain*, Vol.119, pp. 593-609, ISSN 0006-8950

*Stroke*, Vol.29, No.4, pp. 785-792, ISSN 00392499

Vol.40, No.4, pp. 98-99, ISSN 00392499

1981 to 2003. *Stroke*, Vol.37, No.1, pp. 56-62, ISSN 00392499

510X

320, ISSN: 0301-0082

0140-0118

paretic hand. *Journal of the Neurological Sciences*, Vol.130, No.1, pp. 59-68, ISSN 0022-

Trends in ethnic disparities in stroke incidence in Auckland, New Zealand, during

rehabilitation approach for chronic stroke. *Progress in Neurobiology,* Vol.75, pp. 309-

Zimmerli, L.; Duff, A.; Schuster, C.; Bassetti, C.; Verschure, P. & Kiper, D. (2007). Interactive visuo-motor therapy system for stroke rehabilitation*. Medical & Biological Engineering & Computing,* Vol.45, No.9, (September), pp. 901-7, ISSN:

(2007). Action observation has a positive impact on rehabilitation of motor deficits

motor impairment and recovery in chronic stroke. *Archives of Physical Medicine and* 

(1998). Effect of a therapeutic intervention for the hemiplegic upper limb in the acute phase after stroke: A single-blind, randomized, controlled multicenter trial.

(2009). Repetitive task training for improving functional ability after stroke. *Stroke*,

patient: 1. A method for evaluation of physical performance. *Scandinavian Journal of* 

reorganization patterns after rehabilitative therapy: An exploratory study with

rehabilitation after stroke using a movement based game controller. Accepted for

Hoydonck, G. (1998). Effect of a Therapeutic Intervention for the Hemiplegic Upper Limb in the Acute Phase After Stroke. *Stroke,* Vol.29, pp. 785-792, ISSN 00392499 Holden, M. (2005). Virtual environments for motor rehabilitation: review. *Cyberpsychology &* 

*Behavior: The Impact of the Internet, Multimedia and Virtual Reality on Behavior and* 

The system has the potential to provide a calibrated range of exercises which can be selected to suit the physical and cognitive abilities of a patient ranging from severe hemiparesis to almost able-bodied or integrated as a systems approach to UL rehabilitation.

#### **9. Acknowledgements**

The authors wish to thank the stroke survivors and clinicians who willingly gave their time and thoughts during development of the interventions described here. All studies involving human participants were approved by the University of Otago, NZ Human Ethics Committee or by the NZ Health and Disabilities, Upper South A, Regional Ethic Committee. We thank Im-Able Ltd., New Zealand, for supporting this research through the supply of Able-X and Able-M devices.

#### **10. References**


http://www.ninds.nih.gov/disorders/stroke/poststrokerehab.htm

Anon. (2009). *Recovery After Stroke: Thinking and Cognition.* National Stroke Association. available from

http://www.stroke.org/site/DocServer/NSAFactSheet\_Cognition.pdf?docID=986


170 Augmented Reality – Some Emerging Application Areas

The system has the potential to provide a calibrated range of exercises which can be selected to suit the physical and cognitive abilities of a patient ranging from severe hemiparesis to

The authors wish to thank the stroke survivors and clinicians who willingly gave their time and thoughts during development of the interventions described here. All studies involving human participants were approved by the University of Otago, NZ Human Ethics Committee or by the NZ Health and Disabilities, Upper South A, Regional Ethic Committee. We thank Im-Able Ltd., New Zealand, for supporting this research through the supply of

Adamovich, S.; August, K.; Merians, A. & Tunik, E. (2009). A virtual reality-based system

Adamovich, S.; Fluet, G.; Merians, A.; Mathai, A. & Qiu, Q. (2009). Incorporating haptic

Adamson, J.; Beswick, A. & Ebrahim, S. (2004). Is stroke the most common cause of

Anon. (2007). *Post-Stroke Rehabilitation Fact Sheet.* National Institute Of Neurological

Anon. (2009). *Recovery After Stroke: Thinking and Cognition.* National Stroke Association.

 http://www.stroke.org/site/DocServer/NSAFactSheet\_Cognition.pdf?docID=986 Bassett, S. (2003). The assessment of patient adherence to physiotherapy rehabilitation. *New Zealand Journal of Physiotherapy,* Vol.31, No.2, pp. 60-66, ISSN 0303-7193 Beaton, D.; Davis, A., Hudak, P. & McConnell S. (2001). The DASH (Disabilities of the Arm,

Broeren, J.; Claesson, L.; Goude, D.; Rydmark, M. & Sunnerhagen, K. (2008). Virtual

Buccino G.; Vogt, S.; Ritzl, A.; Fink, G.; Zilles, K.; Freund H. & Rizzolatti G. (2004). Neural

Bütefisch, C.; Hummelsheim, H.; Denzler, P.; & Mauritz, K. (1995). Repetitive training of

http://www.ninds.nih.gov/disorders/stroke/poststrokerehab.htm

*Journal of Hand Therapy*, Vol.6, No.4, pp. 109-117, ISSN 1758-9983

integrated with fmri to study neural mechanisms of action observation-execution: A proof of concept study. *Restorative Neurology and Neuroscience*, Vol.27, No.3, pp.

effects into three-dimensional virtual environments to train the hemiparetic upper extremity. *IEEE Transactions on Neural Systems and Rehabilitation Engineering*, Vol.17,

disability? *Journal of Stroke and Cerebrovascular Diseases*, Vol.13, No.4, pp. 171-177,

Shoulder and Hand) outcome measure: what do we know about it now? *British* 

rehabilitation in an activity centre for community-dwelling persons with stroke: The possibilities of 3-dimensional computer games. *Cerebrovascular Diseases*, Vol.26,

Circuits Underlying Imitation Learning of Hand Actions: An Event-Related fMRI

isolated movements improves the outcome of motor rehabilitation of the centrally

almost able-bodied or integrated as a systems approach to UL rehabilitation.

**9. Acknowledgements** 

Able-X and Able-M devices.

209-223, ISSN 0922-6028

ISSN 1052-3057

available from

No.5, pp. 512-20, ISSN 1534-4320

Disorders And Stroke. Available from

No.3, pp. 289-296, ISSN: 1015-9770

Study. *Neuron,* Vol.42, pp 323–334, ISSN1748-2968

**10. References** 

paretic hand. *Journal of the Neurological Sciences*, Vol.130, No.1, pp. 59-68, ISSN 0022- 510X


Augmented Reality Assisted Upper Limb Rehabilitation Following Stroke 173

Peattie, A.; Korevaar, A.; Wilson, J.; Sandilands, B.; Chen, X. & King, M. (2009). Automated

Prange, G.; Jannink, M.; Groothuis-Oudshoorn, C.; Hermens, H. & Ijzerman, M. (2006).

Pridmore, T.; Hilton, D.; Green, J.; Eastgate, R. & Cobb, S. (2004). Mixed reality

Pyk, P.; Wille, D.; Chevrier, E.; Hauser, Y.; Holper, L.; Fatton, I.; Greipl, R.; Schlegel, S.;

Ramachandran, V.; Rogers-Ramachandran, D. & Cobb, S. (1995). Touching the phantom.,

Ramachandran, V. & Altschuler, E. (2009). The use of visual feedback, in particular mirror

Rand,D.; Kizony, R.; Feintuch, U.; Katz, N.; Josman, N.; Rizzo, A. & Weiss, P. (2005).

Renner, M. & Rosenzweig, M. (1987). *Enriched and Impoverished Environments.Effects on Brain* 

Rizzolatti, G. & Craighero, L. (2004). The mirror-neuron system. *Annual Review of* 

Rose, F.; Attree, E.; Brooks, B. & Johnson, D. (1998). Virtual Environments In Brain Damage

Rose, F.; Brooks, B. & Rizzo, A. (2005). Virtual Reality in Brain Damage Rehabilitation: Review. *CyberPsychology & Behavior*, Vol.8, No.3, pp. 241-262, ISSN 1094-9313 Sampson, M.; Shau, Y. & King, M. (2011). Bilateral Upper Limb Trainer with Virtual Reality

Sathian, K. Greenspan, A. & Wolf, S. (2000). Doing it with mirrors: a case study of a novel

Schimmelpfennig, L. (2000) Ohne Kopf geht`s nicht. Geistig Fit. Vol.2, pp. 6-9, ISBN

Schultheis, M. & Rizzo, A. (2001). The application of virtual reality technology in rehabilitation. *Rehabilitation Psychology*, Vol.46, No.3, pp. 296–311, ISSN 0090-5550

*and Behavior.* Springer-Verlag, New York, ISBN 978-3540965237

*Neuroscience,* Vol.27, pp. 169–192, ISSN 0147-006X

Ios Press: Amsterdam, Netherlands, ISBN-10 905199429X

*Technology*. (doi:10.3109/17483107.2011.562959), ISSN 1748-3107

Sydney, Australia, December 2-4, 2009

Vol.43, No.2, pp. 171-183, ISSN 0748-7711

*Nature*, Vol.377, pp. 489–490, ISSN 0028-0836

25-27, 2008.

ISSN 0006-8950

73-76, ISSN 1545-9683

3211836756

7460

Variable Resistance System for Upper Limb Rehabilitation, *Proceedings of Australasian Conference on Robotics and Automation 2009*, ISBN 978-0-9807404-0-0

Systematic review of the effect of robot-aided therapy on recovery of the hemiparetic arm after stroke. *Journal of Rehabilitation Research and Development*,

environments in stroke rehabilitation: interfaces across the real/virtual divide. *Proceedings of International Conference on Virtual Reality, Disability and Associated Technologies*. ISBN-10: 0704911426 Oxford, United Kingdom, September 20-22, 2004

Ottiger, L.; Rückreim, B.; Pescatore, A.; Meyer-Heim, A.; Kiper, D. & Eng, K. (2008). A Paediatric Interactive Therapy System for Arm and Hand Rehabilitation. *Virtual Rehabilitation 2008,* pp. 127-132, ISBN 978-1-4244-2700-0 Vancouver, Canada August

visual feedback, in restoring brain function. *Brain*, Vol.132, No.7, pp. 1693-1710,

Comparison of Two VR Platforms for Rehabilitation: Video Capture versus HMD. *Presence Teleoperators and Virtual Environments,* Vol.14, No.2, pp. 147-160, ISSN 1054-

Rehabilitation: A Rationale From Basic Neuroscience, In *Virtual Environments in Clinical Psychology and Neuroscience.* Riva, G.; Wiederhold, B. & Molinari, E. (Eds.)

for poststroke rehabilitation: case series report. *Disability and Rehabilitation: Assistive* 

approach to neurorehabilitation. *Neurorehabilitation and Neural Repair,* Vol.14, pp.


172 Augmented Reality – Some Emerging Application Areas

Housman, S.; Scott, K. & Reinkensmeyer, D. (2009). A randomized controlled trial of

IJsselsteijn, W.; Nap, H.; de Kort, Y.; & Poels, K. (2007). Digital Game Design for Elderly

Johnson, M.; Feng, X.; Johnson, L. & Winters, J. (2007). Potential of a suite of

King, M.; Hale, L.; Pekkari, A.; Persson, M.; Gregorsson, M. & Nilsson, M. (2010). An

Krebs, H. I., Volpe, B. T., Ferraro, M., Fasoli, S., Palazzolo, J., Rohrer, B., Edelstein, L., et al.

Krebs, H.; Mernoff, S.; Fasoli, S.; Hughes, R.; Stein, J. & Hogan, N. (2008). A comparison of

Kwakkel, G.; Kollen, B.; Van der Grond, J. & Prevo, A. (2003). Probability of regaining

Kwakkel, G.; Kollen, B. & Krebs, H. (2008). Effects of Robot-assisted therapy on upper limb

Lang, C.; MacDonald, J.; Reisman, D.; Boyd, L.; Kimberley, T.; Schindler-Ivens, S.; Hornby,

McAuley, E.; Duncan, T. & Tammen, V. (1987). Psychometric properties of the Intrinsic

Parkinson, M. (ed.) (2000). *Aids to examination of the peripheral nervous system.* 4th ed. W. B.

*Research Quarterly for Exercise & Sport*, Vol.60, pp. 48-58, ISSN 0270-1367 McCombe-Waller, S. & Whitall, J. (2008). Bilateral arm training: Why and who benefits?

*Rehabilitation,* Vol.90, No.10, (October), pp. 1692–1698, ISSN 0003-9993 Levin, M. (2011). Can virtual reality offer enriched environments for rehabilitation? *Expert Review of Neurotherapeutics,* Vol.11, No.2, (February), pp.153-5, ISSN: 1473-7175 Merians, A.; Jack, D.; Boian, R.; Tremaine, M.; Burdea, G.; Adamovich, S.; Recce, M. et al.

*Lancet,* Vol.354, No.9174, pp. 191-196, ISSN: 0140-6736

Vol.22, No.2, pp. 111-121, ISSN: 1545-9683

onset in acute stroke. *Stroke*, *34*(9), 2181-2186, ISSN 00392499

*Physical Therapy*, Vol.82, No.9, pp. 898-915, ISSN 0031-9023

*NeuroRehabilitation*, Vol.23, pp. 29-41, ISSN 1053-8135

Saunders, ISBN0-7020-2512-7, Edinburgh, United Kingdom

1545-9683

17430003

1748-3107

pp. 17-22, ISBN 978-1-59593-943-2

gravity-supported, computer-enhanced arm exercise for individuals with severe hemiparesis. *Neurorehabilitation and Neural Repair*, Vol.23, No.5, pp. 505-514, ISSN

Users, In *Proceedings of Futureplay 2007*. (Toronto, Canada, 14-18 November 2007),

robot/computer-assisted motivating systems for personalized, home-based, stroke rehabilitation. *Journal of NeuroEngineering and Rehabilitation*, Vol.4, pp. 6, ISSN:

affordable, computerized, table-based exercise system for stroke survivors. *Disability and Rehabilitation Assistive Technology*, Vol.5, No.4, (July), pp. 288-93, ISSN:

(2002). Robot-aided neuro-rehabilitation: From evidence-based to science-based rehabilitation. *Topics in Stroke Rehabilitation*, Vol.8, No.4, pp. 54-70, ISSN: 1074-9357

functional and impairment-based robotic training in severe to moderate chronic stroke: a pilot study. *NeuroRehabilitation,* Vol.23, No.1, pp. 81–87, ISSN 1053-8135 Kwakkel, G.; Wagenaar, R.; Twisk, J.; Lankhorst, G. & Koetsier, J. (1999). Intensity of leg and

arm training after primary middle-cerebral-artery stroke: A randomised trial.

dexterity in the flaccid upper limb: Impact of severity of paresis and time since

recovery after stroke: A Systematic Review. *Neurorehabilitation and neural repair*,

G.; Ross, S. & Scheets, P. (2009). Observation of amounts of movement practice provided during stroke rehabilitation. *Archives of Physical Medicine and* 

(2002). Virtual reality-augmented rehabilitation for patients following stroke.

Motivation Inventory in a competitive sport setting: A confirmatory analysis.


**9** 

Julian Stadon

*Australia* 

*Curtin University of Technology,* 

**Post-Biological Agency in** 

**Real-Time Mixed Reality Data Transfer** 

This research offers a contribution, through art theory and practice, to an emerging culturally orientated discourse regarding mixed reality interaction. It seeks to deconstruct and analyse syncretic, hybridized agency, particularly in mixed reality data transfer systems. Recent developments in bridging autonomous relationships with digital representation, through mixed reality interfacing, have brought about the need for further analysis of these new 'postbiological', hybridized states of being that traverse traditional paradigms of time and space. Roy Ascott's reconceptualisation of syncretic dialogues may facilitate further understanding of multi-layered world views, both material and metaphysical, that are emerging from our engagement with such pervasive computational technologies and post-biological systems. Syncretism has traditionally been regarded as an attempt to harmonise and analogise disparate or opposing veiwpoints (Ascott, 2010). Citing recent examples of research, this will discuss what Deleuze and Guattari have called the 'deterritorialisation' of the human body through its dispersion into multiple reality manifestations, in relation to how mixed reality data transfer might constitute a 'reterritorialising' effect on our syncretic understanding of

The research analyses systems that allow for the bridging of the body with its virtual incarnations, through unique transfers of biological and physical data, such as 3D bioimaging, biofeedback data, motion tracking and bio/nano microscopy. These processes involve interfacing that allows for biological and physical data to be transferred into a (digitally mediated) mixed reality state. Giannachi calls this 'hypersurfacing' (Giannachi, 2008) and recent technologies provide further opportunity for hypersurfaced systems to be substantially developed. It is a popular belief that we are now, through a media convergent, participatory culture (that is integrated socially through a sub network of platforms) creating what was first coined in 1997 as *collective intelligence* by Pierre Lévy (Levy, 1997), which exists in a global society of knowledge (data) transfer. This perspective evades traditional (mythologically based) notions of anthropomorphic interaction as it moves beyond the individual and into a universal model of (hypersurfaced) open access. Networked agency destabilises traditional orthodoxies of thought through challenging notions of representation, confronting materialism, accelerating and smoothing social engagement and most importantly, demanding participation in these open systems of collaborative engagement. This has redefined our understandings of consciousness and

post-biological digital identity (Delueze and Guatarri, 1980).

**1. Introduction** 

