**Post-Biological Agency in Real-Time Mixed Reality Data Transfer**

Julian Stadon *Curtin University of Technology, Australia* 

## **1. Introduction**

174 Augmented Reality – Some Emerging Application Areas

Sims, N. & Muyderman, H. (2009). Mitochondria, oxidative metabolism and cell death in stroke. *Biochimica et Biophysica Acta,* Vol.1802, No.1, pp. 80–91, ISSN 0006-3002 Stefan, K.; Cohen, L.; Duque, J.; Mazzocchio, R.; Celnik, P.; Sawaki, L.; Ungerleider, L. &

Stein, D.; Brailowski, S. & Will, B. (1995). *Brain Repair,* Oxford University Press, ISBN

Stein, J.; Harvey, R.; Macko, R.; Winstein, C. & Zorowitz, R. (2009). *Stroke recovery and rehabilitation,* Demos Medical Publishing. ISBN 978-1933864129, New York Sterr, A.; Freivogel, S. & Schmalohr, D. (2002). Neurobehavioral aspects of recovery:

Stinear, C.; Barber, P.; Smale, P.; Coxon, J.; Fleming, M. & Byblow, W. (2007). Functional

Sütbeyaz, S.; Yavuzer, G.; Sezer, N. & Koseoglu B. (2007). Mirror Therapy Enhances Lower-

Sveistrup, H. (2004). Motor rehabilitation using virtual reality. *Journal of NeuroEngineering* 

Tinson, D. (1989). How stroke patients spend their days: An observational study of the

Van Peppen, R.; Kwakkel, G.; Wood-Dauphinee, S.; Hendriks, H.; Van der Wees, P. &

Werner, R. & Kessler, S. (1996). Effectiveness of an intensive outpatient rehabilitation

Wolf, S. & Catlin, P. (2001). Assessing Wolf Motor function Test as outcome measure for research in patients after stroke. *Stroke*, Vol.32, pp. 1635-1639, ISSN 00392499 Wyller, T.; Sveen, U.; Sedring, K.; Pettersen, A. & Bautz-Holter, E. (1997). Subjective well-

Yavuzer, G.; Selles, R.; Sezer, N.; Sutbeyaz, S.; Bussmann, J.; Koseoglu, F.; Atay, M. et al.

*Rehabilitation*, Vol.75, No.2, pp. 114-120, ISSN 08949115

*Neuroscience,* Vol.25, pp. 9339–9346, ISSN 0270-6474

0195076427, New York

Vol.130, pp. 170–180, ISSN 0006-8950

*and Rehabilitation*, Vol.1, pp. 10, ISSN 17430003

0003-9993

ISSN 0003-9993

0269-2155

0269-2155

0269-2155

393–398, ISSN 0003-9993

Classen, J. (2005). Formation of a motor memory by action observation. *Journal of* 

Assessment of the learned nonuse phenomenon in hemiparetic adolescents. *Archives of Physical Medicine and Rehabilitation,* Vol.83, No.12, pp. 1726-1731, ISSN

potential in chronic stroke patients depends on corticospinal tract integrity. *Brain*,

Extremity Motor Recovery and Motor Functioning After Stroke: A Randomized Controlled Trial. *Archives of Physical Medicine and Rehabilitation*, Vol.88, pp. 555-559,

treatment regime offered to patients with movement disorders in hospitals following stroke. *International Disability Studies*, Vol.11, pp. 45-49, ISSN:0259-9147 van der Lee, J.; Snels, I; Beckerman, H.; Lankhorst, G.; Wagenaar, R. & Bouter, L. (2001).

Exercise therapy for arm function in stroke patients: A systematic review of randomized controlled trials. *Clinical Rehabilitation*, Vol.15, No.1, pp. 20-31, ISSN

Dekker, J. (2004). The impact of physical therapy on functional outcomes after stroke: What's the evidence? *Clinical Rehabilitation*, Vol.18, No.8, pp. 833-862, ISSN

program for postacute stroke patients. *American Journal of Physical Medicine and* 

being one year after stroke. *Clinical Rehabilitation*, Vol.11, No.2, pp. 139-145, ISSN

(2008). Mirror therapy improves hand function in subacute stroke: a randomized controlled trial. *Archives of Physical Medicine and Rehabilitation*, Vol.89, No.3, pp. This research offers a contribution, through art theory and practice, to an emerging culturally orientated discourse regarding mixed reality interaction. It seeks to deconstruct and analyse syncretic, hybridized agency, particularly in mixed reality data transfer systems. Recent developments in bridging autonomous relationships with digital representation, through mixed reality interfacing, have brought about the need for further analysis of these new 'postbiological', hybridized states of being that traverse traditional paradigms of time and space. Roy Ascott's reconceptualisation of syncretic dialogues may facilitate further understanding of multi-layered world views, both material and metaphysical, that are emerging from our engagement with such pervasive computational technologies and post-biological systems. Syncretism has traditionally been regarded as an attempt to harmonise and analogise disparate or opposing veiwpoints (Ascott, 2010). Citing recent examples of research, this will discuss what Deleuze and Guattari have called the 'deterritorialisation' of the human body through its dispersion into multiple reality manifestations, in relation to how mixed reality data transfer might constitute a 'reterritorialising' effect on our syncretic understanding of post-biological digital identity (Delueze and Guatarri, 1980).

The research analyses systems that allow for the bridging of the body with its virtual incarnations, through unique transfers of biological and physical data, such as 3D bioimaging, biofeedback data, motion tracking and bio/nano microscopy. These processes involve interfacing that allows for biological and physical data to be transferred into a (digitally mediated) mixed reality state. Giannachi calls this 'hypersurfacing' (Giannachi, 2008) and recent technologies provide further opportunity for hypersurfaced systems to be substantially developed. It is a popular belief that we are now, through a media convergent, participatory culture (that is integrated socially through a sub network of platforms) creating what was first coined in 1997 as *collective intelligence* by Pierre Lévy (Levy, 1997), which exists in a global society of knowledge (data) transfer. This perspective evades traditional (mythologically based) notions of anthropomorphic interaction as it moves beyond the individual and into a universal model of (hypersurfaced) open access. Networked agency destabilises traditional orthodoxies of thought through challenging notions of representation, confronting materialism, accelerating and smoothing social engagement and most importantly, demanding participation in these open systems of collaborative engagement. This has redefined our understandings of consciousness and

Seeking Syncretism: Post-Biological Agency in Real-Time Mixed Reality Data Transfer 177

engagement (rather than entering traditional text or numerical based data sets and command sequences). A good example for mixed reality would be would be the Layar application for smartphones. This application, available for free download to any smartphone user, provides an advanced augmented reality platform capable of reliably delivering many different AR experience, though largely focusing on using geolocation to augment the user's physical surroundings. For example, Manifest.AR, an international artist's collective working with emergent forms of augmented reality as public art, use the technology to transform public space for users. They install virtual objects and artworks that respond to and overlay the configuration of located physical meaning. The application uses geolocation software to superimpose computer generated three-dimensional art objects, enabling the public to see the work integrated into the physical location as if it existed in the real world (Manifest.AR, 2011). Thus, the Layar application reterritorialises information primarily through geolocating individuals, in order to provide a richer engagement with

their physical surrounds through the layering of virtual content over real time video.

reproducing itself (Deleuze and Guatarri, 1980)."

environments.

Massively Multi User Online Worlds (MMOs) are another example of an open virtual environment that allows for the contribution to and manipulation of private and publicly owned virtual space through a variety of methods. The content of these environments is dependent on the participants, due to this open interaction, and therefore relies on the quality of information transfer methods being used. Through collaborative creative production MMOs facilitate social engagement and further collaborative production by its participants. Spatial developments define the environments and the (real or virtual) individuals inhabiting such spaces through their participation in and response to them. The collective construction of such virtual meeting sites, for remote interpersonal interaction acts as an instrument of location and orientation, referential to the real world of knowledge. Implementing biological and physical data into MMOs through augmented reality, contributes new knowledge in regards to bridged mixed reality states, under a paradigm of post-biological deterritorialisation and reterritorialisation of the body. Deleuze and Guatarri discuss deterritorialisation in terms of dispersed resemblance and identity. In *Difference and Repetition* Deleuze introduces the notion of deterritorialisation (through dispersion) as a "dark precursor" that "relates heterogeneous systems and even completely disparate things (Deleuze, 1993)." In order for deterritorialisation to occur there must be some form of agent that can remain constant and self-referent. Deleuze and Guatarri state that: "The alignment of the code or linearity of the nucleic sequence in fact marks a threshold of deterritorialisation of the "sign" that gives it a new ability to be copied and makes the organism more deterritorialised than a crystal: only something deterritorialised is capable of

Virtual reality's hybridization with physical and biological architecture is constructed by the methods used to connect the environments. The combination and cohesion of heterogeneous elements is generally problematic, particularly when a three dimensional space is primarily viewed on a two dimensional plane. The integration of virtual elements and physical environments relies on bridging the two spaces with dynamic interfaces that are simultaneously accessible and able to be openly engaged with, edited and developed. To create integration systems that network physical and virtual data 'shared locations' are required in order to represent the data in a meaningful way that is inclusive of both

presence in way that requires rethinking everything from spirituality, time and space, esotericism, agency, emergence, quantum coherence through to eroticism (Ascott, 2010). This fusion of real and representation, linking cyber and real worlds constitutes mixed reality interaction as experienced by humans in the physical world, their avatars, agents, and virtual humans.

Remote Procedure Call Interfaces (RPCI) and real-time data transfer enhance the experience of the hypersurface for the audience beyond any previous virtual media types, such as, hypertext, HTML, VRML, virtual reality, etc. Unlike traditional sites for communication and cultural exchange, virtual platforms rely on actions and conversations to shape not only the social and cultural environments, but also the spatial environment. XML RPC interfaces further this to (re)include the physical environment, from the perspective of the viewer. The following research scopes unique autonomous relationships between authors and agents, through augmenting biological and virtual (natural and technical) information into mixed reality states that allow for their mediation through networked systems interfacing. Such systems allow participants to physically interact with virtual biological components through real-time data transfer and mediate through physical engagement, rather than entering traditional text or numerical based data sets and command sequences.

Previously, the majority of research in mixed reality focused on specific technical aspects of the field encompassing computer science, cognitive science, robotics, etc. with arts, media and humanities only recently contributing to a mixed reality knowledge base. Along with several examples from computer science, I will primarily discuss three recent projects: *Organtrader2010*, *Promethean Alchemist* and *Terra(socio)sonica:Pouvoir/Poussanse* which use innovative mixed reality data transfer methods, in a culturally positioned way in order to demonstrate the 'deterritorialisation' (Deleuze and Guatarri, 1980) of the human body and its' 'reterritorialising' effect on postbiological digital identity construction. The examples chosen deconstructs the notion of 'post-biological' digital identity through creating innovative actualisation examples. Through increasing the level of physical and social involvement that viewers have in the creation of a mixed reality system of exchange, the works included demonstrate a new model of representation and experience that stretches, disperses and merges the position of both artwork and viewer in order to question postbiological identity and the body

## **2. Background**

Current research in mixed reality and interactive workspaces that use the concept of a bridge for visual data transfer have continued the development of new knowledge in this emerging field, however the majority of previous research in this area has been in the field of computer science. The rigorous application of cultural and philosophical discourse to recent developments in screen based technologies and software applications will suggest new modes of representation, that are concerned with the affective capacities of art in a way that articulates a sense of dispersed embodiment.

Unlike traditional sites for communication and cultural exchange, digital platforms rely on actions and conversations to shape not only the social and cultural environments, but also the spatial environments. Such systems allow participants to physically interact with virtual (deterritorialised) biological representations and mediate (reterritorialise) through physical 176 Augmented Reality – Some Emerging Application Areas

presence in way that requires rethinking everything from spirituality, time and space, esotericism, agency, emergence, quantum coherence through to eroticism (Ascott, 2010). This fusion of real and representation, linking cyber and real worlds constitutes mixed reality interaction as experienced by humans in the physical world, their avatars, agents,

Remote Procedure Call Interfaces (RPCI) and real-time data transfer enhance the experience of the hypersurface for the audience beyond any previous virtual media types, such as, hypertext, HTML, VRML, virtual reality, etc. Unlike traditional sites for communication and cultural exchange, virtual platforms rely on actions and conversations to shape not only the social and cultural environments, but also the spatial environment. XML RPC interfaces further this to (re)include the physical environment, from the perspective of the viewer. The following research scopes unique autonomous relationships between authors and agents, through augmenting biological and virtual (natural and technical) information into mixed reality states that allow for their mediation through networked systems interfacing. Such systems allow participants to physically interact with virtual biological components through real-time data transfer and mediate through physical engagement, rather than entering

Previously, the majority of research in mixed reality focused on specific technical aspects of the field encompassing computer science, cognitive science, robotics, etc. with arts, media and humanities only recently contributing to a mixed reality knowledge base. Along with several examples from computer science, I will primarily discuss three recent projects: *Organtrader2010*, *Promethean Alchemist* and *Terra(socio)sonica:Pouvoir/Poussanse* which use innovative mixed reality data transfer methods, in a culturally positioned way in order to demonstrate the 'deterritorialisation' (Deleuze and Guatarri, 1980) of the human body and its' 'reterritorialising' effect on postbiological digital identity construction. The examples chosen deconstructs the notion of 'post-biological' digital identity through creating innovative actualisation examples. Through increasing the level of physical and social involvement that viewers have in the creation of a mixed reality system of exchange, the works included demonstrate a new model of representation and experience that stretches, disperses and merges the position of both artwork and viewer in order to question post-

Current research in mixed reality and interactive workspaces that use the concept of a bridge for visual data transfer have continued the development of new knowledge in this emerging field, however the majority of previous research in this area has been in the field of computer science. The rigorous application of cultural and philosophical discourse to recent developments in screen based technologies and software applications will suggest new modes of representation, that are concerned with the affective capacities of art in a way

Unlike traditional sites for communication and cultural exchange, digital platforms rely on actions and conversations to shape not only the social and cultural environments, but also the spatial environments. Such systems allow participants to physically interact with virtual (deterritorialised) biological representations and mediate (reterritorialise) through physical

traditional text or numerical based data sets and command sequences.

and virtual humans.

biological identity and the body

that articulates a sense of dispersed embodiment.

**2. Background** 

engagement (rather than entering traditional text or numerical based data sets and command sequences). A good example for mixed reality would be would be the Layar application for smartphones. This application, available for free download to any smartphone user, provides an advanced augmented reality platform capable of reliably delivering many different AR experience, though largely focusing on using geolocation to augment the user's physical surroundings. For example, Manifest.AR, an international artist's collective working with emergent forms of augmented reality as public art, use the technology to transform public space for users. They install virtual objects and artworks that respond to and overlay the configuration of located physical meaning. The application uses geolocation software to superimpose computer generated three-dimensional art objects, enabling the public to see the work integrated into the physical location as if it existed in the real world (Manifest.AR, 2011). Thus, the Layar application reterritorialises information primarily through geolocating individuals, in order to provide a richer engagement with their physical surrounds through the layering of virtual content over real time video.

Massively Multi User Online Worlds (MMOs) are another example of an open virtual environment that allows for the contribution to and manipulation of private and publicly owned virtual space through a variety of methods. The content of these environments is dependent on the participants, due to this open interaction, and therefore relies on the quality of information transfer methods being used. Through collaborative creative production MMOs facilitate social engagement and further collaborative production by its participants. Spatial developments define the environments and the (real or virtual) individuals inhabiting such spaces through their participation in and response to them. The collective construction of such virtual meeting sites, for remote interpersonal interaction acts as an instrument of location and orientation, referential to the real world of knowledge. Implementing biological and physical data into MMOs through augmented reality, contributes new knowledge in regards to bridged mixed reality states, under a paradigm of post-biological deterritorialisation and reterritorialisation of the body. Deleuze and Guatarri discuss deterritorialisation in terms of dispersed resemblance and identity. In *Difference and Repetition* Deleuze introduces the notion of deterritorialisation (through dispersion) as a "dark precursor" that "relates heterogeneous systems and even completely disparate things (Deleuze, 1993)." In order for deterritorialisation to occur there must be some form of agent that can remain constant and self-referent. Deleuze and Guatarri state that: "The alignment of the code or linearity of the nucleic sequence in fact marks a threshold of deterritorialisation of the "sign" that gives it a new ability to be copied and makes the organism more deterritorialised than a crystal: only something deterritorialised is capable of reproducing itself (Deleuze and Guatarri, 1980)."

Virtual reality's hybridization with physical and biological architecture is constructed by the methods used to connect the environments. The combination and cohesion of heterogeneous elements is generally problematic, particularly when a three dimensional space is primarily viewed on a two dimensional plane. The integration of virtual elements and physical environments relies on bridging the two spaces with dynamic interfaces that are simultaneously accessible and able to be openly engaged with, edited and developed. To create integration systems that network physical and virtual data 'shared locations' are required in order to represent the data in a meaningful way that is inclusive of both environments.

Seeking Syncretism: Post-Biological Agency in Real-Time Mixed Reality Data Transfer 179

These research examples articulate a range of different solutions that have been proposed for technological developments in the field of computer science, and often neglect the philosophical and theoretical impact of such technologies on human subjectivity, representation, identity and social discourse. The collaboration between computer science and art establishes a transdisciplinary practice, that is capable of traversing disparate fields, such as: biology, cybernetics, philosophy, spirituality, nanotechnology, distributed cognition, robotics, game theory, virtual environments and aesthetics. This produces a syncretic dialogue orientated towards existence and consciousness that is (potentially) more thorough and rigorous than any previous methodology relating to our engagement with digital systems. Emerging technologies often develop faster than we have the ability to understand them. When these technologies, particularly imaging systems across science and lived experience

Art and science have, of course shared a long history of collaboration, particularly in the medical, biological and astrological sciences. Historically, scientific findings have been represented to society through artistic representation, for example anatomy of the body, drawings of the galaxy (via telescopes), or that which was accessed (beyond natural vision) through the microscope. It is important to note here that representation has historically been further facilitated by technology that enables us to see beyond the limitations of the body. Whether due to the technicization of scientific imaging or academia's need to quantify/qualify research under set paradigms, the significant (collaborative) involvement of artists in scientific inquiry became quite a rare occurrence for some years. In recent times we have seen artists become much more flexible and adventurous with the mediums they use and this has seen a return to sciencific methods and technologies. Unlike previously where the artists' role primarily was to document proceedings, today artists are much more likely to collaborate with a scientist/team in order to produce outcomes that traverse the respective fields in order to

become creative mediums, they redefine the ways by which we define humanity.

question the cultural impact such technologies and scientific fields have on society.

where traditional hierarchies between the body and virtual agents are redefined.

An example of a researcher that creates shared mixed reality systems of exchange under a cultural predisposition is telematic artist Paul Sermon. Sermon's early work explored the emergence of user-determined narrative by bringing remote participants together in a shared telepresent environment. Through the use of live chromakey and video conferencing technology, two public rooms or installations and their audiences are joined in a virtual duplicate that turns into a mutual space of activity. Currently Sermon's practice examines the concepts of presence and performance within Second Life and what he calls 'first life', and attempts to bridge these two spaces through mixed reality techniques and interfaces (Sermon and Gould, 2009). The notion of telepresence is explored through a blurring between 'online' and 'offline' identities, and the signifiers and conditions that make us feel present in this world. His research questions how subjectivity is articulated in relation to embodiment and disembodiment. Sermon creates hypersurfaces through which data can oscillate between two reality states in an autonomous way: present and telepresent. The development of such a method of data exchange creates an interesting situation where both the user and an autonomous agent (their avatar) can now affect visual data in a mixed reality environment. The important aspect to this exchange is that Sermon does not privilege either the online agent or the physical beings that participate, thus alluding to a post-biological system of engagement

**3. Examples from artistic practice** 

For the construction and exploration of mixed reality to occur interfacing is required to bridge the virtual environment with the physical so that both spaces can be mediated in an autonomous manner. The hypersurface is the site on which bridges are built: where the real and virtual, material and textual, author and agent can meet and interact with each other. Performance technology theorist Gabriella Giannachi states that, "The hypersurface is a zone of exchange between consciousness (language and text) and levels of the inorganic… Able to present dichotomous relationships, between representation and matter, inside and outside, organic and inorganic, the hypersurface is the site of virtual performance (Giannachi)." For the construction and exploration of mixed reality to occur interfacing is required to bridge the virtual environment with the physical so that both spaces can be mediated in an autonomous manner. The hypersurface is the site on which bridges are built: where the real and virtual, material and textual, author and agent can meet and interact with each other.

#### **2.1 Examples from computer science**

The earliest example of a research project that proposed a hypersurfaced system for data transfer using mixed reality was in 1999. Butz et al. proposed a drag and drop technique between an augmented reality space to a screen space within the EMMIE system. Using a mirror metaphor, virtual objects would change representation and dimensionality by passing through screen boundaries, with the approach focused on transferring documents.

Recently Lang et al. (Lang et al., 2008) from Georgia Tech University modified Second Life to create mixed reality experiences the purpose being the creation of a novel augmented reality environment for entertainment. This example bridged reality states in a way that facilitated a further inquiry into the socio-cultural implications of such systems, but was never addressed in the research publications.

The VTT Technical Research Center Finland has also recently worked with hypersurfacing Second Life avatars within physical experience through the *Meeting Avatars* joint project with IBM and Nokia (Kantonen et al., 2010). By using the Second Life engine, virtual avatars had the same appearance and behavior as in the virtual world but in their context be represented in a physical meeting room

Barakonyi and Schmalstieg (2008) created two pilot systems in order to facilitate proactive multi-user interface adaptation and user interface migration. The system was developed in order to migrate tasks across a range of autonomous agents and a number of users, rather than a single avatar being used by each individual. The goal was to increase the versatility of ubiquitous agency through mixed reality data bridging (hypersurfacing). By increasing the number of agents (in various reality states) that can autonomously perform tasks set by users, the bridge defines a dedicated space where the viewer can transfer objects and images between worlds, spaces, and contexts

Koleva et al (2006) explored navigation between real, augmented and virtual worlds by establishing "mixed reality boundaries" and proposed a model of how space, boundaries can be represented. Schnädelbach et al (2006) further generalized the concept to any architectural construct, how collaboration and communication can be established in this type of environment. Finally, Grasset et al proposed a general conceptual model on how to represent spaces, navigation and the different steps of a transition between contexts (Grasset et al. 2005) (Grassett et al 2006).

These research examples articulate a range of different solutions that have been proposed for technological developments in the field of computer science, and often neglect the philosophical and theoretical impact of such technologies on human subjectivity, representation, identity and social discourse. The collaboration between computer science and art establishes a transdisciplinary practice, that is capable of traversing disparate fields, such as: biology, cybernetics, philosophy, spirituality, nanotechnology, distributed cognition, robotics, game theory, virtual environments and aesthetics. This produces a syncretic dialogue orientated towards existence and consciousness that is (potentially) more thorough and rigorous than any previous methodology relating to our engagement with digital systems. Emerging technologies often develop faster than we have the ability to understand them. When these technologies, particularly imaging systems across science and lived experience become creative mediums, they redefine the ways by which we define humanity.

## **3. Examples from artistic practice**

178 Augmented Reality – Some Emerging Application Areas

For the construction and exploration of mixed reality to occur interfacing is required to bridge the virtual environment with the physical so that both spaces can be mediated in an autonomous manner. The hypersurface is the site on which bridges are built: where the real and virtual, material and textual, author and agent can meet and interact with each other. Performance technology theorist Gabriella Giannachi states that, "The hypersurface is a zone of exchange between consciousness (language and text) and levels of the inorganic… Able to present dichotomous relationships, between representation and matter, inside and outside, organic and inorganic, the hypersurface is the site of virtual performance (Giannachi)." For the construction and exploration of mixed reality to occur interfacing is required to bridge the virtual environment with the physical so that both spaces can be mediated in an autonomous manner. The hypersurface is the site on which bridges are built: where the real and virtual,

The earliest example of a research project that proposed a hypersurfaced system for data transfer using mixed reality was in 1999. Butz et al. proposed a drag and drop technique between an augmented reality space to a screen space within the EMMIE system. Using a mirror metaphor, virtual objects would change representation and dimensionality by passing through screen boundaries, with the approach focused on transferring documents. Recently Lang et al. (Lang et al., 2008) from Georgia Tech University modified Second Life to create mixed reality experiences the purpose being the creation of a novel augmented reality environment for entertainment. This example bridged reality states in a way that facilitated a further inquiry into the socio-cultural implications of such systems, but was

The VTT Technical Research Center Finland has also recently worked with hypersurfacing Second Life avatars within physical experience through the *Meeting Avatars* joint project with IBM and Nokia (Kantonen et al., 2010). By using the Second Life engine, virtual avatars had the same appearance and behavior as in the virtual world but in their context be

Barakonyi and Schmalstieg (2008) created two pilot systems in order to facilitate proactive multi-user interface adaptation and user interface migration. The system was developed in order to migrate tasks across a range of autonomous agents and a number of users, rather than a single avatar being used by each individual. The goal was to increase the versatility of ubiquitous agency through mixed reality data bridging (hypersurfacing). By increasing the number of agents (in various reality states) that can autonomously perform tasks set by users, the bridge defines a dedicated space where the viewer can transfer objects and images

Koleva et al (2006) explored navigation between real, augmented and virtual worlds by establishing "mixed reality boundaries" and proposed a model of how space, boundaries can be represented. Schnädelbach et al (2006) further generalized the concept to any architectural construct, how collaboration and communication can be established in this type of environment. Finally, Grasset et al proposed a general conceptual model on how to represent spaces, navigation and the different steps of a transition between contexts (Grasset

material and textual, author and agent can meet and interact with each other.

**2.1 Examples from computer science** 

never addressed in the research publications.

represented in a physical meeting room

between worlds, spaces, and contexts

et al. 2005) (Grassett et al 2006).

Art and science have, of course shared a long history of collaboration, particularly in the medical, biological and astrological sciences. Historically, scientific findings have been represented to society through artistic representation, for example anatomy of the body, drawings of the galaxy (via telescopes), or that which was accessed (beyond natural vision) through the microscope. It is important to note here that representation has historically been further facilitated by technology that enables us to see beyond the limitations of the body. Whether due to the technicization of scientific imaging or academia's need to quantify/qualify research under set paradigms, the significant (collaborative) involvement of artists in scientific inquiry became quite a rare occurrence for some years. In recent times we have seen artists become much more flexible and adventurous with the mediums they use and this has seen a return to sciencific methods and technologies. Unlike previously where the artists' role primarily was to document proceedings, today artists are much more likely to collaborate with a scientist/team in order to produce outcomes that traverse the respective fields in order to question the cultural impact such technologies and scientific fields have on society.

An example of a researcher that creates shared mixed reality systems of exchange under a cultural predisposition is telematic artist Paul Sermon. Sermon's early work explored the emergence of user-determined narrative by bringing remote participants together in a shared telepresent environment. Through the use of live chromakey and video conferencing technology, two public rooms or installations and their audiences are joined in a virtual duplicate that turns into a mutual space of activity. Currently Sermon's practice examines the concepts of presence and performance within Second Life and what he calls 'first life', and attempts to bridge these two spaces through mixed reality techniques and interfaces (Sermon and Gould, 2009). The notion of telepresence is explored through a blurring between 'online' and 'offline' identities, and the signifiers and conditions that make us feel present in this world. His research questions how subjectivity is articulated in relation to embodiment and disembodiment. Sermon creates hypersurfaces through which data can oscillate between two reality states in an autonomous way: present and telepresent. The development of such a method of data exchange creates an interesting situation where both the user and an autonomous agent (their avatar) can now affect visual data in a mixed reality environment. The important aspect to this exchange is that Sermon does not privilege either the online agent or the physical beings that participate, thus alluding to a post-biological system of engagement where traditional hierarchies between the body and virtual agents are redefined.

Seeking Syncretism: Post-Biological Agency in Real-Time Mixed Reality Data Transfer 181

The work is inspired by previously discussed Deleuzian notions of deterritorialisation, with particular regards to second-order cybernetic feedback systems (within the context of mixed reality social interaction). Deleuze and Guatarri discuss deterritorialisation in terms of dispersed resemblance and identity. In order for deterritorialisation to occur there must be some form of agent that can remain constant and self-referent and in this case it is the audience that acts as agent and the self referent being the various resulting feedback (mixed reality image and sound). This feedback affects physical participation with the installation

In practice, the work models global deterritorialisation by destabilising established social groups through intuitive interaction with virtual space, reconfiguring them in the soundscape. The work also investigates the relationship between traditional (postcolonial)

socio-technic hierarchies, and those developing in current evolving cyber cities

and therefore facilitates a second-order cybernetic feedback loop.

Fig. 2. Technical specifications diagram.

## **3.1 Terra(socio)sonica: Pouvoir/Puissance**

*Terra(socio)sonica: Pouvoir/Puissance*, by Julian Stadon (in collaboration with Steven Berrick) realises a sonic soundscape constructed via the movements of communities that inhabit two current landscape realities that constitute the University of Western Australia (UWA) cultural precinct. Through the translation of movement into sound in both the physical and virtual realms, the work explores the notion of unspoken 'silent dialogues' under a paradigm of social engagement. Individuals and large clusters of people produce amplified sounds and shadows based on their oscillating movements within a defined social landscape.

The work uses real-time motion-tracking technologies with a unique pipeline application to create a mixed-reality soundscape. This audio environment is mediated through interactions between the viewer, the physical environment, and other participants within a hypersurfaced mixed-reality feedback loop. As visitors negotiate a traditional public environment –the entrance and surround to the Somerville Auditorium at UWA- data regarding their movements and interaction with others present is gathered and translated into sonic outputs, both in the physical and virtual environments. In the physical, the output is via stereo speakers installed in the space and in the virtual, a three dimensional representation shadows and echoes sonic and visual traces of the real-time dialogues into UWA's Second Life Environment. In this case the physical augments into the virtual and the virtual into the physical in an ongoing representational archive of what has occurred in that shared space.

Fig. 1. Second Life view of virtual 'shadows' from physical data.

The individual experiences an intimate interaction with the work and social environment where they control the soundscape through their actions, thereby conducting their own personal song. Meanwhile, each community that forms also produces unique tones. Movements of individuals between groups result in a sonic symphony of social interaction that shifts dynamically according to the social dialogues that occur in the space. The audience movement in the space produces stepped tonal outputs, for example a stroll to the right will cause the pitch to drop with each step. The pace of the movement determines the speed of the notes; lingering conversations produce long lingering sounds while the rush of busy passersby results in fleeting melodies that come and go just as quickly.

180 Augmented Reality – Some Emerging Application Areas

*Terra(socio)sonica: Pouvoir/Puissance*, by Julian Stadon (in collaboration with Steven Berrick) realises a sonic soundscape constructed via the movements of communities that inhabit two current landscape realities that constitute the University of Western Australia (UWA) cultural precinct. Through the translation of movement into sound in both the physical and virtual realms, the work explores the notion of unspoken 'silent dialogues' under a paradigm of social engagement. Individuals and large clusters of people produce amplified sounds and shadows based on their oscillating movements within a defined social

The work uses real-time motion-tracking technologies with a unique pipeline application to create a mixed-reality soundscape. This audio environment is mediated through interactions between the viewer, the physical environment, and other participants within a hypersurfaced mixed-reality feedback loop. As visitors negotiate a traditional public environment –the entrance and surround to the Somerville Auditorium at UWA- data regarding their movements and interaction with others present is gathered and translated into sonic outputs, both in the physical and virtual environments. In the physical, the output is via stereo speakers installed in the space and in the virtual, a three dimensional representation shadows and echoes sonic and visual traces of the real-time dialogues into UWA's Second Life Environment. In this case the physical augments into the virtual and the virtual into the physical in an ongoing representational archive of what has occurred in that

Fig. 1. Second Life view of virtual 'shadows' from physical data.

busy passersby results in fleeting melodies that come and go just as quickly.

The individual experiences an intimate interaction with the work and social environment where they control the soundscape through their actions, thereby conducting their own personal song. Meanwhile, each community that forms also produces unique tones. Movements of individuals between groups result in a sonic symphony of social interaction that shifts dynamically according to the social dialogues that occur in the space. The audience movement in the space produces stepped tonal outputs, for example a stroll to the right will cause the pitch to drop with each step. The pace of the movement determines the speed of the notes; lingering conversations produce long lingering sounds while the rush of

**3.1 Terra(socio)sonica: Pouvoir/Puissance** 

landscape.

shared space.

The work is inspired by previously discussed Deleuzian notions of deterritorialisation, with particular regards to second-order cybernetic feedback systems (within the context of mixed reality social interaction). Deleuze and Guatarri discuss deterritorialisation in terms of dispersed resemblance and identity. In order for deterritorialisation to occur there must be some form of agent that can remain constant and self-referent and in this case it is the audience that acts as agent and the self referent being the various resulting feedback (mixed reality image and sound). This feedback affects physical participation with the installation and therefore facilitates a second-order cybernetic feedback loop.

In practice, the work models global deterritorialisation by destabilising established social groups through intuitive interaction with virtual space, reconfiguring them in the soundscape. The work also investigates the relationship between traditional (postcolonial) socio-technic hierarchies, and those developing in current evolving cyber cities

Fig. 2. Technical specifications diagram.

Seeking Syncretism: Post-Biological Agency in Real-Time Mixed Reality Data Transfer 183

system, standard Second Life interfaces and mobile platforms. **organtrader2010** uses the organ trade metaphor to question the meaning of ownership and the relationship between content and property. In regards to (unregulated) machines of production and the subversion of power hierarchies, organtrader2010 examines the roles of media artist/supplier, gallery/distributor and participant/trader. In doing so, this project also explores deterritorialisation of the body and post-biological identity in mixed realities.

**organtrader2010** uses a narrative representational structure in a mixed reality context where a participant, wearing a camera mounted HMD (head mounted display) can transfer real CT scanned organs to an augmented organ 'trader.' This augmented Second Life avatar can exist in both physical and virtual space simultaneously, so when the participant hands over one of their organs to the 'augmented trader' they are also giving their organs unsuspectingly to the 'in-world avatar.' This avatar is linked to a network of organ trader avatars that all have ownership permissions to clone and steal organs from the augmented

trader and sell them to other Second Life avatars.

Fig. 3. Installation setup at Banff New Media Institute (2010).

As mentioned above, the organs in **organtrader2010** are obtained through real CT scans and are made by converting data into a 3D model, then converting this model to Open Scene Graph. They are then included in a Python-based application that uses the OSGSWIG

Conceptually, the work attempts to explore Bernard Steigler's historicisation of technical objects and western philosophy in a way that embraces Deleuze's translation of Bergsonism, particularly regarding intuitive method. For Deleuze, intuitive thought is more like a narrative than an instantaneous 'eureka' moment. In order to understand Deleuzian intuition it is important to situate it within the Bergsonian departure from a structuralist notion of reality in terms of space, in favour of the metaphysical idea that reality should be understood in terms of time. In contrast to Descartes' 'instantaneous glance' or moment of clarity-distinctness that separates the intuitive idea, Deleuzian intuition is rather a "progressive description of the whole," more like a multi-lateral trajectory than a moment of illumination. For Deleuze, intuition "reconcatenates thought to beings as the co-presence of a being of the simulacrum and of a simulacrum of Being" (Deleuze, 1991).

Deleuze's concept of intuitive thought is in line with Bernard Steigler's notion of technics defining what it is to be human. As mentioned earlier, Steigler has posited the important contribution to metaphysical thinking that it is technics -the artificial realm of symbols, systems, tools, etc.- that makes humans functional, speaking, meaning-making creatures; that is, what makes humans human. It is in Deleuze's distinction between different degrees of 'power' that I see Bergsonism, that is, the intuitive method, and Steigler's technics coming together. In looking at 'power' one must distinguish between the two French words puissance and pouvoir. "In social terms, puissance is immanent power, power to act rather than power to dominate another; we could say that puissance is praxis (in which equals clash or act together) rather than poiesis (in which others are matter to be formed by the command of a superior, a sense of transcendent power that matches what pouvoir indicates for Deleuze)." (Stiegler, 1998). Our intuitive interaction with technology is, as demonstrated by the hypersurface, the enactment of puissance, that is, immanent power to create meaning through our interactions as opposed to the passing-down of preestablished ideas through transcendent power (pouvoir). Intuitive interaction challenges the academy model, which essentially treats education as transcendent power, the passing on of information by solving problems that already have answers. Instead, intuitive interaction creates opportunity for true knowledge production; an instance of power that puissance indicates for Deleuze.

It is through observing human intuitive interaction with technologies that one might theorize a plausible post-human or postbiological digital identity. As summarized by Daniel Smith and John Protevi, "Here we see the empiricist theme of the 'externality of relations': in an assemblage or consistency, the 'becoming' or relation of the terms attains its own independent ontological status. In Deleuze's favourite example, the wasp and orchid create a "becoming" or symbiotic emergent unit." In *Terra(socio)sonica:Pouvior/Puissance*, human interaction with the hypersurfaced mixed-reality feedback loop becomes an emergent unit in the soundscape, physical and virtual worlds.

#### **3.2 Organtrader2010**

**organtrader2010** is a novel mixed reality interface that allows for the transfer of real CT scanned organs into augmented reality and Second Life. Using the metaphor of organ trade to allude to traditional gallery hierarchies, organtrader2010 allows the participant to donate, sell, buy or steal virtual organs across platforms including an interactive mixed reality 182 Augmented Reality – Some Emerging Application Areas

Conceptually, the work attempts to explore Bernard Steigler's historicisation of technical objects and western philosophy in a way that embraces Deleuze's translation of Bergsonism, particularly regarding intuitive method. For Deleuze, intuitive thought is more like a narrative than an instantaneous 'eureka' moment. In order to understand Deleuzian intuition it is important to situate it within the Bergsonian departure from a structuralist notion of reality in terms of space, in favour of the metaphysical idea that reality should be understood in terms of time. In contrast to Descartes' 'instantaneous glance' or moment of clarity-distinctness that separates the intuitive idea, Deleuzian intuition is rather a "progressive description of the whole," more like a multi-lateral trajectory than a moment of illumination. For Deleuze, intuition "reconcatenates thought to beings as the co-presence of a being of the simulacrum and of a simulacrum of Being"

Deleuze's concept of intuitive thought is in line with Bernard Steigler's notion of technics defining what it is to be human. As mentioned earlier, Steigler has posited the important contribution to metaphysical thinking that it is technics -the artificial realm of symbols, systems, tools, etc.- that makes humans functional, speaking, meaning-making creatures; that is, what makes humans human. It is in Deleuze's distinction between different degrees of 'power' that I see Bergsonism, that is, the intuitive method, and Steigler's technics coming together. In looking at 'power' one must distinguish between the two French words puissance and pouvoir. "In social terms, puissance is immanent power, power to act rather than power to dominate another; we could say that puissance is praxis (in which equals clash or act together) rather than poiesis (in which others are matter to be formed by the command of a superior, a sense of transcendent power that matches what pouvoir indicates for Deleuze)." (Stiegler, 1998). Our intuitive interaction with technology is, as demonstrated by the hypersurface, the enactment of puissance, that is, immanent power to create meaning through our interactions as opposed to the passing-down of preestablished ideas through transcendent power (pouvoir). Intuitive interaction challenges the academy model, which essentially treats education as transcendent power, the passing on of information by solving problems that already have answers. Instead, intuitive interaction creates opportunity for true knowledge production; an instance of power that

It is through observing human intuitive interaction with technologies that one might theorize a plausible post-human or postbiological digital identity. As summarized by Daniel Smith and John Protevi, "Here we see the empiricist theme of the 'externality of relations': in an assemblage or consistency, the 'becoming' or relation of the terms attains its own independent ontological status. In Deleuze's favourite example, the wasp and orchid create a "becoming" or symbiotic emergent unit." In *Terra(socio)sonica:Pouvior/Puissance*, human interaction with the hypersurfaced mixed-reality feedback loop becomes an emergent unit

**organtrader2010** is a novel mixed reality interface that allows for the transfer of real CT scanned organs into augmented reality and Second Life. Using the metaphor of organ trade to allude to traditional gallery hierarchies, organtrader2010 allows the participant to donate, sell, buy or steal virtual organs across platforms including an interactive mixed reality

(Deleuze, 1991).

puissance indicates for Deleuze.

**3.2 Organtrader2010** 

in the soundscape, physical and virtual worlds.

system, standard Second Life interfaces and mobile platforms. **organtrader2010** uses the organ trade metaphor to question the meaning of ownership and the relationship between content and property. In regards to (unregulated) machines of production and the subversion of power hierarchies, organtrader2010 examines the roles of media artist/supplier, gallery/distributor and participant/trader. In doing so, this project also explores deterritorialisation of the body and post-biological identity in mixed realities.

**organtrader2010** uses a narrative representational structure in a mixed reality context where a participant, wearing a camera mounted HMD (head mounted display) can transfer real CT scanned organs to an augmented organ 'trader.' This augmented Second Life avatar can exist in both physical and virtual space simultaneously, so when the participant hands over one of their organs to the 'augmented trader' they are also giving their organs unsuspectingly to the 'in-world avatar.' This avatar is linked to a network of organ trader avatars that all have ownership permissions to clone and steal organs from the augmented trader and sell them to other Second Life avatars.

Fig. 3. Installation setup at Banff New Media Institute (2010).

As mentioned above, the organs in **organtrader2010** are obtained through real CT scans and are made by converting data into a 3D model, then converting this model to Open Scene Graph. They are then included in a Python-based application that uses the OSGSWIG

Seeking Syncretism: Post-Biological Agency in Real-Time Mixed Reality Data Transfer 185

Perhaps most explicitly exemplifying this mixed reality structure is *Promethean Alchemist.* This project was conceptualized during the Liminal Screens Co-Production Residency at the Banff New Media Institute (BNMI), The Banff Centre, Canada, and is currently under development. *Promethean Alchemist* is a mixed-reality interactive data transfer system that engages participants in mythological creation and DNA sequencing. Biological DNA is translated into computer data using DNA sequencing technology, which is further manipulated into 2D and 3D visual graphic appearance in the ARToolkit. The visualization form is the classic DNA double helix, which re-enters the physical world through an XML RPC (Remote Procedure Call) interface. When two sections of DNA are brought into close proximity, the system recognizes their proximity and 'splices' (combines) the data set into

Thus, wearing a HMD (Head Mounted Display) and using these cards, participants are able to pick up segments of DNA in mixed-reality space and throw them into the 'mixing pot' (visualized as a brazier burning with Promethean Fire) to create constantly evolving artificial life forms. Participants are able to create life in an augmented environment and then retrieve their creation in the virtual. Here they can export the form to 3D printing or

*Promethean Alchemist* is an artistic rendition of the implications of DNA code translated into the Deleuzian 'fold'. That is, the in-betweenness of spaces; "able to represent dialectical opposites, such as organic and inorganic, inside and outside."(Deleuze, 1993). The viewers' participation in the layers of code, which constitutes the work, implicates them in a complex exchange between organic and inorganic information where the differences between the two dissolve. The *Promethean Alchemist* is therefore an example of Giannachi hypersurface: *"*Able to present dichotomous relationships, between representation and matter, inside and outside, organic and inorganic, the hypersurface is the site of virtual performance." (Giannachi, 2008). My participation in the exchange is the threefold translation from organic DNA to computational code to visual representation in the ARToolkit. The viewer pushes this further by becoming the 'alchemist' stirring a virtual primordial soup. While the viewer in firmly planted in the physical realm, they still experience the dissolve between the

*Promethean Alchemist* pays homage both visually and conceptually to the Ancient Greek creation myth of Prometheus and Epimetheus, which philosopher Bernard Stiegler references in his text *Technics and Time 1* (Stiegler, 1998). His text seeks to define technics -a technical entity arising out of, but distinct from, mechanical and biological entities- and to delineate the relationship between technics and humans. In the Greek creation story, Prometheus and Epimetheus were given the task of allotting suitable powers to all mortal creatures. Epimetheus did the initial distribution, which was then to be reviewed by Prometheus. Epimetheus set about giving creatures equal shares of positive and negative traits (small beings were given flight, weaker beings speed, etc); "Thus he made his whole distribution on a principle of compensation, being careful by these devices that no species should be destroyed." (Stiegler, 1998). However, when Prometheus came to inspect the work, he realized that Epimetheus had forgotten humans and had nothing left to bestow upon the naked, unshod, and unarmed creatures. Distraught, Prometheus stole from the

**3.3 Promethean alchemist** 

one.

animation platforms.

organic and inorganic, humanity and technology.

python wrapper for the ARToolkit to enable the augmented reality system to occur. To bridge this application with Second Life, data is streamed in to the Linden Scripting Language via the PHP server using XML. PHP provides the potential to extend the application network to include mobile devices and multiple reality environments with the system. The actual **organtrader2010** application can even be installed to Python enabled platforms for multiple mixed reality participation.

Fig. 4. Technical overview diagram for **organtrader2010**.

The system uses an XML RPC to link an augmented reality application with Second Life via a PHP server. This pipeline allows for a method of real-time transfer of 3D visual material, linking the body with the augmented and virtual representations of it. The use of real CT scanned organs with fiducial marker and proximity tracking adds to the viewer's experience of agency within the processes involved in the simulated organ trade and in the process of media art creation, display and dissemination.

While being more technically advanced than the previous example, this project challenges the same ideas under a more specific paradigm of the body. Here the work was designed to engage the viewer in a fictitious situation where they participate in a system of monetary exchange for virtual organs. This quite literally questions the value society places on biological elements within virtual and mixed reality systems as the audience dictates the market.

#### **3.3 Promethean alchemist**

184 Augmented Reality – Some Emerging Application Areas

python wrapper for the ARToolkit to enable the augmented reality system to occur. To bridge this application with Second Life, data is streamed in to the Linden Scripting Language via the PHP server using XML. PHP provides the potential to extend the application network to include mobile devices and multiple reality environments with the system. The actual **organtrader2010** application can even be installed to Python enabled

platforms for multiple mixed reality participation.

Fig. 4. Technical overview diagram for **organtrader2010**.

media art creation, display and dissemination.

market.

The system uses an XML RPC to link an augmented reality application with Second Life via a PHP server. This pipeline allows for a method of real-time transfer of 3D visual material, linking the body with the augmented and virtual representations of it. The use of real CT scanned organs with fiducial marker and proximity tracking adds to the viewer's experience of agency within the processes involved in the simulated organ trade and in the process of

While being more technically advanced than the previous example, this project challenges the same ideas under a more specific paradigm of the body. Here the work was designed to engage the viewer in a fictitious situation where they participate in a system of monetary exchange for virtual organs. This quite literally questions the value society places on biological elements within virtual and mixed reality systems as the audience dictates the Perhaps most explicitly exemplifying this mixed reality structure is *Promethean Alchemist.* This project was conceptualized during the Liminal Screens Co-Production Residency at the Banff New Media Institute (BNMI), The Banff Centre, Canada, and is currently under development. *Promethean Alchemist* is a mixed-reality interactive data transfer system that engages participants in mythological creation and DNA sequencing. Biological DNA is translated into computer data using DNA sequencing technology, which is further manipulated into 2D and 3D visual graphic appearance in the ARToolkit. The visualization form is the classic DNA double helix, which re-enters the physical world through an XML RPC (Remote Procedure Call) interface. When two sections of DNA are brought into close proximity, the system recognizes their proximity and 'splices' (combines) the data set into one.

Thus, wearing a HMD (Head Mounted Display) and using these cards, participants are able to pick up segments of DNA in mixed-reality space and throw them into the 'mixing pot' (visualized as a brazier burning with Promethean Fire) to create constantly evolving artificial life forms. Participants are able to create life in an augmented environment and then retrieve their creation in the virtual. Here they can export the form to 3D printing or animation platforms.

*Promethean Alchemist* is an artistic rendition of the implications of DNA code translated into the Deleuzian 'fold'. That is, the in-betweenness of spaces; "able to represent dialectical opposites, such as organic and inorganic, inside and outside."(Deleuze, 1993). The viewers' participation in the layers of code, which constitutes the work, implicates them in a complex exchange between organic and inorganic information where the differences between the two dissolve. The *Promethean Alchemist* is therefore an example of Giannachi hypersurface: *"*Able to present dichotomous relationships, between representation and matter, inside and outside, organic and inorganic, the hypersurface is the site of virtual performance." (Giannachi, 2008). My participation in the exchange is the threefold translation from organic DNA to computational code to visual representation in the ARToolkit. The viewer pushes this further by becoming the 'alchemist' stirring a virtual primordial soup. While the viewer in firmly planted in the physical realm, they still experience the dissolve between the organic and inorganic, humanity and technology.

*Promethean Alchemist* pays homage both visually and conceptually to the Ancient Greek creation myth of Prometheus and Epimetheus, which philosopher Bernard Stiegler references in his text *Technics and Time 1* (Stiegler, 1998). His text seeks to define technics -a technical entity arising out of, but distinct from, mechanical and biological entities- and to delineate the relationship between technics and humans. In the Greek creation story, Prometheus and Epimetheus were given the task of allotting suitable powers to all mortal creatures. Epimetheus did the initial distribution, which was then to be reviewed by Prometheus. Epimetheus set about giving creatures equal shares of positive and negative traits (small beings were given flight, weaker beings speed, etc); "Thus he made his whole distribution on a principle of compensation, being careful by these devices that no species should be destroyed." (Stiegler, 1998). However, when Prometheus came to inspect the work, he realized that Epimetheus had forgotten humans and had nothing left to bestow upon the naked, unshod, and unarmed creatures. Distraught, Prometheus stole from the

Seeking Syncretism: Post-Biological Agency in Real-Time Mixed Reality Data Transfer 187

modern digital alchemist points to the ways in which fruitless occult pursuits can sometimes evolve into transformative practices unforeseen in the former utopian, self-deluded view.

At the recent First International Conference on Transdisciplinary Imaging at the Intersections between Art, Science and Culture (TIIC) Roy Ascott gave a keynote in which he described Syncretism as a possible method by which to classify mixed reality interaction (Ascott 2010). He used Second Life as an example of a metaverse that allowed for an

Second Life, like all virtual environments uses an avatar (agent) to navigate users through the space. While these agents are, usually controlled by the user (avatars can be automated and left on their own, plus there are bots being regularly created and used), they function as independent to their 'master' and are therefore autonomous. Looking to social media and the examples of Facebook, LinkedIn and YouTube to name only three, this can also be said for user profiles on all platforms. Who we represent ourselves as on social networks is not necessarily a true articulation of our identity by any means and therefore it is autonomous. Avatars represent a transient, continually altered identity, usually that of its author and acts as an agent, through which users can engage with virtual platforms. This becomes particularly interesting in unique autonomous systems where participants can physically interact with a virtual deterritorialised 'self' and mediate it through physical engagement. The dispersion of multiple autonomous virtual agents via mixed reality constructs and expands deterritorialisation to include reterritorialisation, by facilitating a dispersive relationship between the body and its virtual self-referent. In the same way that a digital device deterritorialises and reterritorialises information through binary code, the augmentation of an autonomous agent into a shared space with the body, creates new

Critical literary theorist Donna Haraway relates the body's augmentation through digital technology to the notion of the cyborg. In *A Cyborg Manifesto* she argues that the body can be viewed as a conglomerate where its components can be separated, combined with new elements and put together again in ways that violate its traditional boundaries Haraway, 1991). This rhetoric implies a fractured identity that articulates a 'cyborg' reality. *In Chaos Bound*, literary theorist N. Katherine Hayles refers to the notion of dispersed self in light of virtual bodies and narrative, arguing that by turning bodiless information into narratives, the teleology of disembodiment is replaced with contests with ambiguous outcomes: "As I have argued, human being is first of all embodied being, and the complexities of this embodiment mean that human awareness unfolds in very different ways than intelligence in

The advent of nanobiology has called for a rethinking of Hayles and Harraways' posthuman discourse through it shifting our perception of organisms from micro to nano scale. Charles Ostman suggests: "[T]he very definition of life itself may be perched on the edge of the next great revolution in medicine- nanobiology. What is emerging now are technologies and applications in the arenas of biomolecular 'components' integrated into microscale systems, . . . synthetically engineered quasi-viral components, modified DNA and related pseudoproteins, biomolecular prosthetics, and biomolecular organelle component 'entities' .

opportunities for investigation into technology, the body and identity.

**4. Syncretic post biological identity** 

embodied syncretic participatory experience.

cybernetic machines (Hayles, 1999)."

Gods the gifts of skill in the arts and fire, for without fire there was no way to use the skill. Thus through kinship with the Gods by their possession of such God-like powers, humans were able to create things such as clothing, shoes and weapons and could obtain food from the earth.

Thus what constitutes humanity is something that is outside of humanity itself: imagination, discovery and realization through technics; what could be called *Promethean Fire*. While animals are granted predestination in their origin, humans must create their qualities, "Humanity is without qualities, without predestination: it must invent, realize, produce qualities, and nothing indicates that, once produced these qualities will bring about humanity that they will become *its* qualities; for they may rather become those of technics." (Stiegler, 1998). In other words, humanity *is* its inventions, its tools, its technology. *Promethean Alchemist* demonstrates this concept in that each entity that exists in the virtual environment of the system is a translation and an archive of a particular moment in the evolution of humanity mediated between genetic and computational code.

It is through this definition of being in the divine gift of Promethean Fire that humans experience their mortality. "To partake in the lot of immortals means to endure one's mortality by the fact of being in (privative) relation with immortality." (Stiegler, 1998). Indeed, *Promethean Alchemist* represents, in a literal way, the desire of many scientists and philosophers to locate immortality in humanity's technological existence. The translation of DNA code into computational code and re-created in the augmented environment of the digital realm, demonstrates an instance of, 'transcendence through technology.' (Hayles, 1991). The ability to manipulate DNA mediated through the hypersurface represents the hope of many scientists to, in essence, dupe Zeus once again by locating the essence of humanity *within* that which defines it: in technics, thereby attaining immortality.

This hope has practical grounding, for example, in conservation biology. Currently, many scientists are extracting DNA samples from endangered animal species bred in captivity and preserving them for a future time when the organism might be safely reintroduced into its restored habitat, or even into another ecosystem on another planet. The assumption is that organisms can be preserved intact in DNA to be re-substantiated in physical form at a later time; the organism is thought to exist in essence in its DNA. Immortality is gleaned through the ability to store and preserve the organism in this way indefinitely. *Promethean Alchemist*  actively uses the kind of technology that further abstracts and compresses the organism. That the viewer can only retrieve their creation through the limited methods of 3D printing or animation platforms, rather than re-substantiating into living tissue is a testament to the current limitations of translating from biological to virtual environments and back.

Alchemy is, ostensibly, the precursor to modern medicine and science, though its methods may have been closer to magic than scientific method. Sixteenth and Seventeenth Century scientists and swindlers dabbled for years in an alchemical obsession, the search for the philosopher's stone — the long-sought agent for transmuting lead to gold and unlocking other material and spiritual secrets. The stone was the unified theory of everything in that time, including the search for immortality and ways to keep the body young. [5] In *Promethean Alchemist* the viewer becomes the alchemist searching for immortality in the primordial soup of virtual DNA, combining and recombining combinations of computational 'DNA' code to create immortal virtual life forms. The metaphor of the modern digital alchemist points to the ways in which fruitless occult pursuits can sometimes evolve into transformative practices unforeseen in the former utopian, self-deluded view.

## **4. Syncretic post biological identity**

186 Augmented Reality – Some Emerging Application Areas

Gods the gifts of skill in the arts and fire, for without fire there was no way to use the skill. Thus through kinship with the Gods by their possession of such God-like powers, humans were able to create things such as clothing, shoes and weapons and could obtain food from

Thus what constitutes humanity is something that is outside of humanity itself: imagination, discovery and realization through technics; what could be called *Promethean Fire*. While animals are granted predestination in their origin, humans must create their qualities, "Humanity is without qualities, without predestination: it must invent, realize, produce qualities, and nothing indicates that, once produced these qualities will bring about humanity that they will become *its* qualities; for they may rather become those of technics." (Stiegler, 1998). In other words, humanity *is* its inventions, its tools, its technology. *Promethean Alchemist* demonstrates this concept in that each entity that exists in the virtual environment of the system is a translation and an archive of a particular moment in the

It is through this definition of being in the divine gift of Promethean Fire that humans experience their mortality. "To partake in the lot of immortals means to endure one's mortality by the fact of being in (privative) relation with immortality." (Stiegler, 1998). Indeed, *Promethean Alchemist* represents, in a literal way, the desire of many scientists and philosophers to locate immortality in humanity's technological existence. The translation of DNA code into computational code and re-created in the augmented environment of the digital realm, demonstrates an instance of, 'transcendence through technology.' (Hayles, 1991). The ability to manipulate DNA mediated through the hypersurface represents the hope of many scientists to, in essence, dupe Zeus once again by locating the essence of

This hope has practical grounding, for example, in conservation biology. Currently, many scientists are extracting DNA samples from endangered animal species bred in captivity and preserving them for a future time when the organism might be safely reintroduced into its restored habitat, or even into another ecosystem on another planet. The assumption is that organisms can be preserved intact in DNA to be re-substantiated in physical form at a later time; the organism is thought to exist in essence in its DNA. Immortality is gleaned through the ability to store and preserve the organism in this way indefinitely. *Promethean Alchemist*  actively uses the kind of technology that further abstracts and compresses the organism. That the viewer can only retrieve their creation through the limited methods of 3D printing or animation platforms, rather than re-substantiating into living tissue is a testament to the

evolution of humanity mediated between genetic and computational code.

humanity *within* that which defines it: in technics, thereby attaining immortality.

current limitations of translating from biological to virtual environments and back.

Alchemy is, ostensibly, the precursor to modern medicine and science, though its methods may have been closer to magic than scientific method. Sixteenth and Seventeenth Century scientists and swindlers dabbled for years in an alchemical obsession, the search for the philosopher's stone — the long-sought agent for transmuting lead to gold and unlocking other material and spiritual secrets. The stone was the unified theory of everything in that time, including the search for immortality and ways to keep the body young. [5] In *Promethean Alchemist* the viewer becomes the alchemist searching for immortality in the primordial soup of virtual DNA, combining and recombining combinations of computational 'DNA' code to create immortal virtual life forms. The metaphor of the

the earth.

At the recent First International Conference on Transdisciplinary Imaging at the Intersections between Art, Science and Culture (TIIC) Roy Ascott gave a keynote in which he described Syncretism as a possible method by which to classify mixed reality interaction (Ascott 2010). He used Second Life as an example of a metaverse that allowed for an embodied syncretic participatory experience.

Second Life, like all virtual environments uses an avatar (agent) to navigate users through the space. While these agents are, usually controlled by the user (avatars can be automated and left on their own, plus there are bots being regularly created and used), they function as independent to their 'master' and are therefore autonomous. Looking to social media and the examples of Facebook, LinkedIn and YouTube to name only three, this can also be said for user profiles on all platforms. Who we represent ourselves as on social networks is not necessarily a true articulation of our identity by any means and therefore it is autonomous. Avatars represent a transient, continually altered identity, usually that of its author and acts as an agent, through which users can engage with virtual platforms. This becomes particularly interesting in unique autonomous systems where participants can physically interact with a virtual deterritorialised 'self' and mediate it through physical engagement. The dispersion of multiple autonomous virtual agents via mixed reality constructs and expands deterritorialisation to include reterritorialisation, by facilitating a dispersive relationship between the body and its virtual self-referent. In the same way that a digital device deterritorialises and reterritorialises information through binary code, the augmentation of an autonomous agent into a shared space with the body, creates new opportunities for investigation into technology, the body and identity.

Critical literary theorist Donna Haraway relates the body's augmentation through digital technology to the notion of the cyborg. In *A Cyborg Manifesto* she argues that the body can be viewed as a conglomerate where its components can be separated, combined with new elements and put together again in ways that violate its traditional boundaries Haraway, 1991). This rhetoric implies a fractured identity that articulates a 'cyborg' reality. *In Chaos Bound*, literary theorist N. Katherine Hayles refers to the notion of dispersed self in light of virtual bodies and narrative, arguing that by turning bodiless information into narratives, the teleology of disembodiment is replaced with contests with ambiguous outcomes: "As I have argued, human being is first of all embodied being, and the complexities of this embodiment mean that human awareness unfolds in very different ways than intelligence in cybernetic machines (Hayles, 1999)."

The advent of nanobiology has called for a rethinking of Hayles and Harraways' posthuman discourse through it shifting our perception of organisms from micro to nano scale. Charles Ostman suggests: "[T]he very definition of life itself may be perched on the edge of the next great revolution in medicine- nanobiology. What is emerging now are technologies and applications in the arenas of biomolecular 'components' integrated into microscale systems, . . . synthetically engineered quasi-viral components, modified DNA and related pseudoproteins, biomolecular prosthetics, and biomolecular organelle component 'entities' .

Seeking Syncretism: Post-Biological Agency in Real-Time Mixed Reality Data Transfer 189

occur, situated though, in an entirely digitally mediated environment. The projects discussed serve as practical research outcomes that present an investigation into the social and cultural impacts of such systems, through the utilisation of traditional explanations of how we integrate ourselves as individuals in a greater social context. The works serve to function as social experiments that propose questions without answers, in order to allow viewers to investigate these ideologies from a personally discursive position. Mixed reality environments exist as mirrors for society and as such should represent the current state of human existence within such networked systems of social and cultural exchange. The ability to incorporate real-time date transfer methods into such systems redefines our very

Ascott, R. (2008). Cybernetic, Technoetic, Syncretic : The Prospect for Art), *Leonardo* Vol.41,

Ascott, R. (2010). Syncretic Dialogues: Keynote address at Proceedings of The First

A. Butz, T. H¨ollerer, S. Feiner, B. MacIntyre, and C. Beshers. (1999). Enveloping Users and

Deleuze, G. (1993, original 1968) *Difference and Repetition.* Columbia University Press, New

Deleuze, G. and Guatarri, F. (1987, original 1980) *A Thousand Plateaus: Capitalisim and* 

Haraway, D.1(991,) Simians Cyborgs and Women, The Reinvention of Nature, Routledge,

Koleva, B. Schndelbach, H. Benford, S. and Greenhalgh, C.( 2000) Traversable interfaces

Deleuze, G. (1991, original 1966) *Bergsonism*. Zone Books, Brooklyn, NY.

*Schizophrenia.* University of Minnesota Press, Minneapolis, MN. Giannachi, G. (2008) *Virtual Theatres: An Introduction*, Routledge, London and New York. Grasset, R. Lamb, P. Billinghurst, M. (2005). Evaluation of Mixed-Space Collaboration. In

International Conference on Transdisciplinary Imaging at the Intersections between Art, Science and Culture, pp. 4, ISBN: 978-0-9807186-6-9, Sydney Australia,

Computers in a Collaborative 3d Augmented Reality. *IWAR'99: Proceedingsof the 2nd IEEE and ACM International Workshop on Augmented Reality,* page35, Washington,

*Proceedings of the 4th IEEE/ACM International Symposium on Mixed and Augmented Reality* (ISMAR '05). IEEE Computer Society, pp. 90-99, Washington, DC, USA,. Grasset, R. Looser, J and Billinghurst, M. (2006). Transitional Interface: Concept, Issues and

Framework. In *Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality* (ISMAR '06). IEEE Computer Society, pp. 231-

New York. Hayles, N.K. (1990). Chaos Bound: Orderly Disorder in Contemporary Literature and Science. Cornell University Press, Ithaca, NY Kantonen T. Woodward C., Katz N. (2010) *Mixed reality in virtual world teleconferencing*. *Proceedings IEEE Virtual Reality 2010*. pp. 179-182 Waltham, Massachusetts, USA,

between real and virtual worlds. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (CHI '00), pp. 233-240.ACM, New York, NY, USA,

understanding of what constitutes identity and humanity

No.3, (June 2008) pp. 204.

232,Washington, DC, USA.

March 20 - 24,

2000.

November 2010

DC, USA, 1999.

York.

**6. References** 

. . [that] will redefine the very essence of what is commonly referred to as 'life'." (Ostman 2003). Critical theorist Colin Milburn relates nanotechnology to virtual environments, stating: "Nanotechnology thrives in the realm of the virtual. Throughout its history, the field has been shaped by futuristic visions of technological revolution, hyperbolic promises of scientific convergence at the molecular scale, and science fiction stories of the world rebuilt atom by atom." (Milburn, 2008). Today nanotechnology exists as a living sensation of the future of human existence: In Milburn's words, "[A] bodily registration of potential for global change."(Milburn, 2008). Brian Massumi states, "The body,sensor of change, is a transducer of the virtual." Through existing in these virtual representations, that are directly linked to living bio-systems, we effectively sense, feel and think in a way that hybridizes the virtual with scientific inquiry, and therefore we require a discourse that addresses whether this does in fact make us post-biological (Massumi, 2002).

#### **5. Conclusion**

Art has historically had a strong preoccupation with the body and with consciousness. When dealing with such a massive area of inquiry there is a need to look to a wide variety of fields, As Ascott has stated, "[H]owever eccentric or esoteric, any culture, immediate or distant in space or time, any technology, ancient or modern, to find ideas and processes that allow for the navigation of mind and its open-ended exploration."(Ascott, 2008). Roy Ascott proposes a syncretic approach to this issue: "Just as cybernetics analogizes differences between systems, so syncretism finds likeness between unlike things. Syncretic thinking breaches boundaries and subverts protocols. Thinking out of the box, testing the limits of language, behaviour and thought puts the artist on the edge of social norms but at the centre of human development."(Ascott, 2008). Second-order cybernetics did very well to explain our early relationships with machines in terms of interactivity and connectivity, however the incorporation of more open networked systems of autonomous/anthropomorphic based interactions have created a less physical and paradigmatic situation. Networked agency destabilises traditional orthodoxies of thought through challenging notions of representation, confronting materialism, accelerating and smoothing social engagement and most importantly, demanding participation in these open systems of collaborative engagement.

As art is fundamentally an articulation of the human condition it can therefore be said that syncretism is also a valid method for analysing identity within the post-biological discourse. If we are indeed post-biological then we must exist in syncretic mixed reality state. The hybridisation of augmented reality and virtual environments with physical/biological systems calls for a rethinking of not only posthuman ideologies, but also the way that cybernetic systems function. This paper has scoped a range of examples from varying fields of inquiry that have influenced the author's own practice, which is articulated in order to provide a range of practical outcomes to what has been discussed. Through the creation of systems that engage the viewer in a hybridized participatory interaction with mixed reality data transfer, these notions of deterritorialisation, reterritorialisation, syncretism and post biological identity can be explored in a more intuitive and involved fashion.

Mixed Reality data transfer allows for the oscillation between different realities in a way that is seamless and intuitive as it incorporates traditional paradigms of physical engagement to occur, situated though, in an entirely digitally mediated environment. The projects discussed serve as practical research outcomes that present an investigation into the social and cultural impacts of such systems, through the utilisation of traditional explanations of how we integrate ourselves as individuals in a greater social context. The works serve to function as social experiments that propose questions without answers, in order to allow viewers to investigate these ideologies from a personally discursive position. Mixed reality environments exist as mirrors for society and as such should represent the current state of human existence within such networked systems of social and cultural exchange. The ability to incorporate real-time date transfer methods into such systems redefines our very understanding of what constitutes identity and humanity

#### **6. References**

188 Augmented Reality – Some Emerging Application Areas

. . [that] will redefine the very essence of what is commonly referred to as 'life'." (Ostman 2003). Critical theorist Colin Milburn relates nanotechnology to virtual environments, stating: "Nanotechnology thrives in the realm of the virtual. Throughout its history, the field has been shaped by futuristic visions of technological revolution, hyperbolic promises of scientific convergence at the molecular scale, and science fiction stories of the world rebuilt atom by atom." (Milburn, 2008). Today nanotechnology exists as a living sensation of the future of human existence: In Milburn's words, "[A] bodily registration of potential for global change."(Milburn, 2008). Brian Massumi states, "The body,sensor of change, is a transducer of the virtual." Through existing in these virtual representations, that are directly linked to living bio-systems, we effectively sense, feel and think in a way that hybridizes the virtual with scientific inquiry, and therefore we require a discourse that addresses whether

Art has historically had a strong preoccupation with the body and with consciousness. When dealing with such a massive area of inquiry there is a need to look to a wide variety of fields, As Ascott has stated, "[H]owever eccentric or esoteric, any culture, immediate or distant in space or time, any technology, ancient or modern, to find ideas and processes that allow for the navigation of mind and its open-ended exploration."(Ascott, 2008). Roy Ascott proposes a syncretic approach to this issue: "Just as cybernetics analogizes differences between systems, so syncretism finds likeness between unlike things. Syncretic thinking breaches boundaries and subverts protocols. Thinking out of the box, testing the limits of language, behaviour and thought puts the artist on the edge of social norms but at the centre of human development."(Ascott, 2008). Second-order cybernetics did very well to explain our early relationships with machines in terms of interactivity and connectivity, however the incorporation of more open networked systems of autonomous/anthropomorphic based interactions have created a less physical and paradigmatic situation. Networked agency destabilises traditional orthodoxies of thought through challenging notions of representation, confronting materialism, accelerating and smoothing social engagement and most importantly, demanding participation in these open systems of collaborative

As art is fundamentally an articulation of the human condition it can therefore be said that syncretism is also a valid method for analysing identity within the post-biological discourse. If we are indeed post-biological then we must exist in syncretic mixed reality state. The hybridisation of augmented reality and virtual environments with physical/biological systems calls for a rethinking of not only posthuman ideologies, but also the way that cybernetic systems function. This paper has scoped a range of examples from varying fields of inquiry that have influenced the author's own practice, which is articulated in order to provide a range of practical outcomes to what has been discussed. Through the creation of systems that engage the viewer in a hybridized participatory interaction with mixed reality data transfer, these notions of deterritorialisation, reterritorialisation, syncretism and post

Mixed Reality data transfer allows for the oscillation between different realities in a way that is seamless and intuitive as it incorporates traditional paradigms of physical engagement to

biological identity can be explored in a more intuitive and involved fashion.

this does in fact make us post-biological (Massumi, 2002).

**5. Conclusion** 

engagement.


**10** 

*Singapore* 

**Intraoperative Visual Guidance and Control** 

Minimally invasive surgery (MIS) with its small artificial incisions imposed on the patient skin for operation is becoming a preferred surgical treatment comparing with the conventional open surgery. The open surgeries usually create large traumas on the patient bodies to expose their internal anatomy for operational access. The MIS offers its distinct advantages of small injuries, less blood loss and pain and faster recovery during the surgery procedures (Puangmali et al., 2008). However, the indirectly accessed operating causes problems such as restricted vision and difficult hand-eye coordination through the small key holes. The recent development of the modern optics, computer graphics, computer vision and robotics provides possible chances to resolve the problems described above.

Virtual Reality (VR) is a technology providing users an artificial environment to simulate the physical world and objects for displaying, training or gaming etc. Users are allowed to interact with the VR environment through human-computer interface that consists of computer graphic models and various sensors. The major applications of virtual reality in surgery can be divided into three areas: virtual humans for training, virtual telemedicine shared decision environments for training of multiple players and the fusion of virtual humans with real humans for performing surgery (Rosen et al., 1996). VR can create individualized treatment needs and training protocols by providing an environment in which the intensity of feedback and training can be systematically manipulated and enhanced (Anderson et al., 2002; Wang et al., 1999; Cai et al., 2003). VR simulators also allow users to compare their performance with that of their peers (Pearson et al., 2002; Watterson et al., 2002). VR assisted surgical planning facilitates preoperative and postoperative diagnosis with sufficient details of 3-D medical models which are commonly constructed from the computed tomography (CT) or magnetic resonance (MR) images. For example, surgical planning was conducted using a three-dimensional VR based interface to provide a quantitative osteotomy-simulated bone model and prediction of postoperative appearance with photorealistic quality (Xia, et al., 2001). Additionally, there were medical researches based on VR technology specifically conducted to assist surgeons during neurologic, orthopedic, arthroscopic, and urologic surgery (Akay & Marsh, 2001; Burdea et al., 2000; Heng et al., 2004; Kuo et al., 2001; Chui et al., 2006). Recently, there is an increasing

**1. Introduction**

**Interface for Augmented Reality** 

Rong Wen, Chee-Kong Chui and Kah-Bin Lim

**Robotic Surgery** 

*National University of Singapore,* 

