**Intraoperative Visual Guidance and Control Interface for Augmented Reality Robotic Surgery**

Rong Wen, Chee-Kong Chui and Kah-Bin Lim *National University of Singapore, Singapore* 

### **1. Introduction**

190 Augmented Reality – Some Emerging Application Areas

Lang, T. MacIntyre, B. and Zugaza, I. (2008). Massively Multiplayer Online Worlds as a

Levy, P. (1997), *Collective Intelligence.* trans. Bononno, R. Perseus Books, Cambridge, Brian

Milburn, C. (2008). Atoms and Avatars: Virtual Worlds as Massively-Multiplayer

Ostman,O. (2003). Nanobiology? Where Nanotechnology and Biology Come Together.

Sermon, P and Gould, C (2011). Liberate your Avatar: The Revolution Will Be Socially

Stiegler, B. (1998). *Technics and Time, 1: The Fault of Epimetheus*, trans. Richard Beardsworth &

Laboratories. *Spontaneous Generations*. University of Toronto

www.biota.org/ostman/nanobio.htm (accessed June 3, 2004)

George Collins Stanford University Press, 1998

(VR'08 IEEE), March 8-12 2008.

Massachusetts. MIT Press

New York, USA.

Platform for Augmented Reality Experiences. Virtual Reality Conference, 2008

Massumi, B. (2002) *Parables for the Virtual: Movement, Affect, Sensation*. Duke University Press, Durham, NC, McLuhan, M. (1994) *Understanding Media*.

Networked , *Creating Second Lives* , pp. 15-31 Routledge, Taylor & Francis Group,

Minimally invasive surgery (MIS) with its small artificial incisions imposed on the patient skin for operation is becoming a preferred surgical treatment comparing with the conventional open surgery. The open surgeries usually create large traumas on the patient bodies to expose their internal anatomy for operational access. The MIS offers its distinct advantages of small injuries, less blood loss and pain and faster recovery during the surgery procedures (Puangmali et al., 2008). However, the indirectly accessed operating causes problems such as restricted vision and difficult hand-eye coordination through the small key holes. The recent development of the modern optics, computer graphics, computer vision and robotics provides possible chances to resolve the problems described above.

Virtual Reality (VR) is a technology providing users an artificial environment to simulate the physical world and objects for displaying, training or gaming etc. Users are allowed to interact with the VR environment through human-computer interface that consists of computer graphic models and various sensors. The major applications of virtual reality in surgery can be divided into three areas: virtual humans for training, virtual telemedicine shared decision environments for training of multiple players and the fusion of virtual humans with real humans for performing surgery (Rosen et al., 1996). VR can create individualized treatment needs and training protocols by providing an environment in which the intensity of feedback and training can be systematically manipulated and enhanced (Anderson et al., 2002; Wang et al., 1999; Cai et al., 2003). VR simulators also allow users to compare their performance with that of their peers (Pearson et al., 2002; Watterson et al., 2002). VR assisted surgical planning facilitates preoperative and postoperative diagnosis with sufficient details of 3-D medical models which are commonly constructed from the computed tomography (CT) or magnetic resonance (MR) images. For example, surgical planning was conducted using a three-dimensional VR based interface to provide a quantitative osteotomy-simulated bone model and prediction of postoperative appearance with photorealistic quality (Xia, et al., 2001). Additionally, there were medical researches based on VR technology specifically conducted to assist surgeons during neurologic, orthopedic, arthroscopic, and urologic surgery (Akay & Marsh, 2001; Burdea et al., 2000; Heng et al., 2004; Kuo et al., 2001; Chui et al., 2006). Recently, there is an increasing

Intraoperative Visual Guidance and Control Interface for Augmented Reality Robotic Surgery 193

Depending on different ways of augmenting the environment, there are three approaches in AR display devices: Optical See Through devices, Video See Through devices, and Direct Augmentations. Direct Augmentations is also called spatial augmented reality (SAR) using projector-camera system to present the virtual information directly in the 3D real world environment (Raskar et al., 2001; Bimber & Raskar, 2004). Based on the above different approaches of augmentations, four kinds of AR platform are popularly used in present surgical simulation and treatment: head-mounted display (HMD) based AR (Cakmakci & Rolland, 2006), semi-transparent mirror based AR (Liao et al., 2010) and projector based AR

Comparing with the other augmentation approaches, SAR offers the distinct advantages that allow users not wearing heavy helmet, overcoming the limited vision area, sharing the AR intraoperative information and communicating freely with their fellow users. The projector-based AR, which is considered to be the most natural way combining the virtual environment and real environment together, is expected to serve as an effective integrated interface to overcome the aforementioned constraints of the MIS. In addition, the surgical robot and interactive hand gesture operation, considered as dynamic components in the real environment, can be easily integrated in this AR interface for robot assisted MIS surgery.

The general work flow using this advanced user interface is illustrated in Fig. 1. It first involves the handling of 3D models that can be derived from the CT images and the preoperative surgical planning. The construction of the static 3D AR environment includes

Static AR model

4D AR model

Inspection/Monitoring

Robotic

Registration

Hand gestures

Hand gestures

Intraoperative tracking/ recongnition

GPU rendering

CT scan 2D CT images Medical model

Signal from medical device

Intraoperative tracking/recognition

Fig. 1. Work flow of the advanced interface for augmented reality surgery.

**2. Spatial AR interface for robot assisted MIS** 

(Bimber & Raskar, 2004).

emphasis on data fusion involving integration of virtual patient with real patient as a navigational aid in surgery (Rosen et al., 1996; Wen et al., 2010).

Augmented reality (AR) does not rely solely on artificially created virtual environments but expands the virtual computer-generated models with real world environment. AR is characterized as being a fusion of real and virtual data within a real world environment (Milgram et al., 1994). Medical augmented reality is proposed to create the virtual scenes on the related physical tissues, organs or patient skin to assist surgeons visualizing the internal anatomy or invisible surgical planning such as operational distance, trajectories etc.. Due to the advancement in tracking, visualization and display technology, computer-aided medical procedure based AR solutions was examined in the context of minimally invasive surgery (Navab et al., 2007). AR system has been developed to enhance the endoscopic or laparoscopic view to extend surgeons' visibility of hidden anatomical structures that were beneath the then surgical scene. This augmentation facilitates the surgical operation to avoid risk regions like arteries and nerves. On the other hand the pathology regions are easily to be reached. The needle's insertion happened within the target organ was guided to reach the tumor volume (Konishia et al., 2005; Thoranaghatte et al., 2008; Lee et al., 2010). Thus a more intuitive visual assistance during the surgery procedure is provided for surgeon's response. Integration of laparoscopic and 3D-ultrosound images that provided anatomical information inside the target organs was studied by Konishia et al. (Konishia et al., 2005). With head-mounted display (HMD) based AR interface, numerous studies have demonstrated the potential efficiency of MRI-guided tumour extraction and therapy (Liao et al., 2010). Projector-based visualization system for clinically intraoperative navigation was reported (Hoppe, 2003). Grimson et al. (Grimson et al., 1996) showed its potentials to surgical navigation systems that enhance physical scenes with the augmentation of overlaying virtual internal structures. Another medical application of marking the surgical object with a virtual-pen was based on direct AR interface (Seo et al., 2007). It was proposed to supersede the ink pen which was widely used in the surgical environment. With the virtual-pen, the surgical markings were achieved by directly projecting them onto human body. The introduction of AR to surgical treatment creates a virtual medium between preoperative plan and intraoperative environment.

In order to enhance manual dexterity and accuracy of instrument manipulation, robotic assistance has been introduced in the MIS to solve operation constraint problems. The da Vinci™ Surgical System, a master–slave integrated robot, has been applied for treating a range of surgical conditions improving operational flexibility and accuracy (Guthart & Salisbury, 2000). For the sophisticated minimally invasive operations like coronary artery bypass and mitral valve repair, robot assisted surgery is expected to achieve satisfying clinical outcomes (Devernay et al., 2001). Surgical robot also demonstrated its distinct advantages in speed, accuracy and consistency for the larger liver tumor treatment with RFA. Yang et al. investigated design and implementation of a novel robotic system for RFA considering both kinematics and clinical requirements (Yang et al., 2010). Other advantages offered by the robotic RFA include minimizing radioactive exposure and overcoming visual restriction as well as spatial constraint for minimally invasive approaches like the laparoscopic or percutaneous surgical procedure. Integrating the imaging guidance and the surgical robot assistance into minimally invasive operation brings the preoperative planning and intraoperative manipulation into perspective.

## **2. Spatial AR interface for robot assisted MIS**

192 Augmented Reality – Some Emerging Application Areas

emphasis on data fusion involving integration of virtual patient with real patient as a

Augmented reality (AR) does not rely solely on artificially created virtual environments but expands the virtual computer-generated models with real world environment. AR is characterized as being a fusion of real and virtual data within a real world environment (Milgram et al., 1994). Medical augmented reality is proposed to create the virtual scenes on the related physical tissues, organs or patient skin to assist surgeons visualizing the internal anatomy or invisible surgical planning such as operational distance, trajectories etc.. Due to the advancement in tracking, visualization and display technology, computer-aided medical procedure based AR solutions was examined in the context of minimally invasive surgery (Navab et al., 2007). AR system has been developed to enhance the endoscopic or laparoscopic view to extend surgeons' visibility of hidden anatomical structures that were beneath the then surgical scene. This augmentation facilitates the surgical operation to avoid risk regions like arteries and nerves. On the other hand the pathology regions are easily to be reached. The needle's insertion happened within the target organ was guided to reach the tumor volume (Konishia et al., 2005; Thoranaghatte et al., 2008; Lee et al., 2010). Thus a more intuitive visual assistance during the surgery procedure is provided for surgeon's response. Integration of laparoscopic and 3D-ultrosound images that provided anatomical information inside the target organs was studied by Konishia et al. (Konishia et al., 2005). With head-mounted display (HMD) based AR interface, numerous studies have demonstrated the potential efficiency of MRI-guided tumour extraction and therapy (Liao et al., 2010). Projector-based visualization system for clinically intraoperative navigation was reported (Hoppe, 2003). Grimson et al. (Grimson et al., 1996) showed its potentials to surgical navigation systems that enhance physical scenes with the augmentation of overlaying virtual internal structures. Another medical application of marking the surgical object with a virtual-pen was based on direct AR interface (Seo et al., 2007). It was proposed to supersede the ink pen which was widely used in the surgical environment. With the virtual-pen, the surgical markings were achieved by directly projecting them onto human body. The introduction of AR to surgical treatment creates a virtual medium between

In order to enhance manual dexterity and accuracy of instrument manipulation, robotic assistance has been introduced in the MIS to solve operation constraint problems. The da Vinci™ Surgical System, a master–slave integrated robot, has been applied for treating a range of surgical conditions improving operational flexibility and accuracy (Guthart & Salisbury, 2000). For the sophisticated minimally invasive operations like coronary artery bypass and mitral valve repair, robot assisted surgery is expected to achieve satisfying clinical outcomes (Devernay et al., 2001). Surgical robot also demonstrated its distinct advantages in speed, accuracy and consistency for the larger liver tumor treatment with RFA. Yang et al. investigated design and implementation of a novel robotic system for RFA considering both kinematics and clinical requirements (Yang et al., 2010). Other advantages offered by the robotic RFA include minimizing radioactive exposure and overcoming visual restriction as well as spatial constraint for minimally invasive approaches like the laparoscopic or percutaneous surgical procedure. Integrating the imaging guidance and the surgical robot assistance into minimally invasive operation brings the preoperative planning and

navigational aid in surgery (Rosen et al., 1996; Wen et al., 2010).

preoperative plan and intraoperative environment.

intraoperative manipulation into perspective.

Depending on different ways of augmenting the environment, there are three approaches in AR display devices: Optical See Through devices, Video See Through devices, and Direct Augmentations. Direct Augmentations is also called spatial augmented reality (SAR) using projector-camera system to present the virtual information directly in the 3D real world environment (Raskar et al., 2001; Bimber & Raskar, 2004). Based on the above different approaches of augmentations, four kinds of AR platform are popularly used in present surgical simulation and treatment: head-mounted display (HMD) based AR (Cakmakci & Rolland, 2006), semi-transparent mirror based AR (Liao et al., 2010) and projector based AR (Bimber & Raskar, 2004).

Comparing with the other augmentation approaches, SAR offers the distinct advantages that allow users not wearing heavy helmet, overcoming the limited vision area, sharing the AR intraoperative information and communicating freely with their fellow users. The projector-based AR, which is considered to be the most natural way combining the virtual environment and real environment together, is expected to serve as an effective integrated interface to overcome the aforementioned constraints of the MIS. In addition, the surgical robot and interactive hand gesture operation, considered as dynamic components in the real environment, can be easily integrated in this AR interface for robot assisted MIS surgery.

The general work flow using this advanced user interface is illustrated in Fig. 1. It first involves the handling of 3D models that can be derived from the CT images and the preoperative surgical planning. The construction of the static 3D AR environment includes

Fig. 1. Work flow of the advanced interface for augmented reality surgery.

Intraoperative Visual Guidance and Control Interface for Augmented Reality Robotic Surgery 195

Since the coordinate of the top-left corner *L2* of the projected image is easily detected, the position of *L2* can be adjusted to its expected coordinate. Assume that the *L1* and *L3* are the top-left corner of the source image and the captured image respectively, and *L1'(x, y)* and *L3'(x, y)* are the expected position of the top-left corner of the source image and the captured image respectively. The expected coordinates of the projected image can be acquired by calculating the spatial relationship with the homography between the camera and the world coordinate. The relationship between *L1' (x, y)* and *L3' (x, y)* is shown as Eq. (2) and Eq. (3):

�� � ����

where �� is a affine transformation. A testing projection is conducted to detect the differences between *L3* and *L'3*. With the piecewise linear method described above, �� and

Suppose that the patient is lying on a surgical bed and the camera is placed along the vertical axis from the target organ relative to the surgical bed. A virtual organ model with the same size and angle located as the real one is expected to be projected on the patient body. This bird-eye view of the image of the virtual model can be derived from the CT images. Finally, the geometric correction for the projected image is conducted to register the

���

Fig. 2. Marker based registration for SAR (*M1, M2, M3* are three markers attached onto the

In order to create an overall projected image that appears correct and seamless on arbitrary surfaces in the real world, geometric correction plays an important role in construction of the SAR environment. This task is challenging for projector-based displays because of the

�� � � ����

�� can be replaced by the ��� and ���

virtual model with the real target organ.

*M1(x, y, z)*

*M2(x, y, z)*

*L2*

����

phantom's body.)

following problems:

**4. Correction for projection** 

�

���� �

*M3(x, y, z)*

*L2'*

0 *<sup>i</sup>*

i

*T T <sup>m</sup> RRR m o* . (1)

�� for corresponding piecewise regions.

, (2)

, (3)

Expected projection region

Projection region

Target real organ *O* Virtual organ model *Op*

registration, geometric and radiometric correction. There is an iterative process with AR real-time model rendering and feedback from the real world environment, including hand gesture interaction with the AR model, surgical robot and the feedback of signal from related medical devices during the surgical operation. When the AR model, displayed over the real organ of the patient by the projector-camera system, receives any "stimulus" from the real environment, there will be changes responding in real time on the model to guide the surgeons' decisions and operation.

## **3. Registration**

Registration is a main challenge in AR technology (Pablo et al., 2010). Correct and consistent registration method plays an important role in estimating the position of either the virtual objects relative to the object in the real world or of the virtual objects relative to a common coordinate system. A number of registration methods have been developed ranging from analysis of the real world environment to evaluations of intentionally introduced fiducials (Massoptier & Casciaro, 2008; Teber et al., 2009).

Marker based methods are commonly used to aid intraoperative registration in robot assisted surgeries (Yang et al., 2010). A marker based registration method for projectorcamera system is introduced in this chapter. With the markers attached onto the patient's skin, spatial relationship between the markers and target organ can be derived from CT image of the patient. As shown in Fig. 2, three markers *M1, M2, M3*, are attached onto the patient body to construct a space reference fame *T*. This marker based frame is used to construct the spatial relationship that connects the stereovision frame, world coordinate and CT image frame. The spatial relationship enables the virtual model displayed by the projector-camera system to register with the target organ within the patient body. The two steps to overlay a virtual model of the target organ precisely on the patient with correct position, scale and angle in the projector-camera system are described as follows.

The first step involves establishing the spatial relationship between the markers *Mi* (i=1, 2, 3) and the target organ *O*. CT imaging of the patient with three markers attached is performed prior to the surgery. Appropriate markers deployment is important. The markers should facilitate the registration and avoid inconveniencing the surgery. Their deployment is dependent on the following considerations. Firstly, the surgical tools' motion should not be obstructed by the markers. The surgical path or motion is given a higher priority in preoperative planning compared to that of markers' deployment. Since the markers are used primarily for constructing the space reference frame *T*, this can be achieved as long as the stereovision system can detect the markers during registration. Preoperative planning including obstruction prevention will be discussed in section 5.1. Secondly, the markers should be placed beyond the expected projection region to avoid violating the projected virtual model.

For the second step, the initial calibration of projector-camera system has been described by Bimber & Raskar (Bimber & Raskar, 2004). It establishes the correspondence between the pixels in camera and projector as a lookup table. With the stereo vision system, the reference frame *T* consisted of three markers is achieved from the estimation of space coordinate of the three markers. As mentioned above the spatial relationship between the markers and the target organ based on the image coordinate is derived from the CT images. In this case, the spatial relationship between target organ and the projected virtual model is established as Eq. (1):

$$\mathbf{R}\_0^T = \mathbf{R}\_{m\_i}^T \times \mathbf{R}\_o^{m\_i} \,. \tag{1}$$

Since the coordinate of the top-left corner *L2* of the projected image is easily detected, the position of *L2* can be adjusted to its expected coordinate. Assume that the *L1* and *L3* are the top-left corner of the source image and the captured image respectively, and *L1'(x, y)* and *L3'(x, y)* are the expected position of the top-left corner of the source image and the captured image respectively. The expected coordinates of the projected image can be acquired by calculating the spatial relationship with the homography between the camera and the world coordinate. The relationship between *L1' (x, y)* and *L3' (x, y)* is shown as Eq. (2) and Eq. (3):

$$L\_3 = \mathbf{M}\_\mathbf{A} L\_3 \,'\,,\tag{2}$$

$$L\_1' = \mathbf{H\_p} \mathbf{H\_c^{-1}} L\_3' \,\prime \,\tag{3}$$

where �� is a affine transformation. A testing projection is conducted to detect the differences between *L3* and *L'3*. With the piecewise linear method described above, �� and ���� �� can be replaced by the ��� and ��� ��� �� for corresponding piecewise regions.

Suppose that the patient is lying on a surgical bed and the camera is placed along the vertical axis from the target organ relative to the surgical bed. A virtual organ model with the same size and angle located as the real one is expected to be projected on the patient body. This bird-eye view of the image of the virtual model can be derived from the CT images. Finally, the geometric correction for the projected image is conducted to register the virtual model with the real target organ.

Fig. 2. Marker based registration for SAR (*M1, M2, M3* are three markers attached onto the phantom's body.)

#### **4. Correction for projection**

194 Augmented Reality – Some Emerging Application Areas

registration, geometric and radiometric correction. There is an iterative process with AR real-time model rendering and feedback from the real world environment, including hand gesture interaction with the AR model, surgical robot and the feedback of signal from related medical devices during the surgical operation. When the AR model, displayed over the real organ of the patient by the projector-camera system, receives any "stimulus" from the real environment, there will be changes responding in real time on the model to guide

Registration is a main challenge in AR technology (Pablo et al., 2010). Correct and consistent registration method plays an important role in estimating the position of either the virtual objects relative to the object in the real world or of the virtual objects relative to a common coordinate system. A number of registration methods have been developed ranging from analysis of the real world environment to evaluations of intentionally introduced fiducials

Marker based methods are commonly used to aid intraoperative registration in robot assisted surgeries (Yang et al., 2010). A marker based registration method for projectorcamera system is introduced in this chapter. With the markers attached onto the patient's skin, spatial relationship between the markers and target organ can be derived from CT image of the patient. As shown in Fig. 2, three markers *M1, M2, M3*, are attached onto the patient body to construct a space reference fame *T*. This marker based frame is used to construct the spatial relationship that connects the stereovision frame, world coordinate and CT image frame. The spatial relationship enables the virtual model displayed by the projector-camera system to register with the target organ within the patient body. The two steps to overlay a virtual model of the target organ precisely on the patient with correct

The first step involves establishing the spatial relationship between the markers *Mi* (i=1, 2, 3) and the target organ *O*. CT imaging of the patient with three markers attached is performed prior to the surgery. Appropriate markers deployment is important. The markers should facilitate the registration and avoid inconveniencing the surgery. Their deployment is dependent on the following considerations. Firstly, the surgical tools' motion should not be obstructed by the markers. The surgical path or motion is given a higher priority in preoperative planning compared to that of markers' deployment. Since the markers are used primarily for constructing the space reference frame *T*, this can be achieved as long as the stereovision system can detect the markers during registration. Preoperative planning including obstruction prevention will be discussed in section 5.1. Secondly, the markers should be placed beyond the expected projection region to avoid violating the projected

For the second step, the initial calibration of projector-camera system has been described by Bimber & Raskar (Bimber & Raskar, 2004). It establishes the correspondence between the pixels in camera and projector as a lookup table. With the stereo vision system, the reference frame *T* consisted of three markers is achieved from the estimation of space coordinate of the three markers. As mentioned above the spatial relationship between the markers and the target organ based on the image coordinate is derived from the CT images. In this case, the spatial relationship between target organ and the projected virtual model is established as

position, scale and angle in the projector-camera system are described as follows.

the surgeons' decisions and operation.

(Massoptier & Casciaro, 2008; Teber et al., 2009).

**3. Registration** 

virtual model.

Eq. (1):

In order to create an overall projected image that appears correct and seamless on arbitrary surfaces in the real world, geometric correction plays an important role in construction of the SAR environment. This task is challenging for projector-based displays because of the following problems:

Intraoperative Visual Guidance and Control Interface for Augmented Reality Robotic Surgery 197

T = PC

B

Although Eq. (6) is not directly applicable when the screen is non-at, the function **HpHc**

Undistorted image

C

relates the camera and the projector is proposed in this research. With the piecewise linear method, the source image can be marked piecewise by defined feature points. Consequently, the source image can be operated "piece" by "piece", in which the correction of projected image on non-planar screen surface is approximated by connecting all these piece regions. Quadrilateral piece is used as unit region to compute the homography between source image and projected image, captured image and source image. The region

The point-correspondence between the image pairs could be acquired from the result of registration. For every piece region, four feature points are established with the respective relationship defined by Eq. (9), (10) and (11). *i* is the *i*th piecewise region in the source image, projected image and the captured image. The other parameters have the same meaning as

����� �

����� �

��� �� � ��� ��� 1

�����

��� ��� 1

��� ��� 1

**−1**

**<sup>−</sup>1** by piecewise linear mapping function that

Camera screen

b

a

�, (9)

�, (10)

� (11)

Fig. 3. Viewer-dependent geometric correction.

Planar screen surface

Source image

PC screen

can be found easily. An idea of replacing **HpHc**

that of the equation (4), (5), (6).

connection could be compensated by bilinear interpolation.

P

A

(x0, y0)

� ��� ��� 1

� ��� ��� 1

� ��� ��� 1


In addition, the display quality of the projected image is modulated by the colored and textured arbitrary surface sacrificing its photometric quality. Human eyes are sensitive to this artifact. Radiometric correction can be used to relax this restriction with the SAR system by compensating the spatially varying reflectance.

#### **4.1 Geometric correction**

Assume the screen surface is planar and projection light is perpendicularly to the surface. The projection on the screen surface appears hardly distorted to the observers in front of the screen. When an image is projected on a planar screen surface that is not perpendicular to it, the projective distortion corrupts the projected image. When an image is projected onto a screen surface with irregular unevenness, geometric factor of the screen surfaces also affects distortion of the projected image.

For a planar projection, homography is used to represent the transformation from the source image to the projected image in the world coordinate, from the projected image to the captured image in the camera as Eq. (4) and Eq. (5):

$$
\begin{pmatrix} p\_\chi \\ p\_\mathcal{\mathcal{V}} \\ 1 \end{pmatrix} = \mathbf{H}\_\mathbf{p} \begin{pmatrix} \mathbf{w}\_\mathcal{\mathcal{X}} \\ \mathbf{w}\_\mathcal{\mathcal{Y}} \\ 1 \end{pmatrix} \tag{4}
$$

$$
\begin{pmatrix} c\_{\chi} \\ c\_{\mathcal{V}} \\ 1 \end{pmatrix} = \mathbf{H}\_{\mathbf{c}} \begin{pmatrix} w\_{\chi} \\ w\_{\mathcal{V}} \\ 1 \end{pmatrix} \tag{5}
$$

*(wx, wy, 1)T* is an image point projected on a planar surface. *(px, py, 1)T* and *(cx, cy, 1)T* are the corresponding points located in the projector's and camera's respectively. From *Hp* and *Hc*, a relation from the source image to the captured image can be established in Eq. (6):

$$
\begin{pmatrix} p\_\chi \\ p\_\mathcal{\prime} \\ 1 \end{pmatrix} = \mathbf{H\_p} \mathbf{H\_c^{-1}} \begin{pmatrix} c\_\chi \\ c\_\mathcal{\prime} \\ 1 \end{pmatrix} \tag{6}
$$

The geometric correction process is illustrated in Fig. 3. There are two types of correction models: viewer dependent and viewer independent. For viewer-dependent correction, the source image should be pre-wrapped as *T-1S*, where *S* is the source image and *T* is the mapping function between the source image and the captured image. For viewerindependent correction, the source image should be wrapped as *P-1S,* where *P* is the mapping function between the source image and the projected image. Eq. (7) and Eq. (8) illustrate the correction process**:**

$$PC\left(T^{-1}S\right) = PC\left(C^{-1}P^{-1}\right)S = S \,\, \rho \tag{7}$$

$$P\left(P^{-1}S\right) = P\left(CT^{-1}\right)S = S \ . \tag{8}$$

Fig. 3. Viewer-dependent geometric correction.

196 Augmented Reality – Some Emerging Application Areas

In addition, the display quality of the projected image is modulated by the colored and textured arbitrary surface sacrificing its photometric quality. Human eyes are sensitive to this artifact. Radiometric correction can be used to relax this restriction with the SAR system

Assume the screen surface is planar and projection light is perpendicularly to the surface. The projection on the screen surface appears hardly distorted to the observers in front of the screen. When an image is projected on a planar screen surface that is not perpendicular to it, the projective distortion corrupts the projected image. When an image is projected onto a screen surface with irregular unevenness, geometric factor of the screen surfaces also affects

For a planar projection, homography is used to represent the transformation from the source image to the projected image in the world coordinate, from the projected image to the

���� �

���� �

*(wx, wy, 1)T* is an image point projected on a planar surface. *(px, py, 1)T* and *(cx, cy, 1)T* are the corresponding points located in the projector's and camera's respectively. From *Hp* and *Hc*, a

�� �� 1

�� �� 1

�� � �� �� 1

�, (4)

�, (5)

�, (6)

1 1 <sup>1</sup> *PC T S PC C P S S* , (7)

1 1 *P P S P CT S S* . (8)

� �� �� 1

� �� �� 1

� �� �� 1

relation from the source image to the captured image can be established in Eq. (6):

������

The geometric correction process is illustrated in Fig. 3. There are two types of correction models: viewer dependent and viewer independent. For viewer-dependent correction, the source image should be pre-wrapped as *T-1S*, where *S* is the source image and *T* is the mapping function between the source image and the captured image. For viewerindependent correction, the source image should be wrapped as *P-1S,* where *P* is the mapping function between the source image and the projected image. Eq. (7) and Eq. (8)

2. Geometric distortion arising from lens distortion, arbitrary physical surfaces, etc.. 3. Real-time geometric correction. This is a challenging problem and it may occur during the dynamic projection in spatial space in which screen surface is changing in real-time

1. Perspective distortion arising from projection.

by compensating the spatially varying reflectance.

captured image in the camera as Eq. (4) and Eq. (5):

**4.1 Geometric correction** 

distortion of the projected image.

illustrate the correction process**:**

because of variance of projector's location or angle.

Although Eq. (6) is not directly applicable when the screen is non-at, the function **HpHc −1** can be found easily. An idea of replacing **HpHc <sup>−</sup>1** by piecewise linear mapping function that relates the camera and the projector is proposed in this research. With the piecewise linear method, the source image can be marked piecewise by defined feature points. Consequently, the source image can be operated "piece" by "piece", in which the correction of projected image on non-planar screen surface is approximated by connecting all these piece regions. Quadrilateral piece is used as unit region to compute the homography between source image and projected image, captured image and source image. The region connection could be compensated by bilinear interpolation.

The point-correspondence between the image pairs could be acquired from the result of registration. For every piece region, four feature points are established with the respective relationship defined by Eq. (9), (10) and (11). *i* is the *i*th piecewise region in the source image, projected image and the captured image. The other parameters have the same meaning as that of the equation (4), (5), (6).

$$
\begin{pmatrix} p\_{l\_\chi} \\ p\_{l\_\mathcal{V}} \\ 1 \end{pmatrix} = \mathbf{H}\_{\mathbf{p}\_l} \begin{pmatrix} \mathbf{w}\_{l\_\mathcal{X}} \\ \mathbf{w}\_{l\_\mathcal{V}} \\ 1 \end{pmatrix} \tag{9}
$$

$$
\begin{pmatrix} \mathbf{c}\_{l\_{\mathcal{X}}} \\ \mathbf{c}\_{l\_{\mathcal{Y}}} \\ \mathbf{1} \end{pmatrix} = \mathbf{H}\_{\mathbf{c}\_{l}} \begin{pmatrix} \mathbf{w}\_{l\_{\mathcal{X}}} \\ \mathbf{w}\_{l\_{\mathcal{Y}}} \\ \mathbf{1} \end{pmatrix} \tag{10}
$$

$$
\begin{pmatrix} p\_{l\_\chi} \\ p\_{l\_\chi} \\ 1 \end{pmatrix} = \mathbf{H}\_{\mathbf{p}\_l} \mathbf{H}\_{\mathbf{c}\_l}^{-1} \begin{pmatrix} \mathbf{c}\_{l\_\chi} \\ \mathbf{c}\_{l\_\chi} \\ 1 \end{pmatrix} \tag{11}
$$

Intraoperative Visual Guidance and Control Interface for Augmented Reality Robotic Surgery 199

for the color case, normalization of the elements in the **M** is conducted to eliminate the influence of the surface reflectance. The diagonal elements in the **M** are evaluated to unity, *Mdd* = 1( *d* = R, G, B). By comparing of images with differences in one of the three channels,

Based on the piecewise linear method described in the section A, the matrix **F**, **E** and **M** can

Fig. 5. Projected correction for Augmented Reality environment construction ((a) is a source image of liver model. (c) is the distorted image projected on a phantom body. (b) and (d) are

The dynamic augmented virtual model displayed on the AR interface during surgery is derived from the therapeutic model developed in the preoperative phase. The preoperative plan is based on medical diagnosis, and includes the therapeutic model and surgical planning. The therapeutic model is a stereoscopic cyber-graphic model reconstructed from CT images with relevant organs and tissues segmented. Besides the three dimensional profile of the target tumour and blood vessels, other important circumjacent tissues and organs around the interested region are included in the model as specified by the surgeons. Additionally, markers deployment should also be examined for this therapeutic model

**5. Intraoperative surgical intervention based on the AR interface** 

design to make sure they are beyond the surgical working path.

ܯோభ ൌ οܥభȀοܥோ (k1 = G, B), (13)

ܯீమ ൌ οܥమȀοܥீ) *k2* = R, B), (14)

ܯయ ൌ οܥଷȀοܥ) *k3* = R,G). (15)

**(a) (b)**

**(c) (d)**

the other elements in the *Mij* are acquired for the corresponding channel:

be replaced by **F***i*, **E***i* and **M***i* as (9), (10) and (11).

the corrected images.)

**5.1 Therapeutic model design** 

An example of piecewise geometric correction for chessboard projection is shown as Fig. 4. From Fig. 4, we can observe that the number and location of the selected feature points play important role on the final result.

Fig. 4. Geometric correction on the curved surface with a chessboard pattern ((a) and (d) are side views. (b) and (e) are front views. (c) and (f) are the corrected projection for (a) and (d) respectively from the front views).

#### **4.2 Radiometric correction**

Radiometry distortion is due to angle of projection, screen texture and the environment lightings. Photometric models of interaction with spectra reflectance between projector and camera were proposed by S. K. Nayar et al. (Nayar et al., 2003) and K. Fujii et al. (Fujii et al., 2005). Based on the photometric models, a practical method to apply radiometric compensation by the projector-camera system is presented in this section. Image **C** = *[CR CG CB]T* acquired by camera from the projected image **P** = *[PR PG PB]T* is affected by surface reflectance and environment lighting. Assume projector and camera each have three color channels (R, G, B). The equation (12) illustrates the radiometric transformation from projector to camera via the screen surface based on the above photometric models:

$$\mathbf{C} = \mathbf{F}(\mathbf{E} + \mathbf{M}\mathbf{P}),\tag{12}$$

where :

$$
\begin{bmatrix} \mathbf{C\_R} \\ \mathbf{C\_G} \\ \mathbf{C\_B} \end{bmatrix} = \begin{bmatrix} \mathbf{F\_R} \mathbf{E\_R} \\ \mathbf{F\_G} \mathbf{E\_G} \\ \mathbf{F\_B} \mathbf{E\_B} \end{bmatrix} + \begin{bmatrix} \mathbf{F\_R} \mathbf{M\_{RR}} & \mathbf{M\_{RG}} & \mathbf{M\_{RB}} \\ \mathbf{M\_{GR}} & \mathbf{F\_G} \mathbf{M\_{GG}} & \mathbf{M\_{GB}} \\ \mathbf{M\_{BR}} & \mathbf{M\_{BG}} & \mathbf{F\_B} \mathbf{M\_{BB}} \end{bmatrix} \begin{bmatrix} \mathbf{P\_R} \\ \mathbf{P\_G} \\ \mathbf{P\_B} \end{bmatrix}.
$$

The vector **F** is the reflectance of surface. **E** is the contribution of the environment lighting to the captured image. The color mixing matrix **M** describes the radiometric interaction between the projector and the camera.

In comparison with the brightness of environment lighting, projector is considerably brighter when performing the projection on the textured and colored surface (Fujii et al., 2005). The color mixing matrix **M** can be derived by projecting two gray images with gray values of 255 and 0 respectively for monotonic radiometric calibration. To compute the **M**

for the color case, normalization of the elements in the **M** is conducted to eliminate the influence of the surface reflectance. The diagonal elements in the **M** are evaluated to unity, *Mdd* = 1( *d* = R, G, B). By comparing of images with differences in one of the three channels, the other elements in the *Mij* are acquired for the corresponding channel:

$$\mathcal{M}\_{Rk\_1} = \Delta \mathcal{C}\_{k\_1} / \Delta \mathcal{C}\_R \text{ (}\mathbf{k}\_1 = \mathbf{G} \text{ }\mathbf{B}\text{)}\text{.}\tag{13}$$

$$M\_{Gk\_2} = \Delta \mathcal{C}\_{k\_2} / \Delta \mathcal{C}\_G \text{ (}k\_2 = \text{R, B}\text{)},\tag{14}$$

$$M\_{Bk\_3} = \Delta \mathcal{C}\_{k3} / \Delta \mathcal{C}\_B \text{ (}k\_3 = \text{R,G}\text{)}.\tag{15}$$

Based on the piecewise linear method described in the section A, the matrix **F**, **E** and **M** can be replaced by **F***i*, **E***i* and **M***i* as (9), (10) and (11).

Fig. 5. Projected correction for Augmented Reality environment construction ((a) is a source image of liver model. (c) is the distorted image projected on a phantom body. (b) and (d) are the corrected images.)

#### **5. Intraoperative surgical intervention based on the AR interface**

#### **5.1 Therapeutic model design**

198 Augmented Reality – Some Emerging Application Areas

An example of piecewise geometric correction for chessboard projection is shown as Fig. 4. From Fig. 4, we can observe that the number and location of the selected feature points play

Fig. 4. Geometric correction on the curved surface with a chessboard pattern ((a) and (d) are side views. (b) and (e) are front views. (c) and (f) are the corrected projection for (a) and (d)

**(d) (e)**

**(a)**

Radiometry distortion is due to angle of projection, screen texture and the environment lightings. Photometric models of interaction with spectra reflectance between projector and camera were proposed by S. K. Nayar et al. (Nayar et al., 2003) and K. Fujii et al. (Fujii et al., 2005). Based on the photometric models, a practical method to apply radiometric compensation by the projector-camera system is presented in this section. Image **C** = *[CR CG CB]T* acquired by camera from the projected image **P** = *[PR PG PB]T* is affected by surface reflectance and environment lighting. Assume projector and camera each have three color channels (R, G, B). The equation (12) illustrates the radiometric transformation from

> F�M�� M�� M�� M�� F�M�� M�� M�� M�� F�M��

The vector **F** is the reflectance of surface. **E** is the contribution of the environment lighting to the captured image. The color mixing matrix **M** describes the radiometric interaction

In comparison with the brightness of environment lighting, projector is considerably brighter when performing the projection on the textured and colored surface (Fujii et al., 2005). The color mixing matrix **M** can be derived by projecting two gray images with gray values of 255 and 0 respectively for monotonic radiometric calibration. To compute the **M**

� � ��� � ���, (12)

**(b) (c)**

**(f)**

� � P� P� P� �.

projector to camera via the screen surface based on the above photometric models:

important role on the final result.

respectively from the front views).

� C� C� C� ���

between the projector and the camera.

F�E� F�E� F�E�

���

**4.2 Radiometric correction** 

where :

The dynamic augmented virtual model displayed on the AR interface during surgery is derived from the therapeutic model developed in the preoperative phase. The preoperative plan is based on medical diagnosis, and includes the therapeutic model and surgical planning. The therapeutic model is a stereoscopic cyber-graphic model reconstructed from CT images with relevant organs and tissues segmented. Besides the three dimensional profile of the target tumour and blood vessels, other important circumjacent tissues and organs around the interested region are included in the model as specified by the surgeons. Additionally, markers deployment should also be examined for this therapeutic model design to make sure they are beyond the surgical working path.

Intraoperative Visual Guidance and Control Interface for Augmented Reality Robotic Surgery 201

2. Compute the mean location or centroid in the window based on the probability

2 <sup>20</sup> ( , ), *x y*

2 <sup>02</sup> ( , ), *x y*

<sup>11</sup> ( , ), *x y*

3. Move the center of the window to the location derived from the previous step.

The CamShift algorithm also has the ability to update the size of the search window to fit

For intraoperative interaction, hand gesture is adopted to control display of the AR model and robotic device. Surgeon's hand gestures are detected by a stereovision system over or beside the AR environment. This interface has advantage on sterilization which is an important requirement in operating room. The vision based hand gesture can be integrated into the AR system using projector-camera system or conventional tracking device. The AR model projected on the patient surface can be scaled and rotated using hand gestures.

In order to enhance the speed and accuracy of hand gesture recognition, an innovative 3D model based method is introduced. This method is based on the analysis of model features

where *I(x, y)* is the probability value at coordinate (x, y) in the image, ranging over the entire search window. The coordinates of the centroid are then computed from the

*<sup>M</sup> xIxy* (16)

*<sup>M</sup> yIxy* (17)

*<sup>M</sup> xyI x y* (18)

10 00 / , *<sup>c</sup> xM M* (19)

01 00 / , *<sup>c</sup> y M M* (20)

The procedure of the Mean-Shift tracking algorithm is described as follows::

distribution image. The centroid (xc, yc) is calculated as follows: Firstly, the zeroth and first moments of the image are calculated as:

1. An initial location of a search window is chosen.

moments as Eq. (19) and Eq. (20).

4. Return to Step 2 until convergence is achieved.

the object being tracked as the frames change (Ling et al., 2010).

Fig. 6. RFA needle is tracked based on an AR environment.

Additionally, hand gestures can also be used to guide robotic execution.

For the therapeutic model design based on the robot assisted surgery, optimization algorithms are usually applied to enhance the efficiency of robotic implementation and to decrease complexity during the surgery. For example, an ablation model with a "Voxel Growing" algorithm (Yang et al., 2009) is applied for a prototyped robot assisted radiofrequency ablation (RFA) surgery. This algorithm automatically adapts the different tumour's irregular 3-D prole to produce the ablation points. In order to further reduce the number of required ablation and minimize burning of healthy tissue, the "grown region" is optimally aligned to best t the geometry of the target region. The input for the ablation model is the surface data of the tumour, which are acquired from CT scan in advance. Surgical path planning is another important consideration for the preoperative planning. This method involves the construction of spherical elements dened by inscribed cubes. Each sphere represents the ablation area, and its inscribed cube represents a voxel element. Voxel elements propagate layer by layer along dened directions according to its surface edges of the present layer, covers the entire tumour area coinciding with its contour. As voxel elements grow, the coordinates of centres of every voxel elements are produced and being assigned as the ablation coordinates. After the therapeutic model and the surgical plan are prepared, the model is projected onto the patient in the spatial AR environment.

#### **5.2 Intraoperative tracking and hand gesture recognition**

Intraoperative tracking is to establish the real-time spatial awareness of medical instrument navigation and hand motion necessary for visual guidance and intraoperative control during the surgery. Intraoperative tracking of the operating trajectory of the medical instrument provides surgeons real-time feedback of the operation procedure. This real-time feedback information is usually analysed and dealt with in consideration of the intraoperative process as well as the previous medical preplanning. For example in RFA surgery, when the RFA needle is inserted into the patient body, the distal end of needle is tracked to detect the needle's trajectory. The AR environment is updated in real–time producing a virtual needle part, which is projected on the patient surface, following the needle's treatment plan. The surgeon can visualize the operation within the patient's body and surgical processes.

Numerous tracking methods have been developed for surgical environment including vision based tracking and magnetics based tracking. (Dutkiewicz et al., 2004; Felfoul et al., 2008). After taking into consideration the consistent surgical luminance, distinct background content and usability, a vision-based approach based on Continuously Adaptive Mean-Shift (CamShift) algorithm (Ling et al., 2010) is adopted for intraoperative tracking. CamShift algorithm is built upon the Mean-Shift algorithm to track objects based on probability distribution (Wen et al., 2010; Mustafa & Kalajdziski, 2007). Back projection is a way to find the extent to which the pixel distribution of a region coincides with a histogram model. It can be used to compute the probability distribution image from the original image with the colour histogram. The pixel values in the input image are scaled using the values in the corresponding bin of the color histogram and stored in the probability distribution image. The function computes the probability of each element value according to the empirical probability distribution represented by the histogram. If C is the color of the pixel and F is the probability that a pixel belongs to the object being tracked, then the probability map gives P (C|F). The probability distribution image is used in the algorithm for tracking.

The procedure of the Mean-Shift tracking algorithm is described as follows::

1. An initial location of a search window is chosen.

200 Augmented Reality – Some Emerging Application Areas

For the therapeutic model design based on the robot assisted surgery, optimization algorithms are usually applied to enhance the efficiency of robotic implementation and to decrease complexity during the surgery. For example, an ablation model with a "Voxel Growing" algorithm (Yang et al., 2009) is applied for a prototyped robot assisted radiofrequency ablation (RFA) surgery. This algorithm automatically adapts the different tumour's irregular 3-D prole to produce the ablation points. In order to further reduce the number of required ablation and minimize burning of healthy tissue, the "grown region" is optimally aligned to best t the geometry of the target region. The input for the ablation model is the surface data of the tumour, which are acquired from CT scan in advance. Surgical path planning is another important consideration for the preoperative planning. This method involves the construction of spherical elements dened by inscribed cubes. Each sphere represents the ablation area, and its inscribed cube represents a voxel element. Voxel elements propagate layer by layer along dened directions according to its surface edges of the present layer, covers the entire tumour area coinciding with its contour. As voxel elements grow, the coordinates of centres of every voxel elements are produced and being assigned as the ablation coordinates. After the therapeutic model and the surgical plan are prepared, the model is projected onto the patient in the spatial AR environment.

Intraoperative tracking is to establish the real-time spatial awareness of medical instrument navigation and hand motion necessary for visual guidance and intraoperative control during the surgery. Intraoperative tracking of the operating trajectory of the medical instrument provides surgeons real-time feedback of the operation procedure. This real-time feedback information is usually analysed and dealt with in consideration of the intraoperative process as well as the previous medical preplanning. For example in RFA surgery, when the RFA needle is inserted into the patient body, the distal end of needle is tracked to detect the needle's trajectory. The AR environment is updated in real–time producing a virtual needle part, which is projected on the patient surface, following the needle's treatment plan. The surgeon can visualize the operation within the patient's body

Numerous tracking methods have been developed for surgical environment including vision based tracking and magnetics based tracking. (Dutkiewicz et al., 2004; Felfoul et al., 2008). After taking into consideration the consistent surgical luminance, distinct background content and usability, a vision-based approach based on Continuously Adaptive Mean-Shift (CamShift) algorithm (Ling et al., 2010) is adopted for intraoperative tracking. CamShift algorithm is built upon the Mean-Shift algorithm to track objects based on probability distribution (Wen et al., 2010; Mustafa & Kalajdziski, 2007). Back projection is a way to find the extent to which the pixel distribution of a region coincides with a histogram model. It can be used to compute the probability distribution image from the original image with the colour histogram. The pixel values in the input image are scaled using the values in the corresponding bin of the color histogram and stored in the probability distribution image. The function computes the probability of each element value according to the empirical probability distribution represented by the histogram. If C is the color of the pixel and F is the probability that a pixel belongs to the object being tracked, then the probability map gives P (C|F). The probability distribution image is used in the algorithm for tracking.

**5.2 Intraoperative tracking and hand gesture recognition** 

and surgical processes.

2. Compute the mean location or centroid in the window based on the probability distribution image. The centroid (xc, yc) is calculated as follows:

Firstly, the zeroth and first moments of the image are calculated as:

$$M\_{20} = \sum\_{\mathbf{x}} \sum\_{\mathbf{y}} \mathbf{x}^2 I(\mathbf{x}, \mathbf{y}), \tag{16}$$

$$M\_{02} = \sum\_{x} \sum\_{y} y^2 I(x, y)\_{\prime} \tag{17}$$

$$M\_{11} = \sum\_{\mathbf{x}} \sum\_{\mathbf{y}} \mathbf{x} y I(\mathbf{x}, \mathbf{y})\_{\mathbf{y}} \tag{18}$$

where *I(x, y)* is the probability value at coordinate (x, y) in the image, ranging over the entire search window. The coordinates of the centroid are then computed from the moments as Eq. (19) and Eq. (20).

$$\text{tr}\_c = M\_{10} \;/\; M\_{00\prime} \tag{19}$$

$$y\_c = M\_{01} \; / \; M\_{00} \; \tag{20}$$


The CamShift algorithm also has the ability to update the size of the search window to fit the object being tracked as the frames change (Ling et al., 2010).

Fig. 6. RFA needle is tracked based on an AR environment.

For intraoperative interaction, hand gesture is adopted to control display of the AR model and robotic device. Surgeon's hand gestures are detected by a stereovision system over or beside the AR environment. This interface has advantage on sterilization which is an important requirement in operating room. The vision based hand gesture can be integrated into the AR system using projector-camera system or conventional tracking device. The AR model projected on the patient surface can be scaled and rotated using hand gestures. Additionally, hand gestures can also be used to guide robotic execution.

In order to enhance the speed and accuracy of hand gesture recognition, an innovative 3D model based method is introduced. This method is based on the analysis of model features

Intraoperative Visual Guidance and Control Interface for Augmented Reality Robotic Surgery 203

provides the necessary augmentation in dexterity to provide precision and accuracy even in space constraint situation (Hyosig & Wen, 2001; Kuo & Dai, 2009; Stoianovici et al., 1998; Taylor & Stoianovici, 2003; Taylor, 2008). The Transcutaneous Robot-assisted Ablationdevice Insertion Navigation System (TRAINS) is developed for ablation treatment in liver tumor (Chang et al., 2011; Yang et al., 2010). Technical details and demonstrate the

The integration of the robotic modules requires establishment of the spatial relationship between the multi-body system and surgical field. It begins with the construction of a kinematic model for the manipulator and frame assignments to the various entities for the AR applications. Frame assignment of the multi-body robotic mechanism can be established based on the Denavit-Hartenberg (D-H) convention described by (Fu, 1987). Fig. 8 (left) illustrates the D-H frame assignments for TRAINS prototype. The joint variables and link

**Z5** 

*l*5

Fig. 8. Denavit-Hartenberg frame assignments for TRAINS prototype (left); Robotic needle

**X7, X8** 

*0T1* 0 *q1* 0 0 *1T2 q2* 0 *l1* 0 *2T3 q3* 0 *l2* 90o *3T4* 90o *q4* 0 90o *4T5* 0 *q5* 0 0 *5T6 q6 l4* -90o -90o *6T7 q7* 0 0 90o *7T8* 0 *q8+l5* 0 0

Table 1. Denavit-Hartenberg table for manipulator (*qi* stands for ith joint variable).

*o c*

function of joint variables from their respective axis of control.

**X3, Z4** 

*l*3

*l*<sup>2</sup> **Z7, Z8**  *l*4

**Z6** 

The kinematic model can subsequently be expressed in homogenous transformation matrix as shown in Eq. (21). which represents the orientation and position of the end effector as a

> 0 0 8 8 0 1

*R P T* 

, (21)

*θ D a α*

implementation of robotic modules for AR robotic system are presented as follows.

**X6** 

parameters tabulated in Table 1.

**X2** 

**Z3** 

**X4, X5**

**Z0 , Z1 Z2** 

**X0, X1** 

*l*1

insertion for ablation treatment (right).

produced by a space palm plan and 3D convex hull points. Users are allowed to pose the defined hand gestures without wearing markers or data glove. Due to human hand's flexibility, stereovision system is used to resolve ambiguity on recognizing the hand features.

As shown in Fig. 7, the 3D model is constructed including the space palm plan which is relatively stable, and a brace model of five fingers with the key joints and finger tips. Before starting the hand gestures recognition, the 3D hand model is initialized by characterizing the model with the specific hand parameters such as palm size, fingers layout of different users. This initialization can be achieved by directly measuring the key parameters of the user's hand under the vision system with a defined pose. These key parameters include hand length and width, the ratio between the palm and the fingers, and the palm's location relative to the stereovision system. Palm plan can be acquired by regression of the hand palm coordinates shown as Fig.7 (b). There are several possible ways of tracking the user's hands (Mahmoudi & Parviz, 2006). In consideration of the background of hand operation based on this AR interface is covered by the surgical cloth which is green in color. CamShift based method used for medical instrument tracking can also be integrated with hand tracking. When the hand tracking is implemented, the palm plan is invoked to change its pose in the space coordinate. Following with the hand tracking, the 3D hand model is attached to the user's hand wherever the hand moves. In this case the parameters of the 3D hand model are analysed in real-time and the hand gesture recognition is achieved in real-time.

Fig. 7. 3D model based hand gesture recognition.

#### **5.3 Robotic implementation**

While the introduction of AR in computer aided surgery enhances visualization of preoperative surgical plan, it alone does not necessarily translate to precision and accuracy in intraoperative execution. In the case of minimally invasive surgery, the surgeon is not only deprived of direct visual information but also subjected to dexterous constraints in the manoeuvring of laparoscopic tools or percutaneous ablation devices. Performing such operation manually even with AR guidance can be non-trivial and often expose to uncertainties and inconsistent outcomes. For instance, the challenges in large tumor treatment with percutaneous ablation technique have been discussed (Abolhassani et al.,2007; Choi, et al., 2000; Goldberg, et al., 1998). This lack of intraoperative consistency poses a serious bottleneck for the effectiveness surgical outcome. Robotic assistance 202 Augmented Reality – Some Emerging Application Areas

produced by a space palm plan and 3D convex hull points. Users are allowed to pose the defined hand gestures without wearing markers or data glove. Due to human hand's flexibility, stereovision system is used to resolve ambiguity on recognizing the hand features. As shown in Fig. 7, the 3D model is constructed including the space palm plan which is relatively stable, and a brace model of five fingers with the key joints and finger tips. Before starting the hand gestures recognition, the 3D hand model is initialized by characterizing the model with the specific hand parameters such as palm size, fingers layout of different users. This initialization can be achieved by directly measuring the key parameters of the user's hand under the vision system with a defined pose. These key parameters include hand length and width, the ratio between the palm and the fingers, and the palm's location relative to the stereovision system. Palm plan can be acquired by regression of the hand palm coordinates shown as Fig.7 (b). There are several possible ways of tracking the user's hands (Mahmoudi & Parviz, 2006). In consideration of the background of hand operation based on this AR interface is covered by the surgical cloth which is green in color. CamShift based method used for medical instrument tracking can also be integrated with hand tracking. When the hand tracking is implemented, the palm plan is invoked to change its pose in the space coordinate. Following with the hand tracking, the 3D hand model is attached to the user's hand wherever the hand moves. In this case the parameters of the 3D hand model are analysed in real-time

While the introduction of AR in computer aided surgery enhances visualization of preoperative surgical plan, it alone does not necessarily translate to precision and accuracy in intraoperative execution. In the case of minimally invasive surgery, the surgeon is not only deprived of direct visual information but also subjected to dexterous constraints in the manoeuvring of laparoscopic tools or percutaneous ablation devices. Performing such operation manually even with AR guidance can be non-trivial and often expose to uncertainties and inconsistent outcomes. For instance, the challenges in large tumor treatment with percutaneous ablation technique have been discussed (Abolhassani et al.,2007; Choi, et al., 2000; Goldberg, et al., 1998). This lack of intraoperative consistency poses a serious bottleneck for the effectiveness surgical outcome. Robotic assistance

**(a)**

**(b)**

**(c)**

and the hand gesture recognition is achieved in real-time.

Fig. 7. 3D model based hand gesture recognition.

**5.3 Robotic implementation** 

provides the necessary augmentation in dexterity to provide precision and accuracy even in space constraint situation (Hyosig & Wen, 2001; Kuo & Dai, 2009; Stoianovici et al., 1998; Taylor & Stoianovici, 2003; Taylor, 2008). The Transcutaneous Robot-assisted Ablationdevice Insertion Navigation System (TRAINS) is developed for ablation treatment in liver tumor (Chang et al., 2011; Yang et al., 2010). Technical details and demonstrate the implementation of robotic modules for AR robotic system are presented as follows.

The integration of the robotic modules requires establishment of the spatial relationship between the multi-body system and surgical field. It begins with the construction of a kinematic model for the manipulator and frame assignments to the various entities for the AR applications. Frame assignment of the multi-body robotic mechanism can be established based on the Denavit-Hartenberg (D-H) convention described by (Fu, 1987). Fig. 8 (left) illustrates the D-H frame assignments for TRAINS prototype. The joint variables and link parameters tabulated in Table 1.

Fig. 8. Denavit-Hartenberg frame assignments for TRAINS prototype (left); Robotic needle insertion for ablation treatment (right).


Table 1. Denavit-Hartenberg table for manipulator (*qi* stands for ith joint variable).

The kinematic model can subsequently be expressed in homogenous transformation matrix as shown in Eq. (21). which represents the orientation and position of the end effector as a function of joint variables from their respective axis of control.

$$\begin{aligned} \, \, ^0T\_c = \begin{bmatrix} \, ^0R\_8 & \, ^0P\_8\\ \, & \bar{0} \\ \bar{0} & \, \bar{1} \end{bmatrix} \, ^\prime \end{aligned} \tag{21}$$

Intraoperative Visual Guidance and Control Interface for Augmented Reality Robotic Surgery 205

vision-based tracking technology, other types of intraoperative feedback such as real-time CT scanning etc. are needed to generate more reliable surgical information. Additionally, in order to realize a dynamic observational perspective from any location around the AR environment,

The intraoperative visual guidance and control AR interface is an attempt to integrate the preoperative diagnostic information with real-time augmented visual information produced by the patient's reactions, surgeon's decisions, and robotic implementation. This interface provides surgeons with a mean to extend a person's visual limitations during the MIS and to engage interactively during the surgical robot assisted surgery. Augmented reality technology has the potential as a therapeutic intervention which combines the virtual augmented physical model with locally detected real-time medical information including geometric variance and respiration rate. The interface is coupled with the augmented physical model, surgeon's operation, and robotic implementation through vision based

This intraoperative AR interface will benefit from a stereoscopic vision system with higher accuracy and higher frame rate which plays an important factor in vision based tracking and recognition. While the current tracking algorithm worked satisfactorily at the workstation used for testing, users still had problems with robustness when operating in an environment with a colourful background. In addition, this interface could be expanded to assimilate with more medical devices and equipments such as ultrasound that would integrate more real-time augmented medical information. In this scenario, the interface is a powerful integrated medical AR platform centralized in the network of all the other medical devices

Abolhassani, N.; Patel, R. & Moallem, M. (2007). Needle insertion into soft tissue: A survey. *Medical Engineering & Physics*, Vol.29, No.4, pp. 413-431, ISSN 1350-4533. Akay, M. & Marsh, A. (2001). Neuro orthopedic rehabilitation and disability solutions using

Anderson, J.H.; Chui, C.K.; Cai, Y.; Wang, Y.P.; Li, Z.R.; Ma, X.; Nowinski, W.L.; Solaiyappan,

Burdea, G.; Popescu, V.; Hentz, V. & Colbert, K. (2000). Virtual reality-based orthopedic

Cai, Y.; Chui, C.K.; Ye, X. & Anderson, J. (2003). VR simulated training for less invasive vascular intervention. *Computers & Graphics*, Vol. 27, No.2, pp. 215-221. Cakmakci, O. & Rolland, J.(2006). Head-worn displays: a review. *Display Technology*, Vol. 2,

*Treatment*, Vol. 168, No. 2, pp.1–18, ISSN 1530-437X.

ISBN 9781568812304, Wellesley, MA : A K Peters, 2004.

virtual reality technology information technologies in medicine. *Rehabilitation and* 

M.; Murphy, K. & Gailloud, P. (2002). Virtual reality training in interventional radiology: The Johns Hopkins and Kent Ridge Digital Laboratory Experience. *Seminars in Interventional Radiology*, Vol. 19, No. 2, pp. 179-185, ISSN 0739-9529. Bimber, O. & Raskar, R. (2004). Spatial Augmented Reality: merging real and virtual worlds.

telerehabilitation. *Rehabilitation Engineering, IEEE Transactions*, Vol.8, No.3, pp.430–

more projectors and cameras are required.

**7. Conclusion and future work** 

tracking and hand gesture recognition.

and equipment in the operating room.

432, ISSN 1063-6528.

No.3, pp.199-216, ISSN 1551-319x.

**8. References** 

where

$$\begin{aligned} \;^o R\_8 = \begin{bmatrix} s\_{23}s\_6c\_7 - c\_{23}s\_7 & s\_{23}c\_6 & s\_{23}s\_6s\_7 + c\_{23}c\_7\\ -c\_{23}s\_6c\_7 - s\_{23}s\_7 & c\_{23}c\_6 & -c\_{23}s\_6s\_7 + s\_{23}c\_7\\ c\_6c\_7 & -s\_6 & c\_6s\_7 \end{bmatrix} \end{aligned} \tag{22}$$

$$\begin{aligned} \;^o P\_8 = \begin{bmatrix} s\_{23} s\_6 s\_7 \left( q\_8 + l\_5 \right) + c\_{23} \left[ c\_7 \left( q\_8 + l\_5 \right) + l\_4 \right] + q\_5 c\_{23} + q\_4 s\_{23} + l\_1 c\_2 + l\_2 c\_{23} \\ -c\_{23} s\_6 s\_7 \left( q\_8 + l\_5 \right) + s\_{23} \left[ c\_7 \left( q\_8 + l\_5 \right) + l\_4 \right] + q\_5 c\_{23} - q\_4 s\_{23} + l\_1 s\_2 + l\_2 s\_{23} \\ c\_6 c\_7 \left( q\_8 + l\_5 \right) l\_3 q\_1 \end{bmatrix} \end{aligned} \tag{23}$$

where sj=sin (qi), cij=cos(qi), sij=sin (qi+qj) and cij=cos(qi+qj).

Direct vision is usually absence in minimally invasive surgery or percutaneous interventional procedure. Hence registration between diagnostic medical images, surgical field and its entities has to be done. Preoperative image coordinates can be readily transformed to spatial coordinates based on the system configuration of the imaging module. Intraoperative spatial information however, requires a tracking and navigation system. TRAINS uses a vision-based system for tracking and navigation purpose. Light emitting fiducial markers are used for the registration of entities in the real world to the interventional plan. Fig. 8 (right) depicts the robotic setup in a porcine surgery.

#### **6. Discussion**

The use of the intraoperative visual guidance and AR control interface in surgical procedures provide surgeons with a very exciting viewable integrated information platform. With this platform, the surgeon can concurrently conduct an intraoperative diagnosis, surgical operation and collection of patient's medical information on a single interactive augmented interface. This augmented reality based interactive interface allows surgeons to not only experience a new type of technology that extends their visibility without looking at any other screen or wearing any head-mounted device during the surgery, but also perform the operation with the combination of preoperative and intraoperative medical information. With regards to the surgical robot, not only does it contribute to higher accuracy of the surgical operation and optimizes the procedural flow, it is also easy for this intraoperative AR interface to incorporate the surgical robot assistance into the surgery, which includes the display of the robotic operation inside the patient and interactive control by the surgeons.

Although interactive spatial AR interfaces have been used for some applications such as games and advertisements, the intraoperative visual guidance and interactive AR interface for surgery is still a challenging problem. The reasons for this are as followed. First of all, an accurate display of the human organs and tissues is essential in surgery. The visual corrections, including geometric corrections of a projected virtual organ model, are important steps in constructing the spatial augmented reality environment, especially when projecting on an arbitrary surface. Other than the influence of irregular geometric surfaces, the projection is also affected by dynamic motions in the AR environment such as the patient's aspiration, unexpected surface motions or occlusions etc. Secondly, registration of an AR model with the real organ is especially difficult because of the non-rigid soft tissue. Thirdly, in usability tests with the medical tool, the ablation needle is as assumed to be a rigid needle without bending during the insertion. However, this assumption may not always be true. Therefore, other than vision-based tracking technology, other types of intraoperative feedback such as real-time CT scanning etc. are needed to generate more reliable surgical information. Additionally, in order to realize a dynamic observational perspective from any location around the AR environment, more projectors and cameras are required.

#### **7. Conclusion and future work**

204 Augmented Reality – Some Emerging Application Areas

23 6 7 23 7 23 6 23 6 7 23 7 8 23 6 7 23 7 23 6 23 6 7 23 7 6 7 6 6 7

Direct vision is usually absence in minimally invasive surgery or percutaneous interventional procedure. Hence registration between diagnostic medical images, surgical field and its entities has to be done. Preoperative image coordinates can be readily transformed to spatial coordinates based on the system configuration of the imaging module. Intraoperative spatial information however, requires a tracking and navigation system. TRAINS uses a vision-based system for tracking and navigation purpose. Light emitting fiducial markers are used for the registration of entities in the real world to the

The use of the intraoperative visual guidance and AR control interface in surgical procedures provide surgeons with a very exciting viewable integrated information platform. With this platform, the surgeon can concurrently conduct an intraoperative diagnosis, surgical operation and collection of patient's medical information on a single interactive augmented interface. This augmented reality based interactive interface allows surgeons to not only experience a new type of technology that extends their visibility without looking at any other screen or wearing any head-mounted device during the surgery, but also perform the operation with the combination of preoperative and intraoperative medical information. With regards to the surgical robot, not only does it contribute to higher accuracy of the surgical operation and optimizes the procedural flow, it is also easy for this intraoperative AR interface to incorporate the surgical robot assistance into the surgery, which includes the display of the robotic operation inside the patient and interactive control by the surgeons. Although interactive spatial AR interfaces have been used for some applications such as games and advertisements, the intraoperative visual guidance and interactive AR interface for surgery is still a challenging problem. The reasons for this are as followed. First of all, an accurate display of the human organs and tissues is essential in surgery. The visual corrections, including geometric corrections of a projected virtual organ model, are important steps in constructing the spatial augmented reality environment, especially when projecting on an arbitrary surface. Other than the influence of irregular geometric surfaces, the projection is also affected by dynamic motions in the AR environment such as the patient's aspiration, unexpected surface motions or occlusions etc. Secondly, registration of an AR model with the real organ is especially difficult because of the non-rigid soft tissue. Thirdly, in usability tests with the medical tool, the ablation needle is as assumed to be a rigid needle without bending during the insertion. However, this assumption may not always be true. Therefore, other than

23 6 7 8 5 23 7 8 5 4 5 23 4 23 1 2 2 23 8 23 6 7 8 5 23 7 8 5 4 5 23 4 23 1 2 2 23 67 8 5 31

*s ss q l c c q l l qc qs lc lc P c ss q l s c q l l qc qs ls ls cc q l lq* 

[ ]

, (22)

, (23)

*s sc c s s c s ss c c R c sc s s c c c ss s c cc s cs* 

 

interventional plan. Fig. 8 (right) depicts the robotic setup in a porcine surgery.

[ ] *<sup>o</sup>*

where sj=sin (qi), cij=cos(qi), sij=sin (qi+qj) and cij=cos(qi+qj).

where

**6. Discussion** 

*o*

The intraoperative visual guidance and control AR interface is an attempt to integrate the preoperative diagnostic information with real-time augmented visual information produced by the patient's reactions, surgeon's decisions, and robotic implementation. This interface provides surgeons with a mean to extend a person's visual limitations during the MIS and to engage interactively during the surgical robot assisted surgery. Augmented reality technology has the potential as a therapeutic intervention which combines the virtual augmented physical model with locally detected real-time medical information including geometric variance and respiration rate. The interface is coupled with the augmented physical model, surgeon's operation, and robotic implementation through vision based tracking and hand gesture recognition.

This intraoperative AR interface will benefit from a stereoscopic vision system with higher accuracy and higher frame rate which plays an important factor in vision based tracking and recognition. While the current tracking algorithm worked satisfactorily at the workstation used for testing, users still had problems with robustness when operating in an environment with a colourful background. In addition, this interface could be expanded to assimilate with more medical devices and equipments such as ultrasound that would integrate more real-time augmented medical information. In this scenario, the interface is a powerful integrated medical AR platform centralized in the network of all the other medical devices and equipment in the operating room.

#### **8. References**


Intraoperative Visual Guidance and Control Interface for Augmented Reality Robotic Surgery 207

*Radiology and Surgery*, Vol. 1281, pp. 537-542. Berlin, Germany, Jun. 2005. Kuo, C.H. & Dai, J.S. (2009). Robotics for minimally invasive surgery: a historical review

Kuo, R.L.; Delvecchio, F.C.& Preminger, G.M. (2001). Virtual Reality: Current Urologic

Lee, S.L.; Mirna, L.; Vitiello, V.; Giannarou, S.; Kwok, K.W.; Visentini-Scarzanella, M. &

Liao, H; Inomata, T; Sakuma, I & Dohi, T (2010). Three-dimensional augmented reality for mri-

V1-159 - V1-164, ISBN 978-1-4244-6539-2, Chengdu, China, Aug. 2010. Mahmoudi, F.& Parviz, M. (2006). Visual hand tracking algorithms. *Proc. of GMAI 2006, IEEE Computer Society press*, ISBN 0-7695-2604-7, pp.228–232, London, UK, July 2006. Massoptier, L. & Casciaro, S. (2008). A new fully automatic and robust algorithm for fast

*and Mechanisms*, ISBN 978-1-4020-9484-2, pp.337-354.

117-122, ISSN 0892-7790.

Vol. 34, No. 1, pp. 33-45, ISSN .

No.8, pp.1658-1665, ISSN 1432-1084.

*Technology*, Vol. 2351, pp.282–292.

1-4244-0813-9, Warsaw, Poland, Sept. 2007.

Vol. 8 , No. 4, pp. 371 – 381, ISSN 1530-437X.

tomography: Application to 20 clinical cases. *CARS 2005: Computer Assisted* 

from the perspective of kinematics. *International Symposium on History of Machines* 

Applications and Future Developments. *Journal of Endourology*, Vol. 15, No. 1, pp.

Yang, G.Z. (2010). From medical images to minimally invasive intervention: Computer assistance for robotic surger. *Computerized Medical Imaging and Graphics*,

guided surgery using integral videography auto stereoscopic-image overlay. *IEEE Transactions on Biomedical Engineering*, Vol.57, No.6, pp.1476-1486, ISSN 15582531. Ling, Y.; Zhang, J. & Xing, J. (2010). Video object tracking based on position prediction guide

CAMSHIFT. P*roceeding of Advanced Computer Theory and Engineering (ICACTE)*, pp.

segmentation of liver tissue and tumors from CT scans. *European Radiology*, Vol.18,

displays on the reality-virtuality continuum in proc. *Telemanipulator Telepresence* 

matching. *The International Conference on Computer as a Tool*, pp. 361-368, ISBN 978-

workflow-driven augmented reality for computer-aided medical procedure.

Radiometric Compensation for Screen Imperfections. *Proceeding of IEEE Int'l* 

Edvard N.; Eigil S.; Patricia S.G.; Francisco M.S.M.; Dieter S.; Mauro S.; Thomas S.; Jos V.S. & Enrique J.G. (2010). Augmented reality for minimally invasive surgery: overview and some recent advances. *Augmented Reality*, pp.1-26, In: InTech, Available from: http://www.intechopen.com/articles/show/title/augmentedreality-for-minimally-invasive-surgery-overview-and-some-recent-advances. Pearson, A.M.; Gallagher, A. G.; Rosser, J. C.; & Satava, R. M. (2002). Evaluation of

structured and quantitative training methods for teaching intracorporeal knot

the-art in force and tactile sensing for minimally invasive surgery. *Sensors Journal*,

Milgram, P.; Takemura, H.; Utsumi, A.; & Kishino, F. (1994). Augmentedreality:a class of

Mustafa, B. & Kalajdziski, S. (2007). Spherical mapping based descriptors for 3D object

Navab, N.; Traub, J.; Sielhorst, T.; Feuerstein, M. & Bichlmeier, C. (2007). Action- and

*Computer Graphics and Applications*, Vol.27, No.5, pp. 10–14, ISSN 0272-1716. Nayar, S.K.; Peri, H.; Grossberg, M.D. & Belhumeur, P.N. (2003). A Projection System with

*Workshop Projector-Camera Systems*, pp.97-108 , ISBN 1077-2626, Nov.2007. Pablo L.; Wajid A.; Alicia C.; Jordi C.; Jerome D.; Ole J.E.; Adinda F.; Hugo F.; Denis K.;

tying. *Surgical Endoscopy*, Vol. 16, No. 1, pp.130–137, ISSN 1432-2218. Puangmali, P.; Althoefer, K.; Seneviratne, L.D.; Murphy, D. & Dasgupta, P. (2008). State-of-


206 Augmented Reality – Some Emerging Application Areas

Chang, S.K.Y.; Hlaing, W.W.; Yang, L., & Chui, C.K. (2011). Current technology in

Choi, D.; Lim, H.K.; Kim, S.H.; Lee, W.J.; Jang, H.J. & Lee, J.Y. (2000). Hepatocellular

Devernay, F.; Mourgues, F.& Coste-Maniere, E. (2001). Towards endoscopic augmented

Dutkiewicz, P.; Kielczewski, M. & Kowalski, M. (2004). Visual tracking of surgical tools for

Fu, K.S. (1987). Robotics: control, sensing, vision, and intelligence, Vol. 280227, McGraw-

Fujii, K.; Grossberg, M.D. & Nayar, S.K. (2005). A projector-camera system with real-time

Goldberg, S.N.; Gazelle, G.S.; Solbiati, L.; Livraghi, T.; Tanabe, K.K. & Hahn, P.F.(1998).

Grimson, W.E.L.; L. T.; Wells, W.M.; Ettinger, G.J.; White, S.J. & Kikinis, R. (1996). An

Guthart, G.S. & Salisbury, J.K.Jr. (2000). The intuitive™ telesurgery system: Overview and

Heng, P.A.; Cheng, C.Y.; Wong, T.T; Xu, Y.; Chui, Y.P.; Chan, K.M. & Tso, S.K. (2004). A

*in Biomedicine, IEEE Transactions*, Vol. 8, No.2, pp. 217-227, ISSN 1089-7771. Hoppe, H.; Eggers, G.; Heurich, T.; Raczkowsky, J.; Marmulla, R.; Worn, H.; Hassfeld, S. &

Hyosig, K. & Wen, J.T. (2001). Robotic assistants aid surgeons during minimally invasive

Konishia K.; Hashizumeb M.; Nakamotod M.; Kakejib Y.; Yoshinoc I.; Taketomic A.; Satod

response. *Preliminary Results1. Radiology*, Vol. 217, No.2, pp.558-563. Chui, C.K.; Ong, JSK.; Lian, Z.Y.; Yan, C.H.; Wang, Z.; Ong, SH.; Teo, JCM.; Zhang, J.; Wang,

20, ISBN 0-7695-1113-9, Shatin, N.T., Hong Kong, Jun. 2001.

No.5, pp.231-236, May, 2011.

pp. 438-451, ISSN 1552-826X.

Hill, New York, 1987.

CA , USA, Apr.2000.

104, ISSN 0739-5175.

Vol.27 , No.1, pp. 28 – 35, ISSN 0278-0062.

*Recognition,* Vol. 2, pp.1180, ISSN 1063-6919.

170, No.4, pp.1023-1028 , ISSN 1023-1028.

Vol. 15, No. 2, pp.129-140. , ISSN 0278-0062.

pp.771, ISBN 0-444-51731-6, London, Jun. 2003.

navigation and robotics for liver tumours ablation. *Ann Acad Med Singapore*, Vol.40,

carcinoma treated with percutaneous radio-frequency ablation: usefulness of power doppler us with a microbubble contrast agent in evaluating therapeutic

SC.; Wong, HK.; Teo, C.L. & Teoh, S. H. (2006) Haptics in computer mediated simulation: training in vertebroplasty surgery. *Simulation & Gaming*, Vol. 47, No. 4,

reality for robotically assisted minimally invasive cardiac surgery. *Proceedings of the International Workshop on Medical Imaging and Augmented Reality (MIAR '01)*, pp.16-

laparoscopic surgery. *Proceedings of the Fourth International Workshop on Robot Motion and Control,* pp.23-28 , ISBN 83-714*3-272-0,* Zielona Gora Univ., Poland, June, 2004. Felfoul, O.; Mathieu, J.-B.; Beaudoin, G.; Martel, S. (2008). In vivo mr-tracking based on

magnetic signature selective excitation. *IEEE Transactions on Medical Imaging,*

photometric adaptation for dynamic environments. *Computer Vision and Pattern* 

Ablation of liver tumors using percutaneous RF therapy. *AJR Am J Roentgenol*, Vol.

automatic registration method for frameless stereotaxy, image guided surgery, and enhance reality visualization. *IEEE Transactions on Medical Imaging*, (April 1996),

application, *Robotics and Automation*, pp.618-621, ISBN 0-7803-5886-4, San Francisco,

virtual-reality training system for knee arthroscopic surgery. *Information Technology* 

Moctezuma, J.L. (2003). Projector-based visualization for intraoperative navigation: first clinical results. *Proceedings of the 17th International Congress and Exhibition*,

procedures. *Engineering in Medicine and Biology Magazine*, IEEE, Vol.20, No.1, pp.94-

Y.; Tamurad S. & Maehara Y. (2005). Augmented reality navigation system for endoscopic surgery based on three-dimensional ultrasound and computed tomography: Application to 20 clinical cases. *CARS 2005: Computer Assisted Radiology and Surgery*, Vol. 1281, pp. 537-542. Berlin, Germany, Jun. 2005.


**Part 3** 

**Novel AR Applications** 

**in Daily Living and Learning** 

