**5. Image registration**

This is the process where a medical imaging is aligned with the patient's anatomy to form a visually projected overlay. This can be done through various methods, but the fundamentals rely upon multiple data points being processed to align medical imaging with the best corresponding surface-landmarks on an organ/patient's anatomy.

Image registration can be used in pre-operative planning stage and in the intra-operative stage. The planning stage involves using the combined imaging overlay (of CT scans and MRIs) onto the kidney to identify key structures of importance—hilar vasculature, the spatial attributes of the kidney and their relationship with the renal collecting system. This helps build a roadmap of what the surgery will involve and although it does not require accuracy to the millimetre (as is involved in the intra-operative phase) it allows a pretty good estimation to the planned steps in surgery.

The intra-operative stage requires higher precision image registration and more importantly, dynamic correspondence with the moving organ. This is to allow tumour resection margins that can be accurate to the millimetre.

The surface anatomy can then be correlated with 3D reconstruction imaging of the patient

Augmented Reality in Kidney Cancer http://dx.doi.org/10.5772/intechopen.81890 97

This method is a form of internal tracking relying on a good tracking instrument and accurate computer algorithms to provide the correct position of the organ being operated on. Errors in the estimated spatial position of the instrument tip can create unsafe image registration that

This registration method is more accurate than manual registration as it provides automation and reduces the surgical workload. However, there are still areas for improvement such as better tracking methods. External tracking, where the organ is tracked from the outside is another method. Examples include optical, magnetic and laser but there are inherent drawbacks such as optical tracking requires a direct line of sight and magnetic tracking requiring

A recent advancement in surface-based registration was developed by Edgcumbe et al. where a miniature laser projector called 'Pico lantern' can be dropped into the patient's abdomen. It can then be picked up by laparoscopic instruments and perform surface recognition on the abdominal organ. Using surface-based recognition, it can then project the pre-operative image on the surface of the organ. The projector is visually tracked in the laparoscopic field of view and has been tested in porcine kidneys and in detecting pulsatile motion in carotid arteries [19].

and projected onto the surgeon's operating view.

**Figure 2.** Different methods of image registration [18].

physical placement of a tracer in the organ.

would be inaccurate for tumour margins of partial nephrectomies.

Hughes-Hallet [18] classify image registration used by AR developers into 3 main subtypes: manual registration, surface-based registration and 3D registration (**Figure 2**).

*Manual registration* is the simplest method where the surgeon uses their anatomical knowledge to align a projected imaging onto the organ. Examples of this method have involved: fusing 3D reconstructions with live operative view, projecting intra-abdominal anatomy onto the skin and colour coded projection to highlight "safe zones" of resection margins. These have offered quite a hands-on approach with relatively little planning compared to other registration methods—allowing a low barrier to entry, good anatomical orientation and relatively good awareness of disease-free parenchyma. Manual registration does not allow accurate estimates of tumour margins and is mostly limited to the planning phase.

*Surface based registration* method involves using a tracked instrument to build a topographical map of the internal organ. This has been most extensively used in robotic partial nephrectomy—an example being the da Vinci robot. The laparoscopic instrument tip of the robot can touch a point on the live kidney and this information along with the joint positions sense of the robotic arms can calculate a position in space to build a surface anatomy of the kidney.

**Figure 2.** Different methods of image registration [18].

renal vasculature and ureters can be reduced. AR can also reduce excision margins—to spare as many well-functioning nephrons and reduce the risk and progression of chronic renal

For an AR system to be ideal, the full length of a surgical procedure need to be "augmented." This requires 3 essential features (as adapted by a recent review by Hughes-Hallet et al. [18]): image registration, organ tracking and adapting to intra-operative tissue deformation. In the following chapter, I will describe, in detail, these aspects of AR specific to nephrectomies.

This is the process where a medical imaging is aligned with the patient's anatomy to form a visually projected overlay. This can be done through various methods, but the fundamentals rely upon multiple data points being processed to align medical imaging with the best cor-

Image registration can be used in pre-operative planning stage and in the intra-operative stage. The planning stage involves using the combined imaging overlay (of CT scans and MRIs) onto the kidney to identify key structures of importance—hilar vasculature, the spatial attributes of the kidney and their relationship with the renal collecting system. This helps build a roadmap of what the surgery will involve and although it does not require accuracy to the millimetre (as is involved in the intra-operative phase) it allows a pretty good estimation

The intra-operative stage requires higher precision image registration and more importantly, dynamic correspondence with the moving organ. This is to allow tumour resection margins

Hughes-Hallet [18] classify image registration used by AR developers into 3 main subtypes:

*Manual registration* is the simplest method where the surgeon uses their anatomical knowledge to align a projected imaging onto the organ. Examples of this method have involved: fusing 3D reconstructions with live operative view, projecting intra-abdominal anatomy onto the skin and colour coded projection to highlight "safe zones" of resection margins. These have offered quite a hands-on approach with relatively little planning compared to other registration methods—allowing a low barrier to entry, good anatomical orientation and relatively good awareness of disease-free parenchyma. Manual registration does not allow accurate

*Surface based registration* method involves using a tracked instrument to build a topographical map of the internal organ. This has been most extensively used in robotic partial nephrectomy—an example being the da Vinci robot. The laparoscopic instrument tip of the robot can touch a point on the live kidney and this information along with the joint positions sense of the robotic arms can calculate a position in space to build a surface anatomy of the kidney.

manual registration, surface-based registration and 3D registration (**Figure 2**).

estimates of tumour margins and is mostly limited to the planning phase.

responding surface-landmarks on an organ/patient's anatomy.

insufficiency [17] (**Figure 1**).

96 Evolving Trends in Kidney Cancer

**5. Image registration**

to the planned steps in surgery.

that can be accurate to the millimetre.

The surface anatomy can then be correlated with 3D reconstruction imaging of the patient and projected onto the surgeon's operating view.

This method is a form of internal tracking relying on a good tracking instrument and accurate computer algorithms to provide the correct position of the organ being operated on. Errors in the estimated spatial position of the instrument tip can create unsafe image registration that would be inaccurate for tumour margins of partial nephrectomies.

This registration method is more accurate than manual registration as it provides automation and reduces the surgical workload. However, there are still areas for improvement such as better tracking methods. External tracking, where the organ is tracked from the outside is another method. Examples include optical, magnetic and laser but there are inherent drawbacks such as optical tracking requires a direct line of sight and magnetic tracking requiring physical placement of a tracer in the organ.

A recent advancement in surface-based registration was developed by Edgcumbe et al. where a miniature laser projector called 'Pico lantern' can be dropped into the patient's abdomen. It can then be picked up by laparoscopic instruments and perform surface recognition on the abdominal organ. Using surface-based recognition, it can then project the pre-operative image on the surface of the organ. The projector is visually tracked in the laparoscopic field of view and has been tested in porcine kidneys and in detecting pulsatile motion in carotid arteries [19].

*3D-CT stereoscopic image registration* uses 2 cameras that focus on the same object in space and a combination of the perspectives from 2 separate viewpoints allows a spatial reference for the object. This system has been used for kidneys by isolating a point on the surface of the organ, which can then be aligned to the corresponding point on a 3D reconstruction image. Using this as the centre of rotation, the surgeon can manipulate the operative view for a better understanding of field of operation. This system has shown average target registration error of less than 1 mm and mean registration duration of 48.1 s i.e. more accurate resection margins, faster registration and reduced theatre time.

[24]. More recently, Edgcumbe et al. have developed a tracking device called the Dynamic Augmented Reality Tracker (DART). This is a 3D-printed stainless-steel tracker that can be anchored to a fixed position on the kidney relative to the tumour. This, with the help of an ultrasound transducer, can then be used to track the location of surgical instruments relative to the tumour in real time. This system has been named ARUNS (Augmented Reality Ultrasound Navigation System) and was used in the robotic-assisted excision of a phantom

Augmented Reality in Kidney Cancer http://dx.doi.org/10.5772/intechopen.81890 99

Vávra et al., in their recent review, comment that it may be possible to track organ movement without physical markers in the future. Some of these methods explored include algorithms to predict real-time movement of organs, physics-based deformation models, natural points of reference as tracking points and the use of red-green-blue cameras to perform image registration without markers. They also comment that whilst the average marker associated regis-

The kidney is a dynamic organ and renal surgery involves deformation of the anatomy. Every step in the operation changes the initially projected image registration [26] and for an ideal AR system, there need to be real time feedback for tracing this. An answer to this would be the development of computer algorithms to predict changes in anatomy at crucial steps such as clamping of the renal arteries and surgical dissection at tumour margins. Algorithms can also be developed to predict the effect of ongoing influences on the organ position such as

There have been some tissue deformation models considering renal clamping, incision and external pressure loads to the kidney like intra-operative insufflation. Some of these have shown an improvement of 29% in the registration error when compared to a non-deformation model. However, in these models, the kidney is assumed to have a linear elastic behaviour and the models have been based on ex-vivo kidneys. Another method has been developed taking the diaphragm motion and its influence on kidney motion into consideration—this has used preoperative CT scans during inspiration and expiration and computer errors have been

Although there is a scarcity of studies in this area, one study showed that mathematical models were able to predict up to 52% of the operative deformation in porcine kidneys when

Baumhauer et al. [29] have proposed a system to answer tissue deformation by insertion of custom-designed navigation aids into the kidney and using "inside-out tracking." This is where the CT-scan can provide real-time spatial awareness by identifying the navigation aids and projecting imaging onto the laparoscopic view. Tested in a virtual environment, this system showed a visualisation error of 1.36 mm (adequately accurate). This system was replicated in the clinical setting by Teber et al. [30] where 10 patients had retroperitoneal

tration takes 8 min, a recent marker-less system only took about 5 min [20].

kidney tumour [25].

**7. Tissue deformation**

respiratory patterns and peritoneal insufflation.

compared to pre and post-op CT imaging [28].

shown to be less than 2 mm in predicting kidney positions [27].

There is however, a lack of organ tracking in this system, meaning that each movement of the camera needs a new image registration. Furthermore, stereoscopic cameras are usually very expensive and require large handling computer systems, which is another limitation to adoption of this method.
