**11. Future ideas/solutions**

As Hughes-Hallet mention, there is a "*one-size-fits-all*" approach in most AR developments. Single imaging and imaging registration modalities have been used in isolation for many systems. Every surgical case is different however, and a combination of different modalities can provide a more accurate answer. Below are some examples of adjuncts that could aid current AR:


**3.** Depth perception: whilst current image registration involves imaging modalities such as MRI and CTs, the 2D or 3D imaging does not allow a full understanding of intraabdominal environment. Minimally invasive surgery has especially deprived the depth perception aspect of the surgical experience. Vávra et al. [20] suggest depth-sensing cameras could aid AR in the future.

**4.** Hardware capacities: current AR is limited by the *hardware capacities* and thus processing power of computer systems they are run on. Vizua is a company developing "cloudification" and "application roaming" where the AR applications and data can be remotely managed to get the highest computing power and access to large datasets. A platform such as this could be incorporated into AR systems and answer issues of latency in image

**5.** Other issues mentioned include tissue deformation studies only focusing on one source of deformation, 3D imaging requiring better image registration and the issue of simulation

Regardless of what aspect of AR being explored, there is little quantitative data on in-vivo procedures. Only 20 studies were found by a recent review [27] where AR had been used in clinical practice, and only 9 studies had 10 or more patients in the study. There is a crucial need for *clinical validity* to show improved patient outcomes and safety from using AR in renal

As Hughes-Hallet mention, there is a "*one-size-fits-all*" approach in most AR developments. Single imaging and imaging registration modalities have been used in isolation for many systems. Every surgical case is different however, and a combination of different modalities can provide a more accurate answer. Below are some examples of adjuncts that could aid current AR:

**1.** NIRF—near infrared fluorescence is a type of imaging where indocyanine green (ICG) dye is injected into the body and it can be used to illuminate intravascular renal parenchyma. This can allow the surgeon to detect blood vessels under the organ surface and detect tissue abnormalities. Although NIRF has not shown much promise in predicting malignancy in partial nephrectomies, it has shown a reduction in global renal ischaemia. NIRF has been used in robot-assisted surgery to achieve super-selective arterial clamping—avoiding main arterial clamping in 65% of patients in a recent study. Infusion of ICG dye pre and post arterial clamping ensured that there was selective ischaemia only to the tumour region and adequate renal perfusion was achieved post-clamp removal. This imaging could be used in conjunction with AR to further aid live visual feedback, organ tracking and have

**2.** Imperial College London's iKnife could be used in conjunction with AR [41]. This 'Intelligent knife' is a surgical scalpel that chemically tests the tissue it has contact with. It uses Rapid Evaporative Ionisation Mass Spectrometry (REIMS) for real time analysis of the aerosols created from diathermy of tissues. iKnife has been used in gynaecological

registration and access to good quality imaging (live or pre-operative).

sickness (whilst using heavy current AR headsets).

interventions.

102 Evolving Trends in Kidney Cancer

**11. Future ideas/solutions**

better post-op renal functioning [40].

**Figure 3.** AR of the future. AR involving additional input from live USS imaging, organ trackers and other vital observations and patient data all being fed into the AR headset—providing a hands free platform.

tissues to distinguish between normal, borderline and malignant tissue [42] and this could be used in partial nephrectomies to give real time feedback for a precise tumour margin.

**3.** AR headset—this is a device that is being engineered simultaneously in many major US hospitals. Dr. Varshey and Dr. Murthi, are developing one such headset with the engineering team at the "Augmentarium" (University of Maryland). They hope to develop a system where a headset such as the Microsoft HoloLens can be worn by the surgeon and real-time USS of the patient or vital signs and patient data can be overlaid on the felid of view. This would drastically reduce the number of displays a surgeon has to usually track during an operation. Used in conjunction with dynamic image projection, the AR headset would be a good answer to cover the abovementioned 3 aspects of AR in partial nephrectomies [43].

The AR headset hopes to eliminate any obstructions in the surgeon's view as compared to conventional methods. Furthermore, voice recognition and gesture recognition development would enable hands-free control of the device—which would allow the surgeon to interact with the AR whilst maintaining a sterile environment [20] (**Figure 3**).

**4.** 3D printing—model replications of the patient's kidney can be printed using pre-operative CT/MRIs and these can be used to perform simulated operations prior to placing a knife onto the patient's skin. SIMPeds 3D Print at the Boston Children's Hospital offers exactly this—rapid printing and prototyping for nearly any organ in the human body [44]. Examples of this have been used to replicate and operate on difficult paediatric brain tumours [45], facial reconstructions and orthopaedic surgeries amongst many others [46]. This has allowed surgeons to simulate a realistic assessment of the individual's organ where it can be felt, touched and cut at precise margins. 3D printed surgical planning of partial nephrectomies has been explored by Zhang et al. [46] where face and content validity was obtained by 4 experienced laparoscopic urologists. A pilot study by Silberstein et al. [47] envisioned that 3D models could enhance the surgeon's (and patient's) understanding of the individual's renal malignancy anatomy—this would be especially beneficial in difficulties such as anatomical anomalies and precise segmental artery clamping [48].

**References**

Future [Accessed 01/10/2018]

nih.gov/pubmed/29536200

[1] Interaction Design Foundation. Augmented Reality—The Past, The Present and The

Augmented Reality in Kidney Cancer http://dx.doi.org/10.5772/intechopen.81890 105

[2] Mesko B. The Top 9 Augmented Reality Companies in Healthcare. Available from: http://

[3] Augmented reality system lets doctors see under patients' skin without the scalpel.

[4] Augmented reality helps surgeons to 'see through' tissue and reconnect blood vessels.

[5] Wei NJ, Dougherty B, Myers A, Badawy SM. Using Google glass in surgical settings: Systematic review. JMIR mHealth and uHealth. 2018;**6**(3):e54. DOI: 10.2196/

[6] Moglia A, Ferrari V, Morelli L, Ferrari M, Mosca F, Cuschieri A. A systematic review of virtual reality simulators for robot-assisted surgery. 2016. Available from: http://discovery.dundee.ac.uk/portal/en/theses/a-systematic-review-of-virtual-reality-simulators-

[7] Schulz GB, Grimm T, Buchner A, Jokisch F, Casuscelli J, Kretschmer A, et al. Validation of a high-end virtual reality simulator for training transurethral resection of bladder tumors. Journal of Surgical Education. 2018. DOI: 10.1016/j.jsurg.2018.08.001. Available

[8] Likowski A. Augmenting Reality in the Operating Room. Available from: https://www. umaryland.edu/news/archived-news/march-2017/newspressreleaseshottopics/aug-

[9] Jimenez Y, Cumming S, Wang W, Stuart K, Thwaites D, Lewis S. Patient education using virtual reality increases knowledge and positive experience for breast cancer patients undergoing radiation therapy. Supportive Care in Cancer. 2018;**26**(8):2879-2888. Available from. DOI: 10.1007/s00520-018-4114-4. Available from: https://www.ncbi.nlm.

[10] Rai A, Scovell JM, Xu A, Balasubramanian A, Siller R, Kohn T, et al. Patient-specific virtual simulation—A state of the art approach to teach renal tumor localization. Urology. 2018;**120**:42-48. Available from: doi: 10.1016/j.urology.2018.04.043 Available from: https://

[11] Makiyama K, Nagasaka M, Inuiya T, Takanami K, Ogata M, Kubota Y. Development of a patient-specific simulator for laparoscopic renal surgery. International Journal of

[12] Ueno D, Makiyama K, Yamanaka H, Ijiri T, Yokota H, Kubota Y. Prediction of open urinary tract in laparoscopic partial nephrectomy by virtual resection plane visualization.

www.sciencedirect.com/science/article/pii/S0090429518305405

Urology. 2012;**19**(9):829-835. DOI: 10.1111/j.1442-2042.2012.03053.x

mhealth.9409. Available from: https://www.ncbi.nlm.nih.gov/pubmed/29510969

for-robot-assisted-surgery(00243bee-f90b-486d-95cd-e0fcb51d8d15).html

from: https://www.sciencedirect.com/science/article/pii/S1931720418304021

menting-reality-in-the-operating-room.php [Accessed September 12, 2018]

scholar.aci.info/view/156566a01ed00110002/15e38d4c7c500013f25998b

Medical Devices & Surgical Technology Week. 2018. pp. 164

Medical Devices & Surgical Technology Week. 2018. pp. 213

Extrapolating further from this idea, 3D printed kidneys could also allow remote surgical procedures. An idea in conjunction with this was put forward by Dr. Murthi at a recent VR and AR applications gathering at Newseum, Washington, D.C [8]. She foresaw VR and AR working together to support patient care. An initial AR on the patient including imaging and medical data would allow the clinician to assess the patient's condition (in this case, anatomy) and a remote VR system could allow the clinician to see what the initial AR has shown and consequently advise and provide insight. An augmented nephrectomy system could benefit surgeons where instead of advising, they could operate remotely on replicated 3D models (representing in-vivo kidneys). Initial AR collected locally from the patient could be remotely projected as VR onto a 3D model and a surgeon could perform a partial nephrectomy which could be translated to the real kidney with the help of local robotic da Vinci machines. This model would allow constant feedback between the AR and VR systems and remote operations could be an answer to lack of surgical resources in healthcare deprived areas around the world.
