Augmented reality-based autostereoscopic surgical visualization system for telesurgery

• Published in: International journal for computer assisted radiology and surgery Vol. 16; no. 11; pp. 1985 - 1997
• Main Authors: Huang, Tianqi, Li, Ruiyang, Li, Yangxi, Zhang, Xinran, Liao, Hongen
• Format: Journal Article
• Language: English
• Published: Cham Springer International Publishing 01-11-2021; Springer Nature B.V
• Subjects: Algorithms; Augmented reality; Autostereoscopic; Completion time; Computer Imaging; Computer Science; Computer vision; Display devices; Health Informatics; Image processing; Image reconstruction; Image resolution; Imaging; Medical imaging; Medicine; Medicine & Public Health; Original Article; Pattern Recognition and Graphics; Prototypes; Quantitative analysis; Radiology; Registration; Robotic surgery; Space perception; Surgery; Surgical instruments; System effectiveness; Telemedicine; Telerobotics; Telesurgery; Vision; Visualization; 3D interactive visualization; Percutaneous procedures; Autostereoscopic display; Augmented reality; Telesurgery
• ISSN: 1861-6410; 1861-6429
• DOI: 10.1007/s11548-021-02463-5

Purpose
The visualization of remote surgical scenes is the key to realizing the remote operation of surgical robots. However, current non-endoscopic surgical robot systems lack an effective visualization tool to offer sufficient surgical scene information and depth perception.
Methods
We propose a novel autostereoscopic surgical visualization system integrating 3D intraoperative scene reconstruction, autostereoscopic 3D display, and augmented reality-based image fusion. The preoperative organ structure and the intraoperative surface point cloud are obtained from medical imaging and the RGB-D camera, respectively, and aligned by an automatic marker-free intraoperative registration algorithm. After registration, preoperative meshes with precalculated illumination and intraoperative textured point cloud are blended in real time. Finally, the fused image is shown on a 3D autostereoscopic display device to achieve depth perception.
Results
A prototype of the autostereoscopic surgical visualization system was built. The system had a horizontal image resolution of 1.31 mm, a vertical image resolution of 0.82 mm, an average rendering rate of 33.1 FPS, an average registration rate of 20.5 FPS, and average registration errors of approximately 3 mm. A telesurgical robot prototype based on 3D autostereoscopic display was built. The quantitative evaluation experiments showed that our system achieved similar operational accuracy (1.79 ± 0.87 mm) as the conventional system (1.95 ± 0.71 mm), while having advantages in terms of completion time (with 34.11% reduction) and path length (with 35.87% reduction). Post-experimental questionnaires indicated that the system was user-friendly for novices and experts.
Conclusion
We propose a 3D surgical visualization system with augmented instruction and depth perception for telesurgery. The qualitative and quantitative evaluation results illustrate the accuracy and efficiency of the proposed system. Therefore, it shows great prospects in robotic surgery and telesurgery.