Finding Nano-Ötzi: Cryo-Electron Tomography Visualization Guided by Learned Segmentation

• Published in: IEEE transactions on visualization and computer graphics Vol. 29; no. 10; pp. 4198 - 4214
• Main Authors: Nguyen, Ngan, Bohak, Ciril, Engel, Dominik, Mindek, Peter, Strnad, Ondrej, Wonka, Peter, Li, Sai, Ropinski, Timo, Viola, Ivan
• Format: Journal Article
• Language: English
• Published: New York IEEE 01-10-2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; computer graphics techniques; Data analysis; Data visualization; Deep learning; Design parameters; Frequency analysis; Frequency distribution; Image segmentation; Imaging techniques; Labels; life sciences; Machine learning; machine learning techniques; Noise measurement; Occlusion; scalar field data; Semi-supervised learning; Signal to noise ratio; Task analysis; Three-dimensional displays; Tomography; Transfer functions; Visualization; Volume rendering
• ISSN: 1077-2626; 1941-0506
• DOI: 10.1109/TVCG.2022.3186146

Cryo-electron tomography (cryo-ET) is a new 3D imaging technique with unprecedented potential for resolving submicron structural details. Existing volume visualization methods, however, are not able to reveal details of interest due to low signal-to-noise ratio. In order to design more powerful transfer functions, we propose leveraging soft segmentation as an explicit component of visualization for noisy volumes. Our technical realization is based on semi-supervised learning, where we combine the advantages of two segmentation algorithms. First, the weak segmentation algorithm provides good results for propagating sparse user-provided labels to other voxels in the same volume and is used to generate dense pseudo-labels. Second, the powerful deep-learning-based segmentation algorithm learns from these pseudo-labels to generalize the segmentation to other unseen volumes, a task that the weak segmentation algorithm fails at completely. The proposed volume visualization uses deep-learning-based segmentation as a component for segmentation-aware transfer function design. Appropriate ramp parameters can be suggested automatically through frequency distribution analysis. Furthermore, our visualization uses gradient-free ambient occlusion shading to further suppress the visual presence of noise, and to give structural detail the desired prominence. The cryo-ET data studied in our technical experiments are based on the highest-quality tilted series of intact SARS-CoV-2 virions. Our technique shows the high impact in target sciences for visual data analysis of very noisy volumes that cannot be visualized with existing techniques.