PLUTO: Pathology-Universal Transformer

• Main Authors: Juyal, Dinkar, Padigela, Harshith, Shah, Chintan, Shenker, Daniel, Harguindeguy, Natalia, Liu, Yi, Martin, Blake, Zhang, Yibo, Nercessian, Michael, Markey, Miles, Finberg, Isaac, Luu, Kelsey, Borders, Daniel, Javed, Syed Ashar, Krause, Emma, Biju, Raymond, Sood, Aashish, Ma, Allen, Nyman, Jackson, Shamshoian, John, Chhor, Guillaume, Sanghavi, Darpan, Thibault, Marc, Yu, Limin, Najdawi, Fedaa, Hipp, Jennifer A, Fahy, Darren, Glass, Benjamin, Walk, Eric, Abel, John, Pokkalla, Harsha, Beck, Andrew H, Grullon, Sean
• Format: Journal Article
• Language: English
• Published: 13-05-2024
• Subjects: Computer Science - Computer Vision and Pattern Recognition
• DOI: 10.48550/arxiv.2405.07905

Pathology is the study of microscopic inspection of tissue, and a pathology
diagnosis is often the medical gold standard to diagnose disease. Pathology
images provide a unique challenge for computer-vision-based analysis: a single
pathology Whole Slide Image (WSI) is gigapixel-sized and often contains
hundreds of thousands to millions of objects of interest across multiple
resolutions. In this work, we propose PathoLogy Universal TransfOrmer (PLUTO):
a light-weight pathology FM that is pre-trained on a diverse dataset of 195
million image tiles collected from multiple sites and extracts meaningful
representations across multiple WSI scales that enable a large variety of
downstream pathology tasks. In particular, we design task-specific adaptation
heads that utilize PLUTO's output embeddings for tasks which span pathology
scales ranging from subcellular to slide-scale, including instance
segmentation, tile classification, and slide-level prediction. We compare
PLUTO's performance to other state-of-the-art methods on a diverse set of
external and internal benchmarks covering multiple biologically relevant tasks,
tissue types, resolutions, stains, and scanners. We find that PLUTO matches or
outperforms existing task-specific baselines and pathology-specific foundation
models, some of which use orders-of-magnitude larger datasets and model sizes
when compared to PLUTO. Our findings present a path towards a universal
embedding to power pathology image analysis, and motivate further exploration
around pathology foundation models in terms of data diversity, architectural
improvements, sample efficiency, and practical deployability in real-world
applications.