Case-based similar image retrieval for weakly annotated large histopathological images of malignant lymphoma using deep metric learning

• Published in: Medical image analysis Vol. 85; p. 102752
• Main Authors: Hashimoto, Noriaki, Takagi, Yusuke, Masuda, Hiroki, Miyoshi, Hiroaki, Kohno, Kei, Nagaishi, Miharu, Sato, Kensaku, Takeuchi, Mai, Furuta, Takuya, Kawamoto, Keisuke, Yamada, Kyohei, Moritsubo, Mayuko, Inoue, Kanako, Shimasaki, Yasumasa, Ogura, Yusuke, Imamoto, Teppei, Mishina, Tatsuzo, Tanaka, Ken, Kawaguchi, Yoshino, Nakamura, Shigeo, Ohshima, Koichi, Hontani, Hidekata, Takeuchi, Ichiro
• Format: Journal Article
• Language: English
• Published: Netherlands Elsevier B.V 01-04-2023
• Subjects: Case-based similar image retrieval; Deep metric learning; Digital pathology; Humans; Image Interpretation, Computer-Assisted; Lymphoma - diagnostic imaging; Lymphoma - pathology; Malignant lymphoma; Multiple instance learning; Digital pathology; Case-based similar image retrieval; Malignant lymphoma; Deep metric learning; Multiple instance learning
• ISSN: 1361-8415; 1361-8423
• DOI: 10.1016/j.media.2023.102752

In the present study, we propose a novel case-based similar image retrieval (SIR) method for hematoxylin and eosin (H&E) stained histopathological images of malignant lymphoma. When a whole slide image (WSI) is used as an input query, it is desirable to be able to retrieve similar cases by focusing on image patches in pathologically important regions such as tumor cells. To address this problem, we employ attention-based multiple instance learning, which enables us to focus on tumor-specific regions when the similarity between cases is computed. Moreover, we employ contrastive distance metric learning to incorporate immunohistochemical (IHC) staining patterns as useful supervised information for defining appropriate similarity between heterogeneous malignant lymphoma cases. In the experiment with 249 malignant lymphoma patients, we confirmed that the proposed method exhibited higher evaluation measures than the baseline case-based SIR methods. Furthermore, the subjective evaluation by pathologists revealed that our similarity measure using IHC staining patterns is appropriate for representing the similarity of H&E stained tissue images for malignant lymphoma.
[Display omitted]
•Retrieving similar cases of malignant lymphoma with unannotated whole slide images.•Retrieval focusing on tumor regions by attention-based multiple instance learning.•Deep metric learning using the similarity of immunohistochemical staining patterns.•Performance evaluation of 249 cases through subjective experiment by 10 pathologists.