U-Netmer: U-Net meets Transformer for medical image segmentation

• Main Authors: He, Sheng, Bao, Rina, Grant, P. Ellen, Ou, Yangming
• Format: Journal Article
• Language: English
• Published: 03-04-2023
• Subjects: Computer Science - Computer Vision and Pattern Recognition
• DOI: 10.48550/arxiv.2304.01401

The combination of the U-Net based deep learning models and Transformer is a
new trend for medical image segmentation. U-Net can extract the detailed local
semantic and texture information and Transformer can learn the long-rang
dependencies among pixels in the input image. However, directly adapting the
Transformer for segmentation has ``token-flatten" problem (flattens the local
patches into 1D tokens which losses the interaction among pixels within local
patches) and ``scale-sensitivity" problem (uses a fixed scale to split the
input image into local patches). Compared to directly combining U-Net and
Transformer, we propose a new global-local fashion combination of U-Net and
Transformer, named U-Netmer, to solve the two problems. The proposed U-Netmer
splits an input image into local patches. The global-context information among
local patches is learnt by the self-attention mechanism in Transformer and
U-Net segments each local patch instead of flattening into tokens to solve the
`token-flatten" problem. The U-Netmer can segment the input image with
different patch sizes with the identical structure and the same parameter.
Thus, the U-Netmer can be trained with different patch sizes to solve the
``scale-sensitivity" problem. We conduct extensive experiments in 7 public
datasets on 7 organs (brain, heart, breast, lung, polyp, pancreas and prostate)
and 4 imaging modalities (MRI, CT, ultrasound, and endoscopy) to show that the
proposed U-Netmer can be generally applied to improve accuracy of medical image
segmentation. These experimental results show that U-Netmer provides
state-of-the-art performance compared to baselines and other models. In
addition, the discrepancy among the outputs of U-Netmer with different scales
is linearly correlated to the segmentation accuracy which can be considered as
a confidence score to rank test images by difficulty without ground-truth.