Contextual Transformer Networks for Visual Recognition

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 45; no. 2; pp. 1489 - 1500
• Main Authors: Li, Yehao, Yao, Ting, Pan, Yingwei, Mei, Tao
• Format: Journal Article
• Language: English
• Published: United States IEEE 01-02-2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Computer architecture; Computer vision; Convolution; Feature maps; Image recognition; Image segmentation; Natural language processing; Object detection; Object recognition; Queries; Representations; self-attention; Source code; Task analysis; Transformer; Transformers; vision transformer; Visualization
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2022.3164083

Transformer with self-attention has led to the revolutionizing of natural language processing field, and recently inspires the emergence of Transformer-style architecture design with competitive results in numerous computer vision tasks. Nevertheless, most of existing designs directly employ self-attention over a 2D feature map to obtain the attention matrix based on pairs of isolated queries and keys at each spatial location, but leave the rich contexts among neighbor keys under-exploited. In this work, we design a novel Transformer-style module, i.e., Contextual Transformer ( CoT ) block, for visual recognition. Such design fully capitalizes on the contextual information among input keys to guide the learning of dynamic attention matrix and thus strengthens the capacity of visual representation. Technically, CoT block first contextually encodes input keys via a <inline-formula><tex-math notation="LaTeX">3\times 3</tex-math> <mml:math><mml:mrow><mml:mn>3</mml:mn><mml:mo>×</mml:mo><mml:mn>3</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="pan-ieq1-3164083.gif"/> </inline-formula> convolution, leading to a static contextual representation of inputs. We further concatenate the encoded keys with input queries to learn the dynamic multi-head attention matrix through two consecutive <inline-formula><tex-math notation="LaTeX">1\times 1</tex-math> <mml:math><mml:mrow><mml:mn>1</mml:mn><mml:mo>×</mml:mo><mml:mn>1</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="pan-ieq2-3164083.gif"/> </inline-formula> convolutions. The learnt attention matrix is multiplied by input values to achieve the dynamic contextual representation of inputs. The fusion of the static and dynamic contextual representations are finally taken as outputs. Our CoT block is appealing in the view that it can readily replace each <inline-formula><tex-math notation="LaTeX">3\times 3</tex-math> <mml:math><mml:mrow><mml:mn>3</mml:mn><mml:mo>×</mml:mo><mml:mn>3</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="pan-ieq3-3164083.gif"/> </inline-formula> convolution in ResNet architectures, yielding a Transformer-style backbone named as Contextual Transformer Networks ( CoTNet ). Through extensive experiments over a wide range of applications (e.g., image recognition, object detection, instance segmentation, and semantic segmentation), we validate the superiority of CoTNet as a stronger backbone. Source code is available at https://github.com/JDAI-CV/CoTNet .