SAMSA: Efficient Transformer for Many Data Modalities

• Main Authors: Lenhat, Minh, Nguyen, Viet Anh, Nguyen, Khoa, Hieu, Duong Duc, Hung, Dao Huu, Hy, Truong Son
• Format: Journal Article
• Language: English
• Published: 09-08-2024
• Subjects: Computer Science - Learning
• DOI: 10.48550/arxiv.2408.05391

The versatility of self-attention mechanism earned transformers great success
in almost all data modalities, with limitations on the quadratic complexity and
difficulty of training. Efficient transformers, on the other hand, often rely
on clever data-modality-dependent construction to get over the quadratic
complexity of transformers. This greatly hinders their applications on
different data modalities, which is one of the pillars of contemporary
foundational modeling. In this paper, we lay the groundwork for efficient
foundational modeling by proposing SAMSA - SAMpling-Self-Attention, a
context-aware linear complexity self-attention mechanism that works well on
multiple data modalities. Our mechanism is based on a differentiable sampling
without replacement method we discovered. This enables the self-attention
module to attend to the most important token set, where the importance is
defined by data. Moreover, as differentiability is not needed in inference, the
sparse formulation of our method costs little time overhead, further lowering
computational costs. In short, SAMSA achieved competitive or even SOTA results
on many benchmarks, while being faster in inference, compared to other very
specialized models. Against full self-attention, real inference time
significantly decreases while performance ranges from negligible degradation to
outperformance. We release our source code in the repository:
https://github.com/HySonLab/SAMSA