TEAFormers: TEnsor-Augmented Transformers for Multi-Dimensional Time Series Forecasting

• Main Authors: Kong, Linghang, Chen, Elynn, Chen, Yuzhou, Han, Yuefeng
• Format: Journal Article
• Language: English
• Published: 27-10-2024
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Learning; Statistics - Machine Learning
• DOI: 10.48550/arxiv.2410.20439

Multi-dimensional time series data, such as matrix and tensor-variate time
series, are increasingly prevalent in fields such as economics, finance, and
climate science. Traditional Transformer models, though adept with sequential
data, do not effectively preserve these multi-dimensional structures, as their
internal operations in effect flatten multi-dimensional observations into
vectors, thereby losing critical multi-dimensional relationships and patterns.
To address this, we introduce the Tensor-Augmented Transformer (TEAFormer), a
novel method that incorporates tensor expansion and compression within the
Transformer framework to maintain and leverage the inherent multi-dimensional
structures, thus reducing computational costs and improving prediction
accuracy. The core feature of the TEAFormer, the Tensor-Augmentation (TEA)
module, utilizes tensor expansion to enhance multi-view feature learning and
tensor compression for efficient information aggregation and reduced
computational load. The TEA module is not just a specific model architecture
but a versatile component that is highly compatible with the attention
mechanism and the encoder-decoder structure of Transformers, making it
adaptable to existing Transformer architectures. Our comprehensive experiments,
which integrate the TEA module into three popular time series Transformer
models across three real-world benchmarks, show significant performance
enhancements, highlighting the potential of TEAFormers for cutting-edge time
series forecasting.