Domain Adaptation for Time series Transformers using One-step fine-tuning

• Main Authors: Khanal, Subina, Tirupathi, Seshu, Zizzo, Giulio, Rawat, Ambrish, Pedersen, Torben Bach
• Format: Journal Article
• Language: English
• Published: 12-01-2024
• Subjects: Computer Science - Learning
• DOI: 10.48550/arxiv.2401.06524

The recent breakthrough of Transformers in deep learning has drawn
significant attention of the time series community due to their ability to
capture long-range dependencies. However, like other deep learning models,
Transformers face limitations in time series prediction, including insufficient
temporal understanding, generalization challenges, and data shift issues for
the domains with limited data. Additionally, addressing the issue of
catastrophic forgetting, where models forget previously learned information
when exposed to new data, is another critical aspect that requires attention in
enhancing the robustness of Transformers for time series tasks. To address
these limitations, in this paper, we pre-train the time series Transformer
model on a source domain with sufficient data and fine-tune it on the target
domain with limited data. We introduce the \emph{One-step fine-tuning}
approach, adding some percentage of source domain data to the target domains,
providing the model with diverse time series instances. We then fine-tune the
pre-trained model using a gradual unfreezing technique. This helps enhance the
model's performance in time series prediction for domains with limited data.
Extensive experimental results on two real-world datasets show that our
approach improves over the state-of-the-art baselines by 4.35% and 11.54% for
indoor temperature and wind power prediction, respectively.