A Framework for Automatic Validation and Application of Lossy Data Compression in Ensemble Data Assimilation

• Main Authors: Keller, Kai, Yashiro, Hisashi, Wahib, Mohamed, Gerofi, Balazs, Kestelman, Adrian Cristal, Bautista-Gomez, Leonardo
• Format: Journal Article
• Language: English
• Published: 04-10-2024
• Subjects: Physics - Atmospheric and Oceanic Physics; Physics - Geophysics
• DOI: 10.48550/arxiv.2410.03184

Ensemble data assimilation techniques form an indispensable part of numerical
weather prediction. As the ensemble size grows and model resolution increases,
the amount of required storage becomes a major issue. Data compression schemes
may come to the rescue not only for operational weather prediction, but also
for weather history archives. In this paper, we present the design and
implementation of an easy-to-use framework for evaluating the impact of lossy
data compression in large scale ensemble data assimilation. The framework
leverages robust statistical qualifiers to determine which compression
parameters can be safely applied to the climate variables. Furthermore, our
proposal can be used to apply the best parameters during operation, while
monitoring data integrity. We perform an exemplary study on the Lorenz96 model
to identify viable compression parameters and achieve a 1/3 saving in storage
space and an effective speedup of 6% per assimilation cycle, while monitoring
the state integrity.