Accelerating Communication in Deep Learning Recommendation Model Training with Dual-Level Adaptive Lossy Compression

• Main Authors: Feng, Hao, Zhang, Boyuan, Ye, Fanjiang, Si, Min, Chu, Ching-Hsiang, Tian, Jiannan, Yin, Chunxing, Deng, Summer, Hao, Yuchen, Balaji, Pavan, Geng, Tong, Tao, Dingwen
• Format: Journal Article
• Language: English
• Published: 05-07-2024
• Subjects: Computer Science - Distributed, Parallel, and Cluster Computing; Computer Science - Learning
• DOI: 10.48550/arxiv.2407.04272

DLRM is a state-of-the-art recommendation system model that has gained
widespread adoption across various industry applications. The large size of
DLRM models, however, necessitates the use of multiple devices/GPUs for
efficient training. A significant bottleneck in this process is the
time-consuming all-to-all communication required to collect embedding data from
all devices. To mitigate this, we introduce a method that employs error-bounded
lossy compression to reduce the communication data size and accelerate DLRM
training. We develop a novel error-bounded lossy compression algorithm,
informed by an in-depth analysis of embedding data features, to achieve high
compression ratios. Moreover, we introduce a dual-level adaptive strategy for
error-bound adjustment, spanning both table-wise and iteration-wise aspects, to
balance the compression benefits with the potential impacts on accuracy. We
further optimize our compressor for PyTorch tensors on GPUs, minimizing
compression overhead. Evaluation shows that our method achieves a 1.38$\times$
training speedup with a minimal accuracy impact.