TAPIOCA: An I/O Library for Optimized Topology-Aware Data Aggregation on Large-Scale Supercomputers

• Published in: 2017 IEEE International Conference on Cluster Computing (CLUSTER) pp. 70 - 80
• Main Authors: Tessier, Francois, Vishwanath, Venkatram, Jeannot, Emmanuel
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01-09-2017
• Subjects: Arrays; Bandwidth; Data models; Data movement optimization; Libraries; Network topology; Parallel I/O; Placement; Supercomputers; Topology-aware
• ISSN: 2168-9253
• DOI: 10.1109/CLUSTER.2017.80

Reading and writing data efficiently from storage system is necessary for most scientific simulations to achieve good performance at scale. Many software solutions have been developed to decrease the I/O bottleneck. One well-known strategy, in the context of collective I/O operations, is the two-phase I/O scheme. This strategy consists of selecting a subset of processes to aggregate contiguous pieces of data before performing reads/writes. In this paper, we present TAPIOCA, an MPI-based library implementing an efficient topology-aware two-phase I/O algorithm. We show how TAPIOCA can take advantage of double-buffering and one-sided communication to reduce as much as possible the idle time during data aggregation. We also introduce our cost model leading to a topology-aware aggregator placement optimizing the movements of data. We validate our approach at large scale on two leadership-class supercomputers: Mira (IBM BG/Q) and Theta (Cray XC40). We present the results obtained with TAPIOCA on a micro-benchmark and the I/O kernel of a large-scale simulation. On both architectures, we show a substantial improvement of I/O performance compared with the default MPI I/O implementation. On BG/Q+GPFS, for instance, our algorithm leads to a performance improvement by a factor of twelve while on the Cray XC40 system associated with a Lustre filesystem, we achieve an improvement of four.