Using Object Based Files for High Performance Parallel I/O

• Published in: 2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications pp. 149 - 154
• Main Authors: Logan, J., Dickens, P.M.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01-09-2007
• Subjects: Application software; Checkpointing; Computational modeling; Concurrent computing; Conferences; Costs; Data models; File systems; file view; High performance computing; I/O Performance; Large-scale systems; MPI-IO; object-based cache; object-based files; parallel I/O
• ISBN: 9781424413478; 1424413478
• DOI: 10.1109/IDAACS.2007.4488394

We contend that the scalable I/O problem in high performance computing is largely due to the legacy view of a file as a linear sequence of bytes. In this paper we introduce an alternative to the traditional "flat file " that uses the information contained in file views to partition a file into an optimal set of objects, minimizing locking contention and simplifying the lock management strategy. We illustrate the use of an object based cache added to ROMIO to efficiently and transparently add object-based file capabilities to MPI-IO. We analyze the performance of our system using the FLASH-IO benchmark, and demonstrate a substantial performance improvement over the standard ROMIO implementation.