Clusterfile: a flexible physical layout parallel file system

• Published in: Concurrency and computation Vol. 15; no. 7-8; pp. 653 - 679
• Main Authors: Isaila, Florin, Tichy, Walter F.
• Format: Journal Article
• Language: English
• Published: Chichester, UK John Wiley & Sons, Ltd 01-06-2003
• Subjects: data redistribution; mapping functions; non-contiguous I/O; parallel file system; parallel I/O; physical and logical partitioning
• ISSN: 1532-0626; 1532-0634
• DOI: 10.1002/cpe.724

This paper presents Clusterfile, a parallel file system that provides parallel file access on a cluster of computers. We introduce a file partitioning model that has been used in the design of Clusterfile. The model uses a data representation that is optimized for multidimensional array partitioning while allowing arbitrary partitions. The paper shows how the file model can be employed for file partitioning into both physical subfiles and logical views. We also present how the conversion between two partitions of the same file is implemented using a general memory redistribution algorithm. We show how we use the algorithm to optimize non‐contiguous read and write operations. The experimental results include performance comparisons with the Parallel Virtual File System (PVFS) and an MPI‐IO implementation for PVFS. Copyright © 2003 John Wiley & Sons, Ltd.