Computing on large-scale distributed systems: XtremWeb architecture, programming models, security, tests and convergence with grid

• Published in: Future generation computer systems Vol. 21; no. 3; pp. 417 - 437
• Main Authors: Cappello, Franck, Djilali, Samir, Fedak, Gilles, Herault, Thomas, Magniette, Frédéric, Néri, Vincent, Lodygensky, Oleg
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01-03-2005; Elsevier
• Subjects: Computer Science; Distributed, Parallel, and Cluster Computing; Global Computing systems; Hardware Architecture; Large-scale distributed systems; XtremWeb; Global Computing systems; Large-scale distributed systems; XtremWeb
• ISSN: 0167-739X; 1872-7115
• DOI: 10.1016/j.future.2004.04.011

Global Computing systems belong to the class of large-scale distributed systems. Their properties high computational, storage and communication performance potentials, high resilience make them attractive in academia and industry as computing infrastructures in complement to more classical infrastructures such as clusters or supercomputers. However, generalizing the use of these systems in a multi-user and multi-parallel programming context involves finding solutions and providing mechanisms for many issues such as programming bag of tasks and message passing parallel applications, securing the application, the system itself and the computing nodes, deploying the systems for harnessing resources managed in different ways. In this paper, we present our research, often influenced by user demands, towards a Computational peer-to-peer system called XtremWeb. We describe (a) the architecture of the system and its motivations, (b) the parallel programming paradigms available in XtremWeb and how they are implemented, (c) the deployment issues and what mechanisms are used to harness simultaneously uncoordinated set of resources, and resources managed by batch schedulers and (d) the security issue and how we address, inside XtremWeb, the protection of the computing resources. We present two multi-parametric applications to be used in production: Aires belonging to the high energy physics (HEP) Auger project and a protein conformation predictor using a molecular dynamic simulator. To evaluate the performance and volatility tolerance, we present experiment results for bag of tasks applications and message passing applications. We show that the system can tolerate massive failure and we discuss the performance of the node protection mechanism. Based on the XtremWeb project developments and evolutions, we will discuss the convergence between Global Computing systems and Grid.