Evaluation on the Performance Fluctuation of Hadoop Jobs in the Cloud

• Published in: 2013 IEEE 16th International Conference on Computational Science and Engineering pp. 159 - 166
• Main Authors: Aida, Kento, Abdul-Rahman, Omar, Sakane, Eisaku, Motoyama, Kazutaka
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01-12-2013
• Subjects: Benchmark testing; Cloud computing; Educational institutions; Hadoop; Interference; performance fluctuation; Servers; Virtualization
• DOI: 10.1109/CSE.2013.34

Cloud computing is widely used as a computing platform in business and academic communities. Performance is an important issue when a user runs an application in the cloud. The user may want to estimate the application execution time before running the application in order to guarantee the application performance or choose a suitable cloud for the user application. However, the application performance in the cloud sometime fluctuates, which makes the estimation of the application performance difficult. In this paper, we present the experimental results obtained in investigating the fluctuation of the application execution time in the cloud. We investigated the performance fluctuation of Hadoop jobs in both a public cloud and a private cloud over three months. The obtained experimental results indicate that the fluctuation is smaller than expected, except for some cases for irregular jobs with significantly long execution times. Moreover, the application performance is independent of the time of day and the day of the week. Finally, we also discuss the reasons for the existence of these irregular jobs through a detailed analysis of job execution traces.