Using Message Logs and Resource Use Data for Cluster Failure Diagnosis

• Published in: 2016 IEEE 23rd International Conference on High Performance Computing (HiPC) pp. 232 - 241
• Main Authors: Chuah, Edward, Jhumka, Arshad, Browne, James C., Gurumdimma, Nentawe, Narasimhamurthy, Sai, Barth, Bill
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01-12-2016
• Subjects: Cluster log data; Correlation; Correlation analysis; Data mining; Electronic mail; Error correction codes; Failure diagnosis; Large cluster system; Lustre file-system; Monitoring; Radiation detectors; Software
• DOI: 10.1109/HiPC.2016.035

Failure diagnosis for large compute clusters using only message logs is known to be incomplete. Recent availability of resource use data provides another potentially useful source of data for failure detection and diagnosis. Early work combining message logs and resource use data for failure diagnosis has shown promising results. This paper describes the CRUMEL framework which implements a new approach to combining rationalized message logs and resource use data for failure diagnosis. CRUMEL identifies patterns of errors and resource use and correlates these patterns by time with system failures. Application of CRUMEL to data from the Ranger supercomputer has yielded improved diagnoses over previous research. CRUMEL has: (i) showed that more events correlated with system failures can only be identified by applying different correlation algorithms, (ii) confirmed six groups of errors, (iii) identified Lustre I/O resource use counters which are correlated with occurrence of Lustre faults which are potential flags for online detection of failures, (iv) matched the dates of correlated error events and correlated resource use with the dates of compute node hang-ups and (v) identified two more error groups associated with compute node hang-ups. The pre-processed data will be put on the public domain in September, 2016.