Autonomy Loops for Monitoring, Operational Data Analytics, Feedback, and Response in HPC Operations

• Main Authors: Boito, Francieli, Brandt, Jim, Cardellini, Valeria, Carns, Philip, Ciorba, Florina M, Egan, Hilary, Eleliemy, Ahmed, Gentile, Ann, Gruber, Thomas, Hanson, Jeff, Haus, Utz-Uwe, Huck, Kevin, Ilsche, Thomas, Jakobsche, Thomas, Jones, Terry, Karlsson, Sven, Mueen, Abdullah, Ott, Michael, Patki, Tapasya, Peng, Ivy, Raghavan, Krishnan, Simms, Stephen, Shoga, Kathleen, Showerman, Michael, Tiwari, Devesh, Wilde, Torsten, Yamamoto, Keiji
• Format: Journal Article
• Language: English
• Published: 30-01-2024
• Subjects: Computer Science - Distributed, Parallel, and Cluster Computing
• DOI: 10.48550/arxiv.2401.16971

Many High Performance Computing (HPC) facilities have developed and deployed
frameworks in support of continuous monitoring and operational data analytics
(MODA) to help improve efficiency and throughput. Because of the complexity and
scale of systems and workflows and the need for low-latency response to address
dynamic circumstances, automated feedback and response have the potential to be
more effective than current human-in-the-loop approaches which are laborious
and error prone. Progress has been limited, however, by factors such as the
lack of infrastructure and feedback hooks, and successful deployment is often
site- and case-specific. In this position paper we report on the outcomes and
plans from a recent Dagstuhl Seminar, seeking to carve a path for community
progress in the development of autonomous feedback loops for MODA, based on the
established formalism of similar (MAPE-K) loops in autonomous computing and
self-adaptive systems. By defining and developing such loops for significant
cases experienced across HPC sites, we seek to extract commonalities and
develop conventions that will facilitate interoperability and
interchangeability with system hardware, software, and applications across
different sites, and will motivate vendors and others to provide telemetry
interfaces and feedback hooks to enable community development and pervasive
deployment of MODA autonomy loops.