Application-aware Congestion Mitigation for High-Performance Computing Systems
Authors:
Archit Patke,
Saurabh Jha,
Haoran Qiu,
Jim Brandt,
Ann Gentile,
Joe Greenseid,
Zbigniew Kalbarczyk,
Ravishankar Iyer
Abstract:
High-performance computing (HPC) systems frequently experience congestion leading to significant application performance variation. However, the impact of congestion on application runtime differs from application to application depending on their network characteristics (such as bandwidth and latency requirements). We leverage this insight to develop Netscope, an automated ML-driven framework tha…
▽ More
High-performance computing (HPC) systems frequently experience congestion leading to significant application performance variation. However, the impact of congestion on application runtime differs from application to application depending on their network characteristics (such as bandwidth and latency requirements). We leverage this insight to develop Netscope, an automated ML-driven framework that considers those network characteristics to dynamically mitigate congestion. We evaluate Netscope on four Cray Aries systems, including a production supercomputer on real scientific applications. Netscope has a lower training cost and accurately estimates the impact of congestion on application runtime with a correlation between 0.7and 0.9 for common scientific applications. Moreover, we find that Netscope reduces tail runtime variability by up to 14.9 times while improving median system utility by 12%.
△ Less
Submitted 3 February, 2021; v1 submitted 14 December, 2020;
originally announced December 2020.
