Scale MLPerf-0.6 models on Google TPU-v3 Pods
Authors:
Sameer Kumar,
Victor Bitorff,
Dehao Chen,
Chiachen Chou,
Blake Hechtman,
HyoukJoong Lee,
Naveen Kumar,
Peter Mattson,
Shibo Wang,
Tao Wang,
Yuanzhong Xu,
Zongwei Zhou
Abstract:
The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6 training benchmark demonstrates the scalability of a suite of industry relevant ML models. MLPerf defines a suite of models, datasets and rules to follow when benchmarking to ensure results are comparable across hardware, frameworks and companies. Using this suite of models, we discuss the optimizations and techniques inc…
▽ More
The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6 training benchmark demonstrates the scalability of a suite of industry relevant ML models. MLPerf defines a suite of models, datasets and rules to follow when benchmarking to ensure results are comparable across hardware, frameworks and companies. Using this suite of models, we discuss the optimizations and techniques including choice of optimizer, spatial partitioning and weight update sharding necessary to scale to 1024 TPU chips. Furthermore, we identify properties of models that make scaling them challenging, such as limited data parallelism and unscaled weights. These optimizations contribute to record performance in transformer, Resnet-50 and SSD in the Google MLPerf-0.6 submission.
△ Less
Submitted 2 October, 2019; v1 submitted 20 September, 2019;
originally announced September 2019.
