PartIR: Composing SPMD Partitioning Strategies for Machine Learning
Authors:
Sami Alabed,
Daniel Belov,
Bart Chrzaszcz,
Juliana Franco,
Dominik Grewe,
Dougal Maclaurin,
James Molloy,
Tom Natan,
Tamara Norman,
Xiaoyue Pan,
Adam Paszke,
Norman A. Rink,
Michael Schaarschmidt,
Timur Sitdikov,
Agnieszka Swietlik,
Dimitrios Vytiniotis,
Joel Wee
Abstract:
Training of modern large neural networks (NN) requires a combination of parallelization strategies encompassing data, model, or optimizer sharding. When strategies increase in complexity, it becomes necessary for partitioning tools to be 1) expressive, allowing the composition of simpler strategies, and 2) predictable to estimate performance analytically. We present PartIR, our design for a NN par…
▽ More
Training of modern large neural networks (NN) requires a combination of parallelization strategies encompassing data, model, or optimizer sharding. When strategies increase in complexity, it becomes necessary for partitioning tools to be 1) expressive, allowing the composition of simpler strategies, and 2) predictable to estimate performance analytically. We present PartIR, our design for a NN partitioning system. PartIR is focused on an incremental approach to rewriting and is hardware-and-runtime agnostic. We present a simple but powerful API for composing sharding strategies and a simulator to validate them. The process is driven by high-level programmer-issued partitioning tactics, which can be both manual and automatic. Importantly, the tactics are specified separately from the model code, making them easy to change. We evaluate PartIR on several different models to demonstrate its predictability, expressibility, and ability to reach peak performance..
△ Less
Submitted 3 March, 2024; v1 submitted 20 January, 2024;
originally announced January 2024.
Curvature-enhanced Graph Convolutional Network for Biomolecular Interaction Prediction
Authors:
Cong Shen,
**jian Ding,
Junjie Wee,
Jialin Bi,
Jiawei Luo,
Kelin Xia
Abstract:
Geometric deep learning has demonstrated a great potential in non-Euclidean data analysis. The incorporation of geometric insights into learning architecture is vital to its success. Here we propose a curvature-enhanced graph convolutional network (CGCN) for biomolecular interaction prediction, for the first time. Our CGCN employs Ollivier-Ricci curvature (ORC) to characterize network local struct…
▽ More
Geometric deep learning has demonstrated a great potential in non-Euclidean data analysis. The incorporation of geometric insights into learning architecture is vital to its success. Here we propose a curvature-enhanced graph convolutional network (CGCN) for biomolecular interaction prediction, for the first time. Our CGCN employs Ollivier-Ricci curvature (ORC) to characterize network local structures and to enhance the learning capability of GCNs. More specifically, ORCs are evaluated based on the local topology from node neighborhoods, and further used as weights for the feature aggregation in message-passing procedure. Our CGCN model is extensively validated on fourteen real-world bimolecular interaction networks and a series of simulated data. It has been found that our CGCN can achieve the state-of-the-art results. It outperforms all existing models, as far as we know, in thirteen out of the fourteen real-world datasets and ranks as the second in the rest one. The results from the simulated data show that our CGCN model is superior to the traditional GCN models regardless of the positive-to-negativecurvature ratios, network densities, and network sizes (when larger than 500).
△ Less
Submitted 23 June, 2023;
originally announced June 2023.
