-
Probing the effects of broken symmetries in machine learning
Authors:
Marcel F. Langer,
Sergey N. Pozdnyakov,
Michele Ceriotti
Abstract:
Symmetry is one of the most central concepts in physics, and it is no surprise that it has also been widely adopted as an inductive bias for machine-learning models applied to the physical sciences. This is especially true for models targeting the properties of matter at the atomic scale. Both established and state-of-the-art approaches, with almost no exceptions, are built to be exactly equivaria…
▽ More
Symmetry is one of the most central concepts in physics, and it is no surprise that it has also been widely adopted as an inductive bias for machine-learning models applied to the physical sciences. This is especially true for models targeting the properties of matter at the atomic scale. Both established and state-of-the-art approaches, with almost no exceptions, are built to be exactly equivariant to translations, permutations, and rotations of the atoms. Incorporating symmetries -- rotations in particular -- constrains the model design space and implies more complicated architectures that are often also computationally demanding. There are indications that non-symmetric models can easily learn symmetries from data, and that doing so can even be beneficial for the accuracy of the model. We put a model that obeys rotational invariance only approximately to the test, in realistic scenarios involving simulations of gas-phase, liquid, and solid water. We focus specifically on physical observables that are likely to be affected -- directly or indirectly -- by symmetry breaking, finding negligible consequences when the model is used in an interpolative, bulk, regime. Even for extrapolative gas-phase predictions, the model remains very stable, even though symmetry artifacts are noticeable. We also discuss strategies that can be used to systematically reduce the magnitude of symmetry breaking when it occurs, and assess their impact on the convergence of observables.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
Smooth, exact rotational symmetrization for deep learning on point clouds
Authors:
Sergey N. Pozdnyakov,
Michele Ceriotti
Abstract:
Point clouds are versatile representations of 3D objects and have found widespread application in science and engineering. Many successful deep-learning models have been proposed that use them as input. The domain of chemical and materials modeling is especially challenging because exact compliance with physical constraints is highly desirable for a model to be usable in practice. These constraint…
▽ More
Point clouds are versatile representations of 3D objects and have found widespread application in science and engineering. Many successful deep-learning models have been proposed that use them as input. The domain of chemical and materials modeling is especially challenging because exact compliance with physical constraints is highly desirable for a model to be usable in practice. These constraints include smoothness and invariance with respect to translations, rotations, and permutations of identical atoms. If these requirements are not rigorously fulfilled, atomistic simulations might lead to absurd outcomes even if the model has excellent accuracy. Consequently, dedicated architectures, which achieve invariance by restricting their design space, have been developed. General-purpose point-cloud models are more varied but often disregard rotational symmetry. We propose a general symmetrization method that adds rotational equivariance to any given model while preserving all the other requirements. Our approach simplifies the development of better atomic-scale machine-learning schemes by relaxing the constraints on the design space and making it possible to incorporate ideas that proved effective in other domains. We demonstrate this idea by introducing the Point Edge Transformer (PET) architecture, which is not intrinsically equivariant but achieves state-of-the-art performance on several benchmark datasets of molecules and solids. A-posteriori application of our general protocol makes PET exactly equivariant, with minimal changes to its accuracy.
△ Less
Submitted 6 February, 2024; v1 submitted 30 May, 2023;
originally announced May 2023.
-
Wigner kernels: body-ordered equivariant machine learning without a basis
Authors:
Filippo Bigi,
Sergey N. Pozdnyakov,
Michele Ceriotti
Abstract:
Machine-learning models based on a point-cloud representation of a physical object are ubiquitous in scientific applications and particularly well-suited to the atomic-scale description of molecules and materials. Among the many different approaches that have been pursued, the description of local atomic environments in terms of their neighbor densities has been used widely and very succesfully. W…
▽ More
Machine-learning models based on a point-cloud representation of a physical object are ubiquitous in scientific applications and particularly well-suited to the atomic-scale description of molecules and materials. Among the many different approaches that have been pursued, the description of local atomic environments in terms of their neighbor densities has been used widely and very succesfully. We propose a novel density-based method which involves computing ``Wigner kernels''. These are fully equivariant and body-ordered kernels that can be computed iteratively with a cost that is independent of the radial-chemical basis and grows only linearly with the maximum body-order considered. This is in marked contrast to feature-space models, which comprise an exponentially-growing number of terms with increasing order of correlations. We present several examples of the accuracy of models based on Wigner kernels in chemical applications, for both scalar and tensorial targets, reaching state-of-the-art accuracy on the popular QM9 benchmark dataset, and we discuss the broader relevance of these ideas to equivariant geometric machine-learning.
△ Less
Submitted 7 March, 2023;
originally announced March 2023.
-
Completeness of Atomic Structure Representations
Authors:
Jigyasa Nigam,
Sergey N. Pozdnyakov,
Kevin K. Huguenin-Dumittan,
Michele Ceriotti
Abstract:
In this paper, we address the challenge of obtaining a comprehensive and symmetric representation of point particle groups, such as atoms in a molecule, which is crucial in physics and theoretical chemistry. The problem has become even more important with the widespread adoption of machine-learning techniques in science, as it underpins the capacity of models to accurately reproduce physical relat…
▽ More
In this paper, we address the challenge of obtaining a comprehensive and symmetric representation of point particle groups, such as atoms in a molecule, which is crucial in physics and theoretical chemistry. The problem has become even more important with the widespread adoption of machine-learning techniques in science, as it underpins the capacity of models to accurately reproduce physical relationships while being consistent with fundamental symmetries and conservation laws. However, some of the descriptors that are commonly used to represent point clouds -- most notably those based on discretized correlations of the neighbor density, that underpin most of the existing ML models of matter at the atomic scale -- are unable to distinguish between special arrangements of particles in three dimensions. This makes it impossible to machine learn their properties. Atom-density correlations are provably complete in the limit in which they simultaneously describe the mutual relationship between all atoms, which is impractical. We present a novel approach to construct descriptors of \emph{finite} correlations based on the relative arrangement of particle triplets, which can be employed to create symmetry-adapted models with universal approximation capabilities, which have the resolution of the neighbor discretization as the sole convergence parameter. Our strategy is demonstrated on a class of atomic arrangements that are specifically built to defy a broad class of conventional symmetric descriptors, showcasing its potential for addressing their limitations.
△ Less
Submitted 30 December, 2023; v1 submitted 28 February, 2023;
originally announced February 2023.
-
Comment on "Manifolds of quasi-constant SOAP and ACSF fingerprints and the resulting failure to machine learn four body interactions"
Authors:
Sergey N. Pozdnyakov,
Michael J. Willatt,
Albert P. Bartók,
Christoph Ortner,
Gábor Csányi,
Michele Ceriotti
Abstract:
The "quasi-constant" SOAP and ACSF fingerprint manifolds recently discovered by Parsaeifard and Goedecker are a direct consequence of the presence of degenerate pairs of configurations, a known shortcoming of all low-body-order atom-density correlation representations of molecular structures. Contrary to the configurations that are rigorously singular -- that we demonstrate can only occur in finit…
▽ More
The "quasi-constant" SOAP and ACSF fingerprint manifolds recently discovered by Parsaeifard and Goedecker are a direct consequence of the presence of degenerate pairs of configurations, a known shortcoming of all low-body-order atom-density correlation representations of molecular structures. Contrary to the configurations that are rigorously singular -- that we demonstrate can only occur in finite, discrete sets -- the continuous "quasi-constant" manifolds exhibit low, but non-zero, sensitivity to atomic displacements. Thus, it is possible to build interpolative machine-learning models of high-order interactions along the manifold, even though the numerical instabilities associated with proximity to the exact singularities affect the accuracy and transferability of such models, to an extent that depends on numerical details of the implementation.
△ Less
Submitted 2 May, 2022;
originally announced May 2022.
-
Incompleteness of graph neural networks for points clouds in three dimensions
Authors:
Sergey N. Pozdnyakov,
Michele Ceriotti
Abstract:
Graph neural networks (GNN) are very popular methods in machine learning and have been applied very successfully to the prediction of the properties of molecules and materials. First-order GNNs are well known to be incomplete, i.e., there exist graphs that are distinct but appear identical when seen through the lens of the GNN. More complicated schemes have thus been designed to increase their res…
▽ More
Graph neural networks (GNN) are very popular methods in machine learning and have been applied very successfully to the prediction of the properties of molecules and materials. First-order GNNs are well known to be incomplete, i.e., there exist graphs that are distinct but appear identical when seen through the lens of the GNN. More complicated schemes have thus been designed to increase their resolving power. Applications to molecules (and more generally, point clouds), however, add a geometric dimension to the problem. The most straightforward and prevalent approach to construct graph representation for molecules regards atoms as vertices in a graph and draws a bond between each pair of atoms within a chosen cutoff. Bonds can be decorated with the distance between atoms, and the resulting "distance graph NNs" (dGNN) have empirically demonstrated excellent resolving power and are widely used in chemical ML, with all known indistinguishable configurations being resolved in the fully-connected limit, which is equivalent to infinite or sufficiently large cutoff. Here we present a counterexample that proves that dGNNs are not complete even for the restricted case of fully-connected graphs induced by 3D atom clouds. We construct pairs of distinct point clouds whose associated graphs are, for any cutoff radius, equivalent based on a first-order Weisfeiler-Lehman test. This class of degenerate structures includes chemically-plausible configurations, both for isolated structures and for infinite structures that are periodic in 1, 2, and 3 dimensions. The existence of indistinguishable configurations sets an ultimate limit to the expressive power of some of the well-established GNN architectures for atomistic machine learning. Models that explicitly use angular or directional information in the description of atomic environments can resolve this class of degeneracies.
△ Less
Submitted 7 November, 2022; v1 submitted 18 January, 2022;
originally announced January 2022.
-
Local invertibility and sensitivity of atomic structure-feature map**s
Authors:
Sergey N. Pozdnyakov,
Liwei Zhang,
Christoph Ortner,
Gábor Csányi,
Michele Ceriotti
Abstract:
The increasingly common applications of machine-learning schemes to atomic-scale simulations have triggered efforts to better understand the mathematical properties of the map** between the Cartesian coordinates of the atoms and the variety of representations that can be used to convert them into a finite set of symmetric descriptors or features. Here, we analyze the sensitivity of the map** t…
▽ More
The increasingly common applications of machine-learning schemes to atomic-scale simulations have triggered efforts to better understand the mathematical properties of the map** between the Cartesian coordinates of the atoms and the variety of representations that can be used to convert them into a finite set of symmetric descriptors or features. Here, we analyze the sensitivity of the map** to atomic displacements, showing that the combination of symmetry and smoothness leads to map**s that have singular points at which the Jacobian has one or more null singular values (besides those corresponding to infinitesimal translations and rotations). This is in fact desirable, because it enforces physical symmetry constraints on the values predicted by regression models constructed using such representations. However, besides these symmetry-induced singularities, there are also spurious singular points, that we find to be linked to the incompleteness of the map**, i.e. the fact that, for certain classes of representations, structurally distinct configurations are not guaranteed to be mapped onto different feature vectors. Additional singularities can be introduced by a too aggressive truncation of the infinite basis set that is used to discretize the representations.
△ Less
Submitted 23 September, 2021;
originally announced September 2021.
-
On the Completeness of Atomic Structure Representations
Authors:
Sergey N. Pozdnyakov,
Michael J. Willatt,
Albert P. Bartók,
Christoph Ortner,
Gábor Csányi,
Michele Ceriotti
Abstract:
Many-body descriptors are widely used to represent atomic environments in the construction of machine learned interatomic potentials and more broadly for fitting, classification and embedding tasks on atomic structures. It was generally believed that 3-body descriptors uniquely specify the environment of an atom, up to a rotation and permutation of like atoms. We produce several counterexamples to…
▽ More
Many-body descriptors are widely used to represent atomic environments in the construction of machine learned interatomic potentials and more broadly for fitting, classification and embedding tasks on atomic structures. It was generally believed that 3-body descriptors uniquely specify the environment of an atom, up to a rotation and permutation of like atoms. We produce several counterexamples to this belief, with the consequence that any classifier, regression or embedding model for atom-centred properties that uses 3 (or 4)-body features will incorrectly give identical results for different configurations. Writing global properties (such as total energies) as a sum of many atom-centred contributions mitigates, but does not eliminate, the impact of this fundamental deficiency -- explaining the success of current "machine-learning" force fields. We anticipate the issues that will arise as the desired accuracy increases, and suggest potential solutions.
△ Less
Submitted 5 June, 2020; v1 submitted 31 January, 2020;
originally announced January 2020.
