-
Unnatural Algorithms in Machine Learning
Authors:
Christian Goodbrake
Abstract:
Natural gradient descent has a remarkable property that in the small learning rate limit, it displays an invariance with respect to network reparameterizations, leading to robust training behavior even for highly covariant network parameterizations. We show that optimization algorithms with this property can be viewed as discrete approximations of natural transformations from the functor determini…
▽ More
Natural gradient descent has a remarkable property that in the small learning rate limit, it displays an invariance with respect to network reparameterizations, leading to robust training behavior even for highly covariant network parameterizations. We show that optimization algorithms with this property can be viewed as discrete approximations of natural transformations from the functor determining an optimizer's state space from the diffeomorphism group if its configuration manifold, to the functor determining that state space's tangent bundle from this group. Algorithms with this property enjoy greater efficiency when used to train poorly parameterized networks, as the network evolution they generate is approximately invariant to network reparameterizations. More specifically, the flow generated by these algorithms in the limit as the learning rate vanishes is invariant under smooth reparameterizations, the respective flows of the parameters being determined by equivariant maps. By casting this property a natural transformation, we allow for generalizations beyond equivariance with respect to group actions; this framework can account for non-invertible maps such as projections, creating a framework for the direct comparison of training behavior across non-isomorphic network architectures, and the formal examination of limiting behavior as network size increases by considering inverse limits of these projections, should they exist. We introduce a simple method of introducing this naturality more generally and examine a number of popular machine learning training algorithms, finding that most are unnatural.
△ Less
Submitted 7 December, 2023;
originally announced December 2023.
-
Brain Chains as Topological Signatures for Alzheimer's Disease
Authors:
Christian Goodbrake,
David Beers,
Travis B. Thompson,
Heather A. Harrington,
Alain Goriely
Abstract:
We propose a topological framework to study the evolution of Alzheimer's disease, the most common neurodegenerative disease. The modeling of this disease starts with the representation of the brain connectivity as a graph and the seeding of a toxic protein in a specific region represented by a vertex. Over time, the accumulation of toxic proteins at vertices and their propagation along edges are m…
▽ More
We propose a topological framework to study the evolution of Alzheimer's disease, the most common neurodegenerative disease. The modeling of this disease starts with the representation of the brain connectivity as a graph and the seeding of a toxic protein in a specific region represented by a vertex. Over time, the accumulation of toxic proteins at vertices and their propagation along edges are modeled by a dynamical system on this graph. These dynamics provide an order on the edges of the graph according to the damage created by high concentrations of proteins. This sequence of edges defines a filtration of the graph. We consider different filtrations given by different disease seeding locations. To study this filtration we propose a new combinatorial and topological method. A filtration defines a maximal chain in the partially ordered set of spanning subgraphs ordered by inclusion. To identify similar graphs, and define a topological signature, we quotient this poset by graph homotopy equivalence, which gives maximal chains in a smaller poset. We provide an algorithm to compute this direct quotient without computing all subgraphs and then propose bounds on the total number of graphs up to homotopy equivalence. To compare the maximal chains generated by this method, we extend Kendall's $d_K$ metric for permutations to more general graded posets and establish bounds for this metric. We then demonstrate the utility of this framework on actual brain graphs by studying the dynamics of tau proteins on the structural connectome. {We show that the proposed topological brain chain equivalence classes distinguish different simulated subtypes of Alzheimer's disease.
△ Less
Submitted 18 September, 2023; v1 submitted 22 August, 2022;
originally announced August 2022.
-
Differential elimination for dynamical models via projections with applications to structural identifiability
Authors:
Ruiwen Dong,
Christian Goodbrake,
Heather A Harrington,
Gleb Pogudin
Abstract:
Elimination of unknowns in a system of differential equations is often required when analysing (possibly nonlinear) dynamical systems models, where only a subset of variables are observable. One such analysis, identifiability, often relies on computing input-output relations via differential algebraic elimination. Determining identifiability, a natural prerequisite for meaningful parameter estimat…
▽ More
Elimination of unknowns in a system of differential equations is often required when analysing (possibly nonlinear) dynamical systems models, where only a subset of variables are observable. One such analysis, identifiability, often relies on computing input-output relations via differential algebraic elimination. Determining identifiability, a natural prerequisite for meaningful parameter estimation, is often prohibitively expensive for medium to large systems due to the computationally expensive task of elimination.
We propose an algorithm that computes a description of the set of differential-algebraic relations between the input and output variables of a dynamical system model. The resulting algorithm outperforms general-purpose software for differential elimination on a set of benchmark models from literature.
We use the designed elimination algorithm to build a new randomized algorithm for assessing structural identifiability of a parameter in a parametric model. A parameter is said to be identifiable if its value can be uniquely determined from input-output data assuming the absence of noise and sufficiently exciting inputs. Our new algorithm allows the identification of models that could not be tackled before.
Our implementation is publicly available as a Julia package at https://github.com/SciML/StructuralIdentifiability.jl.
△ Less
Submitted 23 November, 2022; v1 submitted 1 November, 2021;
originally announced November 2021.
