-
Analysis of the Geometric Structure of Neural Networks and Neural ODEs via Morse Functions
Authors:
Christian Kuehn,
Sara-Viola Kuntz
Abstract:
Besides classical feed-forward neural networks, also neural ordinary differential equations (neural ODEs) gained particular interest in recent years. Neural ODEs can be interpreted as an infinite depth limit of feed-forward or residual neural networks. We study the input-output dynamics of finite and infinite depth neural networks with scalar output. In the finite depth case, the input is a state…
▽ More
Besides classical feed-forward neural networks, also neural ordinary differential equations (neural ODEs) gained particular interest in recent years. Neural ODEs can be interpreted as an infinite depth limit of feed-forward or residual neural networks. We study the input-output dynamics of finite and infinite depth neural networks with scalar output. In the finite depth case, the input is a state associated to a finite number of nodes, which maps under multiple non-linear transformations to the state of one output node. In analogy, a neural ODE maps a linear transformation of the input to a linear transformation of its time-$T$ map. We show that depending on the specific structure of the network, the input-output map has different properties regarding the existence and regularity of critical points. These properties can be characterized via Morse functions, which are scalar functions, where every critical point is non-degenerate. We prove that critical points cannot exist, if the dimension of the hidden layer is monotonically decreasing or the dimension of the phase space is smaller or equal to the input dimension. In the case that critical points exist, we classify their regularity depending on the specific architecture of the network. We show that each critical point is non-degenerate, if for finite depth neural networks the underlying graph has no bottleneck, and if for neural ODEs, the linear transformations used have full rank. For each type of architecture, the proven properties are comparable in the finite and in the infinite depth case. The established theorems allow us to formulate results on universal embedding, i.e.\ on the exact representation of maps by neural networks and neural ODEs. Our dynamical systems viewpoint on the geometric structure of the input-output map provides a fundamental understanding, why certain architectures perform better than others.
△ Less
Submitted 15 May, 2024;
originally announced May 2024.
-
Embedding Capabilities of Neural ODEs
Authors:
Christian Kuehn,
Sara-Viola Kuntz
Abstract:
A class of neural networks that gained particular interest in the last years are neural ordinary differential equations (neural ODEs). We study input-output relations of neural ODEs using dynamical systems theory and prove several results about the exact embedding of maps in different neural ODE architectures in low and high dimension. The embedding capability of a neural ODE architecture can be i…
▽ More
A class of neural networks that gained particular interest in the last years are neural ordinary differential equations (neural ODEs). We study input-output relations of neural ODEs using dynamical systems theory and prove several results about the exact embedding of maps in different neural ODE architectures in low and high dimension. The embedding capability of a neural ODE architecture can be increased by adding, for example, a linear layer, or augmenting the phase space. Yet, there is currently no systematic theory available and our work contributes towards this goal by develo** various embedding results as well as identifying situations, where no embedding is possible. The mathematical techniques used include as main components iterative functional equations, Morse functions and suspension flows, as well as several further ideas from analysis. Although practically, mainly universal approximation theorems are used, our geometric dynamical systems viewpoint on universal embedding provides a fundamental understanding, why certain neural ODE architectures perform better than others.
△ Less
Submitted 28 September, 2023; v1 submitted 2 August, 2023;
originally announced August 2023.
-
Concurrent Graph Queries on the Lucata Pathfinder
Authors:
Emory Smith,
Shannon Kuntz,
Jason Riedy,
Martin Deneroff
Abstract:
High-performance analysis of unstructured data like graphs now is critical for applications ranging from business intelligence to genome analysis. Towards this, data centers hold large graphs in memory to serve multiple concurrent queries from different users. Even a single analysis often explores multiple options. Current computing architectures often are not the most time- or energy-efficient so…
▽ More
High-performance analysis of unstructured data like graphs now is critical for applications ranging from business intelligence to genome analysis. Towards this, data centers hold large graphs in memory to serve multiple concurrent queries from different users. Even a single analysis often explores multiple options. Current computing architectures often are not the most time- or energy-efficient solutions. The novel Lucata Pathfinder architecture tackles this problem, combining migratory threads for low-latency reading with memory-side processing for high-performance accumulation. One hundred to 750 concurrent breadth-first searches (BFS) all achieve end-to-end speed-ups of 81% to 97% over one-at-a-time queries on a graph with 522M edges. Comparing to RedisGraph running on a large Intel-based server, the Pathfinder achieves a 19$\times$ speed-up running 128 BFS queries concurrently. The Pathfinder also efficiently supports a mix of concurrent analyses, demonstrated with connected components and BFS.
△ Less
Submitted 23 September, 2022;
originally announced September 2022.
