-
Automated Inference of Graph Transformation Rules
Authors:
Jakob L. Andersen,
Akbar Davoodi,
Rolf Fagerberg,
Christoph Flamm,
Walter Fontana,
Juri Kolčák,
Christophe V. F. P. Laurent,
Daniel Merkle,
Nikolai Nøjgaard
Abstract:
The explosion of data available in life sciences is fueling an increasing demand for expressive models and computational methods. Graph transformation is a model for dynamic systems with a large variety of applications. We introduce a novel method of the graph transformation model construction, combining generative and dynamical viewpoints to give a fully automated data-driven model inference meth…
▽ More
The explosion of data available in life sciences is fueling an increasing demand for expressive models and computational methods. Graph transformation is a model for dynamic systems with a large variety of applications. We introduce a novel method of the graph transformation model construction, combining generative and dynamical viewpoints to give a fully automated data-driven model inference method.
The method takes the input dynamical properties, given as a "snapshot" of the dynamics encoded by explicit transitions, and constructs a compatible model. The obtained model is guaranteed to be minimal, thus framing the approach as model compression (from a set of transitions into a set of rules). The compression is permissive to a lossy case, where the constructed model is allowed to exhibit behavior outside of the input transitions, thus suggesting a completion of the input dynamics.
The task of graph transformation model inference is naturally highly challenging due to the combinatorics involved. We tackle the exponential explosion by proposing a heuristically minimal translation of the task into a well-established problem, set cover, for which highly optimized solutions exist. We further showcase how our results relate to Kolmogorov complexity expressed in terms of graph transformation.
△ Less
Submitted 3 April, 2024;
originally announced April 2024.
-
Representing catalytic mechanisms with rule composition
Authors:
Jakob L. Andersen,
Rolf Fagerberg,
Christoph Flamm,
Walter Fontana,
Juri Kolčák,
Christophe V. F. P. Laurent,
Daniel Merkle,
Nikolai Nøjgaard
Abstract:
Reaction mechanisms are often presented as sequences of elementary steps, such as codified by arrow pushing. We propose an approach for representing such mechanisms using graph transformation. In this framework, each elementary step is a rule for modifying a molecular graph and a mechanism is a sequence of such rules. To generate a compact representation of a multi-step reaction, we compose the ru…
▽ More
Reaction mechanisms are often presented as sequences of elementary steps, such as codified by arrow pushing. We propose an approach for representing such mechanisms using graph transformation. In this framework, each elementary step is a rule for modifying a molecular graph and a mechanism is a sequence of such rules. To generate a compact representation of a multi-step reaction, we compose the rules of individual steps into a composite rule, providing a rigorous and fully automated approach to coarse-graining. While the composite rule retains the graphical conditions necessary for the execution of a mechanism, it also records information about transient changes not visible by comparing educts and products. By projecting the rule onto a single "overlay graph", we generalize Fujita's idea of an Imaginary Transition Structure from elementary reactions to composite reactions. The utility of the overlay graph construct is exemplified in the context of enzyme-catalyzed reactions. In a first application, we exploit mechanistic information in the Mechanism and Catalytic Site Atlas to construct overlay graphs of hydrolase reactions listed in the database. These graphs point at a spectrum of catalytic entanglement of enzyme and substrate, de-emphasizing the notion of a singular catalyst in favor of a collection of catalytic sites that can be distributed across enzyme and substrate. In a second application, we deploy composite rules to search the Rhea database for reactions of known or unknown mechanism that are, in principle, compatible with the mechanisms implied by the composite rules. We believe this work adds to the utility of graph-transformation formalisms in representing and reasoning about chemistry in an automated yet insightful fashion.
△ Less
Submitted 25 August, 2022; v1 submitted 12 January, 2022;
originally announced January 2022.
-
Efficient Modular Graph Transformation Rule Application
Authors:
Jakob L. Andersen,
Rolf Fagerberg,
Juri Kolčák,
Christophe V. F. P. Laurent,
Daniel Merkle,
Nikolai Nøjgaard
Abstract:
Graph transformation formalisms have proven to be suitable tools for the modelling of chemical reactions. They are well established in theoretical studies and increasingly also in practical applications in chemistry. The latter is made feasible via the development of programming frameworks which makes the formalisms executable.
The application of such frameworks to large networks of chemical rea…
▽ More
Graph transformation formalisms have proven to be suitable tools for the modelling of chemical reactions. They are well established in theoretical studies and increasingly also in practical applications in chemistry. The latter is made feasible via the development of programming frameworks which makes the formalisms executable.
The application of such frameworks to large networks of chemical reactions, however, poses unique computational challenges. One such characteristic is the inherent combinatorial nature of the graphs involved. The graphs consist of many connected components, representing individual molecules. While the existing methods for implementing graph transformations can be applied to such graphs, the combinatorics of constructing graph matches quickly becomes a computational bottleneck as the size of the chemical reaction network grows.
In this contribution, we develop a new method of enumerating graph matches during graph transformation rule application. The method is designed to improve performance in such scenarios and is based on constructing graph matches in an iterative, component-wise fashion which allows redundant applications to be detected early and pruned. We further extend the algorithm with an efficient heuristic based on local symmetries of the graphs, which allow us to detect and discard isomorphic applications early. Finally, we conduct chemical network generation experiments on real-life as well as synthetic data and compare against the state-of-the-art algorithm in the field.
△ Less
Submitted 25 August, 2022; v1 submitted 12 January, 2022;
originally announced January 2022.
-
Low-Order Modeling and High-Fidelity Simulations for the Prediction of Combustion Instabilities in Liquid Rocket Engines and Gas Turbines
Authors:
Charlelie Laurent
Abstract:
Combustion instabilities are a major concern in the design of Liquid Rocket Engines (LREs) and gas turbines. During this PhD work, several directions were explored to understand and mitigate their effects. First, more efficient and robust numerical methods for their prediction in complex combustors were designed. In this matter, a novel type of modal expansion, named a frame expansion and comparab…
▽ More
Combustion instabilities are a major concern in the design of Liquid Rocket Engines (LREs) and gas turbines. During this PhD work, several directions were explored to understand and mitigate their effects. First, more efficient and robust numerical methods for their prediction in complex combustors were designed. In this matter, a novel type of modal expansion, named a frame expansion and comparable to the classical Galerkin expansion, was introduced to build more accurate acoustic Low-Order Models (LOMs), able to account for the full geometrical complexity of industrial combustors. In particular, the frame expansion is able to accurately represent the acoustic velocity field near non-rigid-wall boundaries, a crucial ability that the Galerkin method lacks. An entire class of novel numerical methods, based on the frame expansion, were then designed and combined with the state-space formalism to build acoustic networks of complex systems. The second ingredient in the prediction of thermoacoustic instabilities is the flame dynamics modeling. This work dealt with this problem, in the specific case of a cryogenic coaxial jet-flame characteristic of a LRE. Flame dynamics driving phenomena were identified thanks to three-dimensional Large Eddy Simulations (LES) of the Mascotte experimental test rig where both reactants (CH4 and O2) are injected in transcritical conditions. Several LES with harmonic modulation of the fuel inflow at various frequencies and amplitudes were performed in order to evaluate the flame response to acoustic oscillations and compute a Flame Transfer Function (FTF). The stabilization of this flame in the near-injector region, which is of primary importance on the overall flame dynamics, was also investigated thanks to multi-physics two-dimensional Direct Numerical Simulations (DNS), where a conjugate heat transfer problem is resolved at the injector lip.
△ Less
Submitted 1 March, 2021;
originally announced March 2021.
-
Graph Transformation for Enzymatic Mechanisms
Authors:
Jakob L. Andersen,
Rolf Fagerberg,
Christoph Flamm,
Walter Fontana,
Juraj Kolčák,
Christophe V. F. P. Laurent,
Daniel Merkle,
Nikolai Nøjaard
Abstract:
Motivation: The design of enzymes is as challenging as it is consequential for making chemical synthesis in medical and industrial applications more efficient, cost-effective and environmentally friendly. While several aspects of this complex problem are computationally assisted, the drafting of catalytic mechanisms, i.e. the specification of the chemical steps-and hence intermediate states-that t…
▽ More
Motivation: The design of enzymes is as challenging as it is consequential for making chemical synthesis in medical and industrial applications more efficient, cost-effective and environmentally friendly. While several aspects of this complex problem are computationally assisted, the drafting of catalytic mechanisms, i.e. the specification of the chemical steps-and hence intermediate states-that the enzyme is meant to implement, is largely left to human expertise. The ability to capture specific chemistries of multi-step catalysis in a fashion that enables its computational construction and design is therefore highly desirable and would equally impact the elucidation of existing enzymatic reactions whose mechanisms are unknown. Results: We use the mathematical framework of graph transformation to express the distinction between rules and reactions in chemistry. We derive about 1000 rules for amino acid side chain chemistry from the M-CSA database, a curated repository of enzymatic mechanisms. Using graph transformation we are able to propose hundreds of hypothetical catalytic mechanisms for a large number of unrelated reactions in the Rhea database. We analyze these mechanisms to find that they combine in chemically sound fashion individual steps from a variety of known multi-step mechanisms, showing that plausible novel mechanisms for catalysis can be constructed computationally.
△ Less
Submitted 26 March, 2021; v1 submitted 5 February, 2021;
originally announced February 2021.
-
Implicit Regularization via Neural Feature Alignment
Authors:
Aristide Baratin,
Thomas George,
César Laurent,
R Devon Hjelm,
Guillaume Lajoie,
Pascal Vincent,
Simon Lacoste-Julien
Abstract:
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a regularization effect induced by a dynamical alignment of the neural tangent features introduced by Jacot et al, along a small number of task-relevant directions. This can be interpreted as a combined mechanism of feature selection and compression. By extrapolating a new analysis of Rad…
▽ More
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a regularization effect induced by a dynamical alignment of the neural tangent features introduced by Jacot et al, along a small number of task-relevant directions. This can be interpreted as a combined mechanism of feature selection and compression. By extrapolating a new analysis of Rademacher complexity bounds for linear models, we motivate and study a heuristic complexity measure that captures this phenomenon, in terms of sequences of tangent kernel classes along optimization paths.
△ Less
Submitted 16 March, 2021; v1 submitted 3 August, 2020;
originally announced August 2020.
-
Revisiting Loss Modelling for Unstructured Pruning
Authors:
César Laurent,
Camille Ballas,
Thomas George,
Nicolas Ballas,
Pascal Vincent
Abstract:
By removing parameters from deep neural networks, unstructured pruning methods aim at cutting down memory footprint and computational cost, while maintaining prediction accuracy. In order to tackle this otherwise intractable problem, many of these methods model the loss landscape using first or second order Taylor expansions to identify which parameters can be discarded. We revisit loss modelling…
▽ More
By removing parameters from deep neural networks, unstructured pruning methods aim at cutting down memory footprint and computational cost, while maintaining prediction accuracy. In order to tackle this otherwise intractable problem, many of these methods model the loss landscape using first or second order Taylor expansions to identify which parameters can be discarded. We revisit loss modelling for unstructured pruning: we show the importance of ensuring locality of the pruning steps. We systematically compare first and second order Taylor expansions and empirically show that both can reach similar levels of performance. Finally, we show that better preserving the original network function does not necessarily transfer to better performing networks after fine-tuning, suggesting that only considering the impact of pruning on the loss might not be a sufficient objective to design good pruning criteria.
△ Less
Submitted 22 June, 2020;
originally announced June 2020.
-
Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
Authors:
Thomas George,
César Laurent,
Xavier Bouthillier,
Nicolas Ballas,
Pascal Vincent
Abstract:
Optimization algorithms that leverage gradient covariance information, such as variants of natural gradient descent (Amari, 1998), offer the prospect of yielding more effective descent directions. For models with many parameters, the covariance matrix they are based on becomes gigantic, making them inapplicable in their original form. This has motivated research into both simple diagonal approxima…
▽ More
Optimization algorithms that leverage gradient covariance information, such as variants of natural gradient descent (Amari, 1998), offer the prospect of yielding more effective descent directions. For models with many parameters, the covariance matrix they are based on becomes gigantic, making them inapplicable in their original form. This has motivated research into both simple diagonal approximations and more sophisticated factored approximations such as KFAC (Heskes, 2000; Martens & Grosse, 2015; Grosse & Martens, 2016). In the present work we draw inspiration from both to propose a novel approximation that is provably better than KFAC and amendable to cheap partial updates. It consists in tracking a diagonal variance, not in parameter coordinates, but in a Kronecker-factored eigenbasis, in which the diagonal approximation is likely to be more effective. Experiments show improvements over KFAC in optimization speed for several deep network architectures.
△ Less
Submitted 26 July, 2021; v1 submitted 11 June, 2018;
originally announced June 2018.
-
Theano: A Python framework for fast computation of mathematical expressions
Authors:
The Theano Development Team,
Rami Al-Rfou,
Guillaume Alain,
Amjad Almahairi,
Christof Angermueller,
Dzmitry Bahdanau,
Nicolas Ballas,
Frédéric Bastien,
Justin Bayer,
Anatoly Belikov,
Alexander Belopolsky,
Yoshua Bengio,
Arnaud Bergeron,
James Bergstra,
Valentin Bisson,
Josh Bleecher Snyder,
Nicolas Bouchard,
Nicolas Boulanger-Lewandowski,
Xavier Bouthillier,
Alexandre de Brébisson,
Olivier Breuleux,
Pierre-Luc Carrier,
Kyunghyun Cho,
Jan Chorowski,
Paul Christiano
, et al. (88 additional authors not shown)
Abstract:
Theano is a Python library that allows to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. Since its introduction, it has been one of the most used CPU and GPU mathematical compilers - especially in the machine learning community - and has shown steady performance improvements. Theano is being actively and continuously developed since 2008, mu…
▽ More
Theano is a Python library that allows to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. Since its introduction, it has been one of the most used CPU and GPU mathematical compilers - especially in the machine learning community - and has shown steady performance improvements. Theano is being actively and continuously developed since 2008, multiple frameworks have been built on top of it and it has been used to produce many state-of-the-art machine learning models.
The present article is structured as follows. Section I provides an overview of the Theano software and its community. Section II presents the principal features of Theano and how to use them, and compares them with other similar projects. Section III focuses on recently-introduced functionalities and improvements. Section IV compares the performance of Theano against Torch7 and TensorFlow on several machine learning models. Section V discusses current limitations of Theano and potential ways of improving it.
△ Less
Submitted 9 May, 2016;
originally announced May 2016.
-
Recurrent Batch Normalization
Authors:
Tim Cooijmans,
Nicolas Ballas,
César Laurent,
Çağlar Gülçehre,
Aaron Courville
Abstract:
We propose a reparameterization of LSTM that brings the benefits of batch normalization to recurrent neural networks. Whereas previous works only apply batch normalization to the input-to-hidden transformation of RNNs, we demonstrate that it is both possible and beneficial to batch-normalize the hidden-to-hidden transition, thereby reducing internal covariate shift between time steps. We evaluate…
▽ More
We propose a reparameterization of LSTM that brings the benefits of batch normalization to recurrent neural networks. Whereas previous works only apply batch normalization to the input-to-hidden transformation of RNNs, we demonstrate that it is both possible and beneficial to batch-normalize the hidden-to-hidden transition, thereby reducing internal covariate shift between time steps. We evaluate our proposal on various sequential problems such as sequence classification, language modeling and question answering. Our empirical results show that our batch-normalized LSTM consistently leads to faster convergence and improved generalization.
△ Less
Submitted 27 February, 2017; v1 submitted 29 March, 2016;
originally announced March 2016.
-
Batch Normalized Recurrent Neural Networks
Authors:
César Laurent,
Gabriel Pereyra,
Philémon Brakel,
Ying Zhang,
Yoshua Bengio
Abstract:
Recurrent Neural Networks (RNNs) are powerful models for sequential data that have the potential to learn long-term dependencies. However, they are computationally expensive to train and difficult to parallelize. Recent work has shown that normalizing intermediate representations of neural networks can significantly improve convergence rates in feedforward neural networks . In particular, batch no…
▽ More
Recurrent Neural Networks (RNNs) are powerful models for sequential data that have the potential to learn long-term dependencies. However, they are computationally expensive to train and difficult to parallelize. Recent work has shown that normalizing intermediate representations of neural networks can significantly improve convergence rates in feedforward neural networks . In particular, batch normalization, which uses mini-batch statistics to standardize features, was shown to significantly reduce training time. In this paper, we show that applying batch normalization to the hidden-to-hidden transitions of our RNNs doesn't help the training procedure. We also show that when applied to the input-to-hidden transitions, batch normalization can lead to a faster convergence of the training criterion but doesn't seem to improve the generalization performance on both our language modelling and speech recognition tasks. All in all, applying batch normalization to RNNs turns out to be more challenging than applying it to feedforward networks, but certain variants of it can still be beneficial.
△ Less
Submitted 5 October, 2015;
originally announced October 2015.
