-
Lyapunov-Guided Representation of Recurrent Neural Network Performance
Authors:
Ryan Vogt,
Yang Zheng,
Eli Shlizerman
Abstract:
Recurrent Neural Networks (RNN) are ubiquitous computing systems for sequences and multivariate time series data. While several robust architectures of RNN are known, it is unclear how to relate RNN initialization, architecture, and other hyperparameters with accuracy for a given task. In this work, we propose to treat RNN as dynamical systems and to correlate hyperparameters with accuracy through…
▽ More
Recurrent Neural Networks (RNN) are ubiquitous computing systems for sequences and multivariate time series data. While several robust architectures of RNN are known, it is unclear how to relate RNN initialization, architecture, and other hyperparameters with accuracy for a given task. In this work, we propose to treat RNN as dynamical systems and to correlate hyperparameters with accuracy through Lyapunov spectral analysis, a methodology specifically designed for nonlinear dynamical systems. To address the fact that RNN features go beyond the existing Lyapunov spectral analysis, we propose to infer relevant features from the Lyapunov spectrum with an Autoencoder and an embedding of its latent representation (AeLLE). Our studies of various RNN architectures show that AeLLE successfully correlates RNN Lyapunov spectrum with accuracy. Furthermore, the latent representation learned by AeLLE is generalizable to novel inputs from the same task and is formed early in the process of RNN training. The latter property allows for the prediction of the accuracy to which RNN would converge when training is complete. We conclude that representation of RNN through Lyapunov spectrum along with AeLLE provides a novel method for organization and interpretation of variants of RNN architectures.
△ Less
Submitted 27 December, 2023; v1 submitted 11 April, 2022;
originally announced April 2022.
-
Binary Optimal Control Of Single-Flux-Quantum Pulse Sequences
Authors:
Ryan H. Vogt,
N. Anders Petersson
Abstract:
We introduce a binary, relaxed gradient, trust-region method for optimizing pulse sequences for single flux quanta (SFQ) control of a quantum computer. The pulse sequences are optimized with the goal of realizing unitary gate transformations. Each pulse has a fixed amplitude and duration. We model this process as an binary optimal control problem, constrained by Schrödinger's equation, where the b…
▽ More
We introduce a binary, relaxed gradient, trust-region method for optimizing pulse sequences for single flux quanta (SFQ) control of a quantum computer. The pulse sequences are optimized with the goal of realizing unitary gate transformations. Each pulse has a fixed amplitude and duration. We model this process as an binary optimal control problem, constrained by Schrödinger's equation, where the binary variables indicate whether each pulse is on or off. We introduce a first-order trust-region method, which takes advantage of a relaxed gradient to determine an optimal pulse sequence that minimizes the gate infidelity, while also suppressing leakage to higher energy levels. The proposed algorithm has a computational complexity of ${\cal O}(p\log(p)$, where $p$ is the number of pulses in the sequence. We present numerical results for the H and X gates, where the optimized pulse sequences give gate fidelity's better than $99.9\%$, in $\approx 25$ trust-region iterations.
△ Less
Submitted 23 August, 2022; v1 submitted 18 June, 2021;
originally announced June 2021.
-
Solving Bang-Bang Problems Using The Immersed Interface Method and Integer Programming
Authors:
Ryan H. Vogt,
Sarah Strikwerda
Abstract:
In this paper we study numerically solving optimal control problems with bang-bang control functions. We present a formal Lagrangian approach for solving the optimal control problem, and address difficulties encountered when numerically solving the state and adjoint equations by using the immersed interface method. We note that our numerical approach does not approximate the discontinuous control…
▽ More
In this paper we study numerically solving optimal control problems with bang-bang control functions. We present a formal Lagrangian approach for solving the optimal control problem, and address difficulties encountered when numerically solving the state and adjoint equations by using the immersed interface method. We note that our numerical approach does not approximate the discontinuous control function with smooth functions, instead we solve the true bang-bang optimal control problem. Our approach for solving the optimal control problem uses an adjoint-based gradient. We use the gradient in our first-order trust-region method to generate a local minimizing control. We present detailed numerical results to demonstrate the effectiveness of our method.
△ Less
Submitted 17 April, 2021;
originally announced April 2021.
-
On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools
Authors:
Ryan Vogt,
Maximilian Puelma Touzel,
Eli Shlizerman,
Guillaume Lajoie
Abstract:
Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive to parameter initialization, architecture, and optimizer hyperparameters. Considering RNNs as dynamical systems, a natural way to capture stability, i.e., the growth and decay over long iterates, are the Lyapunov Exponents (LEs), which form the Lya…
▽ More
Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive to parameter initialization, architecture, and optimizer hyperparameters. Considering RNNs as dynamical systems, a natural way to capture stability, i.e., the growth and decay over long iterates, are the Lyapunov Exponents (LEs), which form the Lyapunov spectrum. The LEs have a bearing on stability of RNN training dynamics because forward propagation of information is related to the backward propagation of error gradients. LEs measure the asymptotic rates of expansion and contraction of nonlinear system trajectories, and generalize stability analysis to the time-varying attractors structuring the non-autonomous dynamics of data-driven RNNs. As a tool to understand and exploit stability of training dynamics, the Lyapunov spectrum fills an existing gap between prescriptive mathematical approaches of limited scope and computationally-expensive empirical approaches. To leverage this tool, we implement an efficient way to compute LEs for RNNs during training, discuss the aspects specific to standard RNN architectures driven by typical sequential datasets, and show that the Lyapunov spectrum can serve as a robust readout of training stability across hyperparameters. With this exposition-oriented contribution, we hope to draw attention to this understudied, but theoretically grounded tool for understanding training stability in RNNs.
△ Less
Submitted 24 June, 2020;
originally announced June 2020.
-
Some Remarks on Realization of Simplicial Algebras in Cat
Authors:
Zbigniew Fiedorowicz,
Rainer M. Vogt
Abstract:
In this paper we discuss why the passage from simplicial algebras over a Cat operad to algebras over that operad involves apparently unavoidable technicalities.
In this paper we discuss why the passage from simplicial algebras over a Cat operad to algebras over that operad involves apparently unavoidable technicalities.
△ Less
Submitted 21 September, 2014;
originally announced September 2014.
-
Rectification of Weak Product Algebras over an Operad in Cat and Top and Applications
Authors:
Zbigniew Fiedorowicz,
Manfred Stelzer,
Rainer M. Vogt
Abstract:
We develop an alternative to the May-Thomason construction used to compare operad based infinite loop machines to that of Segal, which relies on weak products. Our construction has the advantage that it can be carried out in $Cat$, whereas their construction gives rise to simplicial categories. As an application we show that a simplicial algebra over a $Σ$-free $Cat$ operad $\mathcal{O}$ is functo…
▽ More
We develop an alternative to the May-Thomason construction used to compare operad based infinite loop machines to that of Segal, which relies on weak products. Our construction has the advantage that it can be carried out in $Cat$, whereas their construction gives rise to simplicial categories. As an application we show that a simplicial algebra over a $Σ$-free $Cat$ operad $\mathcal{O}$ is functorially weakly equivalent to a $Cat$ algebra over $\mathcal{O}$. When combined with the results of a previous paper, this allows us to conclude that up to weak equivalences the category of $\mathcal{O}$-categories is equivalent to the category of $B\mathcal{O}$-spaces, where $B:Cat\to Top$ is the classifying space functor. In particular, $n$-fold loop spaces (and more generally $E_n$ spaces) are functorially weakly equivalent to classifying spaces of $n$-fold monoidal categories. Another application is a change of operads construction within $Cat$.
△ Less
Submitted 4 June, 2015; v1 submitted 12 November, 2013;
originally announced November 2013.
-
Homotopy homomorphisms and the classifying space functor
Authors:
R. M. Vogt
Abstract:
We show that the classifying space functor $B: Mon \to Top*$ from the category of topological monoids to the category of based spaces is left adjoint to the Moore loop space functor $Ω': Top*\to Mon$ after we have localized $Mon$ with respect to all homomorphisms whose underlying maps are homotopy equivalences and $Top*$ with respect to all based maps which are (not necessarily based) homotopy equ…
▽ More
We show that the classifying space functor $B: Mon \to Top*$ from the category of topological monoids to the category of based spaces is left adjoint to the Moore loop space functor $Ω': Top*\to Mon$ after we have localized $Mon$ with respect to all homomorphisms whose underlying maps are homotopy equivalences and $Top*$ with respect to all based maps which are (not necessarily based) homotopy equivalences. It is well-known that this localization of $Top*$ exists, and we show that the localization of $Mon$ is the category of monoids and homotopy classes of homotopy homomorphisms. To make this statement precise we have to modify the classical definition of a homotopy homomorphism, and we discuss the necessary changes. The adjunction is induced by an adjunction up to homotopy between the category of well-pointed monoids and homotopy homomorphisms and the category of well-pointed spaces. This adjunction is shown to lift to diagrams. As a consequence, the well-known derived adjunction between the homotopy colimit and the constant diagram functor can also be seen to be induced by an adjuction up to homotopy before taking homotopy classes. As applications we among other things deduce a more algebraic version of the group completion theorem and show that the classifying space functor preserves homotopy colimits up to natural homotopy equivalences.
△ Less
Submitted 25 June, 2014; v1 submitted 22 March, 2012;
originally announced March 2012.
-
Homotopy Colimits of Algebras Over Cat-Operads and Iterated Loop Spaces
Authors:
Zbigniew Fiedorowicz,
Manfred Stelzer,
Rainer M. Vogt
Abstract:
We extend Thomason's homotopy colimit construction in the category of permutative categories to categories of algebras over an arbitrary $\Cat$ operad and analyze its properties. We then use this homotopy colimit to prove that the classifying space functor induces an equivalence between the category of $n$-fold monoidal categories and the category of $\mathcal{C}_n$-spaces after formally inverting…
▽ More
We extend Thomason's homotopy colimit construction in the category of permutative categories to categories of algebras over an arbitrary $\Cat$ operad and analyze its properties. We then use this homotopy colimit to prove that the classifying space functor induces an equivalence between the category of $n$-fold monoidal categories and the category of $\mathcal{C}_n$-spaces after formally inverting certain classes of weak equivalences, where $\mathcal{C}_n$ is the little $n$-cubes operad. As a consequence we obtain an equivalence of the categories of $n$-fold monoidal categories and the category of $n$-fold loop spaces and loop maps after localization with respect to some other class of weak equivalences. We recover Thomason's corresponding result about infinite loop spaces and obtain related results about braided monoidal categories and 2-fold loop spaces.
△ Less
Submitted 30 July, 2013; v1 submitted 1 September, 2011;
originally announced September 2011.
-
An Additivity Theorem for the Interchange of E_n Structures
Authors:
Zbigniew Fiedorowicz,
Rainer M. Vogt
Abstract:
The notion of interchange of two multiplicative structures on a topological space is encoded by the tensor product of the two operads parametrizing these structures. Intuitively one might thus expect that the tensor product of an E_m and an E_n operad (which encode the muliplicative structures of m-fold, respectively n-fold loop spaces) ought to be an E_{m+n} operad. However there are easy counter…
▽ More
The notion of interchange of two multiplicative structures on a topological space is encoded by the tensor product of the two operads parametrizing these structures. Intuitively one might thus expect that the tensor product of an E_m and an E_n operad (which encode the muliplicative structures of m-fold, respectively n-fold loop spaces) ought to be an E_{m+n} operad. However there are easy counterexamples to this naive conjecture. In this paper we show that the tensor product of a cofibrant E_m operad and a cofibrant E_n operad is an E_{m+n} operad. It follows that if A_i are E_{m_i} operads for i=1,2,...,k, then there is an E_{m_1+m_2+...+m_k} operad which maps into their tensor product.
△ Less
Submitted 11 November, 2013; v1 submitted 7 February, 2011;
originally announced February 2011.
-
A model structure for coloured operads in symmetric spectra
Authors:
Javier J. Gutiérrez,
Rainer M. Vogt
Abstract:
We describe a model structure for coloured operads with values in the category of symmetric spectra (with the positive model structure), in which fibrations and weak equivalences are defined at the level of the underlying collections. This allows us to treat R-module spectra (where R is a cofibrant ring spectrum) as algebras over a cofibrant spectrum-valued operad with R as its first term. Using t…
▽ More
We describe a model structure for coloured operads with values in the category of symmetric spectra (with the positive model structure), in which fibrations and weak equivalences are defined at the level of the underlying collections. This allows us to treat R-module spectra (where R is a cofibrant ring spectrum) as algebras over a cofibrant spectrum-valued operad with R as its first term. Using this model structure, we give suficient conditions for homotopical localizations in the category of symmetric spectra to preserve module structures.
△ Less
Submitted 22 September, 2010; v1 submitted 11 June, 2010;
originally announced June 2010.
-
Associahedra and Weak Monoidal Structures on Categories
Authors:
Zbigniew Fiedorowicz,
Steven Gubkin,
Rainer M. Vogt
Abstract:
This paper answers the following question: what algebraic structure on a category corresponds to an $A_n$ structure (in the sense of Stasheff) on the geometric realization of its nerve?
This paper answers the following question: what algebraic structure on a category corresponds to an $A_n$ structure (in the sense of Stasheff) on the geometric realization of its nerve?
△ Less
Submitted 10 September, 2011; v1 submitted 21 May, 2010;
originally announced May 2010.
-
The HELP-Lemma and its converse in Quillen model categories
Authors:
R. M. Vogt
Abstract:
We show that a map between fibrant objects in a closed model category is a weak equivalence if and only if it has the right homotopy extension lifting property with respect to all cofibrations. The dual statement holds for maps between cofibrant objects.
We show that a map between fibrant objects in a closed model category is a weak equivalence if and only if it has the right homotopy extension lifting property with respect to all cofibrations. The dual statement holds for maps between cofibrant objects.
△ Less
Submitted 29 April, 2010;
originally announced April 2010.
-
Numerably Contractible Spaces
Authors:
E. Schwamberger,
R. Vogt
Abstract:
Numerably contractible spaces play an important role in the theory of homotopy pushouts and pullbacks. The corresponding results imply that a number of well known weak homotopy equivalences are genuine ones if numerably contractible spaces are involved. In this paper we give a first systematic investigation of numerably contractible spaces. We list the elementary properties of the category of th…
▽ More
Numerably contractible spaces play an important role in the theory of homotopy pushouts and pullbacks. The corresponding results imply that a number of well known weak homotopy equivalences are genuine ones if numerably contractible spaces are involved. In this paper we give a first systematic investigation of numerably contractible spaces. We list the elementary properties of the category of these spaces. We then study simplicial objects in this category. In particular, we show that the topological realization functor preserves fibration sequences if the base is path-connected and numerably contractible in each dimension. Consequently, the loop space functor commutes with realization up to homotopy. We give simple conditions which assure that free algebras over a topological operad are numerably contractible.
△ Less
Submitted 30 October, 2008;
originally announced October 2008.
-
Localization of algebras over coloured operads
Authors:
Carles Casacuberta,
Javier J. Gutierrez,
Ieke Moerdijk,
Rainer M. Vogt
Abstract:
We give sufficient conditions for homotopical localization functors to preserve algebras over coloured operads in monoidal model categories. Our approach encompasses a number of previous results about preservation of structures under localizations, such as loop spaces or infinite loop spaces, and provides new results of the same kind. For instance, under suitable assumptions, homotopical localiz…
▽ More
We give sufficient conditions for homotopical localization functors to preserve algebras over coloured operads in monoidal model categories. Our approach encompasses a number of previous results about preservation of structures under localizations, such as loop spaces or infinite loop spaces, and provides new results of the same kind. For instance, under suitable assumptions, homotopical localizations preserve ring spectra (in the strict sense, not only up to homotopy), modules over ring spectra, and algebras over commutative ring spectra, as well as ring maps, module maps, and algebra maps. It is principally the treatment of module spectra and their maps that led us to the use of coloured operads (also called enriched multicategories) in this context.
△ Less
Submitted 24 June, 2008;
originally announced June 2008.
-
On the Multiplicative Structure of Topological Hochschild Homology
Authors:
M. Brun,
Z. Fiedorowicz,
R. M. Vogt
Abstract:
We show that the topological Hochschild homology THH(R of an E_n-ring spectrum R is an E_{n-1}-ring spectrum. The proof is based on the fact that the tensor product of the operad Ass for monoid structures and the the little n-cubes operad is an E_{n+1}-operad, a result which is of independent interest.
We show that the topological Hochschild homology THH(R of an E_n-ring spectrum R is an E_{n-1}-ring spectrum. The proof is based on the fact that the tensor product of the operad Ass for monoid structures and the the little n-cubes operad is an E_{n+1}-operad, a result which is of independent interest.
△ Less
Submitted 19 June, 2007; v1 submitted 16 October, 2004;
originally announced October 2004.
-
Colimits, Stanley-Reisner algebras, and loop spaces
Authors:
Taras Panov,
Nigel Ray,
Rainer Vogt
Abstract:
We study diagrams associated with a finite simplicial complex K, in various algebraic and topological categories. We relate their colimits to familiar structures in algebra, combinatorics, geometry and topology. These include: right-angled Artin and Coxeter groups (and their complex analogues, which we call circulation groups); Stanley-Reisner algebras and coalgebras; Davis and Januszkiewicz's s…
▽ More
We study diagrams associated with a finite simplicial complex K, in various algebraic and topological categories. We relate their colimits to familiar structures in algebra, combinatorics, geometry and topology. These include: right-angled Artin and Coxeter groups (and their complex analogues, which we call circulation groups); Stanley-Reisner algebras and coalgebras; Davis and Januszkiewicz's spaces DJ(K) associated with toric manifolds and their generalisations; and coordinate subspace arrangements. When K is a flag complex, we extend well-known results on Artin and Coxeter groups by confirming that the relevant circulation group is homotopy equivalent to the space of loops $ΩDJ(K)$. We define homotopy colimits for diagrams of topological monoids and topological groups, and show they commute with the formation of classifying spaces in a suitably generalised sense. We deduce that the homotopy colimit of the appropriate diagram of topological groups is a model for $ΩDJ(K)$ for an arbitrary complex K, and that the natural projection onto the original colimit is a homotopy equivalence when K is flag. In this case, the two models are compatible.
△ Less
Submitted 9 February, 2002;
originally announced February 2002.
-
Iterated Monoidal Categories
Authors:
C. Balteanu,
Z. Fiedorowicz,
R. Schwaenzl,
R. Vogt
Abstract:
We develop a notion of iterated monoidal category and show that this notion corresponds in a precise way to the notion of iterated loop space. Specifically the group completion of the nerve of such a category is an iterated loop space and free iterated monoidal categories give rise to finite simplicial operads of the same homotopy type as the classical little cubes operads used to parametrize th…
▽ More
We develop a notion of iterated monoidal category and show that this notion corresponds in a precise way to the notion of iterated loop space. Specifically the group completion of the nerve of such a category is an iterated loop space and free iterated monoidal categories give rise to finite simplicial operads of the same homotopy type as the classical little cubes operads used to parametrize the higher H-space structure of iterated loop spaces. Iterated monoidal categories encompass, as a special case, the notion of braided tensor categories, as used in the theory of quantum groups.
△ Less
Submitted 18 August, 1998;
originally announced August 1998.
