-
Moment matching based reduced closed-loop design to achieve asymptotic performance
Authors:
Tudor C. Ionescu
Abstract:
In this paper, the moment matching techniques are adopted to obtain reduced-order closed-loop systems with reduced-order controllers that maintain the closed-loop stability and guarantee desired asymptotic performance, after revealing the relationship between the Internal Model Principle used in control design and the time-domain moment matching problem. As a result, the design of a low order cont…
▽ More
In this paper, the moment matching techniques are adopted to obtain reduced-order closed-loop systems with reduced-order controllers that maintain the closed-loop stability and guarantee desired asymptotic performance, after revealing the relationship between the Internal Model Principle used in control design and the time-domain moment matching problem. As a result, the design of a low order controller can be done starting from considering the achieving of asymptotic performance as a moment matching problem, resulting in a reduced order closed-loop system.
△ Less
Submitted 3 May, 2024;
originally announced May 2024.
-
About the Cohen-Macaulay defect and almost Cohen-Macaulay rings
Authors:
Cristodor Ionescu
Abstract:
We notice the connection between almost Cohen-Macaulay rings and the Cohen-Macaulay defect. We introduce a Serre-type condition for modules, that is connected to the Cohen-Macaulay defect in the same way that the condition $(S_n)$ is connected to Cohen-Macaulay modules.
We notice the connection between almost Cohen-Macaulay rings and the Cohen-Macaulay defect. We introduce a Serre-type condition for modules, that is connected to the Cohen-Macaulay defect in the same way that the condition $(S_n)$ is connected to Cohen-Macaulay modules.
△ Less
Submitted 9 February, 2024;
originally announced February 2024.
-
Learning from One Continuous Video Stream
Authors:
João Carreira,
Michael King,
Viorica Pătrăucean,
Dilara Gokay,
Cătălin Ionescu,
Yi Yang,
Daniel Zoran,
Joseph Heyward,
Carl Doersch,
Yusuf Aytar,
Dima Damen,
Andrew Zisserman
Abstract:
We introduce a framework for online learning from a single continuous video stream -- the way people and animals learn, without mini-batches, data augmentation or shuffling. This poses great challenges given the high correlation between consecutive video frames and there is very little prior work on it. Our framework allows us to do a first deep dive into the topic and includes a collection of str…
▽ More
We introduce a framework for online learning from a single continuous video stream -- the way people and animals learn, without mini-batches, data augmentation or shuffling. This poses great challenges given the high correlation between consecutive video frames and there is very little prior work on it. Our framework allows us to do a first deep dive into the topic and includes a collection of streams and tasks composed from two existing video datasets, plus methodology for performance evaluation that considers both adaptation and generalization. We employ pixel-to-pixel modelling as a practical and flexible way to switch between pre-training and single-stream evaluation as well as between arbitrary tasks, without ever requiring changes to models and always using the same pixel loss. Equipped with this framework we obtained large single-stream learning gains from pre-training with a novel family of future prediction tasks, found that momentum hurts, and that the pace of weight updates matters. The combination of these insights leads to matching the performance of IID learning with batch size 1, when using the same architecture and without costly replay buffers.
△ Less
Submitted 28 March, 2024; v1 submitted 1 December, 2023;
originally announced December 2023.
-
Time-Domain Moment Matching for Second-Order Systems
Authors:
Xiaodong Cheng,
Tudor C. Ionescu,
Monica Pătraşcu
Abstract:
This paper studies a structure-preserving model reduction problem for large-scale second-order dynamical systems via the framework of time-domain moment matching. The moments of a second-order system are interpreted as the solutions of second-order Sylvester equations, which leads to families of parameterized second-order reduced models that match the moments of an original second-order system at…
▽ More
This paper studies a structure-preserving model reduction problem for large-scale second-order dynamical systems via the framework of time-domain moment matching. The moments of a second-order system are interpreted as the solutions of second-order Sylvester equations, which leads to families of parameterized second-order reduced models that match the moments of an original second-order system at selected interpolation points. Based on this, a two-sided moment matching problem is addressed, providing a unique second-order reduced system that match two distinct sets interpolation points. Furthermore, we also construct the reduced second-order systems that matches the moments of both zero and first order derivative of the original second-order system. Finally, the Loewner framework is extended to the second-order systems, where two parameterized families of models are presented that retain the second-order structure and interpolate sets of tangential data.
△ Less
Submitted 2 May, 2023;
originally announced May 2023.
-
HiP: Hierarchical Perceiver
Authors:
Joao Carreira,
Skanda Koppula,
Daniel Zoran,
Adria Recasens,
Catalin Ionescu,
Olivier Henaff,
Evan Shelhamer,
Relja Arandjelovic,
Matt Botvinick,
Oriol Vinyals,
Karen Simonyan,
Andrew Zisserman,
Andrew Jaegle
Abstract:
General perception systems such as Perceivers can process arbitrary modalities in any combination and are able to handle up to a few hundred thousand inputs. They achieve this generality by using exclusively global attention operations. This however hinders them from scaling up to the inputs sizes required to process raw high-resolution images or video. In this paper, we show that some degree of l…
▽ More
General perception systems such as Perceivers can process arbitrary modalities in any combination and are able to handle up to a few hundred thousand inputs. They achieve this generality by using exclusively global attention operations. This however hinders them from scaling up to the inputs sizes required to process raw high-resolution images or video. In this paper, we show that some degree of locality can be introduced back into these models, greatly improving their efficiency while preserving their generality. To scale them further, we introduce a self-supervised approach that enables learning dense low-dimensional positional embeddings for very large signals. We call the resulting model a Hierarchical Perceiver (HiP). In sum our contributions are: 1) scaling Perceiver-type models to raw high-resolution images and audio+video, 2) showing the feasibility of learning 1M+ positional embeddings from scratch using masked auto-encoding, 3) demonstrating competitive performance on raw data from ImageNet, AudioSet, PASCAL VOC, ModelNet40 and Kinetics datasets with the same exact, unchanged model and without specialized preprocessing or any tokenization.
△ Less
Submitted 3 November, 2022; v1 submitted 22 February, 2022;
originally announced February 2022.
-
ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions
Authors:
Zheng Li,
Yue Zhao,
Xiyang Hu,
Nicola Botta,
Cezar Ionescu,
George H. Chen
Abstract:
Outlier detection refers to the identification of data points that deviate from a general data distribution. Existing unsupervised approaches often suffer from high computational cost, complex hyperparameter tuning, and limited interpretability, especially when working with large, high-dimensional datasets. To address these issues, we present a simple yet effective algorithm called ECOD (Empirical…
▽ More
Outlier detection refers to the identification of data points that deviate from a general data distribution. Existing unsupervised approaches often suffer from high computational cost, complex hyperparameter tuning, and limited interpretability, especially when working with large, high-dimensional datasets. To address these issues, we present a simple yet effective algorithm called ECOD (Empirical-Cumulative-distribution-based Outlier Detection), which is inspired by the fact that outliers are often the "rare events" that appear in the tails of a distribution. In a nutshell, ECOD first estimates the underlying distribution of the input data in a nonparametric fashion by computing the empirical cumulative distribution per dimension of the data. ECOD then uses these empirical distributions to estimate tail probabilities per dimension for each data point. Finally, ECOD computes an outlier score of each data point by aggregating estimated tail probabilities across dimensions. Our contributions are as follows: (1) we propose a novel outlier detection method called ECOD, which is both parameter-free and easy to interpret; (2) we perform extensive experiments on 30 benchmark datasets, where we find that ECOD outperforms 11 state-of-the-art baselines in terms of accuracy, efficiency, and scalability; and (3) we release an easy-to-use and scalable (with distributed support) Python implementation for accessibility and reproducibility.
△ Less
Submitted 24 August, 2022; v1 submitted 2 January, 2022;
originally announced January 2022.
-
Perceiver IO: A General Architecture for Structured Inputs & Outputs
Authors:
Andrew Jaegle,
Sebastian Borgeaud,
Jean-Baptiste Alayrac,
Carl Doersch,
Catalin Ionescu,
David Ding,
Skanda Koppula,
Daniel Zoran,
Andrew Brock,
Evan Shelhamer,
Olivier Hénaff,
Matthew M. Botvinick,
Andrew Zisserman,
Oriol Vinyals,
Joāo Carreira
Abstract:
A central goal of machine learning is the development of systems that can solve many problems in as many data domains as possible. Current architectures, however, cannot be applied beyond a small set of stereotyped settings, as they bake in domain & task assumptions or scale poorly to large inputs or outputs. In this work, we propose Perceiver IO, a general-purpose architecture that handles data f…
▽ More
A central goal of machine learning is the development of systems that can solve many problems in as many data domains as possible. Current architectures, however, cannot be applied beyond a small set of stereotyped settings, as they bake in domain & task assumptions or scale poorly to large inputs or outputs. In this work, we propose Perceiver IO, a general-purpose architecture that handles data from arbitrary settings while scaling linearly with the size of inputs and outputs. Our model augments the Perceiver with a flexible querying mechanism that enables outputs of various sizes and semantics, doing away with the need for task-specific architecture engineering. The same architecture achieves strong results on tasks spanning natural language and visual understanding, multi-task and multi-modal reasoning, and StarCraft II. As highlights, Perceiver IO outperforms a Transformer-based BERT baseline on the GLUE language benchmark despite removing input tokenization and achieves state-of-the-art performance on Sintel optical flow estimation with no explicit mechanisms for multiscale correspondence.
△ Less
Submitted 15 March, 2022; v1 submitted 30 July, 2021;
originally announced July 2021.
-
COPOD: Copula-Based Outlier Detection
Authors:
Zheng Li,
Yue Zhao,
Nicola Botta,
Cezar Ionescu,
Xiyang Hu
Abstract:
Outlier detection refers to the identification of rare items that are deviant from the general data distribution. Existing approaches suffer from high computational complexity, low predictive capability, and limited interpretability. As a remedy, we present a novel outlier detection algorithm called COPOD, which is inspired by copulas for modeling multivariate data distribution. COPOD first constr…
▽ More
Outlier detection refers to the identification of rare items that are deviant from the general data distribution. Existing approaches suffer from high computational complexity, low predictive capability, and limited interpretability. As a remedy, we present a novel outlier detection algorithm called COPOD, which is inspired by copulas for modeling multivariate data distribution. COPOD first constructs an empirical copula, and then uses it to predict tail probabilities of each given data point to determine its level of "extremeness". Intuitively, we think of this as calculating an anomalous p-value. This makes COPOD both parameter-free, highly interpretable, and computationally efficient. In this work, we make three key contributions, 1) propose a novel, parameter-free outlier detection algorithm with both great performance and interpretability, 2) perform extensive experiments on 30 benchmark datasets to show that COPOD outperforms in most cases and is also one of the fastest algorithms, and 3) release an easy-to-use Python implementation for reproducibility.
△ Less
Submitted 20 September, 2020;
originally announced September 2020.
-
A locally F-finite Noetherian domain that is not F-finite
Authors:
Tiberiu Dumitrescu,
Cristodor Ionescu
Abstract:
Using an old example of Nagata, we construct a Noetherian ring of prime characteristic p, whose Frobenius morphism is locally finite, but not finite.
Using an old example of Nagata, we construct a Noetherian ring of prime characteristic p, whose Frobenius morphism is locally finite, but not finite.
△ Less
Submitted 19 June, 2020; v1 submitted 10 June, 2020;
originally announced June 2020.
-
Model reduction with pole-zero placement and high order moment matching
Authors:
Tudor C. Ionescu,
Orest V. Iftime,
Ion Necoara
Abstract:
In this paper, we compute a low order approximation of a system of large order $n$ that matches $ν$ moments of order $j_i$ of the transfer function, at $ν$ interpolation points, has $\ell$ poles and $k$ zeros fixed and also matches $ν-(\ell +k)$ moments of order $j_i+1$, where $j_i+1$ is the multiplicity of the $i$-th interpolation point. We derive explicit linear systems in the free parameters to…
▽ More
In this paper, we compute a low order approximation of a system of large order $n$ that matches $ν$ moments of order $j_i$ of the transfer function, at $ν$ interpolation points, has $\ell$ poles and $k$ zeros fixed and also matches $ν-(\ell +k)$ moments of order $j_i+1$, where $j_i+1$ is the multiplicity of the $i$-th interpolation point. We derive explicit linear systems in the free parameters to simultaneously achieve the required pole-zero placement and match the desired high order moments. We compute the closed form of the free parameters that meet the constraints, as the solution of a $ν$ order linear system. Furthermore, for data-driven model reduction, we generalize the construction of the Loewner matrices to include the data and the imposed pole and higher order moment constraints. The resulting approximations achieve a trade-off between the good norm approximation and the preservation of the dynamics of the original system in a region of interest.
△ Less
Submitted 24 February, 2021; v1 submitted 12 March, 2020;
originally announced March 2020.
-
Making Sense of Reinforcement Learning and Probabilistic Inference
Authors:
Brendan O'Donoghue,
Ian Osband,
Catalin Ionescu
Abstract:
Reinforcement learning (RL) combines a control problem with statistical estimation: The system dynamics are not known to the agent, but can be learned through experience. A recent line of research casts `RL as inference' and suggests a particular framework to generalize the RL problem as probabilistic inference. Our paper surfaces a key shortcoming in that approach, and clarifies the sense in whic…
▽ More
Reinforcement learning (RL) combines a control problem with statistical estimation: The system dynamics are not known to the agent, but can be learned through experience. A recent line of research casts `RL as inference' and suggests a particular framework to generalize the RL problem as probabilistic inference. Our paper surfaces a key shortcoming in that approach, and clarifies the sense in which RL can be coherently cast as an inference problem. In particular, an RL agent must consider the effects of its actions upon future rewards and observations: The exploration-exploitation tradeoff. In all but the most simple settings, the resulting inference is computationally intractable so that practical RL algorithms must resort to approximation. We demonstrate that the popular `RL as inference' approximation can perform poorly in even very basic problems. However, we show that with a small modification the framework does yield algorithms that can provably perform well, and we show that the resulting algorithm is equivalent to the recently proposed K-learning, which we further connect with Thompson sampling.
△ Less
Submitted 4 November, 2020; v1 submitted 3 January, 2020;
originally announced January 2020.
-
Finite generation of Andre-Quillen (co-)homology of F-finite algebras
Authors:
Cristodor Ionescu
Abstract:
We prove that the Andre-Quillen homology and cohomology modules of F-finite Z(p)-algebras are finitely generated.
We prove that the Andre-Quillen homology and cohomology modules of F-finite Z(p)-algebras are finitely generated.
△ Less
Submitted 27 September, 2019;
originally announced September 2019.
-
Examples and Results from a BSc-level Course on Domain Specific Languages of Mathematics
Authors:
Patrik Jansson,
Sólrún Halla Einarsdóttir,
Cezar Ionescu
Abstract:
At the workshop on Trends in Functional Programming in Education (TFPIE) in 2015 Ionescu and Jansson presented the approach underlying the "Domain Specific Languages of Mathematics" (DSLsofMath) course even before the first course instance. We were then encouraged to come back to present our experience and the student results. Now, three years later, we have seen three groups of learners attend th…
▽ More
At the workshop on Trends in Functional Programming in Education (TFPIE) in 2015 Ionescu and Jansson presented the approach underlying the "Domain Specific Languages of Mathematics" (DSLsofMath) course even before the first course instance. We were then encouraged to come back to present our experience and the student results. Now, three years later, we have seen three groups of learners attend the course, and the first two groups have also continued on to take challenging courses in the subsequent year. In this paper we present three examples from the course material to set the scene, and we present an evaluation of the student results showing improvements in the pass rates and grades in later courses.
△ Less
Submitted 26 June, 2019;
originally announced August 2019.
-
Unsupervised Learning of Object Keypoints for Perception and Control
Authors:
Tejas Kulkarni,
Ankush Gupta,
Catalin Ionescu,
Sebastian Borgeaud,
Malcolm Reynolds,
Andrew Zisserman,
Volodymyr Mnih
Abstract:
The study of object representations in computer vision has primarily focused on develo** representations that are useful for image classification, object detection, or semantic segmentation as downstream tasks. In this work we aim to learn object representations that are useful for control and reinforcement learning (RL). To this end, we introduce Transporter, a neural network architecture for d…
▽ More
The study of object representations in computer vision has primarily focused on develo** representations that are useful for image classification, object detection, or semantic segmentation as downstream tasks. In this work we aim to learn object representations that are useful for control and reinforcement learning (RL). To this end, we introduce Transporter, a neural network architecture for discovering concise geometric object representations in terms of keypoints or image-space coordinates. Our method learns from raw video frames in a fully unsupervised manner, by transporting learnt image features between video frames using a keypoint bottleneck. The discovered keypoints track objects and object parts across long time-horizons more accurately than recent similar methods. Furthermore, consistent long-term tracking enables two notable results in control domains -- (1) using the keypoint co-ordinates and corresponding image features as inputs enables highly sample-efficient reinforcement learning; (2) learning to explore by controlling keypoint locations drastically reduces the search space, enabling deep exploration (leading to states unreachable through random action exploration) without any extrinsic rewards.
△ Less
Submitted 19 November, 2019; v1 submitted 19 June, 2019;
originally announced June 2019.
-
H2 model reduction of linear network systems by moment matching and optimization
Authors:
I. Necoara,
T. C. Ionescu
Abstract:
In this paper we study the problem of model reduction of linear network systems. We aim at computing a reduced order stable approximation of the network with the same topology and optimal w.r.t. H2 norm error approximation. Our approach is based on time-domain moment matching framework, where we optimize over families of parameterized reduced order models matching a set of moments at arbitrary int…
▽ More
In this paper we study the problem of model reduction of linear network systems. We aim at computing a reduced order stable approximation of the network with the same topology and optimal w.r.t. H2 norm error approximation. Our approach is based on time-domain moment matching framework, where we optimize over families of parameterized reduced order models matching a set of moments at arbitrary interpolation points. The parameterization of the low order models is in terms of the free parameters and of the interpolation points. For this family of parameterized models we formulate an optimization-based model reduction problem with the H2 norm of error approximation as objective function while the preservation of some structural and physical properties yields the constraints. This problem is nonconvex and we write it in terms of the Gramians of a minimal realization of the error system. We propose two solutions for this problem. The first solution assumes that the error system admits a block diagonal observability Gramian, allowing for a simple convex reformulation as semidefinite programming, but at the cost of some performance loss. We also derive sufficient conditions to guarantee block diagonalization of the Gramian. The second solution employs a gradient projection method for a smooth reformulation yielding (locally) optimal interpolation points and free parameters. The potential of the methods is illustrated on a power network.
△ Less
Submitted 19 May, 2019; v1 submitted 8 February, 2019;
originally announced February 2019.
-
Unsupervised Control Through Non-Parametric Discriminative Rewards
Authors:
David Warde-Farley,
Tom Van de Wiele,
Tejas Kulkarni,
Catalin Ionescu,
Steven Hansen,
Volodymyr Mnih
Abstract:
Learning to control an environment without hand-crafted rewards or expert data remains challenging and is at the frontier of reinforcement learning research. We present an unsupervised learning algorithm to train agents to achieve perceptually-specified goals using only a stream of observations and actions. Our agent simultaneously learns a goal-conditioned policy and a goal achievement reward fun…
▽ More
Learning to control an environment without hand-crafted rewards or expert data remains challenging and is at the frontier of reinforcement learning research. We present an unsupervised learning algorithm to train agents to achieve perceptually-specified goals using only a stream of observations and actions. Our agent simultaneously learns a goal-conditioned policy and a goal achievement reward function that measures how similar a state is to the goal state. This dual optimization leads to a co-operative game, giving rise to a learned reward function that reflects similarity in controllable aspects of the environment instead of distance in the space of observations. We demonstrate the efficacy of our agent to learn, in an unsupervised manner, to reach a diverse set of goals on three domains -- Atari, the DeepMind Control Suite and DeepMind Lab.
△ Less
Submitted 27 November, 2018;
originally announced November 2018.
-
Optimal H2 moment matching-based model reduction for linear systems by (non)convex optimization
Authors:
I. Necoara,
T. C. Ionescu
Abstract:
In this paper we compute families of reduced order models that match a prescribed set of moments of a highly dimensional linear time-invariant system. First, we fully parametrize the models in the interpolation points and in the free parameters, and then we fix the set of interpolation points and parametrize the models only in the free parameters. Based on these two parametrizations and using as o…
▽ More
In this paper we compute families of reduced order models that match a prescribed set of moments of a highly dimensional linear time-invariant system. First, we fully parametrize the models in the interpolation points and in the free parameters, and then we fix the set of interpolation points and parametrize the models only in the free parameters. Based on these two parametrizations and using as objective function the H2-norm of the error approximation we derive non-convex optimization problems, i.e., we search for the optimal free parameters and even the interpolation points to determine the approximation model yielding the minimal H2-norm error. Further, we provide the necessary first-order optimality conditions for these optimization problems given explicitly in terms of the controllability and the observability Gramians of a minimal realization of the error system. Using the optimality conditions, we propose gradient type methods for solving the corresponding optimization problems, with mathematical guarantees on their convergence. We also derive convex SDP relaxations for these problems and analyze when the convex relaxations are exact. We illustrate numerically the efficiency of our results on several test examples.
△ Less
Submitted 18 November, 2018;
originally announced November 2018.
-
Tensor products and direct limits of almost Cohen-Macaulay modules
Authors:
Cristodor Ionescu,
Samaneh Tabejamaat
Abstract:
We investigate the almost Cohen-Macaulay property and the Serre-type condition $(C_n),\ n\in\mathbb{N},$ for noetherian algebras and modules. More precisely, we find permanence properties of these conditions with respect to tensor products and direct limits.
We investigate the almost Cohen-Macaulay property and the Serre-type condition $(C_n),\ n\in\mathbb{N},$ for noetherian algebras and modules. More precisely, we find permanence properties of these conditions with respect to tensor products and direct limits.
△ Less
Submitted 25 December, 2016;
originally announced December 2016.
-
Domain-Specific Languages of Mathematics: Presenting Mathematical Analysis Using Functional Programming
Authors:
Cezar Ionescu,
Patrik Jansson
Abstract:
We present the approach underlying a course on "Domain-Specific Languages of Mathematics", currently being developed at Chalmers in response to difficulties faced by third-year students in learning and applying classical mathematics (mainly real and complex analysis). The main idea is to encourage the students to approach mathematical domains from a functional programming perspective: to identify…
▽ More
We present the approach underlying a course on "Domain-Specific Languages of Mathematics", currently being developed at Chalmers in response to difficulties faced by third-year students in learning and applying classical mathematics (mainly real and complex analysis). The main idea is to encourage the students to approach mathematical domains from a functional programming perspective: to identify the main functions and types involved and, when necessary, to introduce new abstractions; to give calculational proofs; to pay attention to the syntax of the mathematical expressions; and, finally, to organise the resulting functions and types in domain-specific languages.
△ Less
Submitted 28 November, 2016;
originally announced November 2016.
-
Sequential decision problems, dependent types and generic solutions
Authors:
Nicola Botta,
Patrik Jansson,
Cezar Ionescu,
David R. Christiansen,
Edwin Brady
Abstract:
We present a computer-checked generic implementation for solving finite-horizon sequential decision problems. This is a wide class of problems, including inter-temporal optimizations, knapsack, optimal bracketing, scheduling, etc. The implementation can handle time-step dependent control and state spaces, and monadic representations of uncertainty (such as stochastic, non-deterministic, fuzzy, or…
▽ More
We present a computer-checked generic implementation for solving finite-horizon sequential decision problems. This is a wide class of problems, including inter-temporal optimizations, knapsack, optimal bracketing, scheduling, etc. The implementation can handle time-step dependent control and state spaces, and monadic representations of uncertainty (such as stochastic, non-deterministic, fuzzy, or combinations thereof). This level of genericity is achievable in a programming language with dependent types (we have used both Idris and Agda). Dependent types are also the means that allow us to obtain a formalization and computer-checked proof of the central component of our implementation: Bellman's principle of optimality and the associated backwards induction algorithm. The formalization clarifies certain aspects of backwards induction and, by making explicit notions such as viability and reachability, can serve as a starting point for a theory of controllability of monadic dynamical systems, commonly encountered in, e.g., climate impact research.
△ Less
Submitted 22 March, 2017; v1 submitted 23 October, 2016;
originally announced October 2016.
-
Using Fast Weights to Attend to the Recent Past
Authors:
Jimmy Ba,
Geoffrey Hinton,
Volodymyr Mnih,
Joel Z. Leibo,
Catalin Ionescu
Abstract:
Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs. There is no good reason for this restriction. Synapses have dynamics at many different time-scales and this suggests that artificial ne…
▽ More
Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs. There is no good reason for this restriction. Synapses have dynamics at many different time-scales and this suggests that artificial neural networks might benefit from variables that change slower than activities but much faster than the standard weights. These "fast weights" can be used to store temporary memories of the recent past and they provide a neurally plausible way of implementing the type of attention to the past that has recently proved very helpful in sequence-to-sequence models. By using fast weights we can avoid the need to store copies of neural activity patterns.
△ Less
Submitted 4 December, 2016; v1 submitted 19 October, 2016;
originally announced October 2016.
-
Training Deep Networks with Structured Layers by Matrix Backpropagation
Authors:
Catalin Ionescu,
Orestis Vantzos,
Cristian Sminchisescu
Abstract:
Deep neural network architectures have recently produced excellent results in a variety of areas in artificial intelligence and visual recognition, well surpassing traditional shallow architectures trained using hand-designed features. The power of deep networks stems both from their ability to perform local computations followed by pointwise non-linearities over increasingly larger receptive fiel…
▽ More
Deep neural network architectures have recently produced excellent results in a variety of areas in artificial intelligence and visual recognition, well surpassing traditional shallow architectures trained using hand-designed features. The power of deep networks stems both from their ability to perform local computations followed by pointwise non-linearities over increasingly larger receptive fields, and from the simplicity and scalability of the gradient-descent training procedure based on backpropagation. An open problem is the inclusion of layers that perform global, structured matrix computations like segmentation (e.g. normalized cuts) or higher-order pooling (e.g. log-tangent space metrics defined over the manifold of symmetric positive definite matrices) while preserving the validity and efficiency of an end-to-end deep training framework. In this paper we propose a sound mathematical apparatus to formally integrate global structured computation into deep computation architectures. At the heart of our methodology is the development of the theory and practice of backpropagation that generalizes to the calculus of adjoint matrix variations. The proposed matrix backpropagation methodology applies broadly to a variety of problems in machine learning or computational perception. Here we illustrate it by performing visual segmentation experiments using the BSDS and MSCOCO benchmarks, where we show that deep networks relying on second-order pooling and normalized cuts layers, trained end-to-end using matrix backpropagation, outperform counterparts that do not take advantage of such global layers.
△ Less
Submitted 14 April, 2016; v1 submitted 25 September, 2015;
originally announced September 2015.
-
Revisiting Large Scale Distributed Machine Learning
Authors:
Radu Cristian Ionescu
Abstract:
Nowadays, with the widespread of smartphones and other portable gadgets equipped with a variety of sensors, data is ubiquitous available and the focus of machine learning has shifted from being able to infer from small training samples to dealing with large scale high-dimensional data. In domains such as personal healthcare applications, which motivates this survey, distributed machine learning is…
▽ More
Nowadays, with the widespread of smartphones and other portable gadgets equipped with a variety of sensors, data is ubiquitous available and the focus of machine learning has shifted from being able to infer from small training samples to dealing with large scale high-dimensional data. In domains such as personal healthcare applications, which motivates this survey, distributed machine learning is a promising line of research, both for scaling up learning algorithms, but mostly for dealing with data which is inherently produced at different locations. This report offers a thorough overview of and state-of-the-art algorithms for distributed machine learning, for both supervised and unsupervised learning, ranging from simple linear logistic regression to graphical models and clustering. We propose future directions for most categories, specific to the potential personal healthcare applications. With this in mind, the report focuses on how security and low communication overhead can be assured in the specific case of a strictly client-server architectural model. As particular directions we provides an exhaustive presentation of an empirical clustering algorithm, k-windows, and proposed an asynchronous distributed machine learning algorithm that would scale well and also would be computationally cheap and easy to implement.
△ Less
Submitted 6 July, 2015;
originally announced July 2015.
-
A scalable system for primal-dual optimization
Authors:
Radu Cristian Ionescu
Abstract:
We present some of the most widely used architectures for Big Data, \textit{Hadoop} and \textit{Spark}, and develop several implementations exploiting, the advantages of each. We implement a simplified version of the primal-dual optimization algorithm, described briefly in this paper, by choosing the smoothing functions to be $\Vert \cdot \Vert^2$ with a zero center point. Under the assumption tha…
▽ More
We present some of the most widely used architectures for Big Data, \textit{Hadoop} and \textit{Spark}, and develop several implementations exploiting, the advantages of each. We implement a simplified version of the primal-dual optimization algorithm, described briefly in this paper, by choosing the smoothing functions to be $\Vert \cdot \Vert^2$ with a zero center point. Under the assumption that data is provided as a sparse matrix, we assess the scalability of the designed systems empirically by running them on sample tests.
△ Less
Submitted 7 August, 2015; v1 submitted 6 July, 2015;
originally announced July 2015.
-
Flat local morphisms of rings with prescribed depth and dimension
Authors:
Cristodor Ionescu
Abstract:
For pairs of integers (n,m) and (d,e) satisfying some nedesary conditions, we construct a local flat ring morphism of noetherian local rings u:A -->B such that dim(A)=n, depth(A)=d, dim(B)=m, depth(B)=e.
For pairs of integers (n,m) and (d,e) satisfying some nedesary conditions, we construct a local flat ring morphism of noetherian local rings u:A -->B such that dim(A)=n, depth(A)=d, dim(B)=m, depth(B)=e.
△ Less
Submitted 15 October, 2013; v1 submitted 13 October, 2013;
originally announced October 2013.
-
More properties of almost Cohen-Macaulay rings
Authors:
Cristodor Ionescu
Abstract:
Some interesting properties of almost Cohen-Macaulay rings are investigated and a Serre type property connected with this class of rings is studied.
Some interesting properties of almost Cohen-Macaulay rings are investigated and a Serre type property connected with this class of rings is studied.
△ Less
Submitted 27 April, 2013;
originally announced April 2013.
-
Families of moment matching based, structure preserving approximations for linear port Hamiltonian systems
Authors:
Tudor C. Ionescu,
Alessandro Astolfi
Abstract:
In this paper we propose a solution to the problem of moment matching with preservation of the port Hamiltonian structure, in the framework of time-domain moment matching. We characterize several families of parameterized port Hamiltonian models that match the moments of a given port Hamiltonian system, at a set of finite interpolation points. We also discuss the problem of Markov parameters match…
▽ More
In this paper we propose a solution to the problem of moment matching with preservation of the port Hamiltonian structure, in the framework of time-domain moment matching. We characterize several families of parameterized port Hamiltonian models that match the moments of a given port Hamiltonian system, at a set of finite interpolation points. We also discuss the problem of Markov parameters matching for linear systems as a moment matching problem for descriptor representations associated to the given system, at zero interpolation points. Solving this problem yields families of parameterized reduced order models that achieve Markov parameter matching. Finally, we apply these results to the port Hamiltonian case, resulting in families of parameterized reduced order port Hamiltonian approximations.
△ Less
Submitted 18 April, 2013;
originally announced April 2013.
-
Some examples of two-dimensional regular rings
Authors:
Tiberiu Dumitrescu,
Cristodor Ionescu
Abstract:
Let B be a ring and $A=B[X,Y]/(aX^2+bXY+cY^2-1)$ where $a,b,c\in B$. We study the smoothness of A over B, and the regularity of B when B is a ring of algebraic integers.
Let B be a ring and $A=B[X,Y]/(aX^2+bXY+cY^2-1)$ where $a,b,c\in B$. We study the smoothness of A over B, and the regularity of B when B is a ring of algebraic integers.
△ Less
Submitted 11 January, 2013;
originally announced January 2013.
-
Extended BRST symmetries. Quantum approach
Authors:
Radu Constantinescu,
Carmen Ionescu
Abstract:
The aim of this lecture is to present in a comprehensible way what the BRST quantization means and how the "classical" master equation, action and BRST transformations have to be prolonged towards the same "quantum" items. The presentation will focus not only on the standard BRST symmetry, but on larger symmetries as sp(2), both in the Lagrangean and in the Hamiltonian formalisms. How to find answ…
▽ More
The aim of this lecture is to present in a comprehensible way what the BRST quantization means and how the "classical" master equation, action and BRST transformations have to be prolonged towards the same "quantum" items. The presentation will focus not only on the standard BRST symmetry, but on larger symmetries as sp(2), both in the Lagrangean and in the Hamiltonian formalisms. How to find answers to these questions in more sophisticated cases will be illustrated by the example of a nonlinear system with open superalgebra.
△ Less
Submitted 3 December, 2011;
originally announced December 2011.
-
A note on smoothness and differential bases in positive characteristic
Authors:
Cristodor Ionescu
Abstract:
Let $u:A\to B$ be a morphism of noetherian local rings. We obtain smoothness criteria for algebras with differential bases, in the case of rings containing a field of characteristic $p>0.$ We also give smoothness criteria for reduced morphisms.
Let $u:A\to B$ be a morphism of noetherian local rings. We obtain smoothness criteria for algebras with differential bases, in the case of rings containing a field of characteristic $p>0.$ We also give smoothness criteria for reduced morphisms.
△ Less
Submitted 19 May, 2009; v1 submitted 26 September, 2008;
originally announced September 2008.
-
Theory for the phase behaviour of a colloidal fluid with competing interactions
Authors:
A. J. Archer,
C. Ionescu,
D. Pini,
L. Reatto
Abstract:
We study the phase behaviour of a fluid composed of particles which interact via a pair potential that is repulsive for large inter-particle distances, is attractive at intermediate distances and is strongly repulsive at short distances (the particles have a hard core). As well as exhibiting gas-liquid phase separation, this system also exhibits phase transitions from the uniform fluid phases to…
▽ More
We study the phase behaviour of a fluid composed of particles which interact via a pair potential that is repulsive for large inter-particle distances, is attractive at intermediate distances and is strongly repulsive at short distances (the particles have a hard core). As well as exhibiting gas-liquid phase separation, this system also exhibits phase transitions from the uniform fluid phases to modulated inhomogeneous fluid phases. Starting from a microscopic density functional theory, we develop an order parameter theory for the phase transition in order to examine in detail the phase behaviour. The amplitude of the density modulations is the order parameter in our theory. The theory predicts that the phase transition from the uniform to the modulated fluid phase can be either first order or second order (continuous). The phase diagram exhibits two tricritical points, joined to one another by the line of second order transitions.
△ Less
Submitted 29 August, 2008;
originally announced August 2008.
-
Some algebraic invariants of mixed product ideals
Authors:
Cristodor Ionescu,
Giancarlo Rinaldo
Abstract:
We compute some algebraic invariants (e.g. depth, Castelnuovo - Mumford regularity) for a special class of monomial ideals, namely the ideals of mixed products. As a consequence, we characterize the Cohen-Macaulay ideals of mixed products.
We compute some algebraic invariants (e.g. depth, Castelnuovo - Mumford regularity) for a special class of monomial ideals, namely the ideals of mixed products. As a consequence, we characterize the Cohen-Macaulay ideals of mixed products.
△ Less
Submitted 20 November, 2007;
originally announced November 2007.
-
Smooth cutoff formulation of hierarchical reference theory for a scalar phi4 field theory
Authors:
Cristian D. Ionescu,
Alberto Parola,
Davide Pini,
Luciano Reatto
Abstract:
The phi4 scalar field theory in three dimensions, prototype for the study of phase transitions, is investigated by means of the hierarchical reference theory (HRT) in its smooth cutoff formulation. The critical behavior is described by scaling laws and critical exponents which compare favorably with the known values of the Ising universality class. The inverse susceptibility vanishes identically…
▽ More
The phi4 scalar field theory in three dimensions, prototype for the study of phase transitions, is investigated by means of the hierarchical reference theory (HRT) in its smooth cutoff formulation. The critical behavior is described by scaling laws and critical exponents which compare favorably with the known values of the Ising universality class. The inverse susceptibility vanishes identically inside the coexistence curve, providing a first principle implementation of the Maxwell construction, and shows the expected discontinuity across the phase boundary, at variance with the usual sharp cutoff implementation of HRT. The correct description of first and second order phase transitions within a microscopic, nonperturbative approach is thus achieved in the smooth cutoff HRT.
△ Less
Submitted 13 September, 2007;
originally announced September 2007.
-
Some applications of Andre-Quillen homology to classes of arithmetic rings
Authors:
Tiberiu Dumitrescu,
Cristodor Ionescu
Abstract:
We compute the first Andre-Quillen homology modules for the simple over-rings of integrally closed domains and study an ideal theoretic condition arising from the vanishing of the first homology module.
We compute the first Andre-Quillen homology modules for the simple over-rings of integrally closed domains and study an ideal theoretic condition arising from the vanishing of the first homology module.
△ Less
Submitted 2 February, 2007;
originally announced February 2007.
-
Regularity and finite injective dimension in characteristic p>0
Authors:
Tiberiu Dumitrescu,
Cristodor Ionescu
Abstract:
Recently, the regular local rings of prime characteristic were characterized in terms of the finiteness of injective dimension of the Frobenius map. We obtain relative versions of this result.
Recently, the regular local rings of prime characteristic were characterized in terms of the finiteness of injective dimension of the Frobenius map. We obtain relative versions of this result.
△ Less
Submitted 27 October, 2006;
originally announced October 2006.
-
Bi-partite and global entanglement in a many-particle system with collective spin coupling
Authors:
R. G. Unanyan,
C. Ionescu,
M. Fleischhauer
Abstract:
Bipartite and global entanglement are analyzed for the ground state of a system of $N$ spin 1/2 particles interacting via a collective spin-spin coupling described by the Lipkin-Meshkov-Glick (LMG) Hamiltonian. Under certain conditions which includes the special case of a super-symmetry, the ground state can be constructed analytically. In the case of an anti-ferromagnetic coupling and for an ev…
▽ More
Bipartite and global entanglement are analyzed for the ground state of a system of $N$ spin 1/2 particles interacting via a collective spin-spin coupling described by the Lipkin-Meshkov-Glick (LMG) Hamiltonian. Under certain conditions which includes the special case of a super-symmetry, the ground state can be constructed analytically. In the case of an anti-ferromagnetic coupling and for an even number of particles this state undergoes a smooth crossover as a function of the continuous anisotropy parameter $γ$ from a separable ($γ=\infty $) to a maximally entangled many-particle state ($γ=0$). From the analytic expression for the ground state, bipartite and global entanglement are calculated. In the thermodynamic limit a discontinuous change of the scaling behavior of the bipartite entanglement is found at the isotropy point $γ=0$. For $% γ=0$ the entanglement grows logarithmically with the system size with no upper bound, for $γ\neq 0$ it saturates at a level only depending on $γ$. For finite systems with total spin $J=N/2$ the scaling behavior changes at $γ=γ_{\mathrm{crit}}=1/J$.
△ Less
Submitted 21 December, 2004;
originally announced December 2004.
-
NP - P is not empty
Authors:
Marius Constantin Ionescu
Abstract:
We present the MEoP problem that decides the existence of solutions to certain modular equations over prime numbers and show how this separates the complexity class NP from its subclass P
We present the MEoP problem that decides the existence of solutions to certain modular equations over prime numbers and show how this separates the complexity class NP from its subclass P
△ Less
Submitted 23 September, 2016; v1 submitted 21 September, 2004;
originally announced September 2004.
