-
Mathematical Challenges in Deep Learning
Authors:
Vahid Partovi Nia,
Guojun Zhang,
Ivan Kobyzev,
Michael R. Metel,
Xinlin Li,
Ke Sun,
Sobhan Hemati,
Masoud Asgharian,
Linglong Kong,
Wulong Liu,
Boxing Chen
Abstract:
Deep models are dominating the artificial intelligence (AI) industry since the ImageNet challenge in 2012. The size of deep models is increasing ever since, which brings new challenges to this field with applications in cell phones, personal computers, autonomous cars, and wireless base stations. Here we list a set of problems, ranging from training, inference, generalization bound, and optimizati…
▽ More
Deep models are dominating the artificial intelligence (AI) industry since the ImageNet challenge in 2012. The size of deep models is increasing ever since, which brings new challenges to this field with applications in cell phones, personal computers, autonomous cars, and wireless base stations. Here we list a set of problems, ranging from training, inference, generalization bound, and optimization with some formalism to communicate these challenges with mathematicians, statisticians, and theoretical computer scientists. This is a subjective view of the research questions in deep learning that benefits the tech industry in long run.
△ Less
Submitted 24 March, 2023;
originally announced March 2023.
-
Normalizing Flows: An Introduction and Review of Current Methods
Authors:
Ivan Kobyzev,
Simon J. D. Prince,
Marcus A. Brubaker
Abstract:
Normalizing Flows are generative models which produce tractable distributions where both sampling and density evaluation can be efficient and exact. The goal of this survey article is to give a coherent and comprehensive review of the literature around the construction and use of Normalizing Flows for distribution learning. We aim to provide context and explanation of the models, review current st…
▽ More
Normalizing Flows are generative models which produce tractable distributions where both sampling and density evaluation can be efficient and exact. The goal of this survey article is to give a coherent and comprehensive review of the literature around the construction and use of Normalizing Flows for distribution learning. We aim to provide context and explanation of the models, review current state-of-the-art literature, and identify open questions and promising future directions.
△ Less
Submitted 5 June, 2020; v1 submitted 25 August, 2019;
originally announced August 2019.
-
Tails of Lipschitz Triangular Flows
Authors:
Priyank Jaini,
Ivan Kobyzev,
Yaoliang Yu,
Marcus Brubaker
Abstract:
We investigate the ability of popular flow based methods to capture tail-properties of a target density by studying the increasing triangular maps used in these flow methods acting on a tractable source density. We show that the density quantile functions of the source and target density provide a precise characterization of the slope of transformation required to capture tails in a target density…
▽ More
We investigate the ability of popular flow based methods to capture tail-properties of a target density by studying the increasing triangular maps used in these flow methods acting on a tractable source density. We show that the density quantile functions of the source and target density provide a precise characterization of the slope of transformation required to capture tails in a target density. We further show that any Lipschitz-continuous transport map acting on a source density will result in a density with similar tail properties as the source, highlighting the trade-off between a complex source density and a sufficiently expressive transformation to capture desirable properties of a target density. Subsequently, we illustrate that flow models like Real-NVP, MAF, and Glow as implemented originally lack the ability to capture a distribution with non-Gaussian tails. We circumvent this problem by proposing tail-adaptive flows consisting of a source distribution that can be learned simultaneously with the triangular map to capture tail-properties of a target density. We perform several synthetic and real-world experiments to compliment our theoretical findings.
△ Less
Submitted 18 September, 2020; v1 submitted 9 July, 2019;
originally announced July 2019.
-
Representation Learning for Dynamic Graphs: A Survey
Authors:
Seyed Mehran Kazemi,
Rishab Goel,
Kshitij Jain,
Ivan Kobyzev,
Akshay Sethi,
Peter Forsyth,
Pascal Poupart
Abstract:
Graphs arise naturally in many real-world applications including social networks, recommender systems, ontologies, biology, and computational finance. Traditionally, machine learning models for graphs have been mostly designed for static graphs. However, many applications involve evolving graphs. This introduces important challenges for learning and inference since nodes, attributes, and edges cha…
▽ More
Graphs arise naturally in many real-world applications including social networks, recommender systems, ontologies, biology, and computational finance. Traditionally, machine learning models for graphs have been mostly designed for static graphs. However, many applications involve evolving graphs. This introduces important challenges for learning and inference since nodes, attributes, and edges change over time. In this survey, we review the recent advances in representation learning for dynamic graphs, including dynamic knowledge graphs. We describe existing models from an encoder-decoder perspective, categorize these encoders and decoders based on the techniques they employ, and analyze the approaches in each category. We also review several prominent applications and widely used datasets and highlight directions for future research.
△ Less
Submitted 27 April, 2020; v1 submitted 27 May, 2019;
originally announced May 2019.
