-
Kernelized Cumulants: Beyond Kernel Mean Embeddings
Authors:
Patric Bonnier,
Harald Oberhauser,
Zoltán Szabó
Abstract:
In $\mathbb R^d$, it is well-known that cumulants provide an alternative to moments that can achieve the same goals with numerous benefits such as lower variance estimators. In this paper we extend cumulants to reproducing kernel Hilbert spaces (RKHS) using tools from tensor algebras and show that they are computationally tractable by a kernel trick. These kernelized cumulants provide a new set of…
▽ More
In $\mathbb R^d$, it is well-known that cumulants provide an alternative to moments that can achieve the same goals with numerous benefits such as lower variance estimators. In this paper we extend cumulants to reproducing kernel Hilbert spaces (RKHS) using tools from tensor algebras and show that they are computationally tractable by a kernel trick. These kernelized cumulants provide a new set of all-purpose statistics; the classical maximum mean discrepancy and Hilbert-Schmidt independence criterion arise as the degree one objects in our general construction. We argue both theoretically and empirically (on synthetic, environmental, and traffic data analysis) that going beyond degree one has several advantages and can be achieved with the same computational complexity and minimal overhead in our experiments.
△ Less
Submitted 29 October, 2023; v1 submitted 29 January, 2023;
originally announced January 2023.
-
Proper Scoring Rules, Gradients, Divergences, and Entropies for Paths and Time Series
Authors:
Patric Bonnier,
Harald Oberhauser
Abstract:
Many forecasts consist not of point predictions but concern the evolution of quantities. For example, a central bank might predict the interest rates during the next quarter, an epidemiologist might predict trajectories of infection rates, a clinician might predict the behaviour of medical markers over the next day, etc. The situation is further complicated since these forecasts sometimes only con…
▽ More
Many forecasts consist not of point predictions but concern the evolution of quantities. For example, a central bank might predict the interest rates during the next quarter, an epidemiologist might predict trajectories of infection rates, a clinician might predict the behaviour of medical markers over the next day, etc. The situation is further complicated since these forecasts sometimes only concern the approximate "shape of the future evolution" or "order of events". Formally, such forecasts can be seen as probability measures on spaces of equivalence classes of paths modulo time-parametrization. We leverage the statistical framework of proper scoring rules with classical mathematical results to derive a principled approach to decision making with such forecasts. In particular, we introduce notions of gradients, entropy, and divergence that are tailor-made to respect the underlying non-Euclidean structure.
△ Less
Submitted 11 November, 2021;
originally announced November 2021.
-
Seq2Tens: An Efficient Representation of Sequences by Low-Rank Tensor Projections
Authors:
Csaba Toth,
Patric Bonnier,
Harald Oberhauser
Abstract:
Sequential data such as time series, video, or text can be challenging to analyse as the ordered structure gives rise to complex dependencies. At the heart of this is non-commutativity, in the sense that reordering the elements of a sequence can completely change its meaning. We use a classical mathematical object -- the tensor algebra -- to capture such dependencies. To address the innate computa…
▽ More
Sequential data such as time series, video, or text can be challenging to analyse as the ordered structure gives rise to complex dependencies. At the heart of this is non-commutativity, in the sense that reordering the elements of a sequence can completely change its meaning. We use a classical mathematical object -- the tensor algebra -- to capture such dependencies. To address the innate computational complexity of high degree tensors, we use compositions of low-rank tensor projections. This yields modular and scalable building blocks for neural networks that give state-of-the-art performance on standard benchmarks such as multivariate time series classification and generative models for video.
△ Less
Submitted 30 July, 2021; v1 submitted 12 June, 2020;
originally announced June 2020.
-
Adapted Topologies and Higher Rank Signatures
Authors:
Patric Bonnier,
Chong Liu,
Harald Oberhauser
Abstract:
The topology of weak convergence does not account for the growth of information over time that is captured in the filtration of an adapted stochastic process. For example, two adapted stochastic processes can have very similar laws but give completely different results in applications such as optimal stop**, queuing theory, or stochastic programming. To address such discontinuities, Aldous intro…
▽ More
The topology of weak convergence does not account for the growth of information over time that is captured in the filtration of an adapted stochastic process. For example, two adapted stochastic processes can have very similar laws but give completely different results in applications such as optimal stop**, queuing theory, or stochastic programming. To address such discontinuities, Aldous introduced the extended weak topology, and subsequently, Hoover and Keisler showed that both, weak topology and extended weak topology, are just the first two topologies in a sequence of topologies that get increasingly finer. We use higher rank expected signatures to embed adapted processes into graded linear spaces and show that these embeddings induce the adapted topologies of Hoover--Keisler.
△ Less
Submitted 13 April, 2021; v1 submitted 18 May, 2020;
originally announced May 2020.
-
Signature Cumulants, Ordered Partitions, and Independence of Stochastic Processes
Authors:
Patric Bonnier,
Harald Oberhauser
Abstract:
The sequence of so-called signature moments describes the laws of many stochastic processes in analogy with how the sequence of moments describes the laws of vector-valued random variables. However, even for vector-valued random variables, the sequence of cumulants is much better suited for many tasks than the sequence of moments. This motivates us to study so-called signature cumulants. To do so,…
▽ More
The sequence of so-called signature moments describes the laws of many stochastic processes in analogy with how the sequence of moments describes the laws of vector-valued random variables. However, even for vector-valued random variables, the sequence of cumulants is much better suited for many tasks than the sequence of moments. This motivates us to study so-called signature cumulants. To do so, we develop an elementary combinatorial approach and show that in the same way that cumulants relate to the lattice of partitions, signature cumulants relate to the lattice of so-called "ordered partitions". We use this to give a new characterisation of independence of multivariate stochastic processes; finally we construct a family of unbiased minimum-variance estimators of signature cumulants.
△ Less
Submitted 1 September, 2020; v1 submitted 18 August, 2019;
originally announced August 2019.
-
Deep Signature Transforms
Authors:
Patric Bonnier,
Patrick Kidger,
Imanol Perez Arribas,
Cristopher Salvi,
Terry Lyons
Abstract:
The signature is an infinite graded sequence of statistics known to characterise a stream of data up to a negligible equivalence class. It is a transform which has previously been treated as a fixed feature transformation, on top of which a model may be built. We propose a novel approach which combines the advantages of the signature transform with modern deep learning frameworks. By learning an a…
▽ More
The signature is an infinite graded sequence of statistics known to characterise a stream of data up to a negligible equivalence class. It is a transform which has previously been treated as a fixed feature transformation, on top of which a model may be built. We propose a novel approach which combines the advantages of the signature transform with modern deep learning frameworks. By learning an augmentation of the stream prior to the signature transform, the terms of the signature may be selected in a data-dependent way. More generally, we describe how the signature transform may be used as a layer anywhere within a neural network. In this context it may be interpreted as a pooling operation. We present the results of empirical experiments to back up the theoretical justification. Code available at https://github.com/patrick-kidger/Deep-Signature-Transforms.
△ Less
Submitted 26 October, 2019; v1 submitted 21 May, 2019;
originally announced May 2019.
