-
Finite-Sample Identification of Linear Regression Models with Residual-Permuted Sums
Authors:
Szabolcs Szentpéteri,
Balázs Csanád Csáji
Abstract:
This letter studies a distribution-free, finite-sample data perturbation (DP) method, the Residual-Permuted Sums (RPS), which is an alternative of the Sign-Perturbed Sums (SPS) algorithm, to construct confidence regions. While SPS assumes independent (but potentially time-varying) noise terms which are symmetric about zero, RPS gets rid of the symmetricity assumption, but assumes i.i.d. noises. Th…
▽ More
This letter studies a distribution-free, finite-sample data perturbation (DP) method, the Residual-Permuted Sums (RPS), which is an alternative of the Sign-Perturbed Sums (SPS) algorithm, to construct confidence regions. While SPS assumes independent (but potentially time-varying) noise terms which are symmetric about zero, RPS gets rid of the symmetricity assumption, but assumes i.i.d. noises. The main idea is that RPS permutes the residuals instead of perturbing their signs. This letter introduces RPS in a flexible way, which allows various design-choices. RPS has exact finite sample coverage probabilities and we provide the first proof that these permutation-based confidence regions are uniformly strongly consistent under general assumptions. This means that the RPS regions almost surely shrink around the true parameters as the sample size increases. The ellipsoidal outer-approximation (EOA) of SPS is also extended to RPS, and the effectiveness of RPS is validated by numerical experiments, as well.
△ Less
Submitted 8 June, 2024;
originally announced June 2024.
-
Signed-Perturbed Sums Estimation of ARX Systems: Exact Coverage and Strong Consistency (Extended Version)
Authors:
Algo Carè,
Erik Weyer,
Balázs Cs. Csáji,
Marco C. Campi
Abstract:
Sign-Perturbed Sums (SPS) is a system identification method that constructs confidence regions for the unknown system parameters. In this paper, we study SPS for ARX systems, and establish that the confidence regions are guaranteed to include the true model parameter with exact, user-chosen, probability under mild statistical assumptions, a property that holds true for any finite number of observe…
▽ More
Sign-Perturbed Sums (SPS) is a system identification method that constructs confidence regions for the unknown system parameters. In this paper, we study SPS for ARX systems, and establish that the confidence regions are guaranteed to include the true model parameter with exact, user-chosen, probability under mild statistical assumptions, a property that holds true for any finite number of observed input-output data. Furthermore, we prove the strong consistency of the method, that is, as the number of data points increases, the confidence region gets smaller and smaller and will asymptotically almost surely exclude any parameter value different from the true one. In addition, we also show that, asymptotically, the SPS region is included in an ellipsoid which is marginally larger than the confidence ellipsoid obtained from the asymptotic theory of system identification. The results are theoretically proven and illustrated in a simulation example.
△ Less
Submitted 18 February, 2024;
originally announced February 2024.
-
Sample Complexity of the Sign-Perturbed Sums Identification Method: Scalar Case
Authors:
Szabolcs Szentpéteri,
Balázs Csanád Csáji
Abstract:
Sign-Perturbed Sum (SPS) is a powerful finite-sample system identification algorithm which can construct confidence regions for the true data generating system with exact coverage probabilities, for any finite sample size. SPS was developed in a series of papers and it has a wide range of applications, from general linear systems, even in a closed-loop setup, to nonlinear and nonparametric approac…
▽ More
Sign-Perturbed Sum (SPS) is a powerful finite-sample system identification algorithm which can construct confidence regions for the true data generating system with exact coverage probabilities, for any finite sample size. SPS was developed in a series of papers and it has a wide range of applications, from general linear systems, even in a closed-loop setup, to nonlinear and nonparametric approaches. Although several theoretical properties of SPS were proven in the literature, the sample complexity of the method was not analysed so far. This paper aims to fill this gap and provides the first results on the sample complexity of SPS. Here, we focus on scalar linear regression problems, that is we study the behaviour of SPS confidence intervals. We provide high probability upper bounds, under three different sets of assumptions, showing that the sizes of SPS confidence intervals shrink at a geometric rate around the true parameter, if the observation noises are subgaussian. We also show that similar bounds hold for the previously proposed outer approximation of the confidence region. Finally, we present simulation experiments comparing the theoretical and the empirical convergence rates.
△ Less
Submitted 28 January, 2024;
originally announced January 2024.
-
Improving Kernel-Based Nonasymptotic Simultaneous Confidence Bands
Authors:
Balázs Csanád Csáji,
Bálint Horváth
Abstract:
The paper studies the problem of constructing nonparametric simultaneous confidence bands with nonasymptotic and distribition-free guarantees. The target function is assumed to be band-limited and the approach is based on the theory of Paley-Wiener reproducing kernel Hilbert spaces. The starting point of the paper is a recently developed algorithm to which we propose three types of improvements. F…
▽ More
The paper studies the problem of constructing nonparametric simultaneous confidence bands with nonasymptotic and distribition-free guarantees. The target function is assumed to be band-limited and the approach is based on the theory of Paley-Wiener reproducing kernel Hilbert spaces. The starting point of the paper is a recently developed algorithm to which we propose three types of improvements. First, we relax the assumptions on the noises by replacing the symmetricity assumption with a weaker distributional invariance principle. Then, we propose a more efficient way to estimate the norm of the target function, and finally we enhance the construction of the confidence bands by tightening the constraints of the underlying convex optimization problems. The refinements are also illustrated through numerical experiments.
△ Less
Submitted 28 January, 2024;
originally announced January 2024.
-
On Rate-Optimal Partitioning Classification from Observable and from Privatised Data
Authors:
Balázs Csanád Csáji,
László Györfi,
Ambrus Tamás,
Harro Walk
Abstract:
In this paper we revisit the classical method of partitioning classification and study its convergence rate under relaxed conditions, both for observable (non-privatised) and for privatised data. Let the feature vector $X$ take values in $\mathbb{R}^d$ and denote its label by $Y$. Previous results on the partitioning classifier worked with the strong density assumption, which is restrictive, as we…
▽ More
In this paper we revisit the classical method of partitioning classification and study its convergence rate under relaxed conditions, both for observable (non-privatised) and for privatised data. Let the feature vector $X$ take values in $\mathbb{R}^d$ and denote its label by $Y$. Previous results on the partitioning classifier worked with the strong density assumption, which is restrictive, as we demonstrate through simple examples. We assume that the distribution of $X$ is a mixture of an absolutely continuous and a discrete distribution, such that the absolutely continuous component is concentrated to a $d_a$ dimensional subspace. Here, we study the problem under much milder assumptions: in addition to the standard Lipschitz and margin conditions, a novel characteristic of the absolutely continuous component is introduced, by which the exact convergence rate of the classification error probability is calculated, both for the binary and for the multi-label cases. Interestingly, this rate of convergence depends only on the intrinsic dimension $d_a$.
The privacy constraints mean that the data $(X_1,Y_1), \dots ,(X_n,Y_n)$ cannot be directly observed, and the classifiers are functions of the randomised outcome of a suitable local differential privacy mechanism. The statistician is free to choose the form of this privacy mechanism, and here we add Laplace distributed noises to the discontinuations of all possible locations of the feature vector $X_i$ and to its label $Y_i$. Again, tight upper bounds on the rate of convergence of the classification error probability are derived, without the strong density assumption, such that this rate depends on $2\,d_a$.
△ Less
Submitted 29 February, 2024; v1 submitted 22 December, 2023;
originally announced December 2023.
-
Robust Independence Tests with Finite Sample Guarantees for Synchronous Stochastic Linear Systems
Authors:
Ambrus Tamás,
Dániel Ágoston Bálint,
Balázs Csanád Csáji
Abstract:
The paper introduces robust independence tests with non-asymptotically guaranteed significance levels for stochastic linear time-invariant systems, assuming that the observed outputs are synchronous, which means that the systems are driven by jointly i.i.d. noises. Our method provides bounds for the type I error probabilities that are distribution-free, i.e., the innovations can have arbitrary dis…
▽ More
The paper introduces robust independence tests with non-asymptotically guaranteed significance levels for stochastic linear time-invariant systems, assuming that the observed outputs are synchronous, which means that the systems are driven by jointly i.i.d. noises. Our method provides bounds for the type I error probabilities that are distribution-free, i.e., the innovations can have arbitrary distributions. The algorithm combines confidence region estimates with permutation tests and general dependence measures, such as the Hilbert-Schmidt independence criterion and the distance covariance, to detect any nonlinear dependence between the observed systems. We also prove the consistency of our hypothesis tests under mild assumptions and demonstrate the ideas through the example of autoregressive systems.
△ Less
Submitted 3 August, 2023;
originally announced August 2023.
-
Non-Asymptotic State-Space Identification of Closed-Loop Stochastic Linear Systems using Instrumental Variables
Authors:
Szabolcs Szentpéteri,
Balázs Csanád Csáji
Abstract:
The paper suggests a generalization of the Sign-Perturbed Sums (SPS) finite sample system identification method for the identification of closed-loop observable stochastic linear systems in state-space form. The solution builds on the theory of matrix-variate regression and instrumental variable methods to construct distribution-free confidence regions for the state-space matrices. Both direct and…
▽ More
The paper suggests a generalization of the Sign-Perturbed Sums (SPS) finite sample system identification method for the identification of closed-loop observable stochastic linear systems in state-space form. The solution builds on the theory of matrix-variate regression and instrumental variable methods to construct distribution-free confidence regions for the state-space matrices. Both direct and indirect identification are studied, and the exactness as well as the strong consistency of the construction are proved. Furthermore, a new, computationally efficient ellipsoidal outer-approximation algorithm for the confidence regions is proposed. The new construction results in a semidefinite optimization problem which has an order-of-magnitude smaller number of constraints, as if one applied the ellipsoidal outer-approximation after vectorization. The effectiveness of the approach is also demonstrated empirically via a series of numerical experiments.
△ Less
Submitted 8 June, 2024; v1 submitted 29 January, 2023;
originally announced January 2023.
-
Poisson Equations, Lipschitz Continuity and Controlled Queues
Authors:
Algo Carè,
Balázs Csanád Csáji,
Balázs Gerencsér,
László Gerencsér,
Miklós Rásonyi
Abstract:
The objective of the paper is to revisit a key mathematical technology within the theory of stochastic approximation in a Markovian framework, elaborated in detail by Benveniste, Métivier, and Priouret (1990): the existence, uniqueness and Lipschitz continuity of the solutions of a parameter-dependent Poisson equation associated with a collection of Markov chains on general state spaces. The setup…
▽ More
The objective of the paper is to revisit a key mathematical technology within the theory of stochastic approximation in a Markovian framework, elaborated in detail by Benveniste, Métivier, and Priouret (1990): the existence, uniqueness and Lipschitz continuity of the solutions of a parameter-dependent Poisson equation associated with a collection of Markov chains on general state spaces. The setup and the methodology of our investigation is based on an elegant stability theory for Markov chains, developed by Hairer and Mattingly (2011). The paper provides a transparent analysis of parameter-dependent Poisson equations with convenient conditions. The validity of the proposed conditions is verified for a class of controlled queues.
△ Less
Submitted 13 November, 2022; v1 submitted 22 June, 2019;
originally announced June 2019.
-
Score Permutation Based Finite Sample Inference for Generalized AutoRegressive Conditional Heteroskedasticity (GARCH) Models
Authors:
Balázs Csanád Csáji
Abstract:
A standard model of (conditional) heteroscedasticity, i.e., the phenomenon that the variance of a process changes over time, is the Generalized AutoRegressive Conditional Heteroskedasticity (GARCH) model, which is especially important for economics and finance. GARCH models are typically estimated by the Quasi-Maximum Likelihood (QML) method, which works under mild statistical assumptions. Here, w…
▽ More
A standard model of (conditional) heteroscedasticity, i.e., the phenomenon that the variance of a process changes over time, is the Generalized AutoRegressive Conditional Heteroskedasticity (GARCH) model, which is especially important for economics and finance. GARCH models are typically estimated by the Quasi-Maximum Likelihood (QML) method, which works under mild statistical assumptions. Here, we suggest a finite sample approach, called ScoPe, to construct distribution-free confidence regions around the QML estimate, which have exact coverage probabilities, despite no additional assumptions about moments are made. ScoPe is inspired by the recently developed Sign-Perturbed Sums (SPS) method, which however cannot be applied in the GARCH case. ScoPe works by perturbing the score function using randomly permuted residuals. This produces alternative samples which lead to exact confidence regions. Experiments on simulated and stock market data are also presented, and ScoPe is compared with the asymptotic theory and bootstrap approaches.
△ Less
Submitted 22 July, 2018;
originally announced July 2018.
-
Sign-Perturbed Sums (SPS) with Instrumental Variables for the Identification of ARX Systems - Extended Version
Authors:
Valerio Volpe,
Balázs Cs. Csáji,
Algo Carè,
Erik Weyer,
Marco C. Campi
Abstract:
We propose a generalization of the recently developed system identification method called Sign-Perturbed Sums (SPS). The proposed construction is based on the instrumental variables estimate and, unlike the original SPS, it can construct non-asymptotic confidence regions for linear regression models where the regressors contain past values of the output. Hence, it is applicable to ARX systems, as…
▽ More
We propose a generalization of the recently developed system identification method called Sign-Perturbed Sums (SPS). The proposed construction is based on the instrumental variables estimate and, unlike the original SPS, it can construct non-asymptotic confidence regions for linear regression models where the regressors contain past values of the output. Hence, it is applicable to ARX systems, as well as systems with feedback. We show that this approach provides regions with exact confidence under weak assumptions, i.e., the true parameter is included in the regions with a (user-chosen) exact probability for any finite sample. The paper also proves the strong consistency of the method and proposes a computationally efficient generalization of the previously proposed ellipsoidal outer-approximation. Finally, the new method is demonstrated through numerical experiments, using both real-world and simulated data.
△ Less
Submitted 15 September, 2015;
originally announced September 2015.
-
Cooperative Control in Production and Logistics
Authors:
László Monostori,
Paul Valckenaers,
Alexandre Dolgui,
Hervé Panetto,
Mietek Brdys,
Balázs Csanád Csáji
Abstract:
Classical applications of control engineering and information and communication technology (ICT) in production and logistics are often done in a rigid, centralized and hierarchical way. These inflexible approaches are typically not able to cope with the complexities of the manufacturing environment, such as the instabilities, uncertainties and abrupt changes caused by internal and external disturb…
▽ More
Classical applications of control engineering and information and communication technology (ICT) in production and logistics are often done in a rigid, centralized and hierarchical way. These inflexible approaches are typically not able to cope with the complexities of the manufacturing environment, such as the instabilities, uncertainties and abrupt changes caused by internal and external disturbances, or a large number and variety of interacting, interdependent elements. A paradigm shift, e.g., novel organizing principles and methods, is needed for supporting the interoperability of dynamic alliances of agile and networked systems. Several solution proposals argue that the future of manufacturing and logistics lies in network-like, dynamic, open and reconfigurable systems of cooperative autonomous entities.
The paper overviews various distributed approaches and technologies of control engineering and ICT that can support the realization of cooperative structures from the resource level to the level of networked enterprises. Standard results as well as recent advances from control theory, through cooperative game theory, distributed machine learning to holonic systems, cooperative enterprise modelling, system integration, and autonomous logistics processes are surveyed. A special emphasis is put on the theoretical developments and industrial applications of Robustly Feasible Model Predictive Control (RFMPC). Two case studies are also discussed: i) a holonic, PROSA-based approach to generate short-term forecasts for an additive manufacturing system by means of a delegate multi-agent system (D-MAS); and ii) an application of distributed RFMPC to a drinking water distribution system.
△ Less
Submitted 18 June, 2015;
originally announced June 2015.
