Showing 1–4 of 4 results for author: Sterkenburg, T F

Statistical learning theory and Occam's razor: The argument from empirical risk minimization

Authors: Tom F. Sterkenburg

Abstract: This paper considers the epistemic justification for a simplicity preference in inductive inference that may be obtained from the machine learning framework of statistical learning theory. Uniting elements from both earlier arguments suggesting and rejecting such a justification, the paper spells out a qualified means-ends and model-relative justificatory argument, built on statistical learning th… ▽ More This paper considers the epistemic justification for a simplicity preference in inductive inference that may be obtained from the machine learning framework of statistical learning theory. Uniting elements from both earlier arguments suggesting and rejecting such a justification, the paper spells out a qualified means-ends and model-relative justificatory argument, built on statistical learning theory's central mathematical learning guarantee for the method of empirical risk minimization. △ Less

Submitted 21 December, 2023; originally announced December 2023.

On characterizations of learnability with computable learners

Authors: Tom F. Sterkenburg

Abstract: We study computable PAC (CPAC) learning as introduced by Agarwal et al. (2020). First, we consider the main open question of finding characterizations of proper and improper CPAC learning. We give a characterization of a closely related notion of strong CPAC learning, and provide a negative answer to the COLT open problem posed by Agarwal et al. (2021) whether all decidably representable VC classe… ▽ More We study computable PAC (CPAC) learning as introduced by Agarwal et al. (2020). First, we consider the main open question of finding characterizations of proper and improper CPAC learning. We give a characterization of a closely related notion of strong CPAC learning, and provide a negative answer to the COLT open problem posed by Agarwal et al. (2021) whether all decidably representable VC classes are improperly CPAC learnable. Second, we consider undecidability of (computable) PAC learnability. We give a simple general argument to exhibit such ndecidability, and initiate a study of the arithmetical complexity of learnability. We briefly discuss the relation to the undecidability result of Ben-David et al. (2019), that motivated the work of Agarwal et al. △ Less

Submitted 18 July, 2022; v1 submitted 10 February, 2022; originally announced February 2022.

Comments: Final version, as accepted for COLT 2022. Changes w.r.t. previous arXiv version: In response to reviewer comments, major revision of the discussion in Section 3.2, incl. reformulation of Question 2 (that in its original form had an easy answer) and retraction of Definition 6 and Proposition 2 (that lost their relevance in the revised discussion). Minor textual revisions elsewhere

Journal ref: Proceedings of Thirty Fifth Conference on Learning Theory (COLT 2022), PMLR 178:3365-3379, 2022

The no-free-lunch theorems of supervised learning

Authors: Tom F. Sterkenburg, Peter D. Grünwald

Abstract: The no-free-lunch theorems promote a skeptical conclusion that all possible machine learning algorithms equally lack justification. But how could this leave room for a learning theory, that shows that some algorithms are better than others? Drawing parallels to the philosophy of induction, we point out that the no-free-lunch results presuppose a conception of learning algorithms as purely data-dri… ▽ More The no-free-lunch theorems promote a skeptical conclusion that all possible machine learning algorithms equally lack justification. But how could this leave room for a learning theory, that shows that some algorithms are better than others? Drawing parallels to the philosophy of induction, we point out that the no-free-lunch results presuppose a conception of learning algorithms as purely data-driven. On this conception, every algorithm must have an inherent inductive bias, that wants justification. We argue that many standard learning algorithms should rather be understood as model-dependent: in each application they also require for input a model, representing a bias. Generic algorithms themselves, they can be given a model-relative justification. △ Less

Submitted 9 February, 2022; originally announced February 2022.

Journal ref: Synthese 199:9979-10015 (2021)

A generalized characterization of algorithmic probability

Authors: Tom F. Sterkenburg

Abstract: An a priori semimeasure (also known as "algorithmic probability" or "the Solomonoff prior" in the context of inductive inference) is defined as the transformation, by a given universal monotone Turing machine, of the uniform measure on the infinite strings. It is shown in this paper that the class of a priori semimeasures can equivalently be defined as the class of transformations, by all compatib… ▽ More An a priori semimeasure (also known as "algorithmic probability" or "the Solomonoff prior" in the context of inductive inference) is defined as the transformation, by a given universal monotone Turing machine, of the uniform measure on the infinite strings. It is shown in this paper that the class of a priori semimeasures can equivalently be defined as the class of transformations, by all compatible universal monotone Turing machines, of any continuous computable measure in place of the uniform measure. Some consideration is given to possible implications for the prevalent association of algorithmic probability with certain foundational statistical principles. △ Less

Submitted 28 June, 2016; v1 submitted 24 August, 2015; originally announced August 2015.