Showing 51–85 of 85 results for author: Grünwald, P

When Ignorance is Bliss

Authors: Peter D. Grunwald, Joseph Y. Halpern

Abstract: It is commonly-accepted wisdom that more information is better, and that information should never be ignored. Here we argue, using both a Bayesian and a non-Bayesian analysis, that in some situations you are better off ignoring information if your uncertainty is represented by a set of probability measures. These include situations in which the information is relevant for the prediction task at ha… ▽ More It is commonly-accepted wisdom that more information is better, and that information should never be ignored. Here we argue, using both a Bayesian and a non-Bayesian analysis, that in some situations you are better off ignoring information if your uncertainty is represented by a set of probability measures. These include situations in which the information is relevant for the prediction task at hand. In the non-Bayesian analysis, we show how ignoring information avoids dilation, the phenomenon that additional pieces of information sometimes lead to an increase in uncertainty. In the Bayesian analysis, we show that for small sample sizes and certain prediction tasks, the Bayesian posterior based on a noninformative prior yields worse predictions than simply ignoring the given information. △ Less

Submitted 27 July, 2014; originally announced July 2014.

Comments: Appears in Proceedings of the Twentieth Conference on Uncertainty in Artificial Intelligence (UAI2004)

Report number: UAI-P-2004-PG-226-234

Updating Probabilities

Authors: Peter D. Grunwald, Joseph Y. Halpern

Abstract: As examples such as the Monty Hall puzzle show, applying conditioning to update a probability distribution on a ``naive space', which does not take into account the protocol used, can often lead to counterintuitive results. Here we examine why. A criterion known as CAR (coarsening at random) in the statistical literature characterizes when ``naive' conditioning in a naive space works. We show… ▽ More As examples such as the Monty Hall puzzle show, applying conditioning to update a probability distribution on a ``naive space', which does not take into account the protocol used, can often lead to counterintuitive results. Here we examine why. A criterion known as CAR (coarsening at random) in the statistical literature characterizes when ``naive' conditioning in a naive space works. We show that the CAR condition holds rather infrequently. We then consider more generalized notions of update such as Jeffrey conditioning and minimizing relative entropy (MRE). We give a generalization of the CAR condition that characterizes when Jeffrey conditioning leads to appropriate answers, but show that there are no such conditions for MRE. This generalizes and interconnects previous results obtained in the literature on CAR and MRE. △ Less

Submitted 27 July, 2014; originally announced July 2014.

Comments: Appears in Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence (UAI2002)

Report number: UAI-P-2002-PG-187-196

Quantum-entangled light from localized emitters

Authors: P. Grünwald, W. Vogel

Abstract: Localized radiation sources are analyzed with respect to the relation of nonclassicality and quantum entanglement of the emitted light. The source field parts of the radiation emitted in different directions are closely related to each other. As a consequence, nonclassicality of the light fields in one direction directly implies entanglement of the field modes in different directions. This implica… ▽ More Localized radiation sources are analyzed with respect to the relation of nonclassicality and quantum entanglement of the emitted light. The source field parts of the radiation emitted in different directions are closely related to each other. As a consequence, nonclassicality of the light fields in one direction directly implies entanglement of the field modes in different directions. This implication can be extended to multipartite-entanglement and multi-time quantum correlations. Given that a nonclassical effect is observed, our approach explicitly yields the multipartite entanglement witnesses. Two examples are considered, the fluorescence radiation of a system of two-level atoms and of excitons in a semiconductor quantum well. △ Less

Submitted 29 August, 2014; v1 submitted 13 May, 2014; originally announced May 2014.

Journal ref: Phys. Rev. A 90, 022334 (2014)

Enhanced squeezing by absorption

Authors: P. Grünwald, W. Vogel

Abstract: Absorption is usually expected to be detrimental to quantum coherence effects. However, the situation for complex absorption spectra has been little studied yet. We consider the resonance fluorescence of excitons in a semiconductor quantum well. The creation of excitons requires absorption of the incoming pump-laser light. Thus, the absorption spectrum of the medium acts as a spectral filter for t… ▽ More Absorption is usually expected to be detrimental to quantum coherence effects. However, the situation for complex absorption spectra has been little studied yet. We consider the resonance fluorescence of excitons in a semiconductor quantum well. The creation of excitons requires absorption of the incoming pump-laser light. Thus, the absorption spectrum of the medium acts as a spectral filter for the emitted light. Surprisingly, absorption can even improve quantum effects, as is demonstrated for the squeezing of the resonance fluorescence of the quantum-well system. This effect can be explained by an improved phase matching due to absorption. △ Less

Submitted 23 March, 2016; v1 submitted 6 May, 2014; originally announced May 2014.

Comments: Substantially rewritten, new focus on squeezing enhancement

Journal ref: Phys. Scr. 91 (2016) 043001

Making Decisions Using Sets of Probabilities: Updating, Time Consistency, and Calibration

Authors: Peter D Grunwald, Joseph Y Halpern

Abstract: We consider how an agent should update her beliefs when her beliefs are represented by a set P of probability distributions, given that the agent makes decisions using the minimax criterion, perhaps the best-studied and most commonly-used criterion in the literature. We adopt a game-theoretic framework, where the agent plays against a bookie, who chooses some distribution from P. We consider two r… ▽ More We consider how an agent should update her beliefs when her beliefs are represented by a set P of probability distributions, given that the agent makes decisions using the minimax criterion, perhaps the best-studied and most commonly-used criterion in the literature. We adopt a game-theoretic framework, where the agent plays against a bookie, who chooses some distribution from P. We consider two reasonable games that differ in what the bookie knows when he makes his choice. Anomalies that have been observed before, like time inconsistency, can be understood as arising because different games are being played, against bookies with different information. We characterize the important special cases in which the optimal decision rules according to the minimax criterion amount to either conditioning or simply ignoring the information. Finally, we consider the relationship between updating and calibration when uncertainty is described by sets of probabilities. Our results emphasize the key role of the rectangularity condition of Epstein and Schneider. △ Less

Submitted 16 January, 2014; originally announced January 2014.

Journal ref: Journal Of Artificial Intelligence Research, Volume 42, pages 393-426, 2011

Superfluorescence spectra of excitons in quantum wells

Authors: P. Grünwald, G. K. G. Burau, H. Stolz, W. Vogel

Abstract: We study the fluorescence light emitted from GaAs excitons in semiconductor quantum wells. The excitons are modeled as interacting bosons. By combining quantum optical methods for the excitonic emission spectrum with many particle descriptions of the transmission through the medium, we can evaluate the spectra outside the well. Comparing with experimental spectra, we get a very good agreement. The… ▽ More We study the fluorescence light emitted from GaAs excitons in semiconductor quantum wells. The excitons are modeled as interacting bosons. By combining quantum optical methods for the excitonic emission spectrum with many particle descriptions of the transmission through the medium, we can evaluate the spectra outside the well. Comparing with experimental spectra, we get a very good agreement. The method helps explaining the main features of the observed spectra. It is demonstrated that the observed spectra show clear evidence of superfluorescent emission. △ Less

Submitted 6 November, 2013; v1 submitted 11 June, 2013; originally announced June 2013.

Comments: revised version; accepted for publication in PRB

Journal ref: Phys. Rev. B vol 88, 195308 (2013)

Optimal Squeezing in the Resonance Fluorescence of Single Photon Emitters

Authors: P. Grünwald, W. Vogel

Abstract: Annual Review of Astronomy and Astrophysics, Volume 51, page 511e studied for coherently driven single photon sources, such as atoms and quantum dots. Maximal squeezing is realized, if the electronic subsystem of the emitter is in a pure quantum state. The purification is achieved by using a cavity as a second decay channel, besides the incoherent coupling to the electromagnetic vacuum. For realis… ▽ More Annual Review of Astronomy and Astrophysics, Volume 51, page 511e studied for coherently driven single photon sources, such as atoms and quantum dots. Maximal squeezing is realized, if the electronic subsystem of the emitter is in a pure quantum state. The purification is achieved by using a cavity as a second decay channel, besides the incoherent coupling to the electromagnetic vacuum. For realistic cavities this yields a purity of the electronic state of more than 99%. Aside from numerical calculations, we also derive approximate analytical results. Based on the approximations, effects are studied which originate from the environment of the emitter, including radiationless dephasing and incoherent pum** of the emitter and the cavity mode. The fragility of squeezing against decoherence is substantially reduced, so that squeezing persists even under hostile conditions. The measurement of squeezing from such light sources is also considered. △ Less

Submitted 29 August, 2013; v1 submitted 11 June, 2013; originally announced June 2013.

Comments: 11 pages, 9 figures, published version

Journal ref: Phys. Rev. A 88, 023837 (2013)

Horizon-Independent Optimal Prediction with Log-Loss in Exponential Families

Authors: Peter Bartlett, Peter Grunwald, Peter Harremoes, Fares Hedayati, Wojciech Kotlowski

Abstract: We study online learning under logarithmic loss with regular parametric models. Hedayati and Bartlett (2012b) showed that a Bayesian prediction strategy with Jeffreys prior and sequential normalized maximum likelihood (SNML) coincide and are optimal if and only if the latter is exchangeable, and if and only if the optimal strategy can be calculated without knowing the time horizon in advance. They… ▽ More We study online learning under logarithmic loss with regular parametric models. Hedayati and Bartlett (2012b) showed that a Bayesian prediction strategy with Jeffreys prior and sequential normalized maximum likelihood (SNML) coincide and are optimal if and only if the latter is exchangeable, and if and only if the optimal strategy can be calculated without knowing the time horizon in advance. They put forward the question what families have exchangeable SNML strategies. This paper fully answers this open problem for one-dimensional exponential families. The exchangeability can happen only for three classes of natural exponential family distributions, namely the Gaussian, Gamma, and the Tweedie exponential family of order 3/2. Keywords: SNML Exchangeability, Exponential Family, Online Learning, Logarithmic Loss, Bayesian Strategy, Jeffreys Prior, Fisher Information1 △ Less

Submitted 19 May, 2013; originally announced May 2013.

Comments: 23 pages

Minimum Encoding Approaches for Predictive Modeling

Authors: Peter D Grunwald, Petri Kontkanen, Petri Myllymaki, Tomi Silander, Henry Tirri

Abstract: We analyze differences between two information-theoretically motivated approaches to statistical inference and model selection: the Minimum Description Length (MDL) principle, and the Minimum Message Length (MML) principle. Based on this analysis, we present two revised versions of MML: a pointwise estimator which gives the MML-optimal single parameter model, and a volumewise estimator which give… ▽ More We analyze differences between two information-theoretically motivated approaches to statistical inference and model selection: the Minimum Description Length (MDL) principle, and the Minimum Message Length (MML) principle. Based on this analysis, we present two revised versions of MML: a pointwise estimator which gives the MML-optimal single parameter model, and a volumewise estimator which gives the MML-optimal region in the parameter space. Our empirical results suggest that with small data sets, the MDL approach yields more accurate predictions than the MML estimators. The empirical results also demonstrate that the revised MML estimators introduced here perform better than the original MML estimator suggested by Wallace and Freeman. △ Less

Submitted 30 January, 2013; originally announced January 2013.

Comments: Appears in Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence (UAI1998)

Report number: UAI-P-1998-PG-183-192

Maximum Entropy and the Glasses You Are Looking Through

Authors: Peter D. Grunwald

Abstract: We give an interpretation of the Maximum Entropy (MaxEnt) Principle in game-theoretic terms. Based on this interpretation, we make a formal distinction between different ways of {em applying/} Maximum Entropy distributions. MaxEnt has frequently been criticized on the grounds that it leads to highly representation dependent results. Our distinction allows us to avoid this problem in many cases. We give an interpretation of the Maximum Entropy (MaxEnt) Principle in game-theoretic terms. Based on this interpretation, we make a formal distinction between different ways of {em applying/} Maximum Entropy distributions. MaxEnt has frequently been criticized on the grounds that it leads to highly representation dependent results. Our distinction allows us to avoid this problem in many cases. △ Less

Submitted 16 January, 2013; originally announced January 2013.

Comments: Appears in Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence (UAI2000)

Report number: UAI-P-2000-PG-238-246

Follow the Leader If You Can, Hedge If You Must

Authors: Steven de Rooij, Tim van Erven, Peter D. Grünwald, Wouter M. Koolen

Abstract: Follow-the-Leader (FTL) is an intuitive sequential prediction strategy that guarantees constant regret in the stochastic setting, but has terrible performance for worst-case data. Other hedging strategies have better worst-case guarantees but may perform much worse than FTL if the data are not maximally adversarial. We introduce the FlipFlop algorithm, which is the first method that provably combi… ▽ More Follow-the-Leader (FTL) is an intuitive sequential prediction strategy that guarantees constant regret in the stochastic setting, but has terrible performance for worst-case data. Other hedging strategies have better worst-case guarantees but may perform much worse than FTL if the data are not maximally adversarial. We introduce the FlipFlop algorithm, which is the first method that provably combines the best of both worlds. As part of our construction, we develop AdaHedge, which is a new way of dynamically tuning the learning rate in Hedge without using the doubling trick. AdaHedge refines a method by Cesa-Bianchi, Mansour and Stoltz (2007), yielding slightly improved worst-case guarantees. By interleaving AdaHedge and FTL, the FlipFlop algorithm achieves regret within a constant factor of the FTL regret, without sacrificing AdaHedge's worst-case guarantees. AdaHedge and FlipFlop do not need to know the range of the losses in advance; moreover, unlike earlier methods, both have the intuitive property that the issued weights are invariant under rescaling and translation of the losses. The losses are also allowed to be negative, in which case they may be interpreted as gains. △ Less

Submitted 17 January, 2013; v1 submitted 3 January, 2013; originally announced January 2013.

Comments: under submission

Journal ref: Journal of Machine Learning Research 15(37):1281-1316, 2014

Optimal Squeezing in Resonance Fluorescence via Atomic-State Purification

Authors: P. Grünwald, W. Vogel

Abstract: Squeezing of atomic resonance fluorescence is shown to be optimized by a properly designed environment, which can be realized by a quasi-resonant cavity. Optimal squeezing is achieved if the atomic coherence is maximized, corresponding to a pure atomic quantum state. The atomic-state purification is achieved by the backaction of the cavity field on the atom, which increases the atomic coherence an… ▽ More Squeezing of atomic resonance fluorescence is shown to be optimized by a properly designed environment, which can be realized by a quasi-resonant cavity. Optimal squeezing is achieved if the atomic coherence is maximized, corresponding to a pure atomic quantum state. The atomic-state purification is achieved by the backaction of the cavity field on the atom, which increases the atomic coherence and decreases the atomic excitation. For realistic cavities, the coupling of the atom to the cavity field yields a purity of the atomic state of more than 99%. The fragility of squeezing against dephasing is substantially reduced in this scenario, which may be important for various applications. △ Less

Submitted 27 June, 2012; originally announced June 2012.

Comments: 6 pages including supplemental information, 3 figures. Accepted for PRL

Journal ref: Phys. Rev. Lett. 109, 013601 (2012)

Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence (2010)

Authors: Peter Grunwald, Peter Spirtes

Abstract: This is the Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, which was held on Catalina Island, CA, July 8 - 11 2010. This is the Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, which was held on Catalina Island, CA, July 8 - 11 2010. △ Less

Submitted 28 August, 2014; v1 submitted 11 May, 2012; originally announced May 2012.

Report number: UAI2010

Adaptive Hedge

Authors: Tim van Erven, Peter Grünwald, Wouter M. Koolen, Steven de Rooij

Abstract: Most methods for decision-theoretic online learning are based on the Hedge algorithm, which takes a parameter called the learning rate. In most previous analyses the learning rate was carefully tuned to obtain optimal worst-case performance, leading to suboptimal performance on easy instances, for example when there exists an action that is significantly better than all others. We propose a new wa… ▽ More Most methods for decision-theoretic online learning are based on the Hedge algorithm, which takes a parameter called the learning rate. In most previous analyses the learning rate was carefully tuned to obtain optimal worst-case performance, leading to suboptimal performance on easy instances, for example when there exists an action that is significantly better than all others. We propose a new way of setting the learning rate, which adapts to the difficulty of the learning problem: in the worst case our procedure still guarantees optimal performance, but on easy instances it achieves much smaller regret. In particular, our adaptive method achieves constant regret in a probabilistic setting, when there exists an action that on average obtains strictly smaller loss than all other actions. We also provide a simulation study comparing our approach to existing methods. △ Less

Submitted 28 October, 2011; originally announced October 2011.

Comments: This is the full version of the paper with the same name that will appear in Advances in Neural Information Processing Systems 24 (NIPS 2011), 2012. The two papers are identical, except that this version contains an extra section of Additional Material

Journal ref: Advances in Neural Information Processing Systems 24, pages 1656-1664, December 2011

Probability-free pricing of adjusted American lookbacks

Authors: A. Philip Dawid, Steven de Rooij, Peter Grunwald, Wouter M. Koolen, Glenn Shafer, Alexander Shen, Nikolai Vereshchagin, Vladimir Vovk

Abstract: Consider an American option that pays G(X^*_t) when exercised at time t, where G is a positive increasing function, X^*_t := \sup_{s\le t}X_s, and X_s is the price of the underlying security at time s. Assuming zero interest rates, we show that the seller of this option can hedge his position by trading in the underlying security if he begins with initial capital X_0\int_{X_0}^{\infty}G(x)x^{-2}dx… ▽ More Consider an American option that pays G(X^*_t) when exercised at time t, where G is a positive increasing function, X^*_t := \sup_{s\le t}X_s, and X_s is the price of the underlying security at time s. Assuming zero interest rates, we show that the seller of this option can hedge his position by trading in the underlying security if he begins with initial capital X_0\int_{X_0}^{\infty}G(x)x^{-2}dx (and this is the smallest initial capital that allows him to hedge his position). This leads to strategies for trading that are always competitive both with a given strategy's current performance and, to a somewhat lesser degree, with its best performance so far. It also leads to methods of statistical testing that avoid sacrificing too much of the maximum statistical significance that they achieve in the course of accumulating data. △ Less

Submitted 20 August, 2011; originally announced August 2011.

Comments: 28 pages, 1 figure

MSC Class: 91G20; 60G42

Explicit Bounds for Entropy Concentration under Linear Constraints

Authors: Kostas N. Oikonomou, Peter D. Grunwald

Abstract: Consider the set of all sequences of $n$ outcomes, each taking one of $m$ values, that satisfy a number of linear constraints. If $m$ is fixed while $n$ increases, most sequences that satisfy the constraints result in frequency vectors whose entropy approaches that of the maximum entropy vector satisfying the constraints. This well-known "entropy concentration" phenomenon underlies the maximum ent… ▽ More Consider the set of all sequences of $n$ outcomes, each taking one of $m$ values, that satisfy a number of linear constraints. If $m$ is fixed while $n$ increases, most sequences that satisfy the constraints result in frequency vectors whose entropy approaches that of the maximum entropy vector satisfying the constraints. This well-known "entropy concentration" phenomenon underlies the maximum entropy method. Existing proofs of the concentration phenomenon are based on limits or asymptotics and unrealistically assume that constraints hold precisely, supporting maximum entropy inference more in principle than in practice. We present, for the first time, non-asymptotic, explicit lower bounds on $n$ for a number of variants of the concentration result to hold to any prescribed accuracies, with the constraints holding up to any specified tolerance, taking into account the fact that allocations of discrete units can satisfy constraints only approximately. Again unlike earlier results, we measure concentration not by deviation from the maximum entropy value, but by the $\ell_1$ and $\ell_2$ distances from the maximum entropy-achieving frequency vector. One of our results holds independently of the alphabet size $m$ and is based on a novel proof technique using the multi-dimensional Berry-Esseen theorem. We illustrate and compare our results using various detailed examples. △ Less

Submitted 30 September, 2015; v1 submitted 29 July, 2011; originally announced July 2011.

Comments: 1) An error affecting sec. 3 has been corrected: the parameters delta and theta cannot be chosen independently. Sec. 3 has been revised up to Theorem 3.15 in sec. 3.6. 2) Some minor updates in sec. 4. 3) Some proofs used in both sec. 3 and sec. 4 have been unified (This version to appear in IEEE Transactions on Information Theory, December 2015)

Raman-assisted Rabi resonances in two-mode cavity QED

Authors: P. Grünwald, S. K. Singh, W. Vogel

Abstract: The dynamics of a vibronic system in a lossy two-mode cavity is studied, with the first mode being resonant to the electronic transition and the second one being nearly resonant due to Raman transitions. We derive analytical solutions for the dynamics of this system. For a properly chosen detuning of the second mode from the exact Raman resonance, we obtain conditions that are closely related to t… ▽ More The dynamics of a vibronic system in a lossy two-mode cavity is studied, with the first mode being resonant to the electronic transition and the second one being nearly resonant due to Raman transitions. We derive analytical solutions for the dynamics of this system. For a properly chosen detuning of the second mode from the exact Raman resonance, we obtain conditions that are closely related to the phenomenon of Rabi resonance as it is well known in laser physics. Such resonances can be observed in the spontaneous emission spectra, where the spectrum of the second mode in the case of weak Raman coupling is enhanced substantially. △ Less

Submitted 10 June, 2011; v1 submitted 10 March, 2011; originally announced March 2011.

Comments: 6 pages, 5 figures

Journal ref: Phys. Rev. A Vol. 83, 063806 (2011)

Prequential Plug-In Codes that Achieve Optimal Redundancy Rates even if the Model is Wrong

Authors: Peter Grünwald, Wojciech Kotłowski

Abstract: We analyse the prequential plug-in codes relative to one-parameter exponential families M. We show that if data are sampled i.i.d. from some distribution outside M, then the redundancy of any plug-in prequential code grows at rate larger than 1/2 ln(n) in the worst case. This means that plug-in codes, such as the Rissanen-Dawid ML code, may behave inferior to other important universal codes such… ▽ More We analyse the prequential plug-in codes relative to one-parameter exponential families M. We show that if data are sampled i.i.d. from some distribution outside M, then the redundancy of any plug-in prequential code grows at rate larger than 1/2 ln(n) in the worst case. This means that plug-in codes, such as the Rissanen-Dawid ML code, may behave inferior to other important universal codes such as the 2-part MDL, Shtarkov and Bayes codes, for which the redundancy is always 1/2 ln(n) + O(1). However, we also show that a slight modification of the ML plug-in code, "almost" in the model, does achieve the optimal redundancy even if the the true distribution is outside M. △ Less

Submitted 3 February, 2010; originally announced February 2010.

Entanglement in atomic resonance fluorescence

Authors: P. Grünwald, W. Vogel

Abstract: The resonance fluorescence from regular atomic systems is shown to represent a continuous source of non-Gaussian entangled radiation propagating in two different directions. For a single atom entanglement occurs under the same conditions as squeezing. For more atoms, the entanglement can be more robust against dephasing than squeezing, hence providing a useful continuous source for various applic… ▽ More The resonance fluorescence from regular atomic systems is shown to represent a continuous source of non-Gaussian entangled radiation propagating in two different directions. For a single atom entanglement occurs under the same conditions as squeezing. For more atoms, the entanglement can be more robust against dephasing than squeezing, hence providing a useful continuous source for various applications of entangled radiation. △ Less

Submitted 10 March, 2010; v1 submitted 14 January, 2010; originally announced January 2010.

Comments: 4 pages, 1 figure

Journal ref: Phys. Rev. Lett. 104, 233602 (2010); Erratum: Phys. Rev. Lett. 106, 119903(E) (2011)

Regret and Jeffreys Integrals in Exp. Families

Authors: Peter Grunwald, Peter Harremoes

Abstract: The problem of whether minimax redundancy, minimax regret and Jeffreys integrals are finite or infinite are discussed. The problem of whether minimax redundancy, minimax regret and Jeffreys integrals are finite or infinite are discussed. △ Less

Submitted 31 March, 2009; originally announced March 2009.

An Algorithmic and a geometric characterization of coarsening at random

Authors: Richard D. Gill, Peter D. Grünwald

Abstract: We show that the class of conditional distributions satisfying the coarsening at random (CAR) property for discrete data has a simple and robust algorithmic description based on randomized uniform multicovers: combinatorial objects generalizing the notion of partition of a set. However, the complexity of a given CAR mechanism can be large: the maximal "height" of the needed multicovers can be ex… ▽ More We show that the class of conditional distributions satisfying the coarsening at random (CAR) property for discrete data has a simple and robust algorithmic description based on randomized uniform multicovers: combinatorial objects generalizing the notion of partition of a set. However, the complexity of a given CAR mechanism can be large: the maximal "height" of the needed multicovers can be exponential in the number of points in the sample space. The results stem from a geometric interpretation of the set of CAR distributions as a convex polytope and a characterization of its extreme points. The hierarchy of CAR models defined in this way could be useful in parsimonious statistical modeling of CAR mechanisms, though the results also raise doubts in applied work as to the meaningfulness of the CAR assumption in its full generality. △ Less

Submitted 5 November, 2008; originally announced November 2008.

Comments: Published in at http://dx.doi.org/10.1214/07-AOS532 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: Note accidental duplicate submission arXiv:0811.0683 MSC Class: 62A01 (Primary); 62N01 (Secondary)

Journal ref: Annals of Statistics 2008, Vol. 36, No. 5, 2409-2422

Algorithmic information theory

Authors: Peter D. Grunwald, Paul M. B. Vitanyi

Abstract: We introduce algorithmic information theory, also known as the theory of Kolmogorov complexity. We explain the main concepts of this quantitative approach to defining `information'. We discuss the extent to which Kolmogorov's and Shannon's information theory have a common purpose, and where they are fundamentally different. We indicate how recent developments within the theory allow one to forma… ▽ More We introduce algorithmic information theory, also known as the theory of Kolmogorov complexity. We explain the main concepts of this quantitative approach to defining `information'. We discuss the extent to which Kolmogorov's and Shannon's information theory have a common purpose, and where they are fundamentally different. We indicate how recent developments within the theory allow one to formally distinguish between `structural' (meaningful) and `random' information as measured by the Kolmogorov structure function, which leads to a mathematical formalization of Occam's razor in inductive inference. We end by discussing some of the philosophical implications of the theory. △ Less

Submitted 17 September, 2008; v1 submitted 16 September, 2008; originally announced September 2008.

Comments: 37 pages, 2 figures, pdf, in: Philosophy of Information, P. Adriaans and J. van Benthem, Eds., A volume in Handbook of the philosophy of science, D. Gabbay, P. Thagard, and J. Woods, Eds., Elsevier, 2008. In version 1 of September 16 the refs are missing. Corrected in version 2 of September 17

Entropy Concentration and the Empirical Coding Game

Authors: Peter Grunwald

Abstract: We give a characterization of Maximum Entropy/Minimum Relative Entropy inference by providing two `strong entropy concentration' theorems. These theorems unify and generalize Jaynes' `concentration phenomenon' and Van Campenhout and Cover's `conditional limit theorem'. The theorems characterize exactly in what sense a prior distribution Q conditioned on a given constraint, and the distribution P… ▽ More We give a characterization of Maximum Entropy/Minimum Relative Entropy inference by providing two `strong entropy concentration' theorems. These theorems unify and generalize Jaynes' `concentration phenomenon' and Van Campenhout and Cover's `conditional limit theorem'. The theorems characterize exactly in what sense a prior distribution Q conditioned on a given constraint, and the distribution P, minimizing the relative entropy D(P ||Q) over all distributions satisfying the constraint, are `close' to each other. We then apply our theorems to establish the relationship between entropy concentration and a game-theoretic characterization of Maximum Entropy Inference due to Topsoe and others. △ Less

Submitted 5 September, 2008; originally announced September 2008.

Comments: A somewhat modified version of this paper was published in Statistica Neerlandica 62(3), pages 374-392, 2008

Catching Up Faster by Switching Sooner: A Prequential Solution to the AIC-BIC Dilemma

Authors: Tim van Erven, Peter Grunwald, Steven de Rooij

Abstract: Bayesian model averaging, model selection and its approximations such as BIC are generally statistically consistent, but sometimes achieve slower rates og convergence than other methods such as AIC and leave-one-out cross-validation. On the other hand, these other methods can br inconsistent. We identify the "catch-up phenomenon" as a novel explanation for the slow convergence of Bayesian method… ▽ More Bayesian model averaging, model selection and its approximations such as BIC are generally statistically consistent, but sometimes achieve slower rates og convergence than other methods such as AIC and leave-one-out cross-validation. On the other hand, these other methods can br inconsistent. We identify the "catch-up phenomenon" as a novel explanation for the slow convergence of Bayesian methods. Based on this analysis we define the switch distribution, a modification of the Bayesian marginal distribution. We show that, under broad conditions,model selection and prediction based on the switch distribution is both consistent and achieves optimal convergence rates, thereby resolving the AIC-BIC dilemma. The method is practical; we give an efficient implementation. The switch distribution has a data compression interpretation, and can thus be viewed as a "prequential" or MDL method; yet it is different from the MDL methods that are usually considered in the literature. We compare the switch distribution to Bayes factor model selection and leave-one-out cross-validation. △ Less

Submitted 7 July, 2008; originally announced July 2008.

Comments: A preliminary version of a part of this paper appeared at the NIPS 2007 conference

MSC Class: 62G99; 94A99

A Game-Theoretic Analysis of Updating Sets of Probabilities

Authors: Peter D. Grunwald, Joseph Y. Halpern

Abstract: We consider how an agent should update her uncertainty when it is represented by a set $¶$ of probability distributions and the agent observes that a random variable $X$ takes on value $x$, given that the agent makes decisions using the minimax criterion, perhaps the best-studied and most commonly-used criterion in the literature. We adopt a game-theoretic framework, where the agent plays agains… ▽ More We consider how an agent should update her uncertainty when it is represented by a set $¶$ of probability distributions and the agent observes that a random variable $X$ takes on value $x$, given that the agent makes decisions using the minimax criterion, perhaps the best-studied and most commonly-used criterion in the literature. We adopt a game-theoretic framework, where the agent plays against a bookie, who chooses some distribution from $¶$. We consider two reasonable games that differ in what the bookie knows when he makes his choice. Anomalies that have been observed before, like time inconsistency, can be understood as arising important because different games are being played, against bookies with different information. We characterize the important special cases in which the optimal decision rules according to the minimax criterion amount to either conditioning or simply ignoring the information. Finally, we consider the relationship between conditioning and calibration when uncertainty is described by sets of probabilities. △ Less

Submitted 20 November, 2007; originally announced November 2007.

ACM Class: I.2.4

An algorithmic and a geometric characterization of Coarsening At Random

Authors: Richard D. Gill, Peter D. Grunwald

Abstract: We show that the class of conditional distributions satisfying the coarsening at Random (CAR) property for discrete data has a simple and robust algorithmic description based on randomized uniform multicovers: combinatorial objects generalizing the notion of partition of a set. However, the complexity of a given CAR mechanism can be large: the maximal "height" of the needed multicovers can be ex… ▽ More We show that the class of conditional distributions satisfying the coarsening at Random (CAR) property for discrete data has a simple and robust algorithmic description based on randomized uniform multicovers: combinatorial objects generalizing the notion of partition of a set. However, the complexity of a given CAR mechanism can be large: the maximal "height" of the needed multicovers can be exponential in the number of points in the sample space. The results stem from a geometric interpretation of the set of CAR distributions as a convex polytope and a characterization of its extreme points. The hierarchy of CAR models defined in this way could be useful in parsimonious statistical modelling of CAR mechanisms, though the results also raise doubts in applied work as to the meaningfulness of the CAR assumption in its full generality. △ Less

Submitted 13 September, 2007; v1 submitted 13 October, 2005; originally announced October 2005.

Comments: 16 pages; accepted in this form for publication by Annals of Statistics

Report number: See also 0811.0683 (duplicate submission) MSC Class: 62A01 (Primary); 62N01; 60A99; 68T37 (Secondary)

Journal ref: The Annals of Statistics 2008, Vol. 36, No. 5, 2409-2422

When Ignorance is Bliss

Authors: Peter D. Grunwald, Joseph Y. Halpern

Abstract: It is commonly-accepted wisdom that more information is better, and that information should never be ignored. Here we argue, using both a Bayesian and a non-Bayesian analysis, that in some situations you are better off ignoring information if your uncertainty is represented by a set of probability measures. These include situations in which the information is relevant for the prediction task at… ▽ More It is commonly-accepted wisdom that more information is better, and that information should never be ignored. Here we argue, using both a Bayesian and a non-Bayesian analysis, that in some situations you are better off ignoring information if your uncertainty is represented by a set of probability measures. These include situations in which the information is relevant for the prediction task at hand. In the non-Bayesian analysis, we show how ignoring information avoids dilation, the phenomenon that additional pieces of information sometimes lead to an increase in uncertainty. In the Bayesian analysis, we show that for small sample sizes and certain prediction tasks, the Bayesian posterior based on a noninformative prior yields worse predictions than simply ignoring the given information. △ Less

Submitted 25 October, 2005; originally announced October 2005.

Comments: In Proceedings of the Twentieth Conference on Uncertainty in AI, 2004, pp. 226-234

ACM Class: I.2.4

Asymptotic Log-loss of Prequential Maximum Likelihood Codes

Authors: Peter Grunwald, Steven de Rooij

Abstract: We analyze the Dawid-Rissanen prequential maximum likelihood codes relative to one-parameter exponential family models M. If data are i.i.d. according to an (essentially) arbitrary P, then the redundancy grows at rate c/2 ln n. We show that c=v1/v2, where v1 is the variance of P, and v2 is the variance of the distribution m* in M that is closest to P in KL divergence. This shows that prequential… ▽ More We analyze the Dawid-Rissanen prequential maximum likelihood codes relative to one-parameter exponential family models M. If data are i.i.d. according to an (essentially) arbitrary P, then the redundancy grows at rate c/2 ln n. We show that c=v1/v2, where v1 is the variance of P, and v2 is the variance of the distribution m* in M that is closest to P in KL divergence. This shows that prequential codes behave quite differently from other important universal codes such as the 2-part MDL, Shtarkov and Bayes codes, for which c=1. This behavior is undesirable in an MDL model selection setting. △ Less

Submitted 1 February, 2005; originally announced February 2005.

Comments: 22 pages, an abstract has been submitted to COLT 2005

ACM Class: E.4

An Empirical Study of MDL Model Selection with Infinite Parametric Complexity

Authors: Steven de Rooij, Peter Grunwald

Abstract: Parametric complexity is a central concept in MDL model selection. In practice it often turns out to be infinite, even for quite simple models such as the Poisson and Geometric families. In such cases, MDL model selection as based on NML and Bayesian inference based on Jeffreys' prior can not be used. Several ways to resolve this problem have been proposed. We conduct experiments to compare and… ▽ More Parametric complexity is a central concept in MDL model selection. In practice it often turns out to be infinite, even for quite simple models such as the Poisson and Geometric families. In such cases, MDL model selection as based on NML and Bayesian inference based on Jeffreys' prior can not be used. Several ways to resolve this problem have been proposed. We conduct experiments to compare and evaluate their behaviour on small sample sizes. We find interestingly poor behaviour for the plug-in predictive code; a restricted NML model performs quite well but it is questionable if the results validate its theoretical motivation. The Bayesian model with the improper Jeffreys' prior is the most dependable. △ Less

Submitted 14 January, 2005; originally announced January 2005.

Comments: 23 pages, 11 graphs

ACM Class: E.3; G.4

Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory

Authors: Peter D. Grunwald, A. Philip Dawid

Abstract: We describe and develop a close relationship between two problems that have customarily been regarded as distinct: that of maximizing entropy, and that of minimizing worst-case expected loss. Using a formulation grounded in the equilibrium theory of zero-sum games between Decision Maker and Nature, these two problems are shown to be dual to each other, the solution to each providing that to th… ▽ More We describe and develop a close relationship between two problems that have customarily been regarded as distinct: that of maximizing entropy, and that of minimizing worst-case expected loss. Using a formulation grounded in the equilibrium theory of zero-sum games between Decision Maker and Nature, these two problems are shown to be dual to each other, the solution to each providing that to the other. Although Topsœdescribed this connection for the Shannon entropy over 20 years ago, it does not appear to be widely known even in that important special case. We here generalize this theory to apply to arbitrary decision problems and loss functions. We indicate how an appropriate generalized definition of entropy can be associated with such a problem, and we show that, subject to certain regularity conditions, the above-mentioned duality continues to apply in this extended context. This simultaneously provides a possible rationale for maximizing entropy and a tool for finding robust Bayes acts. We also describe the essential identity between the problem of maximizing entropy and that of minimizing a related discrepancy or divergence between distributions. This leads to an extension, to arbitrary discrepancies, of a well-known minimax theorem for the case of Kullback-Leibler divergence (the ``redundancy-capacity theorem'' of information theory). For the important case of families of distributions having certain mean values specified, we develop simple sufficient conditions and methods for identifying the desired solutions. △ Less

Submitted 5 October, 2004; originally announced October 2004.

Comments: Published by the Institute of Mathematical Statistics (http://www.imstat.org) in the Annals of Statistics (http://www.imstat.org/aos/) at http://dx.doi.org/10.1214/009053604000000553

Report number: IMS-AOS-AOS231 MSC Class: 62C20 (Primary) 94A17 (Secondary)

Journal ref: Annals of Statistics 2004, Vol. 32, No. 4, 1367-1433

Shannon Information and Kolmogorov Complexity

Authors: Peter Grunwald, Paul Vitanyi

Abstract: We compare the elementary theories of Shannon information and Kolmogorov complexity, the extent to which they have a common purpose, and where they are fundamentally different. We discuss and relate the basic notions of both theories: Shannon entropy versus Kolmogorov complexity, the relation of both to universal coding, Shannon mutual information versus Kolmogorov (`algorithmic') mutual informa… ▽ More We compare the elementary theories of Shannon information and Kolmogorov complexity, the extent to which they have a common purpose, and where they are fundamentally different. We discuss and relate the basic notions of both theories: Shannon entropy versus Kolmogorov complexity, the relation of both to universal coding, Shannon mutual information versus Kolmogorov (`algorithmic') mutual information, probabilistic sufficient statistic versus algorithmic sufficient statistic (related to lossy compression in the Shannon theory versus meaningful information in the Kolmogorov theory), and rate distortion theory versus Kolmogorov's structure function. Part of the material has appeared in print before, scattered through various publications, but this is the first comprehensive systematic comparison. The last mentioned relations are new. △ Less

Submitted 1 October, 2004; originally announced October 2004.

Comments: Survey, LaTeX 54 pages, 3 figures, Submitted to IEEE Trans Information Theory

ACM Class: E.4, H.1.1

Journal ref: There are some errors in this paper draft; when in doubt see the textbook Li, Vitanyi, An Introduction to Kolmogorov Complexity and Its Applications, Springer, 1993, 1997, 2008, 2019

Suboptimal behaviour of Bayes and MDL in classification under misspecification

Authors: Peter Grunwald, John Langford

Abstract: We show that forms of Bayesian and MDL inference that are often applied to classification problems can be *inconsistent*. This means there exists a learning problem such that for all amounts of data the generalization errors of the MDL classifier and the Bayes classifier relative to the Bayesian posterior both remain bounded away from the smallest achievable generalization error. We show that forms of Bayesian and MDL inference that are often applied to classification problems can be *inconsistent*. This means there exists a learning problem such that for all amounts of data the generalization errors of the MDL classifier and the Bayes classifier relative to the Bayesian posterior both remain bounded away from the smallest achievable generalization error. △ Less

Submitted 10 June, 2004; originally announced June 2004.

Comments: This is a slightly longer version of our paper at the COLT (Computational Learning Theory) 2004 Conference, containing two extra pages of discussion of the main results

MSC Class: 62A01; 68T05; 68T10

A tutorial introduction to the minimum description length principle

Authors: Peter Grunwald

Abstract: This tutorial provides an overview of and introduction to Rissanen's Minimum Description Length (MDL) Principle. The first chapter provides a conceptual, entirely non-technical introduction to the subject. It serves as a basis for the technical introduction given in the second chapter, in which all the ideas of the first chapter are made mathematically precise. The main ideas are discussed in gr… ▽ More This tutorial provides an overview of and introduction to Rissanen's Minimum Description Length (MDL) Principle. The first chapter provides a conceptual, entirely non-technical introduction to the subject. It serves as a basis for the technical introduction given in the second chapter, in which all the ideas of the first chapter are made mathematically precise. The main ideas are discussed in great conceptual and technical detail. This tutorial is an extended version of the first two chapters of the collection "Advances in Minimum Description Length: Theory and Application" (edited by P.Grunwald, I.J. Myung and M. Pitt, to be published by the MIT Press, Spring 2005). △ Less

Submitted 4 June, 2004; originally announced June 2004.

Comments: 80 pages 5 figures Report with 2 chapters

MSC Class: 6201; 6801; 68T05; 68T10; 9401

The statistical strength of nonlocality proofs

Authors: Wim van Dam, Peter Grunwald, Richard Gill

Abstract: There exist numerous proofs of Bell's theorem, stating that quantum mechanics is incompatible with local realistic theories of nature. Here we define the strength of such nonlocality proofs in terms of the amount of evidence against local realism provided by the corresponding experiments. This measure tells us how many trials of the experiment we should perform in order to observe a substantial… ▽ More There exist numerous proofs of Bell's theorem, stating that quantum mechanics is incompatible with local realistic theories of nature. Here we define the strength of such nonlocality proofs in terms of the amount of evidence against local realism provided by the corresponding experiments. This measure tells us how many trials of the experiment we should perform in order to observe a substantial violation of local realism. Statistical considerations show that the amount of evidence should be measured by the Kullback-Leibler or relative entropy divergence between the probability distributions over the measurement outcomes that the respective theories predict. The statistical strength of a nonlocality proof is thus determined by the experimental implementation of it that maximizes the Kullback-Leibler divergence from experimental (quantum mechanical) truth to the set of all possible local theories. An implementation includes a specification with which probabilities the different measurement settings are sampled, and hence the maximization is done over all such setting distributions. We analyze two versions of Bell's nonlocality proof (his original proof and an optimized version by Peres), and proofs by Clauser-Horne-Shimony-Holt, Hardy, Mermin, and Greenberger-Horne-Zeilinger. We find that the GHZ proof is at least four and a half times stronger than all other proofs, while of the two-party proofs, the one of CHSH is the strongest. △ Less

Submitted 1 June, 2004; v1 submitted 17 July, 2003; originally announced July 2003.

Comments: 36 pages, no figures, v2: major changes in presentation, no changes in results, conjectures added

Journal ref: IEEE-Transactions on Information Theory 51 (2005), 2812-2835

Updating Probabilities

Authors: Peter D. Grunwald, Joseph Y. Halpern

Abstract: As examples such as the Monty Hall puzzle show, applying conditioning to update a probability distribution on a ``naive space'', which does not take into account the protocol used, can often lead to counterintuitive results. Here we examine why. A criterion known as CAR (``coarsening at random'') in the statistical literature characterizes when ``naive'' conditioning in a naive space works. We s… ▽ More As examples such as the Monty Hall puzzle show, applying conditioning to update a probability distribution on a ``naive space'', which does not take into account the protocol used, can often lead to counterintuitive results. Here we examine why. A criterion known as CAR (``coarsening at random'') in the statistical literature characterizes when ``naive'' conditioning in a naive space works. We show that the CAR condition holds rather infrequently, and we provide a procedural characterization of it, by giving a randomized algorithm that generates all and only distributions for which CAR holds. This substantially extends previous characterizations of CAR. We also consider more generalized notions of update such as Jeffrey conditioning and minimizing relative entropy (MRE). We give a generalization of the CAR condition that characterizes when Jeffrey conditioning leads to appropriate answers, and show that there exist some very simple settings in which MRE essentially never gives the right results. This generalizes and interconnects previous results obtained in the literature on CAR and MRE. △ Less

Submitted 23 June, 2003; originally announced June 2003.

Comments: This is an expanded version of a paper that appeared in Proceedings of the Eighteenth Conference on Uncertainty in AI, 2002, pp. 187--196. to appear, Journal of AI Research

ACM Class: I.2.4