-
More Efficient Exploration with Symbolic Priors on Action Sequence Equivalences
Authors:
Toby Johnstone,
Nathan Grinsztajn,
Johan Ferret,
Philippe Preux
Abstract:
Incorporating prior knowledge in reinforcement learning algorithms is mainly an open question. Even when insights about the environment dynamics are available, reinforcement learning is traditionally used in a tabula rasa setting and must explore and learn everything from scratch. In this paper, we consider the problem of exploiting priors about action sequence equivalence: that is, when different…
▽ More
Incorporating prior knowledge in reinforcement learning algorithms is mainly an open question. Even when insights about the environment dynamics are available, reinforcement learning is traditionally used in a tabula rasa setting and must explore and learn everything from scratch. In this paper, we consider the problem of exploiting priors about action sequence equivalence: that is, when different sequences of actions produce the same effect. We propose a new local exploration strategy calibrated to minimize collisions and maximize new state visitations. We show that this strategy can be computed at little cost, by solving a convex optimization problem. By replacing the usual epsilon-greedy strategy in a DQN, we demonstrate its potential in several environments with various dynamic structures.
△ Less
Submitted 7 November, 2021; v1 submitted 20 October, 2021;
originally announced October 2021.
-
Strongly uplifting cardinals and the boldface resurrection axioms
Authors:
Joel David Hamkins,
Thomas A. Johnstone
Abstract:
We introduce the strongly uplifting cardinals, which are equivalently characterized, we prove, as the superstrongly unfoldable cardinals and also as the almost hugely unfoldable cardinals, and we show that their existence is equiconsistent over ZFC with natural instances of the boldface resurrection axiom, such as the boldface resurrection axiom for proper forcing.
We introduce the strongly uplifting cardinals, which are equivalently characterized, we prove, as the superstrongly unfoldable cardinals and also as the almost hugely unfoldable cardinals, and we show that their existence is equiconsistent over ZFC with natural instances of the boldface resurrection axiom, such as the boldface resurrection axiom for proper forcing.
△ Less
Submitted 30 October, 2014; v1 submitted 11 March, 2014;
originally announced March 2014.
-
On ground model definability
Authors:
Victoria Gitman,
Thomas A. Johnstone
Abstract:
Laver, and Woodin independently, showed that models of ${\rm ZFC}$ are uniformly definable in their set-forcing extensions, using a ground model parameter. We investigate ground model definability for models of fragments of ${\rm ZFC}$, particularly of ${\rm ZF}+{\rm DC}_δ$ and of ${\rm ZFC}^-$, and we obtain both positive and negative results. Generalizing the results of Laver and Woodin, we show…
▽ More
Laver, and Woodin independently, showed that models of ${\rm ZFC}$ are uniformly definable in their set-forcing extensions, using a ground model parameter. We investigate ground model definability for models of fragments of ${\rm ZFC}$, particularly of ${\rm ZF}+{\rm DC}_δ$ and of ${\rm ZFC}^-$, and we obtain both positive and negative results. Generalizing the results of Laver and Woodin, we show that models of ${\rm ZF}+{\rm DC}_δ$ are uniformly definable in their set-forcing extensions by posets admitting a gap at $δ$, using a ground model parameter. In particular, this means that models of ${\rm ZF}+{\rm DC}_δ$ are uniformly definable in their forcing extensions by posets of size less than $δ$. We also show that it is consistent for ground model definability to fail for models of ${\rm ZFC}^-$ of the form $H_{κ^+}$. Using forcing, we produce a ${\rm ZFC}$ universe in which there is a cardinal $κ>\!>ω$ such that $H_{κ^+}$ is not definable in its Cohen forcing extension. As a corollary, we show that there is always a countable transitive model of ${\rm ZFC}^-$ violating ground model definability. These results turn out to have a bearing on ground model definability for models of ${\rm ZFC}$. It follows from our proof methods that the hereditary size of the parameter that Woodin used to define a ${\rm ZFC}$ model in its set-forcing extension is best possible.
△ Less
Submitted 26 November, 2013;
originally announced November 2013.
-
Resurrection axioms and uplifting cardinals
Authors:
Joel David Hamkins,
Thomas A. Johnstone
Abstract:
We introduce the resurrection axioms, a new class of forcing axioms, and the uplifting cardinals, a new large cardinal notion, and prove that various instances of the resurrection axioms are equiconsistent over ZFC with the existence of an uplifting cardinal.
We introduce the resurrection axioms, a new class of forcing axioms, and the uplifting cardinals, a new large cardinal notion, and prove that various instances of the resurrection axioms are equiconsistent over ZFC with the existence of an uplifting cardinal.
△ Less
Submitted 25 February, 2014; v1 submitted 12 July, 2013;
originally announced July 2013.
-
What is the theory ZFC without power set?
Authors:
Victoria Gitman,
Joel David Hamkins,
Thomas A. Johnstone
Abstract:
We show that the theory ZFC-, consisting of the usual axioms of ZFC but with the power set axiom removed-specifically axiomatized by extensionality, foundation, pairing, union, infinity, separation, replacement and the assertion that every set can be well-ordered-is weaker than commonly supposed and is inadequate to establish several basic facts often desired in its context. For example, there are…
▽ More
We show that the theory ZFC-, consisting of the usual axioms of ZFC but with the power set axiom removed-specifically axiomatized by extensionality, foundation, pairing, union, infinity, separation, replacement and the assertion that every set can be well-ordered-is weaker than commonly supposed and is inadequate to establish several basic facts often desired in its context. For example, there are models of ZFC- in which $ω_1$ is singular, in which every set of reals is countable, yet $ω_1$ exists, in which there are sets of reals of every size $\aleph_n$, but none of size $\aleph_ω$, and therefore, in which the collection axiom sceme fails; there are models of ZFC- for which the Los theorem fails, even when the ultrapower is well-founded and the measure exists inside the model; there are models of ZFC- for which the Gaifman theorem fails, in that there is an embedding $j:M\to N$ of ZFC- models that is $Σ_1$-elementary and cofinal, but not elementary; there are elementary embeddings $j:M\to N$ of ZFC- models whose cofinal restriction $j:M\to \bigcup j``M$ is not elementary. Moreover, the collection of formulas that are provably equivalent in ZFC- to a $Σ_1$-formula or a $Π_1$-formula is not closed under bounded quantification. Nevertheless, these deficits of ZFC- are completely repaired by strengthening it to the theory $ZFC^-$, obtained by using collection rather than replacement in the axiomatization above. These results extend prior work of Zarach.
△ Less
Submitted 3 August, 2015; v1 submitted 11 October, 2011;
originally announced October 2011.
