Showing 1–50 of 82 results for author: Diaconis, P

Enumerative Theory for the Tsetlin Library

Authors: Sourav Chatterjee, Persi Diaconis, Gene B. Kim

Abstract: The Tsetlin library is a well-studied Markov chain on the symmetric group $S_n$. It has stationary distribution $π(σ)$ the Luce model, a nonuniform distribution on $S_n$, which appears in psychology, horse race betting, and tournament poker. Simple enumerative questions, such as ``what is the distribution of the top $k$ cards?'' or ``what is the distribution of the bottom $k$ cards?'' are long ope… ▽ More The Tsetlin library is a well-studied Markov chain on the symmetric group $S_n$. It has stationary distribution $π(σ)$ the Luce model, a nonuniform distribution on $S_n$, which appears in psychology, horse race betting, and tournament poker. Simple enumerative questions, such as ``what is the distribution of the top $k$ cards?'' or ``what is the distribution of the bottom $k$ cards?'' are long open. We settle these questions and draw attention to a host of parallel questions on the extension to the chambers of a hyperplane arrangement. △ Less

Submitted 28 June, 2023; originally announced June 2023.

Comments: 23 pages, 1 figure

MSC Class: 60B15; 60J99

On a Markov construction of couplings

Authors: Persi Diaconis, Laurent Miclo

Abstract: For $N\in\mathbb{N}$, let $π_N$ be the law of the number of fixed points of a random permutation of $\{1, 2, ..., N\}$. Let $\mathcal{P}$ be a Poisson law of parameter 1.A classical result shows that $π_N$ converges to $\mathcal{P}$ for large $N$ and indeed in total variation $$\left\Vert π_N-\mathcal{P}\right\Vert_{\mathrm{tv}} \leq \frac{2^N}{(N+1)!}$$ This implies that $π_N$ and $\mathcal{P}$… ▽ More For $N\in\mathbb{N}$, let $π_N$ be the law of the number of fixed points of a random permutation of $\{1, 2, ..., N\}$. Let $\mathcal{P}$ be a Poisson law of parameter 1.A classical result shows that $π_N$ converges to $\mathcal{P}$ for large $N$ and indeed in total variation $$\left\Vert π_N-\mathcal{P}\right\Vert_{\mathrm{tv}} \leq \frac{2^N}{(N+1)!}$$ This implies that $π_N$ and $\mathcal{P}$ can be coupled to at least this accuracy. This paper constructs such a coupling (a long open problem) using the machinery of intertwining of two Markov chains. This method shows promise for related problems of random matrix theory. △ Less

Submitted 4 May, 2023; originally announced May 2023.

Double Coset Markov Chains

Authors: Persi Diaconis, Arun Ram, Mackenzie Simper

Abstract: Let $G$ be a finite group. Let $H, K$ be subgroups of $G$ and $H \backslash G / K$ the double coset space. Let $Q$ be a probability on $G$ which is constant on conjugacy classes ($Q(s^{-1} t s) = Q(t)$). The random walk driven by $Q$ on $G$ projects to a Markov chain on $H \backslash G /K$. This allows analysis of the lumped chain using the representation theory of $G$. Examples include coagulatio… ▽ More Let $G$ be a finite group. Let $H, K$ be subgroups of $G$ and $H \backslash G / K$ the double coset space. Let $Q$ be a probability on $G$ which is constant on conjugacy classes ($Q(s^{-1} t s) = Q(t)$). The random walk driven by $Q$ on $G$ projects to a Markov chain on $H \backslash G /K$. This allows analysis of the lumped chain using the representation theory of $G$. Examples include coagulation-fragmentation processes and natural Markov chains on contingency tables. Our main example projects the random transvections walk on $GL_n(q)$ onto a Markov chain on $S_n$ via the Bruhat decomposition. The chain on $S_n$ has a Mallows stationary distribution and interesting mixing time behavior. The projection illuminates the combinatorics of Gaussian elimination. Along the way, we give a representation of the sum of transvections in the Hecke algebra of double cosets. Some extensions and examples of double coset Markov chains with $G$ a compact group are discussed. △ Less

Submitted 22 August, 2022; originally announced August 2022.

Comments: working draft, comments welcome!

A random walk on the Rado graph

Authors: Sourav Chatterjee, Persi Diaconis, Laurent Miclo

Abstract: The Rado graph, also known as the random graph $G(\infty, p)$, is a classical limit object for finite graphs. We study natural ball walks as a way of understanding the geometry of this graph. For the walk started at $i$, we show that order $\log_2^*i$ steps are sufficient, and for infinitely many $i$, necessary for convergence to stationarity. The proof involves an application of Hardy's inequalit… ▽ More The Rado graph, also known as the random graph $G(\infty, p)$, is a classical limit object for finite graphs. We study natural ball walks as a way of understanding the geometry of this graph. For the walk started at $i$, we show that order $\log_2^*i$ steps are sufficient, and for infinitely many $i$, necessary for convergence to stationarity. The proof involves an application of Hardy's inequality for trees. △ Less

Submitted 13 May, 2022; originally announced May 2022.

Comments: 43 pages

In praise (and search) of J. V. Uspensky

Authors: Persi Diaconis, Sandy Zabell

Abstract: The two of us have shared a fascination with James Victor Uspensky's 1937 textbook $Introduction \, to \, Mathematical \, Probability$ ever since our graduate student days: it contains many interesting results not found in other books on the same subject in the English language, together with many non-trivial examples, all clearly stated with careful proofs. We present some of Uspensky's gems to a… ▽ More The two of us have shared a fascination with James Victor Uspensky's 1937 textbook $Introduction \, to \, Mathematical \, Probability$ ever since our graduate student days: it contains many interesting results not found in other books on the same subject in the English language, together with many non-trivial examples, all clearly stated with careful proofs. We present some of Uspensky's gems to a modern audience ho** to tempt others to read Uspensky for themselves, as well as report on a few of the other mathematical topics he also wrote about (for example, his book on number theory contains early results about perfect shuffles). Uspensky led an interesting life: a member of the Russian Academy of Sciences, he spoke at the 1924 International Congress of Mathematicians in Toronto before leaving Russia in 1929 and coming to the US and Stanford. Comparatively little has been written about him in English; the second half of this paper attempts to remedy this. △ Less

Submitted 31 January, 2022; originally announced January 2022.

Comments: 49 pages

Isomorphisms between random graphs

Authors: Sourav Chatterjee, Persi Diaconis

Abstract: Consider two independent Erdős-Rényi $G(N,1/2)$ graphs. We show that with probability tending to $1$ as $N\to\infty$, the largest induced isomorphic subgraph has size either $\lfloor x_N-\varepsilon_N\rfloor$ or $\lfloor x_N+\varepsilon_N \rfloor$, where $x_N=4\log_2 N -2 \log_2 \log_2 N - 2\log_2(4/e)+1$ and $\varepsilon_N = (4\log_2 N)^{-1/2}$. Using similar techniques, we also show that if… ▽ More Consider two independent Erdős-Rényi $G(N,1/2)$ graphs. We show that with probability tending to $1$ as $N\to\infty$, the largest induced isomorphic subgraph has size either $\lfloor x_N-\varepsilon_N\rfloor$ or $\lfloor x_N+\varepsilon_N \rfloor$, where $x_N=4\log_2 N -2 \log_2 \log_2 N - 2\log_2(4/e)+1$ and $\varepsilon_N = (4\log_2 N)^{-1/2}$. Using similar techniques, we also show that if $Γ_1$ and $Γ_2$ are independent $G(n,1/2)$ and $G(N,1/2)$ random graphs, then $Γ_2$ contains an isomorphic copy of $Γ_1$ as an induced subgraph with high probability if $n\le \lfloor y_N - \varepsilon_N \rfloor$ and does not contain an isomorphic copy of $Γ_1$ as an induced subgraph with high probability if $n>\lfloor y_N+\varepsilon_N \rfloor$, where $y_N=2\log_2 N+1$ and $\varepsilon_N$ is as above. △ Less

Submitted 30 December, 2022; v1 submitted 9 August, 2021; originally announced August 2021.

Comments: 17 pages. To appear in J. Combin. Theory B

MSC Class: 05C80; 05C60; 60C05

Complexity and randomness in the Heisenberg groups (and beyond)

Authors: Persi Diaconis, Maryanthe Malliaris

Abstract: By studying the commuting graphs of conjugacy classes of the sequence of Heisenberg groups $H_{2n+1}(p)$ and their limit $H_\infty(p)$ we find pseudo-random behavior (and the random graph in the limiting case). This makes a nice case study for transfer of information between finite and infinite objects. Some of this behavior transfers to the problem of understanding what makes understanding the ch… ▽ More By studying the commuting graphs of conjugacy classes of the sequence of Heisenberg groups $H_{2n+1}(p)$ and their limit $H_\infty(p)$ we find pseudo-random behavior (and the random graph in the limiting case). This makes a nice case study for transfer of information between finite and infinite objects. Some of this behavior transfers to the problem of understanding what makes understanding the character theory of the uni-upper-triangular group (mod p) "wild". Our investigations in this paper may be seen as a meditation on the question: is randomness simple or is it complicated? △ Less

Submitted 13 August, 2021; v1 submitted 6 July, 2021; originally announced July 2021.

Sequential importance sampling for estimating expectations over the space of perfect matchings

Authors: Yeganeh Alimohammadi, Persi Diaconis, Mohammad Roghani, Amin Saberi

Abstract: This paper makes three contributions to estimating the number of perfect matching in bipartite graphs. First, we prove that the popular sequential importance sampling algorithm works in polynomial time for dense bipartite graphs. More carefully, our algorithm gives a $(1\pmε)$-approximation for the number of perfect matchings of a $λ$-dense bipartite graph, using $O(n^{\frac{1-2λ}λε^{-2}})$ sample… ▽ More This paper makes three contributions to estimating the number of perfect matching in bipartite graphs. First, we prove that the popular sequential importance sampling algorithm works in polynomial time for dense bipartite graphs. More carefully, our algorithm gives a $(1\pmε)$-approximation for the number of perfect matchings of a $λ$-dense bipartite graph, using $O(n^{\frac{1-2λ}λε^{-2}})$ samples. With size $n$ on each side and for $\frac{1}{2}>λ>0$, a $λ$-dense bipartite graph has all degrees greater than $(λ+\frac{1}{2})n$. Second, practical applications of the algorithm require many calls to matching algorithms. A novel preprocessing step is provided which makes significant improvements. Third, three applications are provided. The first is for counting Latin squares, the second is a practical way of computing the greedy algorithm for a card-guessing game with feedback, and the third is for stochastic block models. In all three examples, sequential importance sampling allows treating practical problems of reasonably large sizes. △ Less

Submitted 5 July, 2022; v1 submitted 2 July, 2021; originally announced July 2021.

Comments: To appear in Annals of Applied Probability

MSC Class: 65C05; 05B15

Discussion of `A Gibbs sampler for a class of random convex polytopes'

Authors: Persi Diaconis, Guanyang Wang

Abstract: This is a contribution for the discussion on "A Gibbs sampler for a class of random convex polytopes" by Pierre E. Jacob, Ruobin Gong, Paul T. Edlefsen and Arthur P. Dempster to appear in the Journal of American Statistical Association. This is a contribution for the discussion on "A Gibbs sampler for a class of random convex polytopes" by Pierre E. Jacob, Ruobin Gong, Paul T. Edlefsen and Arthur P. Dempster to appear in the Journal of American Statistical Association. △ Less

Submitted 14 April, 2021; originally announced April 2021.

Comments: 5 pages

Statistical Enumeration of Groups by Double Cosets

Authors: Persi Diaconis, Mackenzie Simper

Abstract: Let $H$ and $K$ be subgroups of a finite group $G$. Pick $g \in G$ uniformly at random. We study the distribution induced on double cosets. Three examples are treated in detail: 1) $H = K = $ the Borel subgroup in $GL_n(\mathbb{F}_q)$. This leads to new theorems for Mallows measure on permutations and new insights into the LU matrix factorization. 2) The double cosets of the hyperoctahedral group… ▽ More Let $H$ and $K$ be subgroups of a finite group $G$. Pick $g \in G$ uniformly at random. We study the distribution induced on double cosets. Three examples are treated in detail: 1) $H = K = $ the Borel subgroup in $GL_n(\mathbb{F}_q)$. This leads to new theorems for Mallows measure on permutations and new insights into the LU matrix factorization. 2) The double cosets of the hyperoctahedral group inside $S_{2n}$, which leads to new applications of the Ewens's sampling formula of mathematical genetics. 3) Finally, if $H$ and $K$ are parabolic subgroups of $S_n$, the double cosets are `contingency tables', studied by statisticians for the past 100 years. △ Less

Submitted 5 March, 2021; v1 submitted 8 February, 2021; originally announced February 2021.

Comments: 46 pages, 1 figure; minor edits

Hahn polynomials and the Burnside process

Authors: Persi Diaconis, Chenyang Zhong

Abstract: We study a natural Markov chain on $\{0,1,\cdots,n\}$ with eigenvectors the Hahn polynomials. This explicit diagonalization makes it possible to get sharp rates of convergence to stationarity. The process, the Burnside process, is a special case of the celebrated `Swendsen-Wang' or `data augmentation' algorithm. The description involves the beta-binomial distribution and Mallows model on permutati… ▽ More We study a natural Markov chain on $\{0,1,\cdots,n\}$ with eigenvectors the Hahn polynomials. This explicit diagonalization makes it possible to get sharp rates of convergence to stationarity. The process, the Burnside process, is a special case of the celebrated `Swendsen-Wang' or `data augmentation' algorithm. The description involves the beta-binomial distribution and Mallows model on permutations. It introduces a useful generalization of the Burnside process. △ Less

Submitted 20 May, 2021; v1 submitted 26 December, 2020; originally announced December 2020.

Comments: 29 pages; added Section 6 on a continuous limit of the Burnside process; improved the range of the parameter in Theorem 5.2; minor edits

Journal ref: Ramanujan J (2021)

Guessing about Guessing: Practical Strategies for Card Guessing with Feedback

Authors: Persi Diaconis, Ron Graham, Sam Spiro

Abstract: In simple card games, cards are dealt one at a time and the player guesses each card sequentially. We study problems where feedback (e.g. correct/incorrect) is given after each guess. For decks with repeated values (as in blackjack where suits do not matter) the optimal strategy differs from the "greedy strategy" (of guessing a most likely card each round). Further, both optimal and greedy strateg… ▽ More In simple card games, cards are dealt one at a time and the player guesses each card sequentially. We study problems where feedback (e.g. correct/incorrect) is given after each guess. For decks with repeated values (as in blackjack where suits do not matter) the optimal strategy differs from the "greedy strategy" (of guessing a most likely card each round). Further, both optimal and greedy strategies are far too complicated for real time use by human players. Our main results show that simple heuristics perform close to optimal. △ Less

Submitted 17 July, 2021; v1 submitted 7 December, 2020; originally announced December 2020.

Comments: 20 pages, minor typos corrected, to appear in the American Mathematical Monthly excluding Section 6

MSC Class: 60C05

Gambler's Ruin and the ICM

Authors: Persi Diaconis, Stewart N. Ethier

Abstract: Consider gambler's ruin with three players, 1, 2, and 3, having initial capitals $A$, $B$, and $C$ units. At each round a pair of players is chosen (uniformly at random) and a fair coin flip is made resulting in the transfer of one unit between these two players. Eventually, one of the players is eliminated and play continues with the remaining two. Let $σ\in S_3$ be the elimination order (e.g.,… ▽ More Consider gambler's ruin with three players, 1, 2, and 3, having initial capitals $A$, $B$, and $C$ units. At each round a pair of players is chosen (uniformly at random) and a fair coin flip is made resulting in the transfer of one unit between these two players. Eventually, one of the players is eliminated and play continues with the remaining two. Let $σ\in S_3$ be the elimination order (e.g., $σ=132$ means player 1 is eliminated first and player 3 is eliminated second, leaving player 2 with $A+B+C$ units). We seek approximations (and exact formulas) for the elimination order probabilities $P_{A,B,C}(σ)$. Exact, as well as arbitrarily precise, computation of these probabilities is possible when $N:=A+B+C$ is not too large. Linear interpolation can then give reasonable approximations for large $N$. One frequently used approximation, the independent chip model (ICM), is shown to be inadequate. A regression adjustment is proposed, which seems to give good approximations to the elimination order probabilities. △ Less

Submitted 18 April, 2021; v1 submitted 15 November, 2020; originally announced November 2020.

Comments: 32 pages, 5 figure files

Card Guessing with Partial Feedback

Authors: Persi Diaconis, Ron Graham, Xiaoyu He, Sam Spiro

Abstract: Consider the following experiment: a deck with $m$ copies of $n$ different card types is randomly shuffled, and a guesser attempts to guess the cards sequentially as they are drawn. Each time a guess is made, some amount of "feedback" is given. For example, one could tell the guesser the true identity of the card they just guessed (the complete feedback model) or they could be told nothing at all… ▽ More Consider the following experiment: a deck with $m$ copies of $n$ different card types is randomly shuffled, and a guesser attempts to guess the cards sequentially as they are drawn. Each time a guess is made, some amount of "feedback" is given. For example, one could tell the guesser the true identity of the card they just guessed (the complete feedback model) or they could be told nothing at all (the no feedback model). In this paper we explore a partial feedback model, where upon guessing a card, the guesser is only told whether or not their guess was correct. We show in this setting that, uniformly in $n$, at most $m+O(m^{3/4}\log m)$ cards can be guessed correctly in expectation. This resolves a question of Diaconis and Graham from 1981, where even the $m=2$ case was open. △ Less

Submitted 18 July, 2022; v1 submitted 10 October, 2020; originally announced October 2020.

Comments: 24 pages; minor errors corrected

MSC Class: 60C05

Journal ref: Combinator. Probab. Comp. 31 (2022) 1-20

The square and add Markov chain

Authors: Persi Diaconis, Jimmy He, I. Martin Isaacs

Abstract: Squaring and adding $\pm 1$ mod p generates a curiously intractable random walk. A similar process over the finite field $\mathbf{F}_q$ (with $q=2^d$) leads to novel connections between elementary Galois theory and probability. Squaring and adding $\pm 1$ mod p generates a curiously intractable random walk. A similar process over the finite field $\mathbf{F}_q$ (with $q=2^d$) leads to novel connections between elementary Galois theory and probability. △ Less

Submitted 25 August, 2020; originally announced August 2020.

Comments: 16 pages. Comments are welcome!

Speeding up Markov chains with deterministic jumps

Authors: Sourav Chatterjee, Persi Diaconis

Abstract: We show that the convergence of finite state space Markov chains to stationarity can often be considerably speeded up by alternating every step of the chain with a deterministic move. Under fairly general conditions, we show that not only do such schemes exist, they are numerous. We show that the convergence of finite state space Markov chains to stationarity can often be considerably speeded up by alternating every step of the chain with a deterministic move. Under fairly general conditions, we show that not only do such schemes exist, they are numerous. △ Less

Submitted 25 August, 2020; v1 submitted 23 April, 2020; originally announced April 2020.

Comments: 22 pages. To appear in Probab. Theory Related Fields, special issue in honor of Harry Kesten. This revision has some minor corrections and a new proposition due to Jimmy He in the section on open problems

MSC Class: 60J10; 60J22

A phase transition for repeated averages

Authors: Sourav Chatterjee, Persi Diaconis, Allan Sly, Lingfu Zhang

Abstract: Let $x_1,\ldots,x_n$ be a fixed sequence of real numbers. At each stage, pick two indices $I$ and $J$ uniformly at random and replace $x_I$, $x_J$ by $(x_I+x_J)/2$, $(x_I+x_J)/2$. Clearly all the coordinates converge to $(x_1+\cdots+x_n)/n$. We determine the rate of convergence, establishing a sharp "cutoff" transition, answering a question of Jean Bourgain. Let $x_1,\ldots,x_n$ be a fixed sequence of real numbers. At each stage, pick two indices $I$ and $J$ uniformly at random and replace $x_I$, $x_J$ by $(x_I+x_J)/2$, $(x_I+x_J)/2$. Clearly all the coordinates converge to $(x_1+\cdots+x_n)/n$. We determine the rate of convergence, establishing a sharp "cutoff" transition, answering a question of Jean Bourgain. △ Less

Submitted 26 March, 2021; v1 submitted 6 November, 2019; originally announced November 2019.

Comments: 21 pages, 2 figures. Final version. To appear in Ann. Probab

MSC Class: 60J05; 60J20

Randomized sequential importance sampling for estimating the number of perfect matchings in bipartite graphs

Authors: Persi Diaconis, Brett Kolesnik

Abstract: We introduce and study randomized sequential importance sampling algorithms for estimating the number of perfect matchings in bipartite graphs. In analyzing their performance, we establish various non-standard central limit theorems. We expect our methods to be useful for other applied problems. We introduce and study randomized sequential importance sampling algorithms for estimating the number of perfect matchings in bipartite graphs. In analyzing their performance, we establish various non-standard central limit theorems. We expect our methods to be useful for other applied problems. △ Less

Submitted 29 July, 2020; v1 submitted 4 July, 2019; originally announced July 2019.

Comments: v2: minor improvements

MSC Class: 65C05 (Primary); 65C60; 60F05; 68Q25; 82B80 (Secondary)

Gambler's ruin estimates on finite inner uniform domains

Authors: Persi Diaconis, Kelsey Houston-Edwards, Laurent Saloff-Coste

Abstract: Gambler's ruin estimates can be viewed as harmonic measure estimates for finite Markov chains which are absorbed (or killed) at boundary points. We relate such estimates to properties of the underlying chain and its Doob transform. Precisely, we show that gambler's ruin estimates reduce to a good understanding of the Perron-Frobenius eigenfunction and eigenvalue whenever the underlying chain and i… ▽ More Gambler's ruin estimates can be viewed as harmonic measure estimates for finite Markov chains which are absorbed (or killed) at boundary points. We relate such estimates to properties of the underlying chain and its Doob transform. Precisely, we show that gambler's ruin estimates reduce to a good understanding of the Perron-Frobenius eigenfunction and eigenvalue whenever the underlying chain and its Doob transform are Harnack Markov chains. Finite inner-uniform domains (say, in the square grid $\mathbb Z^n$) provide a large class of examples where these ideas apply and lead to detailed estimates. In general, understanding the behavior of the Perron-Frobenius eigenfunction remains a challenge. △ Less

Submitted 11 June, 2019; originally announced June 2019.

Analytic-geometric methods for finite Markov chains with applications to quasi-stationarity

Authors: Persi Diaconis, Kelsey Houston-Edwards, Laurent Saloff-Coste

Abstract: For a relatively large class of well-behaved absorbing (or killed) finite Markov chains, we give detailed quantitative estimates regarding the behavior of the chain before it is absorbed (or killed). Typical examples are random walks on box-like finite subsets of the square lattice $\mathbb Z^d$ absorbed (or killed) at the boundary. The analysis is based on Poincaré, Nash, and Harnack inequalities… ▽ More For a relatively large class of well-behaved absorbing (or killed) finite Markov chains, we give detailed quantitative estimates regarding the behavior of the chain before it is absorbed (or killed). Typical examples are random walks on box-like finite subsets of the square lattice $\mathbb Z^d$ absorbed (or killed) at the boundary. The analysis is based on Poincaré, Nash, and Harnack inequalities, moderate growth, and on the notions of John and inner-uniform domains. △ Less

Submitted 11 June, 2019; originally announced June 2019.

Tensor Product Markov Chains

Authors: Georgia Benkart, Persi Diaconis, Martin W. Liebeck, Pham Huu Tiep

Abstract: We analyze families of Markov chains that arise from decomposing tensor products of irreducible representations. This illuminates the Burnside-Brauer Theorem for building irreducible representations, the McKay Correspondence, and Pitman's 2M-X Theorem. The chains are explicitly diagonalizable, and we use the eigenvalues/eigenvectors to give sharp rates of convergence for the associated random walk… ▽ More We analyze families of Markov chains that arise from decomposing tensor products of irreducible representations. This illuminates the Burnside-Brauer Theorem for building irreducible representations, the McKay Correspondence, and Pitman's 2M-X Theorem. The chains are explicitly diagonalizable, and we use the eigenvalues/eigenvectors to give sharp rates of convergence for the associated random walks. For modular representations, the chains are not reversible, and the analytical details are surprisingly intricate. In the quantum group case, the chains fail to be diagonalizable, but a novel analysis using generalized eigenvectors proves successful. △ Less

Submitted 30 September, 2018; originally announced October 2018.

MSC Class: 60B05; 20C20; 20G42

Bayesian Goodness of Fit Tests: A Conversation for David Mumford

Authors: Persi Diaconis, Guanyang Wang

Abstract: The problem of making practical, useful goodness of fit tests in the Bayesian paradigm is largely open. We introduce a class of special cases (testing for uniformity: have the cards been shuffled enough; does my random generator work) and a class of sensible Bayes tests inspired by Mumford, Wu and Zhu. Calculating these tests presents the challenge of 'doubly intractable distributions'. In present… ▽ More The problem of making practical, useful goodness of fit tests in the Bayesian paradigm is largely open. We introduce a class of special cases (testing for uniformity: have the cards been shuffled enough; does my random generator work) and a class of sensible Bayes tests inspired by Mumford, Wu and Zhu. Calculating these tests presents the challenge of 'doubly intractable distributions'. In present circumstances, modern MCMC techniques are up to the challenge. But many other problems remain. Our paper is didactic, we hope to induce the reader to help take it further. △ Less

Submitted 9 April, 2018; v1 submitted 29 March, 2018; originally announced March 2018.

Comments: 22 pages, 5 figures; added references

Reproducing kernel orthogonal polynomials on the multinomial distribution

Authors: Persi Diaconis, Robert Griffiths

Abstract: Diaconis and Griffiths (2014) study the multivariate Krawtchouk polynomials orthogonal on the multinomial distribution. In this paper we derive the reproducing kernel orthogonal polynomials Q_n(x,y};N,p) on the multinomial distribution which are sums of products of orthonormal polynomials in x and y of fixed total degree n=0,1,.., N. sum_{n=0}^N rho^nQ_n(x,y);N,p) arises naturally from a probabi… ▽ More Diaconis and Griffiths (2014) study the multivariate Krawtchouk polynomials orthogonal on the multinomial distribution. In this paper we derive the reproducing kernel orthogonal polynomials Q_n(x,y};N,p) on the multinomial distribution which are sums of products of orthonormal polynomials in x and y of fixed total degree n=0,1,.., N. sum_{n=0}^N rho^nQ_n(x,y);N,p) arises naturally from a probabilistic argument. An application to a multinomial goodness of fit test is developed, where the chi-squared test statistic is decomposed into orthogonal components which test the order of fit. A new duplication formula for the reproducing kernel polynomials in terms of the 1-dimensional Krawtchouk polynomials is derived. The duplication formula allows a Lancaster characterization of all reversible Markov chains with a multinomial stationary distribution whose eigenvectors are multivariate Krawtchouk polynomials and where eigenvalues are repeated within the same total degree. The χ^2 cutoff time, and total variation cutoff time is investigated in such chains. Emphasis throughout the paper is on a probabilistic understanding of the polynomials and their applications, particularly to Markov chains. △ Less

Submitted 4 February, 2019; v1 submitted 20 February, 2018; originally announced February 2018.

Shuffling cards by spatial motion

Authors: Persi Diaconis, Soumik Pal

Abstract: We propose a model of card shuffling where a pack of cards, spread as points on a square table, are repeatedly gathered locally at random spots and then spread towards a random direction. A shuffling of the cards is then obtained by arranging the cards by their increasing $x$-coordinate values. When there are $m$ cards on the table we show that this random ordering gets mixed in time… ▽ More We propose a model of card shuffling where a pack of cards, spread as points on a square table, are repeatedly gathered locally at random spots and then spread towards a random direction. A shuffling of the cards is then obtained by arranging the cards by their increasing $x$-coordinate values. When there are $m$ cards on the table we show that this random ordering gets mixed in time $O\left(\log m\right)$. Explicit constants are evaluated in a diffusion limit when the position of $m$ cards evolves as an interesting $2m$-dimensional non-reversible reflected jump diffusion in time. Our main technique involves the use of multidimensional Skorokhod maps for double reflections in $[0,1]^2$ in taking the discrete to continuous limit. The limiting computations are then based on the planar Brownian motion and properties of Bessel processes. △ Less

Submitted 10 June, 2021; v1 submitted 27 August, 2017; originally announced August 2017.

Comments: 27 pages, 2 figures

MSC Class: 60J10; 60J60

Probabilizing Parking Functions

Authors: Persi Diaconis, Angela Hicks

Abstract: We explore the link between combinatorics and probability generated by the question "What does a random parking function look like?" This gives rise to novel probabilistic interpretations of some elegant, known generating functions. It leads to new combinatorics: how many parking functions begin with $i$? We classify features (e.g., the full descent pattern) of parking functions that have exactly… ▽ More We explore the link between combinatorics and probability generated by the question "What does a random parking function look like?" This gives rise to novel probabilistic interpretations of some elegant, known generating functions. It leads to new combinatorics: how many parking functions begin with $i$? We classify features (e.g., the full descent pattern) of parking functions that have exactly the same distribution among parking functions as among all functions. Finally, we develop the link between parking functions and Brownian excursion theory to give examples where the two ensembles differ. △ Less

Submitted 29 November, 2016; originally announced November 2016.

Comments: 40 pages

MSC Class: 60C05

A central limit theorem for a new statistic on permutations

Authors: Sourav Chatterjee, Persi Diaconis

Abstract: This paper does three things: It proves a central limit theorem for novel permutation statistics (for example, the number of descents plus the number of descents in the inverse). It provides a clear illustration of a new approach to proving central limit theorems more generally. It gives us an opportunity to acknowledge the work of our teacher and friend B. V. Rao. This paper does three things: It proves a central limit theorem for novel permutation statistics (for example, the number of descents plus the number of descents in the inverse). It provides a clear illustration of a new approach to proving central limit theorems more generally. It gives us an opportunity to acknowledge the work of our teacher and friend B. V. Rao. △ Less

Submitted 26 October, 2016; v1 submitted 4 August, 2016; originally announced August 2016.

Comments: 10 pages. To appear in a special issue of the Indian Journal of Pure and Applied Mathematics in honor of Professor B. V. Rao

A. Hurwitz and the origins of random matrix theory in mathematics

Authors: Persi Diaconis, Peter J. Forrester

Abstract: The purpose of this article is to put forward the claim that Hurwitz's paper "Uber die Erzeugung der Invarianten durch Integration." [Gott. Nachrichten (1897), 71-90] should be regarded as the origin of random matrix theory in mathematics. Here Hurwitz introduced and developed the notion of an invariant measure for the matrix groups $SO(N)$ and $U(N)$. He also specified a calculus from which the e… ▽ More The purpose of this article is to put forward the claim that Hurwitz's paper "Uber die Erzeugung der Invarianten durch Integration." [Gott. Nachrichten (1897), 71-90] should be regarded as the origin of random matrix theory in mathematics. Here Hurwitz introduced and developed the notion of an invariant measure for the matrix groups $SO(N)$ and $U(N)$. He also specified a calculus from which the explicit form of these measures could be computed in terms of an appropriate parametrisation - Hurwitz chose to use Euler angles. This enabled him to define and compute invariant group integrals over $SO(N)$ and $U(N)$. His main result can be interpreted probabilistically: the Euler angles of a uniformly distributed matrix are independent with beta distributions (and conversely). We use this interpretation to give some new probability results. How Hurwitz's ideas and methods show themselves in the subsequent work of Weyl, Dyson and others on foundational studies in random matrix theory is detailed. △ Less

Submitted 8 January, 2016; v1 submitted 31 December, 2015; originally announced December 2015.

Comments: 22 pages; V2 updates references, makes minor corrections

Random walk on unipotent matrix groups

Authors: Persi Diaconis, Bob Hough

Abstract: We introduce a new method for proving central limit theorems for random walk on nilpotent groups. The method is illustrated in a local central limit theorem on the Heisenberg group, weakening the necessary conditions on the driving measure. As a second illustration, the method is used to study walks on the $n\times n$ uni-upper triangular group with entries taken modulo $p$. The method allows shar… ▽ More We introduce a new method for proving central limit theorems for random walk on nilpotent groups. The method is illustrated in a local central limit theorem on the Heisenberg group, weakening the necessary conditions on the driving measure. As a second illustration, the method is used to study walks on the $n\times n$ uni-upper triangular group with entries taken modulo $p$. The method allows sharp answers to the behavior of individual coordinates: coordinates immediately above the diagonal require order $p^2$ steps for randomness, coordinates on the second diagonal require order $p$ steps; coordinates on the $k$th diagonal require order $p^{\frac{2}{k}}$ steps. △ Less

Submitted 13 November, 2018; v1 submitted 19 December, 2015; originally announced December 2015.

Comments: Minor corrections

MSC Class: 60F05; 60B15; 20B25; 22E25; 60J10; 60E10; 60F25; 60G42

The sample size required in importance sampling

Authors: Sourav Chatterjee, Persi Diaconis

Abstract: The goal of importance sampling is to estimate the expected value of a given function with respect to a probability measure $ν$ using a random sample of size $n$ drawn from a different probability measure $μ$. If the two measures $μ$ and $ν$ are nearly singular with respect to each other, which is often the case in practice, the sample size required for accurate estimation is large. In this articl… ▽ More The goal of importance sampling is to estimate the expected value of a given function with respect to a probability measure $ν$ using a random sample of size $n$ drawn from a different probability measure $μ$. If the two measures $μ$ and $ν$ are nearly singular with respect to each other, which is often the case in practice, the sample size required for accurate estimation is large. In this article it is shown that in a fairly general setting, a sample of size approximately $\exp(D(ν||μ))$ is necessary and sufficient for accurate estimation by importance sampling, where $D(ν||μ)$ is the Kullback-Leibler divergence of $μ$ from $ν$. In particular, the required sample size exhibits a kind of cut-off in the logarithmic scale. The theory is applied to obtain a general formula for the sample size required in importance sampling for one-parameter exponential families (Gibbs measures). △ Less

Submitted 20 June, 2017; v1 submitted 4 November, 2015; originally announced November 2015.

Comments: 37 pages, 4 figures. To appear in Ann. App. Probab

MSC Class: 65C05; 65C60; 60F05; 82B80

Useful bounds on the extreme eigenvalues and vectors of matrices for Harper's operators

Authors: Daniel Bump, Persi Diaconis, Angela Hicks, Laurent Miclo, Harold Widom

Abstract: In analyzing a simple random walk on the Heisenberg group we encounter the problem of bounding the extreme eigenvalues of an $n\times n$ matrix of the form $M=C+D$ where $C$ is a circulant and $D$ a diagonal matrix. The discrete Schrödinger operators are an interesting special case. The Weyl and Horn bounds are not useful here. This paper develops three different approaches to getting good bounds.… ▽ More In analyzing a simple random walk on the Heisenberg group we encounter the problem of bounding the extreme eigenvalues of an $n\times n$ matrix of the form $M=C+D$ where $C$ is a circulant and $D$ a diagonal matrix. The discrete Schrödinger operators are an interesting special case. The Weyl and Horn bounds are not useful here. This paper develops three different approaches to getting good bounds. The first uses the geometry of the eigenspaces of $C$ and $D$, applying a discrete version of the uncertainty principle. The second shows that, in a useful limit, the matrix $M$ tends to the harmonic oscillator on $L^2(\mathbb{R})$ and the known eigenstructure can be transferred back. The third approach is purely probabilistic, extending $M$ to an absorbing Markov chain and using hitting time arguments to bound the Dirichlet eigenvalues. The approaches allow generalization to other walks on other groups. △ Less

Submitted 6 November, 2015; v1 submitted 24 August, 2015; originally announced August 2015.

MSC Class: 60J10; 60B15

Estimates on the amplitude of the first Dirichlet eigenvector in discrete frameworks

Authors: Persi Diaconis, Laurent Miclo

Abstract: Consider a finite absorbing Markov generator, irreducible on the non-absorbing states. Perron-Frobenius theory ensures the existence of a corresponding positive eigenvector $\varphi$. The goal of the paper is to give bounds on the amplitude $\max \varphi/\min\varphi$. Two approaches are proposed: one using a path method and the other one, restricted to the reversible situation, based on spectral e… ▽ More Consider a finite absorbing Markov generator, irreducible on the non-absorbing states. Perron-Frobenius theory ensures the existence of a corresponding positive eigenvector $\varphi$. The goal of the paper is to give bounds on the amplitude $\max \varphi/\min\varphi$. Two approaches are proposed: one using a path method and the other one, restricted to the reversible situation, based on spectral estimates. The latter approach is extended to denumerable birth and death processes absorbing at 0 for which infinity is an entrance boundary. The interest of estimating the ratio is the reduction of the quantitative study of convergence to quasi-stationarity to the convergence to equilibrium of related ergodic processes, as seen in [7]. △ Less

Submitted 28 May, 2015; originally announced May 2015.

An Exercise (?) in Fourier Analysis on the Heisenberg Group

Authors: Daniel Bump, Persi Diaconis, Angela Hicks, Laurent Miclo, Harold Widom

Abstract: Let H(n) be the group of 3x3 uni-uppertriangular matrices with entries in Z/nZ, the integers mod n. We show that the simple random walk converges to the uniform distribution in order n^2 steps. The argument uses Fourier analysis and is surprisingly challenging. It introduces novel techniques for bounding the spectrum which are useful for a variety of walks on a variety of groups. Let H(n) be the group of 3x3 uni-uppertriangular matrices with entries in Z/nZ, the integers mod n. We show that the simple random walk converges to the uniform distribution in order n^2 steps. The argument uses Fourier analysis and is surprisingly challenging. It introduces novel techniques for bounding the spectrum which are useful for a variety of walks on a variety of groups. △ Less

Submitted 13 February, 2015; originally announced February 2015.

Comments: 24 pages, 6 figures

MSC Class: 60J10; 60B15

Central Limit Theorems for some Set Partition Statistics

Authors: Bobbie Chern, Persi Diaconis, Daniel M. Kane, Robert C. Rhoades

Abstract: We prove the conjectured limiting normality for the number of crossings of a uniformly chosen set partition of [n] = {1,2,...,n}. The arguments use a novel stochastic representation and are also used to prove central limit theorems for the dimension index and the number of levels. We prove the conjectured limiting normality for the number of crossings of a uniformly chosen set partition of [n] = {1,2,...,n}. The arguments use a novel stochastic representation and are also used to prove central limit theorems for the dimension index and the number of levels. △ Less

Submitted 3 February, 2015; originally announced February 2015.

MSC Class: 05A18; 05A16; 60C05

The mathematics of the flip and horseshoe shuffles

Authors: Steve Butler, Persi Diaconis, Ron Graham

Abstract: We consider new types of perfect shuffles wherein a deck is split in half, one half of the deck is "reversed", and then the cards are interlaced. Flip shuffles are when the reversal comes from flip** the half over so that we also need to account for face-up/face-down configurations while horseshoe shuffles are when the order of the cards are reversed but all cards still face the same direction.… ▽ More We consider new types of perfect shuffles wherein a deck is split in half, one half of the deck is "reversed", and then the cards are interlaced. Flip shuffles are when the reversal comes from flip** the half over so that we also need to account for face-up/face-down configurations while horseshoe shuffles are when the order of the cards are reversed but all cards still face the same direction. We show that these shuffles are closely related to faro shuffling and determine the order of the associated shuffling groups. △ Less

Submitted 29 December, 2014; originally announced December 2014.

On quantitative convergence to quasi-stationarity

Authors: Persi Diaconis, Laurent Miclo

Abstract: The quantitative long time behavior of absorbing, finite, irreducible Markov processes is considered. Via Doob transforms, it is shown that only the knowledge of the ratio of the values of the underlying first Dirichlet eigenvector is necessary to come back to the well-investigated situation of the convergence to equilibrium of ergodic finite Markov processes. This leads to explicit estimates on t… ▽ More The quantitative long time behavior of absorbing, finite, irreducible Markov processes is considered. Via Doob transforms, it is shown that only the knowledge of the ratio of the values of the underlying first Dirichlet eigenvector is necessary to come back to the well-investigated situation of the convergence to equilibrium of ergodic finite Markov processes. This leads to explicit estimates on the convergence to quasi-stationarity, in particular via functional inequalities. When the process is reversible, the optimal exponential rate consisting of the spectral gap between the two first Dirichlet eigenvalues is recovered. Several simple examples are provided to illustrate the bounds obtained. △ Less

Submitted 6 June, 2014; originally announced June 2014.

Discussion of "Geodesic Monte Carlo on Embedded Manifolds"

Authors: Simon Byrne, Mark Girolami, Persi Diaconis, Christof Seiler, Susan Holmes, Ian L. Dryden, John T. Kent, Marcelo Pereyra, Babak Shahbaba, Shiwei Lan, Jeffrey Streets, Daniel Simpson

Abstract: Contributed discussion and rejoinder to "Geodesic Monte Carlo on Embedded Manifolds" (arXiv:1301.6064) Contributed discussion and rejoinder to "Geodesic Monte Carlo on Embedded Manifolds" (arXiv:1301.6064) △ Less

Submitted 5 November, 2013; originally announced November 2013.

Comments: Discussion of arXiv:1301.6064. To appear in the Scandinavian Journal of Statistics. 18 pages

Journal ref: Scandinavian Journal of Statistics (2014) 41 (1), pages 1-21

Universal Limit Theorems in Graph Coloring Problems With Connections to Extremal Combinatorics

Authors: Bhaswar B. Bhattacharya, Persi Diaconis, Sumit Mukherjee

Abstract: This paper proves limit theorems for the number of monochromatic edges in uniform random colorings of general random graphs. These can be seen as generalizations of the birthday problem (what is the chance that there are two friends with the same birthday?). It is shown that if the number of colors grows to infinity, the asymptotic distribution is either a Poisson mixture or a Normal depending sol… ▽ More This paper proves limit theorems for the number of monochromatic edges in uniform random colorings of general random graphs. These can be seen as generalizations of the birthday problem (what is the chance that there are two friends with the same birthday?). It is shown that if the number of colors grows to infinity, the asymptotic distribution is either a Poisson mixture or a Normal depending solely on the limiting behavior of the ratio of the number of edges in the graph and the number of colors. This result holds for any graph sequence, deterministic or random. On the other hand, when the number of colors is fixed, a necessary and sufficient condition for asymptotic normality is determined. Finally, using some results from the emerging theory of dense graph limits, the asymptotic (non-normal) distribution is characterized for any converging sequence of dense graphs. The proofs are based on moment calculations which relate to the results of Erd\H os and Alon on extremal subgraph counts. As a consequence, a simpler proof of a result of Alon, estimating the number of isomorphic copies of a cycle of given length in graphs with a fixed number of edges, is presented. △ Less

Submitted 5 December, 2014; v1 submitted 9 October, 2013; originally announced October 2013.

Comments: New results added that give limit theorems for the number of monochromatic edges when the number of colors is fixed. 40 pages, 2 figures

MSC Class: 05C15; 60C05; 60F05; 05D99

Journal ref: Annals of Applied Probability, Vol. 27 (1), 337-394, 2017

Some things we've learned (about Markov chain Monte Carlo)

Authors: Persi Diaconis

Abstract: This paper offers a personal review of some things we've learned about rates of convergence of Markov chains to their stationary distributions. The main topic is ways of speeding up diffusive behavior. It also points to open problems and how much more there is to do. This paper offers a personal review of some things we've learned about rates of convergence of Markov chains to their stationary distributions. The main topic is ways of speeding up diffusive behavior. It also points to open problems and how much more there is to do. △ Less

Submitted 30 September, 2013; originally announced September 2013.

Comments: Published in at http://dx.doi.org/10.3150/12-BEJSP09 the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm)

Report number: IMS-BEJ-BEJSP09

Journal ref: Bernoulli 2013, Vol. 19, No. 4, 1294-1305

Combinatorics of balanced carries

Authors: Persi Diaconis, Jason Fulman

Abstract: We study the combinatorics of addition using balanced digits, deriving an analog of Holte's "amazing matrix" for carries in usual addition. The eigenvalues of this matrix for base b balanced addition of n numbers are found to be 1,1/b,...,1/b^n, and formulas are given for its left and right eigenvectors. It is shown that the left eigenvectors can be identified with hyperoctahedral Foulkes characte… ▽ More We study the combinatorics of addition using balanced digits, deriving an analog of Holte's "amazing matrix" for carries in usual addition. The eigenvalues of this matrix for base b balanced addition of n numbers are found to be 1,1/b,...,1/b^n, and formulas are given for its left and right eigenvectors. It is shown that the left eigenvectors can be identified with hyperoctahedral Foulkes characters, and that the right eigenvectors can be identified with hyperoctahedral Eulerian idempotents. We also examine the carries that occur when a column of balanced digits is added, showing this process to be determinantal. The transfer matrix method and a serendipitous diagonalization are used to study this determinantal process. △ Less

Submitted 19 September, 2013; originally announced September 2013.

Comments: 18 pages

Carries, group theory, and additive combinatorics

Authors: Persi Diaconis, Xuancheng Shao, Kannan Soundararajan

Abstract: Given a group G and a normal subgroup H we study the problem of choosing coset representatives with few carries. Given a group G and a normal subgroup H we study the problem of choosing coset representatives with few carries. △ Less

Submitted 2 September, 2013; originally announced September 2013.

An introduction to multivariate Krawtchouk polynomials and their applications

Authors: Persi Diaconis, Robert Griffiths

Abstract: Orthogonal polynomials for the multinomial distribution m(x, p) of N balls dropped into d boxes (box i has probability p(i)) are called multivariate Krawtchouk polynomials. This paper gives an introduction to their properties, collections of natural Markov chains which they explicitly diagonalize and associated bivariate multinomial distributions. Orthogonal polynomials for the multinomial distribution m(x, p) of N balls dropped into d boxes (box i has probability p(i)) are called multivariate Krawtchouk polynomials. This paper gives an introduction to their properties, collections of natural Markov chains which they explicitly diagonalize and associated bivariate multinomial distributions. △ Less

Submitted 8 February, 2014; v1 submitted 31 August, 2013; originally announced September 2013.

Comments: 26 pages

Unseparated pairs and fixed points in random permutations

Authors: Persi Diaconis, Steven N. Evans, Ron Graham

Abstract: In a uniform random permutation Πof [n] := {1,2,...,n}, the set of elements k in [n-1] such that Π(k+1) = Π(k) + 1 has the same distribution as the set of fixed points of Πthat lie in [n-1]. We give three different proofs of this fact using, respectively, an enumeration relying on the inclusion-exclusion principle, the introduction of two different Markov chains to generate uniform random permutat… ▽ More In a uniform random permutation Πof [n] := {1,2,...,n}, the set of elements k in [n-1] such that Π(k+1) = Π(k) + 1 has the same distribution as the set of fixed points of Πthat lie in [n-1]. We give three different proofs of this fact using, respectively, an enumeration relying on the inclusion-exclusion principle, the introduction of two different Markov chains to generate uniform random permutations, and the construction of a combinatorial bijection. We also obtain the distribution of the analogous set for circular permutations that consists of those k in [n] such that Π(k+1 mod n) = Π(k) + 1 mod n. This latter random set is just the set of fixed points of the commutator [ρ, Π], where ρis the n-cycle (1,2,...,n). We show for a general permutation ηthat, under weak conditions on the number of fixed points and 2-cycles of η, the total variation distance between the distribution of the number of fixed points of [η,Π] and a Poisson distribution with expected value 1 is small when n is large. △ Less

Submitted 27 April, 2014; v1 submitted 25 August, 2013; originally announced August 2013.

Comments: revised to incorporate corrections suggested by referee

MSC Class: 60C05; 60B15; 60F05

Fluctuations of the Bose-Einstein condensate

Authors: Sourav Chatterjee, Persi Diaconis

Abstract: This article gives a rigorous analysis of the fluctuations of the Bose-Einstein condensate for a system of non-interacting bosons in an arbitrary potential, assuming that the system is governed by the canonical ensemble. As a result of the analysis, we are able to tell the order of fluctuations of the condensate fraction as well as its limiting distribution upon proper centering and scaling. This… ▽ More This article gives a rigorous analysis of the fluctuations of the Bose-Einstein condensate for a system of non-interacting bosons in an arbitrary potential, assuming that the system is governed by the canonical ensemble. As a result of the analysis, we are able to tell the order of fluctuations of the condensate fraction as well as its limiting distribution upon proper centering and scaling. This yields interesting results. For example, for a system of $n$ bosons in a 3D harmonic trap near the transition temperature, the order of fluctuations of the condensate fraction is $n^{-1/2}$ and the limiting distribution is normal, whereas for the 3D uniform Bose gas, the order of fluctuations is $n^{-1/3}$ and the limiting distribution is an explicit non-normal distribution. For a 2D harmonic trap, the order of fluctuations is $n^{-1/2}(\log n)^{1/2}$, which is larger than $n^{-1/2}$ but the limiting distribution is still normal. All of these results come as easy consequences of a general theorem. △ Less

Submitted 14 January, 2014; v1 submitted 16 June, 2013; originally announced June 2013.

Comments: 26 pages. Minor changes in new version

MSC Class: 82B10; 81V25; 60F05

Closed expressions for averages of set partition statistics

Authors: Bobbie Chern, Persi Diaconis, Daniel M. Kane, Robert C. Rhoades

Abstract: In studying the enumerative theory of super characters' of the group of upper triangular matrices over a finite field we found that the moments (mean, variance and higher moments) of novel statistics on set partitions have simple closed expressions as linear combinations of shifted bell numbers. It is shown here that families of other statistics have similar moments. The coefficients in the linear… ▽ More In studying the enumerative theory of super characters' of the group of upper triangular matrices over a finite field we found that the moments (mean, variance and higher moments) of novel statistics on set partitions have simple closed expressions as linear combinations of shifted bell numbers. It is shown here that families of other statistics have similar moments. The coefficients in the linear combinations are polynomials in $n$. This allows exact enumeration of the moments for small $n$ to determine exact formulae for all $n$. △ Less

Submitted 10 June, 2013; v1 submitted 15 April, 2013; originally announced April 2013.

MSC Class: 05A18; 05A15

Sampling From A Manifold

Authors: Persi Diaconis, Susan Holmes, Mehrdad Shahshahani

Abstract: We develop algorithms for sampling from a probability distribution on a submanifold embedded in Rn. Applications are given to the evaluation of algorithms in 'Topological Statistics'; to goodness of fit tests in exponential families and to Neyman's smooth test. This article is partially expository, giving an introduction to the tools of geometric measure theory. We develop algorithms for sampling from a probability distribution on a submanifold embedded in Rn. Applications are given to the evaluation of algorithms in 'Topological Statistics'; to goodness of fit tests in exponential families and to Neyman's smooth test. This article is partially expository, giving an introduction to the tools of geometric measure theory. △ Less

Submitted 4 July, 2012; v1 submitted 28 June, 2012; originally announced June 2012.

Hopf algebras and Markov chains: Two examples and a theory

Authors: Persi Diaconis, C. Y. Amy Pang, Arun Ram

Abstract: The operation of squaring (coproduct followed by product) in a combinatorial Hopf algebra is shown to induce a Markov chain in natural bases. Chains constructed in this way include widely studied methods of card shuffling, a natural "rock-breaking" process, and Markov chains on simplicial complexes. Many of these chains can be explictly diagonalized using the primitive elements of the algebra and… ▽ More The operation of squaring (coproduct followed by product) in a combinatorial Hopf algebra is shown to induce a Markov chain in natural bases. Chains constructed in this way include widely studied methods of card shuffling, a natural "rock-breaking" process, and Markov chains on simplicial complexes. Many of these chains can be explictly diagonalized using the primitive elements of the algebra and the combinatorics of the free Lie algebra. For card shuffling, this gives an explicit description of the eigenvectors. For rock-breaking, an explicit description of the quasi-stationary distribution and sharp rates to absorption follow. △ Less

Submitted 20 June, 2013; v1 submitted 15 June, 2012; originally announced June 2012.

Comments: 51 pages, 17 figures. (Typographical errors corrected. Further fixes will only appear on the version on Amy Pang's website, the arXiv version will not be updated.)

Convolution powers of complex functions on Z

Authors: Persi Diaconis, Laurent Saloff-Coste

Abstract: Repeated convolution of a probability measure on Z leads to the central limit theorem and other limit theorems. This paper investigates what kinds of results remain without positivity. It reviews theorems due to Schoenberg, Greville, and Thomee which are motivated by applications to data smoothing (Schoenberg and Greville) and finite difference schemes (Thomee). Using Fourier transform arguments,… ▽ More Repeated convolution of a probability measure on Z leads to the central limit theorem and other limit theorems. This paper investigates what kinds of results remain without positivity. It reviews theorems due to Schoenberg, Greville, and Thomee which are motivated by applications to data smoothing (Schoenberg and Greville) and finite difference schemes (Thomee). Using Fourier transform arguments, we prove detailed decay bounds for convolution powers of finitely supported complex functions on Z. If M is an hermitian contraction, an estimate for the off-diagonal entries of the powers M^n k of M^k = I -(I-M)^k is obtained. This generalizes the Carne-Varopoulos Markov chain estimate. △ Less

Submitted 29 May, 2012; originally announced May 2012.

Comments: 1 figure

MSC Class: 60.F.xx

Note on a partition limit theorem for rank and crank

Authors: Persi Diaconis, Svante Janson, Robert C. Rhoades

Abstract: If L is a partition of n, the rank of L is the size of the largest part minus the number of parts. Under the uniform distribution on partitions, Bringmann, Mahlburg, and Rhoades showed that the rank statistic has a limiting distribution. We identify the limit as the difference between two independent extreme value distributions and as the distribution of B(T) where B(t) is standard Brownian motion… ▽ More If L is a partition of n, the rank of L is the size of the largest part minus the number of parts. Under the uniform distribution on partitions, Bringmann, Mahlburg, and Rhoades showed that the rank statistic has a limiting distribution. We identify the limit as the difference between two independent extreme value distributions and as the distribution of B(T) where B(t) is standard Brownian motion and T is the first time that an independent three-dimensional Brownian motion hits the unit sphere. The same limit holds for the crank. △ Less

Submitted 6 May, 2012; originally announced May 2012.

Comments: 3 pages

MSC Class: 11P55 (Primary) 60E07 (Secondary)

On Dirichlet eigenvectors for neutral two-dimensional Markov chains

Authors: Nicolas Champagnat, Persi Diaconis, Laurent Miclo

Abstract: We consider a general class of discrete, two-dimensional Markov chains modeling the dynamics of a population with two types, without mutation or immigration, and neutral in the sense that type has no influence on each individual's birth or death parameters. We prove that all the eigenvectors of the corresponding transition matrix or infinitesimal generator Π can be expressed as the product of "uni… ▽ More We consider a general class of discrete, two-dimensional Markov chains modeling the dynamics of a population with two types, without mutation or immigration, and neutral in the sense that type has no influence on each individual's birth or death parameters. We prove that all the eigenvectors of the corresponding transition matrix or infinitesimal generator Π can be expressed as the product of "universal" polynomials of two variables, depending on each type's size but not on the specific transitions of the dynamics, and functions depending only on the total population size. These eigenvectors appear to be Dirichlet eigenvectors for Π on the complement of triangular subdomains, and as a consequence the corresponding eigenvalues are ordered in a specific way. As an application, we study the quasistationary behavior of finite, nearly neutral, two-dimensional Markov chains, absorbed in the sense that 0 is an absorbing state for each component of the process. △ Less

Submitted 21 February, 2012; originally announced February 2012.

MSC Class: Primary: 60J10; 60J27; secondary: 15A18; 39A14; 47N30; 92D25

Riffle shuffles with biased cuts

Authors: Sami Assaf, Persi Diaconis, Kannan Soundararajan

Abstract: The well-known Gilbert-Shannon-Reeds model for riffle shuffles assumes that the cards are initially cut 'about in half' and then riffled together. We analyze a natural variant where the initial cut is biased. Extending results of Fulman (1998), we show a sharp cutoff in separation and L-infinity distances. This analysis is possible due to the close connection between shuffling and quasisymmetric f… ▽ More The well-known Gilbert-Shannon-Reeds model for riffle shuffles assumes that the cards are initially cut 'about in half' and then riffled together. We analyze a natural variant where the initial cut is biased. Extending results of Fulman (1998), we show a sharp cutoff in separation and L-infinity distances. This analysis is possible due to the close connection between shuffling and quasisymmetric functions along with some complex analysis of a generating function. △ Less

Submitted 12 December, 2011; originally announced December 2011.

Comments: 10 pages

MSC Class: 60B15; 60C05