Showing 1–9 of 9 results for author: Sklar, M

Incentive-Theoretic Bayesian Inference for Collaborative Science

Authors: Stephen Bates, Michael I. Jordan, Michael Sklar, Jake A. Soloff

Abstract: Contemporary scientific research is a distributed, collaborative endeavor, carried out by teams of researchers, regulatory institutions, funding agencies, commercial partners, and scientific bodies, all interacting with each other and facing different incentives. To maintain scientific rigor, statistical methods should acknowledge this state of affairs. To this end, we study hypothesis testing whe… ▽ More Contemporary scientific research is a distributed, collaborative endeavor, carried out by teams of researchers, regulatory institutions, funding agencies, commercial partners, and scientific bodies, all interacting with each other and facing different incentives. To maintain scientific rigor, statistical methods should acknowledge this state of affairs. To this end, we study hypothesis testing when there is an agent (e.g., a researcher or a pharmaceutical company) with a private prior about an unknown parameter and a principal (e.g., a policymaker or regulator) who wishes to make decisions based on the parameter value. The agent chooses whether to run a statistical trial based on their private prior and then the result of the trial is used by the principal to reach a decision. We show how the principal can conduct statistical inference that leverages the information that is revealed by an agent's strategic behavior -- their choice to run a trial or not. In particular, we show how the principal can design a policy to elucidate partial information about the agent's private prior beliefs and use this to control the posterior probability of the null. One implication is a simple guideline for the choice of significance threshold in clinical trials: the type-I error level should be set to be strictly less than the cost of the trial divided by the firm's profit if the trial is successful. △ Less

Submitted 8 February, 2024; v1 submitted 7 July, 2023; originally announced July 2023.

Relative Probability on Finite Outcome Spaces: A Systematic Examination of its Axiomatization, Properties, and Applications

Authors: Max Sklar

Abstract: This work proposes a view of probability as a relative measure rather than an absolute one. To demonstrate this concept, we focus on finite outcome spaces and develop three fundamental axioms that establish requirements for relative probability functions. We then provide a library of examples of these functions and a system for composing them. Additionally, we discuss a relative version of Bayesia… ▽ More This work proposes a view of probability as a relative measure rather than an absolute one. To demonstrate this concept, we focus on finite outcome spaces and develop three fundamental axioms that establish requirements for relative probability functions. We then provide a library of examples of these functions and a system for composing them. Additionally, we discuss a relative version of Bayesian inference and its digital implementation. Finally, we prove the topological closure of the relative probability space, highlighting its ability to preserve information under limits. △ Less

Submitted 27 May, 2023; v1 submitted 30 December, 2022; originally announced December 2022.

Comments: 30 pages, 11 figures

MSC Class: 60A05

A Rigorous Framework for Automated Design Assessment and Type I Error Control: Methods and Demonstrative Examples (Working Paper)

Authors: James Yang, T. Ben Thompson, Michael Sklar

Abstract: We present a proof-by-simulation framework for rigorously controlling a trial design's operating characteristics over continuous regions of parameter space. We show that proof-by-simulation can achieve two major goals: (1) Calibrate a design for provable Type I Error control at a fixed level alpha. (2) Given a fixed design, bound (with high probability) its operating characteristics, such as the T… ▽ More We present a proof-by-simulation framework for rigorously controlling a trial design's operating characteristics over continuous regions of parameter space. We show that proof-by-simulation can achieve two major goals: (1) Calibrate a design for provable Type I Error control at a fixed level alpha. (2) Given a fixed design, bound (with high probability) its operating characteristics, such as the Type I Error, FDR, or bias of bounded estimators. This framework can handle adaptive sampling, nuisance parameters, administrative censoring, multiple arms, and multiple testing. These techniques, which we call Continuous Simu lation Extension (CSE), were first developed in Sklar (2021) to control Type I Error and FWER for designs where unknown parameters have exponential family likelihood. Our appendix improves those results with more efficient bounding and calibration methods, extends them to general operating characteristics including FDR and bias, and extends applicability to include canonical GLMs and some non-parametric problems. In the main paper we demonstrate our CSE approach and software on 3 examples (1) a gentle introduction, analyzing the z-test (2) a hierarchical Bayesian analysis of 4 treatments, with sample sizes fixed (3) an adaptive Bayesian Phase II-III selection design with 4 arms, where interim drop** and go/no-go decisions are based on a hierarchical model. Trillions of simulations were performed for the latter two examples, enabled by specialized INLA software. Open-source software is maintained at (1) △ Less

Submitted 12 January, 2023; v1 submitted 20 December, 2022; originally announced December 2022.

Adaptive Experiments and a Rigorous Framework for Type I Error Verification and Computational Experiment Design

Authors: Michael Sklar

Abstract: This PhD thesis covers breakthroughs in several areas of adaptive experiment design: (i) (Chapter 2) Novel clinical trial designs and statistical methods in the era of precision medicine. (ii) (Chapter 3) Multi-armed bandit theory, with applications to learning healthcare systems and clinical trials. (iii) (Chapter 4) Bandit and covariate processes, with finite and non-denumerable set of arms. (iv… ▽ More This PhD thesis covers breakthroughs in several areas of adaptive experiment design: (i) (Chapter 2) Novel clinical trial designs and statistical methods in the era of precision medicine. (ii) (Chapter 3) Multi-armed bandit theory, with applications to learning healthcare systems and clinical trials. (iii) (Chapter 4) Bandit and covariate processes, with finite and non-denumerable set of arms. (iv) (Chapter 5) A rigorous framework for simulation-based verification of adaptive design properties. △ Less

Submitted 19 May, 2022; originally announced May 2022.

Comments: See chapter 5 for proof-by-simulation material not published elsewhere. Minor corrections have been made to the 2021 original document. See https://mikesklar.github.io/thesis/ for the most current maintained version

Principal-Agent Hypothesis Testing

Authors: Stephen Bates, Michael I. Jordan, Michael Sklar, Jake A. Soloff

Abstract: Consider the relationship between a regulator (the principal) and an experimenter (the agent) such as a pharmaceutical company. The pharmaceutical company wishes to sell a drug for profit, whereas the regulator wishes to allow only efficacious drugs to be marketed. The efficacy of the drug is not known to the regulator, so the pharmaceutical company must run a costly trial to prove efficacy to the… ▽ More Consider the relationship between a regulator (the principal) and an experimenter (the agent) such as a pharmaceutical company. The pharmaceutical company wishes to sell a drug for profit, whereas the regulator wishes to allow only efficacious drugs to be marketed. The efficacy of the drug is not known to the regulator, so the pharmaceutical company must run a costly trial to prove efficacy to the regulator. Critically, the statistical protocol used to establish efficacy affects the behavior of a strategic, self-interested agent; a lower standard of statistical evidence incentivizes the agent to run more trials that are less likely to be effective. The interaction between the statistical protocol and the incentives of the pharmaceutical company is crucial for understanding this system and designing protocols with high social utility. In this work, we discuss how the regulator can set up a protocol with payoffs based on statistical evidence. We show how to design protocols that are robust to an agent's strategic actions, and derive the optimal protocol in the presence of strategic entrants. △ Less

Submitted 15 April, 2024; v1 submitted 13 May, 2022; originally announced May 2022.

Sampling Bias Correction for Supervised Machine Learning: A Bayesian Inference Approach with Practical Applications

Authors: Max Sklar

Abstract: Given a supervised machine learning problem where the training set has been subject to a known sampling bias, how can a model be trained to fit the original dataset? We achieve this through the Bayesian inference framework by altering the posterior distribution to account for the sampling function. We then apply this solution to binary logistic regression, and discuss scenarios where a dataset mig… ▽ More Given a supervised machine learning problem where the training set has been subject to a known sampling bias, how can a model be trained to fit the original dataset? We achieve this through the Bayesian inference framework by altering the posterior distribution to account for the sampling function. We then apply this solution to binary logistic regression, and discuss scenarios where a dataset might be subject to intentional sample bias such as label imbalance. This technique is widely applicable for statistical inference on big data, from the medical sciences to image recognition to marketing. Familiarity with it will give the practitioner tools to improve their inference pipeline from data collection to model selection. △ Less

Submitted 14 March, 2022; v1 submitted 11 March, 2022; originally announced March 2022.

Comments: 13 pages, 0 figures

Improving the Power of the Randomization Test

Authors: Abba M. Krieger, David Azriel, Michael Sklar, Adam Kapelner

Abstract: We consider the problem of evaluating designs for a two-arm randomized experiment with the criterion being the power of the randomization test for the one-sided null hypothesis. Our evaluation assumes a response that is linear in one observed covariate, an unobserved component and an additive treatment effect where the only randomness comes from the treatment allocations. It is well-known that the… ▽ More We consider the problem of evaluating designs for a two-arm randomized experiment with the criterion being the power of the randomization test for the one-sided null hypothesis. Our evaluation assumes a response that is linear in one observed covariate, an unobserved component and an additive treatment effect where the only randomness comes from the treatment allocations. It is well-known that the power depends on the allocations' imbalance in the observed covariate and this is the reason for the classic restricted designs such as rerandomization. We show that power is also affected by two other design choices: the number of allocations in the design and the degree of linear dependence among the allocations. We prove that the more allocations, the higher the power and the lower the variability in the power. Designs that feature greater independence of allocations are also shown to have higher performance. Our theoretical findings and extensive simulation studies imply that the designs with the highest power provide thousands of highly independent allocations that each provide nominal imbalance in the observed covariates. These high powered designs exhibit less randomization than complete randomization and more randomization than recently proposed designs based on numerical optimization. Model choices for a practicing experimenter are rerandomization and greedy pair switching, where both outperform complete randomization and numerical optimization. The tradeoff we find also provides a means to specify the imbalance threshold parameter when rerandomizing. △ Less

Submitted 13 August, 2020; originally announced August 2020.

Comments: 31 pages, 6 figures

Optimal Rerandomization via a Criterion that Provides Insurance Against Failed Experiments

Authors: Adam Kapelner, Abba M. Krieger, Michael Sklar, David Azriel

Abstract: We present an optimized rerandomization design procedure for a non-sequential treatment-control experiment. Randomized experiments are the gold standard for finding causal effects in nature. But sometimes random assignments result in unequal partitions of the treatment and control group visibly seen as imbalance in observed covariates. There can additionally be imbalance on unobserved covariates.… ▽ More We present an optimized rerandomization design procedure for a non-sequential treatment-control experiment. Randomized experiments are the gold standard for finding causal effects in nature. But sometimes random assignments result in unequal partitions of the treatment and control group visibly seen as imbalance in observed covariates. There can additionally be imbalance on unobserved covariates. Imbalance in either observed or unobserved covariates increases treatment effect estimator error inflating the width of confidence regions and reducing experimental power. "Rerandomization" is a strategy that omits poor imbalance assignments by limiting imbalance in the observed covariates to a prespecified threshold. However, limiting this threshold too much can increase the risk of contracting error from unobserved covariates. We introduce a criterion that combines observed imbalance while factoring in the risk of inadvertently imbalancing unobserved covariates. We then use this criterion to locate the optimal rerandomization threshold based on the practitioner's level of desired insurance against high estimator error. We demonstrate the gains of our designs in simulation and in a dataset from a large randomized experiment in education. We provide an open source R package available on CRAN named OptimalRerandExpDesigns which generates designs according to our algorithm. △ Less

Submitted 25 January, 2021; v1 submitted 8 May, 2019; originally announced May 2019.

Comments: 27 pages, 5 figures, 2 tables, 2 algorithms

Fast MLE Computation for the Dirichlet Multinomial

Authors: Max Sklar

Abstract: Given a collection of categorical data, we want to find the parameters of a Dirichlet distribution which maximizes the likelihood of that data. Newton's method is typically used for this purpose but current implementations require reading through the entire dataset on each iteration. In this paper, we propose a modification which requires only a single pass through the dataset and substantially de… ▽ More Given a collection of categorical data, we want to find the parameters of a Dirichlet distribution which maximizes the likelihood of that data. Newton's method is typically used for this purpose but current implementations require reading through the entire dataset on each iteration. In this paper, we propose a modification which requires only a single pass through the dataset and substantially decreases running time. Furthermore we analyze both theoretically and empirically the performance of the proposed algorithm, and provide an open source implementation. △ Less

Submitted 26 May, 2023; v1 submitted 1 May, 2014; originally announced May 2014.