-
Simulations for Meta-analysis of Magnitude Measures
Authors:
Elena Kulinskaya,
David C. Hoaglin
Abstract:
Meta-analysis aims to combine effect measures from several studies. For continuous outcomes, the most popular effect measures use simple or standardized differences in sample means. However, a number of applications focus on the absolute values of these effect measures (i.e., unsigned magnitude effects). We provide statistical methods for meta-analysis of magnitude effects based on standardized me…
▽ More
Meta-analysis aims to combine effect measures from several studies. For continuous outcomes, the most popular effect measures use simple or standardized differences in sample means. However, a number of applications focus on the absolute values of these effect measures (i.e., unsigned magnitude effects). We provide statistical methods for meta-analysis of magnitude effects based on standardized mean differences. We propose a suitable statistical model for random-effects meta-analysis of absolute standardized mean differences (ASMD), investigate a number of statistical methods for point and interval estimation, and provide practical recommendations for choosing among them.
△ Less
Submitted 29 September, 2023;
originally announced October 2023.
-
Simulations for estimation of heterogeneity variance and overall effect with constant and inverse-variance weights in meta-analysis of difference in standardized means (DSM)
Authors:
Elena Kulinskaya,
David C. Hoaglin
Abstract:
When the individual studies assembled for a meta-analysis report means ($μ_C$, $μ_T$) for their treatment (T) and control (C) arms, but those data are on different scales or come from different instruments, the customary measure of effect is the standardized mean difference (SMD). The SMD is defined as the difference between the means in the treatment and control arms, standardized by the assumed…
▽ More
When the individual studies assembled for a meta-analysis report means ($μ_C$, $μ_T$) for their treatment (T) and control (C) arms, but those data are on different scales or come from different instruments, the customary measure of effect is the standardized mean difference (SMD). The SMD is defined as the difference between the means in the treatment and control arms, standardized by the assumed common standard deviation, $σ$. However, if the variances in the two arms differ, there is no consensus on a definition of SMD. Thus, we propose a new effect measure, the difference of standardized means (DSM), defined as $Δ= μ_T/σ_T - μ_C/σ_C$. The estimated DSM can easily be used as an effect measure in standard meta-analysis.
For random-effects meta-analysis of DSM, we introduce new point and interval estimators of the between-studies variance ($τ^2$) based on the $Q$ statistic with effective-sample-size weights, $Q_F$. We study, by simulation, bias and coverage of these new estimators of $τ^2$ and related estimators of $Δ$. For comparison, we also study bias and coverage of well-known estimators based on the $Q$ statistic with inverse-variance weights, $Q_{IV}$, such as the Mandel-Paule, DerSimonian-Laird, and restricted-maximum-likelihood estimators.
△ Less
Submitted 14 April, 2023;
originally announced April 2023.
-
Simulations for estimation of heterogeneity variance $τ^2$ in constant and inverse variance weights meta-analysis of log-odds-ratios
Authors:
Elena Kulinskaya,
David C. Hoaglin
Abstract:
A number of popular estimators of the between-study variance, $τ^2$, are based on the Cochran's $Q$ statistic for testing heterogeneity in meta analysis. We introduce new point and interval estimators of $τ^2$ for log-odds-ratio. These include new DerSimonian-Kacker-type moment estimators based on the first moment of $Q_F$, the $Q$ statistic with effective-sample-size weights, and novel median-unb…
▽ More
A number of popular estimators of the between-study variance, $τ^2$, are based on the Cochran's $Q$ statistic for testing heterogeneity in meta analysis. We introduce new point and interval estimators of $τ^2$ for log-odds-ratio. These include new DerSimonian-Kacker-type moment estimators based on the first moment of $Q_F$, the $Q$ statistic with effective-sample-size weights, and novel median-unbiased estimators. We study, by simulation, bias and coverage of these new estimators of $τ^2$ and, for comparative purposes, bias and coverage of a number of well-known estimators based on the $Q$ statistic with inverse-variance weights, $Q_{IV}$, such as the Mandel-Paule, DerSimonian-Laird, and restricted-maximum-likelihood estimators, and an estimator based on the Kulinskaya-Dollinger (2015) improved approximation to $Q_{IV}$.
△ Less
Submitted 1 August, 2022;
originally announced August 2022.
-
Simulations for the Q statistic with constant and inverse variance weights for binary effect measures
Authors:
Elena Kulinskaya,
David C. Hoaglin
Abstract:
Cochran's $Q$ statistic is routinely used for testing heterogeneity in meta-analysis. Its expected value (under an incorrect null distribution) is part of several popular estimators of the between-study variance, $τ^2$. Those applications generally do not account for the studies' use of estimated variances in the inverse-variance weights that define $Q$ (more explicitly, $Q_{IV}$). Importantly, th…
▽ More
Cochran's $Q$ statistic is routinely used for testing heterogeneity in meta-analysis. Its expected value (under an incorrect null distribution) is part of several popular estimators of the between-study variance, $τ^2$. Those applications generally do not account for the studies' use of estimated variances in the inverse-variance weights that define $Q$ (more explicitly, $Q_{IV}$). Importantly, those weights make approximating the distribution of $Q_{IV}$ rather complicated.
As an alternative, we are investigating a $Q$ statistic, $Q_F$, whose constant weights use only the studies' arm-level sample sizes. For log-odds-ratio, log-relative-risk, and risk difference as the measure of effect, these simulations study approximations to the distributions of $Q_F$ and $Q_{IV}$, as the basis for tests of heterogeneity.
We present the results in 132 Figures, 153 pages in total.
△ Less
Submitted 8 April, 2023; v1 submitted 17 June, 2022;
originally announced June 2022.
-
Simulation study of Q statistic with constant weights for testing and estimation of heterogeneity of standardized mean differences in meta-analysis
Authors:
Ilyas Bakbergenuly,
David C. Hoaglin,
Elena Kulinskaya
Abstract:
Cochran's $Q$ statistic is routinely used for testing heterogeneity in meta-analysis. Its expected value is also used for estimation of between-study variance $τ^2$. Cochran's $Q$, or $Q_{IV}$, uses estimated inverse-variance weights which makes approximating its distribution rather complicated. As an alternative, we are investigating a new $Q$ statistic, $Q_F$, whose constant weights use only the…
▽ More
Cochran's $Q$ statistic is routinely used for testing heterogeneity in meta-analysis. Its expected value is also used for estimation of between-study variance $τ^2$. Cochran's $Q$, or $Q_{IV}$, uses estimated inverse-variance weights which makes approximating its distribution rather complicated. As an alternative, we are investigating a new $Q$ statistic, $Q_F$, whose constant weights use only the studies' effective sample sizes. For standardized mean difference as the measure of effect, we study, by simulation, approximations to distributions of $Q_{IV}$ and $Q_F$, as the basis for tests of heterogeneity and for new point and interval estimators of the between-study variance $τ^2$. These include new DerSimonian-Kacker (2007)-type moment estimators based on the first moment of $Q_F$, and novel median-unbiased estimators of $τ^2$.
△ Less
Submitted 4 March, 2021;
originally announced March 2021.
-
Simulations for a Q statistic with constant weights to assess heterogeneity in meta-analysis of mean difference
Authors:
Elena Kulinskaya,
David C. Hoaglin,
Joseph Newman,
Ilyas Bakbergenuly
Abstract:
A variety of problems in random-effects meta-analysis arise from the conventional $Q$ statistic, which uses estimated inverse-variance (IV) weights. In previous work on standardized mean difference and log-odds-ratio, we found superior performance with an estimator of the overall effect whose weights use only group-level sample sizes. The $Q$ statistic with those weights has the form proposed by D…
▽ More
A variety of problems in random-effects meta-analysis arise from the conventional $Q$ statistic, which uses estimated inverse-variance (IV) weights. In previous work on standardized mean difference and log-odds-ratio, we found superior performance with an estimator of the overall effect whose weights use only group-level sample sizes. The $Q$ statistic with those weights has the form proposed by DerSimonian and Kacker. The distribution of this $Q$ and the $Q$ with IV weights must generally be approximated. We investigate approximations for those distributions, as a basis for testing and estimating the between-study variance ($τ^2$). Some approximations require the variance and third moment of $Q$, which we derive. We describe the design and results of a simulation study, with mean difference as the effect measure, which provides a framework for assessing accuracy of the approximations, level and power of the tests, and bias in estimating $τ^2$. Use of $Q$ with sample-size-based weights and its exact distribution (available for mean difference and evaluated by Farebrother's algorithm) provides precise levels even for very small and unbalanced sample sizes. The corresponding estimator of $τ^2$ is almost unbiased for 10 or more small studies. Under these circumstances this performance compares favorably with the extremely liberal behavior of the standard tests of heterogeneity and the largely biased estimators based on inverse-variance weights.
△ Less
Submitted 21 October, 2020;
originally announced October 2020.
-
Exploring Consequences of Simulation Design for Apparent Performance of Statistical Methods. 2: Results from simulations with normally and uniformly distributed sample sizes
Authors:
Elena Kulinskaya,
David C. Hoaglin,
Ilyas Bakbergenuly
Abstract:
This report continues our investigation of effects a simulation design may have on the conclusions on performance of statistical methods. In the context of meta-analysis of log-odds-ratios, we consider five generation mechanisms for control probabilities and log-odds-ratios. Our first report (Kulinskaya et al. 2020) considered constant sample sizes. Here we report on the results for normally and u…
▽ More
This report continues our investigation of effects a simulation design may have on the conclusions on performance of statistical methods. In the context of meta-analysis of log-odds-ratios, we consider five generation mechanisms for control probabilities and log-odds-ratios. Our first report (Kulinskaya et al. 2020) considered constant sample sizes. Here we report on the results for normally and uniformly distributed sample sizes.
△ Less
Submitted 7 July, 2020;
originally announced July 2020.
-
Exploring Consequences of Simulation Design for Apparent Performance of Statistical Methods. 1: Results from simulations with constant sample sizes
Authors:
Elena Kulinskaya,
David C. Hoaglin,
Ilyas Bakbergenuly
Abstract:
Contemporary statistical publications rely on simulation to evaluate performance of new methods and compare them with established methods. In the context of meta-analysis of log-odds-ratios, we investigate how the ways in which simulations are implemented affect such conclusions. Choices of distributions for sample sizes and/or control probabilities considerably affect conclusions about statistica…
▽ More
Contemporary statistical publications rely on simulation to evaluate performance of new methods and compare them with established methods. In the context of meta-analysis of log-odds-ratios, we investigate how the ways in which simulations are implemented affect such conclusions. Choices of distributions for sample sizes and/or control probabilities considerably affect conclusions about statistical methods. Here we report on the results for constant sample sizes. Our two subsequent publications will cover normally and uniformly distributed sample sizes.
△ Less
Submitted 3 July, 2020; v1 submitted 30 June, 2020;
originally announced June 2020.
-
Simulation study of estimating between-study variance and overall effect in meta-analyses of log-response-ratio for lognormal data
Authors:
Ilyas Bakbergenuly,
David C. Hoaglin,
Elena Kulinskaya
Abstract:
Methods for random-effects meta-analysis require an estimate of the between-study variance, $τ^2$. The performance of estimators of $τ^2$ (measured by bias and coverage) affects their usefulness in assessing heterogeneity of study-level effects, and also the performance of related estimators of the overall effect. For the effect measure log-response-ratio (LRR, also known as the logarithm of the r…
▽ More
Methods for random-effects meta-analysis require an estimate of the between-study variance, $τ^2$. The performance of estimators of $τ^2$ (measured by bias and coverage) affects their usefulness in assessing heterogeneity of study-level effects, and also the performance of related estimators of the overall effect. For the effect measure log-response-ratio (LRR, also known as the logarithm of the ratio of means, RoM), we review four point estimators of $τ^2$ (the popular methods of DerSimonian-Laird (DL), restricted maximum likelihood, and Mandel and Paule (MP), and the less-familiar method of Jackson), four interval estimators for $τ^2$ (profile likelihood, Q-profile, Biggerstaff and Jackson, and Jackson), five point estimators of the overall effect (the four related to the point estimators of $τ^2$ and an estimator whose weights use only study-level sample sizes), and seven interval estimators for the overall effect (four based on the point estimators for $τ^2$, the Hartung-Knapp-Sidik-Jonkman (HKSJ) interval, a modification of HKSJ that uses the MP estimator of $τ^2$ instead of the DL estimator, and an interval based on the sample-size-weighted estimator). We obtain empirical evidence from extensive simulations of data from lognormal distributions.
△ Less
Submitted 5 July, 2019; v1 submitted 3 May, 2019;
originally announced May 2019.
-
Simulation study of estimating between-study variance and overall effect in meta-analyses of mean difference
Authors:
Ilyas Bakbergenuly,
David C. Hoaglin,
Elena Kulinskaya
Abstract:
Methods for random-effects meta-analysis require an estimate of the between-study variance, $τ^2$. The performance of estimators of $τ^2$ (measured by bias and coverage) affects their usefulness in assessing heterogeneity of study-level effects, and also the performance of related estimators of the overall effect. For the effect measure mean difference (MD), we review five point estimators of…
▽ More
Methods for random-effects meta-analysis require an estimate of the between-study variance, $τ^2$. The performance of estimators of $τ^2$ (measured by bias and coverage) affects their usefulness in assessing heterogeneity of study-level effects, and also the performance of related estimators of the overall effect. For the effect measure mean difference (MD), we review five point estimators of $τ^2$ (the popular methods of DerSimonian-Laird, restricted maximum likelihood, and Mandel and Paule (MP); the less-familiar method of Jackson; and a new method (WT) based on the improved approximation to the distribution of the $Q$ statistic by \cite{kulinskaya2004welch}), five interval estimators for $τ^2$ (profile likelihood, Q-profile, Biggerstaff and Jackson, Jackson, and the new WT method), six point estimators of the overall effect (the five related to the point estimators of $τ^2$ and an estimator whose weights use only study-level sample sizes), and eight interval estimators for the overall effect (five based on the point estimators for $τ^2$, the Hartung-Knapp-Sidik-Jonkman (HKSJ) interval, a modification of HKSJ, and an interval based on the sample-size-weighted estimator). We obtain empirical evidence from extensive simulations and an example.
△ Less
Submitted 1 April, 2019;
originally announced April 2019.
-
Simulation study of estimating between-study variance and overall effect in meta-analysis of standardized mean difference
Authors:
Ilyas Bakbergenuly,
David C. Hoaglin,
Elena Kulinskaya
Abstract:
Methods for random-effects meta-analysis require an estimate of the between-study variance, $τ^2$. The performance of estimators of $τ^2$ (measured by bias and coverage) affects their usefulness in assessing heterogeneity of study-level effects, and also the performance of related estimators of the overall effect. For the effect measure standardized mean difference (SMD), we provide the results fr…
▽ More
Methods for random-effects meta-analysis require an estimate of the between-study variance, $τ^2$. The performance of estimators of $τ^2$ (measured by bias and coverage) affects their usefulness in assessing heterogeneity of study-level effects, and also the performance of related estimators of the overall effect. For the effect measure standardized mean difference (SMD), we provide the results from extensive simulations on five point estimators of $τ^2$ (the popular methods of DerSimonian-Laird, restricted maximum likelihood, and Mandel and Paule (MP); the less-familiar method of Jackson; the new method (KDB) based on the improved approximation to the distribution of the Q statistic by Kulinskaya, Dollinger and Bjørkestøl (2011) ), five interval estimators for $τ^2$ (profile likelihood, Q-profile, Biggerstaff and Jackson, Jackson, and the new KDB method), six point estimators of the overall effect (the five related to the point estimators of $τ^2$ and an estimator whose weights use only study-level sample sizes), and eight interval estimators for the overall effect (five based on the point estimators for $τ^2$; the Hartung-Knapp-Sidik-Jonkman (HKSJ) interval; a modification of HKSJ; and an interval based on the sample-size-weighted estimator).
△ Less
Submitted 4 March, 2019;
originally announced March 2019.
-
Simulation study of estimating between-study variance and overall effect in meta-analysis of odds-ratios
Authors:
Ilyas Bakbergenuly,
David C. Hoaglin,
Elena Kulinskaya
Abstract:
Random-effects meta-analysis requires an estimate of the between-study variance, $τ^2$. We study methods of estimation of $τ^2$ and its confidence interval in meta-analysis of odds ratio, and also the performance of related estimators of the overall effect. We provide results of extensive simulations on five point estimators of $τ^2$ (the popular methods of DerSimonian-Laird, restricted maximum li…
▽ More
Random-effects meta-analysis requires an estimate of the between-study variance, $τ^2$. We study methods of estimation of $τ^2$ and its confidence interval in meta-analysis of odds ratio, and also the performance of related estimators of the overall effect. We provide results of extensive simulations on five point estimators of $τ^2$ (the popular methods of DerSimonian-Laird, restricted maximum likelihood, and Mandel and Paule; the less-familiar method of Jackson; and the new method (KD) based on the improved approximation to the distribution of the Q statistic by Kulinskaya and Dollinger (2015)); five interval estimators for $τ^2$ (profile likelihood, Q-profile, Biggerstaff and Jackson, Jackson, and KD), six point estimators of the overall effect (the five inverse-variance estimators related to the point estimators of $τ^2$ and an estimator (SSW) whose weights use only study-level sample sizes), and eight interval estimators for the overall effect (five based on the point estimators for $τ^2$; the Hartung-Knapp-Sidik-Jonkman (HKSJ) interval; a KD-based modification of HKSJ; and an interval based on the sample-size-weighted estimator). Results of our simulations show that none of the point estimators of $τ^2$ can be recommended, however the new KD estimator provides a reliable coverage of $τ^2$. Inverse-variance estimators of the overall effect are substantially biased. The SSW estimator of the overall effect and the related confidence interval provide the reliable point and interval estimation of log-odds-ratio.
△ Less
Submitted 19 February, 2019;
originally announced February 2019.
