Showing 1–14 of 14 results for author: Leckie, G

Sociodemographic inequalities in student achievement: An intersectional multilevel analysis of individual heterogeneity and discriminatory accuracy (MAIHDA) with application to students in London, England

Authors: Lucy Prior, Clare Evans, Juan Merlo, George Leckie

Abstract: Sociodemographic inequalities in student achievement are a persistent concern for education systems and are increasingly recognized to be intersectional. Intersectionality considers the multidimensional nature of disadvantage, appreciating the interlocking social determinants which shape individual experience. Intersectional multilevel analysis of individual heterogeneity and discriminatory accura… ▽ More Sociodemographic inequalities in student achievement are a persistent concern for education systems and are increasingly recognized to be intersectional. Intersectionality considers the multidimensional nature of disadvantage, appreciating the interlocking social determinants which shape individual experience. Intersectional multilevel analysis of individual heterogeneity and discriminatory accuracy (MAIHDA) is a new approach developed in population health but with limited application in educational research. In this study, we introduce and apply this approach to study sociodemographic inequalities in student achievement across two cohorts of students in London, England. We define 144 intersectional strata arising from combinations of student age, gender, free school meal status, special educational needs, and ethnicity. We find substantial strata-level variation in achievement composed primarily by additive rather than interactive effects with results stubbornly consistent across the cohorts. We conclude that policymakers should pay greater attention to multiply marginalized students and intersectional MAIHDA provides a useful approach to study their experiences. △ Less

Submitted 10 October, 2023; v1 submitted 10 November, 2022; originally announced November 2022.

Comments: 63 pages (main text 39 pages), 10 figures, 10 tables

Decomposing ethnic achievement gaps across multiple levels of analysis and for multiple ethnic groups

Authors: Beatriz Gallo Cordoba, George Leckie, William J. Browne

Abstract: Ethnic achievement gaps are often explained in terms of student and school factors. The decomposition of these gaps into their within- and between-school components has therefore been applied as a strategy to quantify the overall influence of each set of factors. Three competing approaches have previously been proposed, but each is limited to the study of student-school decompositions of the gap b… ▽ More Ethnic achievement gaps are often explained in terms of student and school factors. The decomposition of these gaps into their within- and between-school components has therefore been applied as a strategy to quantify the overall influence of each set of factors. Three competing approaches have previously been proposed, but each is limited to the study of student-school decompositions of the gap between two ethnic groups (e.g., White and Black). The authors show that these approaches can be reformulated as mediation models facilitating new extensions to allow additional levels in the school system (e.g., classrooms, school districts, geographic areas) and multiple ethnic groups (e.g., White, Black, Hispanic, Asian). The authors illustrate these extensions using administrative data for high school students in Colombia and highlight the increased substantive insights and nuanced policy implications they afford. △ Less

Submitted 12 June, 2022; originally announced June 2022.

Comments: Sociological Methodology (2022)

Mixed-effects location scale models for joint modelling school value-added effects on the mean and variance of student achievement

Authors: George Leckie, Richard Parker, Harvey Goldstein, Kate Tilling

Abstract: School value-added models are widely applied to study, monitor, and hold schools to account for school differences in student learning. The traditional model is a mixed-effects linear regression of student current achievement on student prior achievement, background characteristics, and a school random intercept effect. The latter is referred to as the school value-added score and measures the mea… ▽ More School value-added models are widely applied to study, monitor, and hold schools to account for school differences in student learning. The traditional model is a mixed-effects linear regression of student current achievement on student prior achievement, background characteristics, and a school random intercept effect. The latter is referred to as the school value-added score and measures the mean student covariate-adjusted achievement in each school. In this article, we argue that further insights may be gained by additionally studying the variance in this quantity in each school. These include the ability to identify both individual schools and school types that exhibit unusually high or low variability in student achievement, even after accounting for differences in student intakes. We explore and illustrate how this can be done via fitting mixed-effects location scale versions of the traditional school value-added model. We discuss the implications of our work for research and school accountability systems. △ Less

Submitted 30 September, 2023; v1 submitted 5 October, 2021; originally announced October 2021.

A Comparison of Value-Added Models for School Accountability

Authors: George Leckie, Lucy Prior

Abstract: School accountability systems increasingly hold schools to account for their performances using value-added models purporting to measure the effects of schools on student learning. The most common approach is to fit a linear regression of student current achievement on student prior achievement, where the school effects are the school means of the predicted residuals. In the literature, further ad… ▽ More School accountability systems increasingly hold schools to account for their performances using value-added models purporting to measure the effects of schools on student learning. The most common approach is to fit a linear regression of student current achievement on student prior achievement, where the school effects are the school means of the predicted residuals. In the literature, further adjustments are usually made for student sociodemographics and sometimes school composition and 'non-malleable' characteristics. However, accountability systems typically make fewer adjustments: for transparency to end users, because data is unavailable or of insufficient quality, or for ideological reasons. There is therefore considerable interest in understanding the extent to which simpler models give similar school effects to more theoretically justified but complex models. We explore these issues via a case study and empirical analysis of England's 'Progress 8' secondary school accountability system. △ Less

Submitted 14 January, 2022; v1 submitted 20 July, 2021; originally announced July 2021.

A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses, and challenges for 'Progress 8'

Authors: Lucy Prior, John Jerrim, Dave Thomson, George Leckie

Abstract: School performance measures are published annually in England to hold schools to account and to support parental school choice. This article reviews and evaluates the Progress 8 secondary school accountability system for state-funded schools. We assess the statistical strengths and weaknesses of Progress 8 relating to: choice of pupil outcome attainment measures; potential adjustments for pupil in… ▽ More School performance measures are published annually in England to hold schools to account and to support parental school choice. This article reviews and evaluates the Progress 8 secondary school accountability system for state-funded schools. We assess the statistical strengths and weaknesses of Progress 8 relating to: choice of pupil outcome attainment measures; potential adjustments for pupil input attainment and background characteristics; decisions around which schools and pupils are excluded from the measure; presentation of Progress 8 to users, choice of statistical model, and calculation of statistical uncertainty; and issues related to the volatility of school performance over time, including scope for reporting multi-year averages. We then discuss challenges for Progress 8 raised by the COVID-19 pandemic. Six simple recommendations follow to improve Progress 8 and school accountability in England. △ Less

Submitted 15 August, 2022; v1 submitted 13 April, 2021; originally announced April 2021.

Comments: 68 pages, 8 figures

Providing educational accountability for Local Authorities based upon sampling pupils within schools: moving away from simplistic school league tables

Authors: Harvey Goldstein, George Leckie, Lucy Prior

Abstract: This paper proposes an alternative educational accountability system for England that moves away from simplistic comparisons, or league tables, among schools towards a more nuanced reporting at a level of education authorities or other school grou**s. Based upon the sampling of pupils within schools, it proposes the use of quantitative pupil assessment data as screening devices that provide evid… ▽ More This paper proposes an alternative educational accountability system for England that moves away from simplistic comparisons, or league tables, among schools towards a more nuanced reporting at a level of education authorities or other school grou**s. Based upon the sampling of pupils within schools, it proposes the use of quantitative pupil assessment data as screening devices that provide evidence within an integrated accountability system that includes inspection. At the same time, it aims to provide a richer set of data than currently available for research as well as accountability purposes. We argue that if carefully implemented within a context of school improvement, such a system has the potential to largely eliminate the deleterious side effects and curriculum distortions of the present system. While being proposed within the context of the current English secondary school system, the proposals will have relevance for other phases of schooling and similar systems in other countries. △ Less

Submitted 23 February, 2020; originally announced February 2020.

Comments: 5700 words

School value-added models for multivariate academic and non-academic outcomes: A more rounded approach to using student data to inform school accountability

Authors: Lucy Prior, Harvey Goldstein, George Leckie

Abstract: Education systems around the world increasingly rely on school value-added models to hold schools to account. These models typically focus on a limited number of academic outcomes, failing to recognise the broader range of non-academic student outcomes, attitudes and behaviours to which schools contribute. We explore how the traditional multilevel modelling approach to school value-added models ca… ▽ More Education systems around the world increasingly rely on school value-added models to hold schools to account. These models typically focus on a limited number of academic outcomes, failing to recognise the broader range of non-academic student outcomes, attitudes and behaviours to which schools contribute. We explore how the traditional multilevel modelling approach to school value-added models can be extended to simultaneously analyse multiple academic and non-academic outcomes and thereby can potentially provide a more rounded approach to using student data to inform school accountability. We jointly model student attainment, absence and exclusion data for schools in England. We find different results across the three outcomes, in terms of the size and consistency of school effects, and the importance of adjusting for student and school characteristics. The results suggest the three outcomes are capturing fundamentally distinct aspects of school performance, recommending the consideration of non-academic outcomes in systems of school accountability. △ Less

Submitted 7 January, 2020; originally announced January 2020.

Comments: Main Manuscript 38 pages, 4 figures, Supplementary Material 17 pages, 4 figures

Variance partitioning in multilevel models for count data

Authors: George Leckie, William Browne, Harvey Goldstein, Juan Merlo, Peter Austin

Abstract: A first step when fitting multilevel models to continuous responses is to explore the degree of clustering in the data. Researchers fit variance-component models and then report the proportion of variation in the response that is due to systematic differences between clusters. Equally they report the response correlation between units within a cluster. These statistics are popularly referred to as… ▽ More A first step when fitting multilevel models to continuous responses is to explore the degree of clustering in the data. Researchers fit variance-component models and then report the proportion of variation in the response that is due to systematic differences between clusters. Equally they report the response correlation between units within a cluster. These statistics are popularly referred to as variance partition coefficients (VPCs) and intraclass correlation coefficients (ICCs). When fitting multilevel models to categorical (binary, ordinal, or nominal) and count responses, these statistics prove more challenging to calculate. For categorical response models, researchers appeal to their latent response formulations and report VPCs/ICCs in terms of latent continuous responses envisaged to underly the observed categorical responses. For standard count response models, however, there are no corresponding latent response formulations. More generally, there is a paucity of guidance on how to partition the variation. As a result, applied researchers are likely to avoid or inadequately report and discuss the substantive importance of clustering and cluster effects in their studies. A recent article drew attention to a little-known exact algebraic expression for the VPC/ICC for the special case of the two-level random-intercept Poisson model. In this article, we make a substantial new contribution. First, we derive exact VPC/ICC expressions for more flexible negative binomial models that allows for overdispersion, a phenomenon which often occurs in practice. Then we derive exact VPC/ICC expressions for three-level and random-coefficient extensions to these models. We illustrate our work with an application to student absenteeism. △ Less

Submitted 13 February, 2020; v1 submitted 15 November, 2019; originally announced November 2019.

The implications of Labour's plan to scrap Key Stage 2 tests for Progress 8 and secondary school accountability in England

Authors: George Leckie, Lucy Prior, Harvey Goldstein

Abstract: In England, Progress 8 is the Conservative government's headline secondary school performance and accountability measure. Progress 8 attempts to measure the average academic progress pupils make in each school between their KS2 tests and their GCSE Attainment 8 examinations. The Labour opposition recently announced they would scrap the KS2 tests were they to be elected. Such a move, however, would… ▽ More In England, Progress 8 is the Conservative government's headline secondary school performance and accountability measure. Progress 8 attempts to measure the average academic progress pupils make in each school between their KS2 tests and their GCSE Attainment 8 examinations. The Labour opposition recently announced they would scrap the KS2 tests were they to be elected. Such a move, however, would preclude the publication of Progress 8 and would leave schools to be compared in terms of their average Attainment 8 scores or, at best, their Attainment 8 scores only adjusted for school differences in pupil demographic and socioeconomic characteristics. In this paper, we argue and illustrate empirically that this best-case scenario of an 'Adjusted Attainment 8' measure would prove less fair and meaningful than Progress 8 and therefore a backwards step, especially when Progress 8 itself has been criticised as biased against schools teaching educationally disadvantaged intakes. △ Less

Submitted 15 November, 2019; originally announced November 2019.

Methods and Software for the Multilevel Social Relations Model: A Tutorial

Authors: Jeremy Koster, George Leckie, Brandy Aven, Christopher Charlton

Abstract: This tutorial demonstrates the estimation and interpretation of the Multilevel Social Relations Model for dyadic data. The Social Relations Model is appropriate for data structures in which individuals appear multiple times as both the source and recipient of dyadic outcomes. Estimated using Stat-JR statistical software, the models are fitted to multiple outcome types: continuous, count, and binar… ▽ More This tutorial demonstrates the estimation and interpretation of the Multilevel Social Relations Model for dyadic data. The Social Relations Model is appropriate for data structures in which individuals appear multiple times as both the source and recipient of dyadic outcomes. Estimated using Stat-JR statistical software, the models are fitted to multiple outcome types: continuous, count, and binary outcomes. In addition, models are demonstrated for dyadic data from a single group and from multiple groups. The modeling approaches are illustrated via a series of case studies, and the data and software to replicate these analyses are available as supplemental files. △ Less

Submitted 22 July, 2019; originally announced July 2019.

Multilevel models for continuous outcomes

Authors: George Leckie

Abstract: Multilevel models (mixed-effect models or hierarchical linear models) are now a standard approach to analysing clustered and longitudinal data in the social, behavioural and medical sciences. This review article focuses on multilevel linear regression models for continuous responses (outcomes or dependent variables). These models can be viewed as an extension of conventional linear regression mode… ▽ More Multilevel models (mixed-effect models or hierarchical linear models) are now a standard approach to analysing clustered and longitudinal data in the social, behavioural and medical sciences. This review article focuses on multilevel linear regression models for continuous responses (outcomes or dependent variables). These models can be viewed as an extension of conventional linear regression models to account for and learn from the clustering in the data. Common clustered applications include studies of school effects on student achievement, hospital effects on patient health, and neighbourhood effects on respondent attitudes. In all these examples, multilevel models allow one to study how the regression relationships vary across clusters, to identify those cluster characteristics which predict such variation, to disentangle social processes operating at different levels of analysis, and to make cluster-specific predictions. Common longitudinal applications include studies of growth curves of individual height and weight and developmental trajectories of individual behaviours. In these examples, multilevel models allow one to describe and explain variation in growth rates and to simultaneously explore predictors of both of intra- and inter-individual variation. This article introduces and illustrates this powerful class of model. We start by focusing on the most commonly applied two-level random-intercept and -slope models. We illustrate through two detailed examples how these models can be applied to both clustered and longitudinal data and in both observational and experimental settings. We then review more flexible three-level, cross-classified, multiple membership and multivariate response models. We end by recommending a range of further reading on all these topics. △ Less

Submitted 12 July, 2019; originally announced July 2019.

Multiple membership multilevel models

Authors: George Leckie

Abstract: Multiple membership multilevel models are an extension of standard multilevel models for non-hierarchical data that have multiple membership structures. Traditional multilevel models involve hierarchical data structures whereby lower-level units such as students are nested within higher-level units such as schools and where these higher-level units may in turn be nested within further grou**s or… ▽ More Multiple membership multilevel models are an extension of standard multilevel models for non-hierarchical data that have multiple membership structures. Traditional multilevel models involve hierarchical data structures whereby lower-level units such as students are nested within higher-level units such as schools and where these higher-level units may in turn be nested within further grou**s or clusters such as school districts, regions, and countries. With hierarchical data structures, there is an exact nesting of each lower-level unit in one and only one higher-level unit. For example, each student attends one school, each school is located within one school district, and so on. However, social reality is more complicated than this, and so social and behavioural data often do not follow pure or strict hierarchies. Two types of non-hierarchical data structures which often appear in practice are cross-classified and multiple membership structures. In this article, we describe multiple membership data structures and multiple membership models which can be used to analyse them. △ Less

Submitted 4 July, 2019; originally announced July 2019.

Comments: arXiv admin note: substantial text overlap with arXiv:1907.02569

Cross-classified multilevel models

Authors: George Leckie

Abstract: Cross-classified multilevel modelling is an extension of standard multilevel modelling for non-hierarchical data that have cross-classified structures. Traditional multilevel models involve hierarchical data structures whereby lower level units such as students are nested within higher level units such as schools and where these higher level units may in turn be nested within further grou**s or… ▽ More Cross-classified multilevel modelling is an extension of standard multilevel modelling for non-hierarchical data that have cross-classified structures. Traditional multilevel models involve hierarchical data structures whereby lower level units such as students are nested within higher level units such as schools and where these higher level units may in turn be nested within further grou**s or clusters such as school districts, regions, and countries. With hierarchical data structures, there is an exact nesting of each lower level unit in one and only one higher level unit. For example, each student attends one school, each school is located within one school district, and so on. However, social reality is more complicated than this, and so social and behavioural data often do not follow pure or strict hierarchies. Two types of non-hierarchical data structures which often appear in practice are cross-classified and multiple membership structures. In this article, we describe cross-classified data structures and cross-classified hierarchical linear modelling which can be used to analyse them. △ Less

Submitted 4 July, 2019; originally announced July 2019.

Should we adjust for pupil background in school value-added models? A study of Progress 8 and school accountability in England

Authors: George Leckie, Harvey Goldstein

Abstract: In the UK, US and elsewhere, school accountability systems increasingly compare schools using value-added measures of school performance derived from pupil scores in high-stakes standardised tests. Rather than naively comparing school average scores, which largely reflect school intake differences in prior attainment, these measures attempt to compare the average progress or improvement pupils mak… ▽ More In the UK, US and elsewhere, school accountability systems increasingly compare schools using value-added measures of school performance derived from pupil scores in high-stakes standardised tests. Rather than naively comparing school average scores, which largely reflect school intake differences in prior attainment, these measures attempt to compare the average progress or improvement pupils make during a year or phase of schooling. Schools, however, also differ in terms of their pupil demographic and socioeconomic characteristics and these also predict why some schools subsequently score higher than others. Many therefore argue that value-added measures unadjusted for pupil background are biased in favour of schools with more 'educationally advantaged' intakes. But, others worry that adjusting for pupil background entrenches socioeconomic inequities and excuses low performing schools. In this article we explore these theoretical arguments and their practical importance in the context of the 'Progress 8' secondary school accountability system in England which has chosen to ignore pupil background. We reveal how the reported low or high performance of many schools changes dramatically once adjustments are made for pupil background and these changes also affect the reported differential performances of region and of different school types. We conclude that accountability systems which choose to ignore pupil background are likely to reward and punish the wrong schools and this will likely have detrimental effects on pupil learning. These findings, especially when coupled with more general concerns surrounding high-stakes testing and school value-added models, raise serious doubts about their use in school accountability systems. △ Less

Submitted 22 November, 2018; originally announced November 2018.