-
The Impossibility of Fair LLMs
Authors:
Jacy Anthis,
Kristian Lum,
Michael Ekstrand,
Avi Feller,
Alexander D'Amour,
Chenhao Tan
Abstract:
The need for fair AI is increasingly clear in the era of general-purpose systems such as ChatGPT, Gemini, and other large language models (LLMs). However, the increasing complexity of human-AI interaction and its social impacts have raised questions of how fairness standards could be applied. Here, we review the technical frameworks that machine learning researchers have used to evaluate fairness,…
▽ More
The need for fair AI is increasingly clear in the era of general-purpose systems such as ChatGPT, Gemini, and other large language models (LLMs). However, the increasing complexity of human-AI interaction and its social impacts have raised questions of how fairness standards could be applied. Here, we review the technical frameworks that machine learning researchers have used to evaluate fairness, such as group fairness and fair representations, and find that their application to LLMs faces inherent limitations. We show that each framework either does not logically extend to LLMs or presents a notion of fairness that is intractable for LLMs, primarily due to the multitudes of populations affected, sensitive attributes, and use cases. To address these challenges, we develop guidelines for the more realistic goal of achieving fairness in particular use cases: the criticality of context, the responsibility of LLM developers, and the need for stakeholder participation in an iterative process of design and evaluation. Moreover, it may eventually be possible and even necessary to use the general-purpose capabilities of AI systems to address fairness challenges as a form of scalable AI-assisted alignment.
△ Less
Submitted 28 May, 2024;
originally announced June 2024.
-
Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation
Authors:
Kristian Lum,
Jacy Reese Anthis,
Chirag Nagpal,
Alexander D'Amour
Abstract:
Bias benchmarks are a popular method for studying the negative impacts of bias in LLMs, yet there has been little empirical investigation of whether these benchmarks are actually indicative of how real world harm may manifest in the real world. In this work, we study the correspondence between such decontextualized "trick tests" and evaluations that are more grounded in Realistic Use and Tangible…
▽ More
Bias benchmarks are a popular method for studying the negative impacts of bias in LLMs, yet there has been little empirical investigation of whether these benchmarks are actually indicative of how real world harm may manifest in the real world. In this work, we study the correspondence between such decontextualized "trick tests" and evaluations that are more grounded in Realistic Use and Tangible {Effects (i.e. RUTEd evaluations). We explore this correlation in the context of gender-occupation bias--a popular genre of bias evaluation. We compare three de-contextualized evaluations adapted from the current literature to three analogous RUTEd evaluations applied to long-form content generation. We conduct each evaluation for seven instruction-tuned LLMs. For the RUTEd evaluations, we conduct repeated trials of three text generation tasks: children's bedtime stories, user personas, and English language learning exercises. We found no correspondence between trick tests and RUTEd evaluations. Specifically, selecting the least biased model based on the de-contextualized results coincides with selecting the model with the best performance on RUTEd evaluations only as often as random chance. We conclude that evaluations that are not based in realistic use are likely insufficient to mitigate and assess bias and real-world harms.
△ Less
Submitted 19 February, 2024;
originally announced February 2024.
-
Flip** the Script on Criminal Justice Risk Assessment: An actuarial model for assessing the risk the federal sentencing system poses to defendants
Authors:
Mikaela Meyer,
Aaron Horowitz,
Erica Marshall,
Kristian Lum
Abstract:
In the criminal justice system, algorithmic risk assessment instruments are used to predict the risk a defendant poses to society; examples include the risk of recidivating or the risk of failing to appear at future court dates. However, defendants are also at risk of harm from the criminal justice system. To date, there exists no risk assessment instrument that considers the risk the system poses…
▽ More
In the criminal justice system, algorithmic risk assessment instruments are used to predict the risk a defendant poses to society; examples include the risk of recidivating or the risk of failing to appear at future court dates. However, defendants are also at risk of harm from the criminal justice system. To date, there exists no risk assessment instrument that considers the risk the system poses to the individual. We develop a risk assessment instrument that "flips the script." Using data about U.S. federal sentencing decisions, we build a risk assessment instrument that predicts the likelihood an individual will receive an especially lengthy sentence given factors that should be legally irrelevant to the sentencing decision. To do this, we develop a two-stage modeling approach. Our first-stage model is used to determine which sentences were "especially lengthy." We then use a second-stage model to predict the defendant's risk of receiving a sentence that is flagged as especially lengthy given factors that should be legally irrelevant. The factors that should be legally irrelevant include, for example, race, court location, and other socio-demographic information about the defendant. Our instrument achieves comparable predictive accuracy to risk assessment instruments used in pretrial and parole contexts. We discuss the limitations of our modeling approach and use the opportunity to highlight how traditional risk assessment instruments in various criminal justice settings also suffer from many of the same limitations and embedded value systems of their creators.
△ Less
Submitted 13 July, 2022; v1 submitted 26 May, 2022;
originally announced May 2022.
-
Characterizing patterns in police stops by race in Minneapolis from 2016-2021
Authors:
Tuviere Onookome-Okome,
Jonah Gorondensky,
Eric Rose,
Jeffery Sauer,
Kristian Lum,
Erica EM Moodie
Abstract:
The murder of George Floyd centered Minneapolis, Minnesota, in conversations on racial injustice in the US. We leverage open data from the Minneapolis Police Department to analyze individual, geographic, and temporal patterns in more than 170,000 police stops since 2016. We evaluate person and vehicle searches at the individual level by race using generalized estimating equations with neighborhood…
▽ More
The murder of George Floyd centered Minneapolis, Minnesota, in conversations on racial injustice in the US. We leverage open data from the Minneapolis Police Department to analyze individual, geographic, and temporal patterns in more than 170,000 police stops since 2016. We evaluate person and vehicle searches at the individual level by race using generalized estimating equations with neighborhood clustering, directly addressing neighborhood differences in police activity. Minneapolis exhibits clear patterns of disproportionate policing by race, wherein Black people are searched at higher rates compared to White people. Temporal visualizations indicate that police stops declined following the murder of George Floyd. This analysis provides contemporary evidence on the state of policing for a major metropolitan area in the United States.
△ Less
Submitted 12 May, 2022;
originally announced May 2022.
-
De-biasing "bias" measurement
Authors:
Kristian Lum,
Yunfeng Zhang,
Amanda Bower
Abstract:
When a model's performance differs across socially or culturally relevant groups--like race, gender, or the intersections of many such groups--it is often called "biased." While much of the work in algorithmic fairness over the last several years has focused on develo** various definitions of model fairness (the absence of group-wise model performance disparities) and eliminating such "bias," mu…
▽ More
When a model's performance differs across socially or culturally relevant groups--like race, gender, or the intersections of many such groups--it is often called "biased." While much of the work in algorithmic fairness over the last several years has focused on develo** various definitions of model fairness (the absence of group-wise model performance disparities) and eliminating such "bias," much less work has gone into rigorously measuring it. In practice, it important to have high quality, human digestible measures of model performance disparities and associated uncertainty quantification about them that can serve as inputs into multi-faceted decision-making processes. In this paper, we show both mathematically and through simulation that many of the metrics used to measure group-wise model performance disparities are themselves statistically biased estimators of the underlying quantities they purport to represent. We argue that this can cause misleading conclusions about the relative group-wise model performance disparities along different dimensions, especially in cases where some sensitive variables consist of categories with few members. We propose the "double-corrected" variance estimator, which provides unbiased estimates and uncertainty quantification of the variance of model performance across groups. It is conceptually simple and easily implementable without statistical software package or numerical optimization. We demonstrate the utility of this approach through simulation and show on a real dataset that while statistically biased estimators of group-wise model performance disparities indicate statistically significant differences, when accounting for statistical bias in the estimator, the estimated between-group disparities are no longer statistically significant.
△ Less
Submitted 29 June, 2022; v1 submitted 11 May, 2022;
originally announced May 2022.
-
Closer than they appear: A Bayesian perspective on individual-level heterogeneity in risk assessment
Authors:
Kristian Lum,
David B. Dunson,
James Johndrow
Abstract:
Risk assessment instruments are used across the criminal justice system to estimate the probability of some future behavior given covariates. The estimated probabilities are then used in making decisions at the individual level. In the past, there has been controversy about whether the probabilities derived from group-level calculations can meaningfully be applied to individuals. Using Bayesian hi…
▽ More
Risk assessment instruments are used across the criminal justice system to estimate the probability of some future behavior given covariates. The estimated probabilities are then used in making decisions at the individual level. In the past, there has been controversy about whether the probabilities derived from group-level calculations can meaningfully be applied to individuals. Using Bayesian hierarchical models applied to a large longitudinal dataset from the court system in the state of Kentucky, we analyze variation in individual-level probabilities of failing to appear for court and the extent to which it is captured by covariates. We find that individuals within the same risk group vary widely in their probability of the outcome. In practice, this means that allocating individuals to risk groups based on standard approaches to risk assessment, in large part, results in creating distinctions among individuals who are not meaningfully different in terms of their likelihood of the outcome. This is because uncertainty about the probability that any particular individual will fail to appear is large relative to the difference in average probabilities among any reasonable set of risk groups.
△ Less
Submitted 1 February, 2021;
originally announced February 2021.
-
Estimating the number of SARS-CoV-2 infections and the impact of social distancing in the United States
Authors:
James Johndrow,
Kristian Lum,
Maria Gargiulo,
Patrick Ball
Abstract:
Understanding the number of individuals who have been infected with the novel coronavirus SARS-CoV-2, and the extent to which social distancing policies have been effective at limiting its spread, are critical for effective policy going forward. Here we present estimates of the extent to which confirmed cases in the United States undercount the true number of infections, and analyze how effective…
▽ More
Understanding the number of individuals who have been infected with the novel coronavirus SARS-CoV-2, and the extent to which social distancing policies have been effective at limiting its spread, are critical for effective policy going forward. Here we present estimates of the extent to which confirmed cases in the United States undercount the true number of infections, and analyze how effective social distancing measures have been at mitigating or suppressing the virus. Our analysis uses a Bayesian model of COVID-19 fatalities with a likelihood based on an underlying differential equation model of the epidemic. We provide analysis for four states with significant epidemics: California, Florida, New York, and Washington. Our short-term forecasts suggest that these states may be following somewhat different trajectories for growth of the number of cases and fatalities.
△ Less
Submitted 18 April, 2020; v1 submitted 6 April, 2020;
originally announced April 2020.
-
The impact of overbooking on a pre-trial risk assessment tool
Authors:
Kristian Lum,
Chesa Boudin,
Megan Price
Abstract:
Pre-trial risk assessment tools are used to make recommendations to judges about appropriate conditions of pre-trial supervision for people who have been arrested. Increasingly, there is concern about whether these models are operating fairly, including concerns about whether the models' input factors are fair measures of one's criminal activity. In this paper, we assess the impact of booking char…
▽ More
Pre-trial risk assessment tools are used to make recommendations to judges about appropriate conditions of pre-trial supervision for people who have been arrested. Increasingly, there is concern about whether these models are operating fairly, including concerns about whether the models' input factors are fair measures of one's criminal activity. In this paper, we assess the impact of booking charges that do not result in a conviction on a popular risk assessment tool, the Arnold Public Safety Assessment. Using data from a pilot run of the tool in San Francisco, CA, we find that booking charges that do not result in a conviction (i.e. charges that are dropped or end in an acquittal) increased the recommended level of pre-trial supervision in around 27% of cases evaluated by the tool
△ Less
Submitted 23 January, 2020;
originally announced January 2020.
-
Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions
Authors:
Shira Mitchell,
Eric Potash,
Solon Barocas,
Alexander D'Amour,
Kristian Lum
Abstract:
A recent flurry of research activity has attempted to quantitatively define "fairness" for decisions based on statistical and machine learning (ML) predictions. The rapid growth of this new field has led to wildly inconsistent terminology and notation, presenting a serious challenge for cataloguing and comparing definitions. This paper attempts to bring much-needed order.
First, we explicate the…
▽ More
A recent flurry of research activity has attempted to quantitatively define "fairness" for decisions based on statistical and machine learning (ML) predictions. The rapid growth of this new field has led to wildly inconsistent terminology and notation, presenting a serious challenge for cataloguing and comparing definitions. This paper attempts to bring much-needed order.
First, we explicate the various choices and assumptions made---often implicitly---to justify the use of prediction-based decisions. Next, we show how such choices and assumptions can raise concerns about fairness and we present a notationally consistent catalogue of fairness definitions from the ML literature. In doing so, we offer a concise reference for thinking through the choices, assumptions, and fairness considerations of prediction-based decision systems.
△ Less
Submitted 24 April, 2020; v1 submitted 19 November, 2018;
originally announced November 2018.
-
Removing the influence of a group variable in high-dimensional predictive modelling
Authors:
Emanuele Aliverti,
Kristian Lum,
James E. Johndrow,
David B. Dunson
Abstract:
In many application areas, predictive models are used to support or make important decisions. There is increasing awareness that these models may contain spurious or otherwise undesirable correlations. Such correlations may arise from a variety of sources, including batch effects, systematic measurement errors, or sampling bias. Without explicit adjustment, machine learning algorithms trained usin…
▽ More
In many application areas, predictive models are used to support or make important decisions. There is increasing awareness that these models may contain spurious or otherwise undesirable correlations. Such correlations may arise from a variety of sources, including batch effects, systematic measurement errors, or sampling bias. Without explicit adjustment, machine learning algorithms trained using these data can produce poor out-of-sample predictions which propagate these undesirable correlations. We propose a method to pre-process the training data, producing an adjusted dataset that is statistically independent of the nuisance variables with minimum information loss. We develop a conceptually simple approach for creating an adjusted dataset in high-dimensional settings based on a constrained form of matrix decomposition. The resulting dataset can then be used in any predictive algorithm with the guarantee that predictions will be statistically independent of the group variable. We develop a scalable algorithm for implementing the method, along with theory support in the form of independence guarantees and optimality. The method is illustrated on some simulation examples and applied to two case studies: removing machine-specific correlations from brain scan data, and removing race and ethnicity information from a dataset used to predict recidivism. That the motivation for removing undesirable correlations is quite different in the two applications illustrates the broad applicability of our approach.
△ Less
Submitted 19 November, 2019; v1 submitted 18 October, 2018;
originally announced October 2018.
-
The causal impact of bail on case outcomes for indigent defendants
Authors:
Kristian Lum,
Mike Baiocchi
Abstract:
We use near-far matching, a technique for estimating causal relationships, to explore whether bail causes a higher likelihood of conviction. We find evidence of a strong causal impact. This paper was compiled as a submission to the 2017 Fairness, Accountability, and Transparency in Machine Learning (FAT ML) workshop.
We use near-far matching, a technique for estimating causal relationships, to explore whether bail causes a higher likelihood of conviction. We find evidence of a strong causal impact. This paper was compiled as a submission to the 2017 Fairness, Accountability, and Transparency in Machine Learning (FAT ML) workshop.
△ Less
Submitted 14 July, 2017;
originally announced July 2017.
-
An algorithm for removing sensitive information: application to race-independent recidivism prediction
Authors:
James E. Johndrow,
Kristian Lum
Abstract:
Predictive modeling is increasingly being employed to assist human decision-makers. One purported advantage of replacing or augmenting human judgment with computer models in high stakes settings-- such as sentencing, hiring, policing, college admissions, and parole decisions-- is the perceived "neutrality" of computers. It is argued that because computer models do not hold personal prejudice, the…
▽ More
Predictive modeling is increasingly being employed to assist human decision-makers. One purported advantage of replacing or augmenting human judgment with computer models in high stakes settings-- such as sentencing, hiring, policing, college admissions, and parole decisions-- is the perceived "neutrality" of computers. It is argued that because computer models do not hold personal prejudice, the predictions they produce will be equally free from prejudice. There is growing recognition that employing algorithms does not remove the potential for bias, and can even amplify it if the training data were generated by a process that is itself biased. In this paper, we provide a probabilistic notion of algorithmic bias. We propose a method to eliminate bias from predictive models by removing all information regarding protected variables from the data to which the models will ultimately be trained. Unlike previous work in this area, our framework is general enough to accommodate data on any measurement scale. Motivated by models currently in use in the criminal justice system that inform decisions on pre-trial release and parole, we apply our proposed method to a dataset on the criminal histories of individuals at the time of sentencing to produce "race-neutral" predictions of re-arrest. In the process, we demonstrate that a common approach to creating "race-neutral" models-- omitting race as a covariate-- still results in racially disparate predictions. We then demonstrate that the application of our proposed method to these data removes racial disparities from predictions with minimal impact on predictive accuracy.
△ Less
Submitted 15 March, 2017;
originally announced March 2017.
-
Bayesian nonparametric generative models for causal inference with missing at random covariates
Authors:
Jason Roy,
Kirsten J Lum,
Michael J. Daniels,
Bret Zeldow,
Jordan Dworkin,
Vincent Lo Re III
Abstract:
We propose a general Bayesian nonparametric (BNP) approach to causal inference in the point treatment setting. The joint distribution of the observed data (outcome, treatment, and confounders) is modeled using an enriched Dirichlet process. The combination of the observed data model and causal assumptions allows us to identify any type of causal effect - differences, ratios, or quantile effects, e…
▽ More
We propose a general Bayesian nonparametric (BNP) approach to causal inference in the point treatment setting. The joint distribution of the observed data (outcome, treatment, and confounders) is modeled using an enriched Dirichlet process. The combination of the observed data model and causal assumptions allows us to identify any type of causal effect - differences, ratios, or quantile effects, either marginally or for subpopulations of interest. The proposed BNP model is well-suited for causal inference problems, as it does not require parametric assumptions about the distribution of confounders and naturally leads to a computationally efficient Gibbs sampling algorithm. By flexibly modeling the joint distribution, we are also able to impute (via data augmentation) values for missing covariates within the algorithm under an assumption of ignorable missingness, obviating the need to create separate imputed data sets. This approach for imputing the missing covariates has the additional advantage of guaranteeing congeniality between the imputation model and the analysis model, and because we use a BNP approach, parametric models are avoided for imputation. The performance of the method is assessed using simulation studies. The method is applied to data from a cohort study of human immunodeficiency virus/hepatitis C virus co-infected patients.
△ Less
Submitted 27 February, 2017;
originally announced February 2017.
-
A statistical framework for fair predictive algorithms
Authors:
Kristian Lum,
James Johndrow
Abstract:
Predictive modeling is increasingly being employed to assist human decision-makers. One purported advantage of replacing human judgment with computer models in high stakes settings-- such as sentencing, hiring, policing, college admissions, and parole decisions-- is the perceived "neutrality" of computers. It is argued that because computer models do not hold personal prejudice, the predictions th…
▽ More
Predictive modeling is increasingly being employed to assist human decision-makers. One purported advantage of replacing human judgment with computer models in high stakes settings-- such as sentencing, hiring, policing, college admissions, and parole decisions-- is the perceived "neutrality" of computers. It is argued that because computer models do not hold personal prejudice, the predictions they produce will be equally free from prejudice. There is growing recognition that employing algorithms does not remove the potential for bias, and can even amplify it, since training data were inevitably generated by a process that is itself biased. In this paper, we provide a probabilistic definition of algorithmic bias. We propose a method to remove bias from predictive models by removing all information regarding protected variables from the permitted training data. Unlike previous work in this area, our framework is general enough to accommodate arbitrary data types, e.g. binary, continuous, etc. Motivated by models currently in use in the criminal justice system that inform decisions on pre-trial release and paroling, we apply our proposed method to a dataset on the criminal histories of individuals at the time of sentencing to produce "race-neutral" predictions of re-arrest. In the process, we demonstrate that the most common approach to creating "race-neutral" models-- omitting race as a covariate-- still results in racially disparate predictions. We then demonstrate that the application of our proposed method to these data removes racial disparities from predictions with minimal impact on predictive accuracy.
△ Less
Submitted 25 October, 2016;
originally announced October 2016.
-
Estimating the observable population size from biased samples: a new approach to population estimation with capture heterogeneity
Authors:
James E. Johndrow,
Kristian Lum,
Daniel Manrique-Vallier
Abstract:
Capture-recapture methods aim to estimate the size of a closed population on the basis of multiple incomplete enumerations of individuals. In many applications, the individual probability of being recorded is heterogeneous in the population. Previous studies have suggested that it is not possible to reliably estimate the total population size when capture heterogeneity exists. Here we approach pop…
▽ More
Capture-recapture methods aim to estimate the size of a closed population on the basis of multiple incomplete enumerations of individuals. In many applications, the individual probability of being recorded is heterogeneous in the population. Previous studies have suggested that it is not possible to reliably estimate the total population size when capture heterogeneity exists. Here we approach population estimation in the presence of capture heterogeneity as a latent length biased nonparametric density estimation problem on the unit interval. We show that in this setting it is generally impossible to estimate the density on the entire unit interval in finite samples, and that estimators of the population size have high and sometimes unbounded risk when the density has significant mass near zero. As an alternative, we propose estimating the population of individuals with capture probability exceeding some threshold. We provide methods for selecting an appropriate threshold, and show that this approach results in estimators with substantially lower risk than estimators of the total population size, with correspondingly smaller uncertainty, even when the parameter of interest is the total population. The alternative paradigm is demonstrated in extensive simulation studies and an application to snowshoe hare multiple recapture data.
△ Less
Submitted 7 June, 2016;
originally announced June 2016.
-
An agent-based epidemiological model of incarceration
Authors:
Kristian Lum,
Samarth Swarup,
Stephen Eubank,
James Hawdon
Abstract:
We build an agent-based model of incarceration based on the SIS model of infectious disease propagation. Our central hypothesis is that the observed racial disparities in incarceration rates between Black and White Americans can be explained as the result of differential sentencing between the two demographic groups. We demonstrate that if incarceration can be spread through a social influence net…
▽ More
We build an agent-based model of incarceration based on the SIS model of infectious disease propagation. Our central hypothesis is that the observed racial disparities in incarceration rates between Black and White Americans can be explained as the result of differential sentencing between the two demographic groups. We demonstrate that if incarceration can be spread through a social influence network, then even relatively small differences in sentencing can result in the large disparities in incarceration rates. Controlling for effects of transmissibility, susceptibility, and influence network structure, our model reproduces the observed large disparities in incarceration rates given the differences in sentence lengths for White and Black drug offenders in the United States without extensive parameter tuning. We further establish the suitability of the SIS model as applied to incarceration, as the observed structural patterns of recidivism are an emergent property of the model. In fact, our model shows a remarkably close correspondence with California incarceration data, without requiring any parameter tuning. This work advances efforts to combine the theories and methods of epidemiology and criminology.
△ Less
Submitted 5 December, 2013;
originally announced December 2013.
-
Bayesian variable selection for spatially dependent generalized linear models
Authors:
Kristian Lum
Abstract:
Despite the abundance of methods for variable selection and accommodating spatial structure in regression models, there is little precedent for incorporating spatial dependence in covariate inclusion probabilities for regionally varying regression models. The lone existing approach is limited by difficult computation and the requirement that the spatial dependence be represented on a lattice, maki…
▽ More
Despite the abundance of methods for variable selection and accommodating spatial structure in regression models, there is little precedent for incorporating spatial dependence in covariate inclusion probabilities for regionally varying regression models. The lone existing approach is limited by difficult computation and the requirement that the spatial dependence be represented on a lattice, making this method inappropriate for areal models with irregular structures that often arise in ecology, epidemiology, and the social sciences. Here we present a novel method for spatial variable selection in areal generalized linear models that can accommodate arbitrary spatial structures and works with a broad subset of GLM likelihoods. The method uses a latent probit model with a spatial dependence structure where the binary response is taken as a covariate inclusion indicator for area-specific GLMs. The covariate inclusion indicators arise via thresholding of latent standard normals on which we place a conditionally autoregressive prior. We propose an efficient MCMC algorithm for computation that is entirely conjugate in any model with a conditionally Gaussian representation of the likelihood, thereby encompassing logistic, probit, multinomial probit and logit, Gaussian, and negative binomial regressions through the use of existing data augmentation methods. We demonstrate superior parameter recovery and prediction in simulation studies as well as in applications to geographic voting patterns and population estimation. Though the method is very broadly applicable, we note in particular that prior to this work, spatial population estimation/capture-recapture models allowing for varying list dependence structures has not been possible.
△ Less
Submitted 4 September, 2012;
originally announced September 2012.
