-
FairTargetSim: An Interactive Simulator for Understanding and Explaining the Fairness Effects of Target Variable Definition
Authors:
Dalia Gala,
Milo Phillips-Brown,
Naman Goel,
Carinal Prunkl,
Laura Alvarez Jubete,
medb corcoran,
Ray Eitel-Porter
Abstract:
Machine learning requires defining one's target variable for predictions or decisions, a process that can have profound implications on fairness: biases are often encoded in target variable definition itself, before any data collection or training. We present an interactive simulator, FairTargetSim (FTS), that illustrates how target variable definition impacts fairness. FTS is a valuable tool for…
▽ More
Machine learning requires defining one's target variable for predictions or decisions, a process that can have profound implications on fairness: biases are often encoded in target variable definition itself, before any data collection or training. We present an interactive simulator, FairTargetSim (FTS), that illustrates how target variable definition impacts fairness. FTS is a valuable tool for algorithm developers, researchers, and non-technical stakeholders. FTS uses a case study of algorithmic hiring, using real-world data and user-defined target variables. FTS is open-source and available at: http://tinyurl.com/ftsinterface. The video accompanying this paper is here: http://tinyurl.com/ijcaifts.
△ Less
Submitted 9 March, 2024;
originally announced March 2024.
-
LUCID-GAN: Conditional Generative Models to Locate Unfairness
Authors:
Andres Algaba,
Carmen Mazijn,
Carina Prunkl,
Jan Danckaert,
Vincent Ginis
Abstract:
Most group fairness notions detect unethical biases by computing statistical parity metrics on a model's output. However, this approach suffers from several shortcomings, such as philosophical disagreement, mutual incompatibility, and lack of interpretability. These shortcomings have spurred the research on complementary bias detection methods that offer additional transparency into the sources of…
▽ More
Most group fairness notions detect unethical biases by computing statistical parity metrics on a model's output. However, this approach suffers from several shortcomings, such as philosophical disagreement, mutual incompatibility, and lack of interpretability. These shortcomings have spurred the research on complementary bias detection methods that offer additional transparency into the sources of discrimination and are agnostic towards an a priori decision on the definition of fairness and choice of protected features. A recent proposal in this direction is LUCID (Locating Unfairness through Canonical Inverse Design), where canonical sets are generated by performing gradient descent on the input space, revealing a model's desired input given a preferred output. This information about the model's mechanisms, i.e., which feature values are essential to obtain specific outputs, allows exposing potential unethical biases in its internal logic. Here, we present LUCID-GAN, which generates canonical inputs via a conditional generative model instead of gradient-based inverse design. LUCID-GAN has several benefits, including that it applies to non-differentiable models, ensures that canonical sets consist of realistic inputs, and allows to assess proxy and intersectional discrimination. We empirically evaluate LUCID-GAN on the UCI Adult and COMPAS data sets and show that it allows for detecting unethical biases in black-box models without requiring access to the training data.
△ Less
Submitted 28 July, 2023;
originally announced July 2023.
-
LUCID: Exposing Algorithmic Bias through Inverse Design
Authors:
Carmen Mazijn,
Carina Prunkl,
Andres Algaba,
Jan Danckaert,
Vincent Ginis
Abstract:
AI systems can create, propagate, support, and automate bias in decision-making processes. To mitigate biased decisions, we both need to understand the origin of the bias and define what it means for an algorithm to make fair decisions. Most group fairness notions assess a model's equality of outcome by computing statistical metrics on the outputs. We argue that these output metrics encounter intr…
▽ More
AI systems can create, propagate, support, and automate bias in decision-making processes. To mitigate biased decisions, we both need to understand the origin of the bias and define what it means for an algorithm to make fair decisions. Most group fairness notions assess a model's equality of outcome by computing statistical metrics on the outputs. We argue that these output metrics encounter intrinsic obstacles and present a complementary approach that aligns with the increasing focus on equality of treatment. By Locating Unfairness through Canonical Inverse Design (LUCID), we generate a canonical set that shows the desired inputs for a model given a preferred output. The canonical set reveals the model's internal logic and exposes potential unethical biases by repeatedly interrogating the decision-making process. We evaluate LUCID on the UCI Adult and COMPAS data sets and find that some biases detected by a canonical set differ from those of output metrics. The results show that by shifting the focus towards equality of treatment and looking into the algorithm's internal workings, the canonical sets are a valuable addition to the toolbox of algorithmic fairness evaluation.
△ Less
Submitted 26 August, 2022;
originally announced August 2022.
-
Simulation Intelligence: Towards a New Generation of Scientific Methods
Authors:
Alexander Lavin,
David Krakauer,
Hector Zenil,
Justin Gottschlich,
Tim Mattson,
Johann Brehmer,
Anima Anandkumar,
Sanjay Choudry,
Kamil Rocki,
Atılım Güneş Baydin,
Carina Prunkl,
Brooks Paige,
Olexandr Isayev,
Erik Peterson,
Peter L. McMahon,
Jakob Macke,
Kyle Cranmer,
Jiaxin Zhang,
Haruko Wainwright,
Adi Hanuka,
Manuela Veloso,
Samuel Assefa,
Stephan Zheng,
Avi Pfeffer
Abstract:
The original "Seven Motifs" set forth a roadmap of essential methods for the field of scientific computing, where a motif is an algorithmic method that captures a pattern of computation and data movement. We present the "Nine Motifs of Simulation Intelligence", a roadmap for the development and integration of the essential algorithms necessary for a merger of scientific computing, scientific simul…
▽ More
The original "Seven Motifs" set forth a roadmap of essential methods for the field of scientific computing, where a motif is an algorithmic method that captures a pattern of computation and data movement. We present the "Nine Motifs of Simulation Intelligence", a roadmap for the development and integration of the essential algorithms necessary for a merger of scientific computing, scientific simulation, and artificial intelligence. We call this merger simulation intelligence (SI), for short. We argue the motifs of simulation intelligence are interconnected and interdependent, much like the components within the layers of an operating system. Using this metaphor, we explore the nature of each layer of the simulation intelligence operating system stack (SI-stack) and the motifs therein: (1) Multi-physics and multi-scale modeling; (2) Surrogate modeling and emulation; (3) Simulation-based inference; (4) Causal modeling and inference; (5) Agent-based modeling; (6) Probabilistic programming; (7) Differentiable programming; (8) Open-ended optimization; (9) Machine programming. We believe coordinated efforts between motifs offers immense opportunity to accelerate scientific discovery, from solving inverse problems in synthetic biology and climate science, to directing nuclear energy experiments and predicting emergent behavior in socioeconomic settings. We elaborate on each layer of the SI-stack, detailing the state-of-art methods, presenting examples to highlight challenges and opportunities, and advocating for specific ways to advance the motifs and the synergies from their combinations. Advancing and integrating these technologies can enable a robust and efficient hypothesis-simulation-analysis type of scientific method, which we introduce with several use-cases for human-machine teaming and automated science.
△ Less
Submitted 27 November, 2022; v1 submitted 6 December, 2021;
originally announced December 2021.
-
Institutionalising Ethics in AI through Broader Impact Requirements
Authors:
Carina Prunkl,
Carolyn Ashurst,
Markus Anderljung,
Helena Webb,
Jan Leike,
Allan Dafoe
Abstract:
Turning principles into practice is one of the most pressing challenges of artificial intelligence (AI) governance. In this article, we reflect on a novel governance initiative by one of the world's largest AI conferences. In 2020, the Conference on Neural Information Processing Systems (NeurIPS) introduced a requirement for submitting authors to include a statement on the broader societal impacts…
▽ More
Turning principles into practice is one of the most pressing challenges of artificial intelligence (AI) governance. In this article, we reflect on a novel governance initiative by one of the world's largest AI conferences. In 2020, the Conference on Neural Information Processing Systems (NeurIPS) introduced a requirement for submitting authors to include a statement on the broader societal impacts of their research. Drawing insights from similar governance initiatives, including institutional review boards (IRBs) and impact requirements for funding applications, we investigate the risks, challenges and potential benefits of such an initiative. Among the challenges, we list a lack of recognised best practice and procedural transparency, researcher opportunity costs, institutional and social pressures, cognitive biases, and the inherently difficult nature of the task. The potential benefits, on the other hand, include improved anticipation and identification of impacts, better communication with policy and governance experts, and a general strengthening of the norms around responsible research. To maximise the chance of success, we recommend measures to increase transparency, improve guidance, create incentives to engage earnestly with the process, and facilitate public deliberation on the requirement's merits and future. Perhaps the most important contribution from this analysis are the insights we can gain regarding effective community-based governance and the role and responsibility of the AI research community more broadly.
△ Less
Submitted 30 May, 2021;
originally announced June 2021.
-
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Authors:
Miles Brundage,
Shahar Avin,
Jasmine Wang,
Haydn Belfield,
Gretchen Krueger,
Gillian Hadfield,
Heidy Khlaaf,
**gying Yang,
Helen Toner,
Ruth Fong,
Tegan Maharaj,
Pang Wei Koh,
Sara Hooker,
Jade Leung,
Andrew Trask,
Emma Bluemke,
Jonathan Lebensold,
Cullen O'Keefe,
Mark Koren,
Théo Ryffel,
JB Rubinovitz,
Tamay Besiroglu,
Federica Carugati,
Jack Clark,
Peter Eckersley
, et al. (34 additional authors not shown)
Abstract:
With the recent wave of progress in artificial intelligence (AI) has come a growing awareness of the large-scale impacts of AI systems, and recognition that existing regulations and norms in industry and academia are insufficient to ensure responsible AI development. In order for AI developers to earn trust from system users, customers, civil society, governments, and other stakeholders that they…
▽ More
With the recent wave of progress in artificial intelligence (AI) has come a growing awareness of the large-scale impacts of AI systems, and recognition that existing regulations and norms in industry and academia are insufficient to ensure responsible AI development. In order for AI developers to earn trust from system users, customers, civil society, governments, and other stakeholders that they are building AI responsibly, they will need to make verifiable claims to which they can be held accountable. Those outside of a given organization also need effective means of scrutinizing such claims. This report suggests various steps that different stakeholders can take to improve the verifiability of claims made about AI systems and their associated development processes, with a focus on providing evidence about the safety, security, fairness, and privacy protection of AI systems. We analyze ten mechanisms for this purpose--spanning institutions, software, and hardware--and make recommendations aimed at implementing, exploring, or improving those mechanisms.
△ Less
Submitted 20 April, 2020; v1 submitted 15 April, 2020;
originally announced April 2020.
-
Beyond Near- and Long-Term: Towards a Clearer Account of Research Priorities in AI Ethics and Society
Authors:
Carina Prunkl,
Jess Whittlestone
Abstract:
One way of carving up the broad "AI ethics and society" research space that has emerged in recent years is to distinguish between "near-term" and "long-term" research. While such ways of breaking down the research space can be useful, we put forward several concerns about the near/long-term distinction gaining too much prominence in how research questions and priorities are framed. We highlight so…
▽ More
One way of carving up the broad "AI ethics and society" research space that has emerged in recent years is to distinguish between "near-term" and "long-term" research. While such ways of breaking down the research space can be useful, we put forward several concerns about the near/long-term distinction gaining too much prominence in how research questions and priorities are framed. We highlight some ambiguities and inconsistencies in how the distinction is used, and argue that while there are differing priorities within this broad research community, these differences are not well-captured by the near/long-term distinction. We unpack the near/long-term distinction into four different dimensions, and propose some ways that researchers can communicate more clearly about their work and priorities using these dimensions. We suggest that moving towards a more nuanced conversation about research priorities can help establish new opportunities for collaboration, aid the development of more consistent and coherent research agendas, and enable identification of previously neglected research areas.
△ Less
Submitted 21 January, 2020; v1 submitted 13 January, 2020;
originally announced January 2020.
-
Black Hole Entropy is Thermodynamic Entropy
Authors:
Carina E. A. Prunkl,
Christopher G. Timpson
Abstract:
The comparison of geometrical properties of black holes with classical thermodynamic variables reveals surprising parallels between the laws of black hole mechanics and the laws of thermodynamics. Since Hawking's discovery that black holes when coupled to quantum matter fields emit radiation at a temperature proportional to their surface gravity, the idea that black holes are genuine thermodynamic…
▽ More
The comparison of geometrical properties of black holes with classical thermodynamic variables reveals surprising parallels between the laws of black hole mechanics and the laws of thermodynamics. Since Hawking's discovery that black holes when coupled to quantum matter fields emit radiation at a temperature proportional to their surface gravity, the idea that black holes are genuine thermodynamic objects with a well-defined thermodynamic entropy has become more and more popular. Surprisingly, arguments that justify this assumption are both sparse and rarely convincing. Most of them rely on an information-theoretic interpretation of entropy, which in itself is a highly debated topic in the philosophy of physics. We discuss some of the pertinent arguments that aim at establishing the identity of black hole surface area (times a constant) and thermodynamic entropy and show why these arguments are not satisfactory. We then present a simple model of a Black Hole Carnot cycle to establish that black hole entropy is genuine thermodynamic entropy which does not require an information-theoretic interpretation.
△ Less
Submitted 14 March, 2019;
originally announced March 2019.
-
On the Equivalence of von Neumann and Thermodynamic Entropy
Authors:
Carina E. A. Prunkl
Abstract:
In 1939, von Neumann argued for the equivalence of the thermodynamic entropy and $-\text{Tr}ρ\lnρ$, since known as the von Neumann entropy. Hemmo and Shenker (2006) recently challenged this argument by pointing out an alleged discrepancy between the two entropies in the single particle case, concluding that they must be distinct. In this article, their argument is shown to be problematic as it a)…
▽ More
In 1939, von Neumann argued for the equivalence of the thermodynamic entropy and $-\text{Tr}ρ\lnρ$, since known as the von Neumann entropy. Hemmo and Shenker (2006) recently challenged this argument by pointing out an alleged discrepancy between the two entropies in the single particle case, concluding that they must be distinct. In this article, their argument is shown to be problematic as it a) allows for a violation of the second law of thermodynamics and b) is based on an incorrect calculation of the von Neumann entropy.
△ Less
Submitted 8 May, 2019; v1 submitted 22 October, 2018;
originally announced October 2018.
-
On the thermodynamical cost of some interpretations of quantum theory
Authors:
Carina E. A. Prunkl,
Christopher G. Timpson
Abstract:
Cabello et al. claim to have proven the existence of an empirically verifiable difference between two broad classes of quantum interpretations. On the basis of three seemingly uncontentious assumptions, (i) the possibility of randomly selected measurements, (ii) the finiteness of a quantum system's memory, and (iii) the validity of Landauer's principle, and further, by applying computational mecha…
▽ More
Cabello et al. claim to have proven the existence of an empirically verifiable difference between two broad classes of quantum interpretations. On the basis of three seemingly uncontentious assumptions, (i) the possibility of randomly selected measurements, (ii) the finiteness of a quantum system's memory, and (iii) the validity of Landauer's principle, and further, by applying computational mechanics to quantum processes, the authors arrive at the conclusion that some quantum interpretations (including central realist interpretations) are associated with an excess heat cost and are thereby untenable - or at least - that they can be distinguished empirically from their competitors by measuring the heat produced. Here, we provide an explicit counterexample to this claim and demonstrate that their surprising result can be traced back to a lack of distinction between system and external agent. By drawing the distinction carefully, we show that the resulting heat cost is fully accounted for in the external agent, thereby restoring the tenability of the quantum interpretations in question.
△ Less
Submitted 4 June, 2017;
originally announced June 2017.