-
Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives
Authors:
Elizabeth Seger,
Noemi Dreksler,
Richard Moulange,
Emily Dardaman,
Jonas Schuett,
K. Wei,
Christoph Winter,
Mackenzie Arnold,
Seán Ó hÉigeartaigh,
Anton Korinek,
Markus Anderljung,
Ben Bucknall,
Alan Chan,
Eoghan Stafford,
Leonie Koessler,
Aviv Ovadya,
Ben Garfinkel,
Emma Bluemke,
Michael Aird,
Patrick Levermore,
Julian Hazell,
Abhishek Gupta
Abstract:
Recent decisions by leading AI labs to either open-source their models or to restrict access to their models has sparked debate about whether, and how, increasingly capable AI models should be shared. Open-sourcing in AI typically refers to making model architecture and weights freely and publicly accessible for anyone to modify, study, build on, and use. This offers advantages such as enabling ex…
▽ More
Recent decisions by leading AI labs to either open-source their models or to restrict access to their models has sparked debate about whether, and how, increasingly capable AI models should be shared. Open-sourcing in AI typically refers to making model architecture and weights freely and publicly accessible for anyone to modify, study, build on, and use. This offers advantages such as enabling external oversight, accelerating progress, and decentralizing control over AI development and use. However, it also presents a growing potential for misuse and unintended consequences. This paper offers an examination of the risks and benefits of open-sourcing highly capable foundation models. While open-sourcing has historically provided substantial net benefits for most software and AI development processes, we argue that for some highly capable foundation models likely to be developed in the near future, open-sourcing may pose sufficiently extreme risks to outweigh the benefits. In such a case, highly capable foundation models should not be open-sourced, at least not initially. Alternative strategies, including non-open-source model sharing options, are explored. The paper concludes with recommendations for developers, standard-setting bodies, and governments for establishing safe and responsible model sharing practices and preserving open-source benefits where safe.
△ Less
Submitted 29 September, 2023;
originally announced November 2023.
-
Model evaluation for extreme risks
Authors:
Toby Shevlane,
Sebastian Farquhar,
Ben Garfinkel,
Mary Phuong,
Jess Whittlestone,
Jade Leung,
Daniel Kokotajlo,
Nahema Marchal,
Markus Anderljung,
Noam Kolt,
Lewis Ho,
Divya Siddarth,
Shahar Avin,
Will Hawkins,
Been Kim,
Iason Gabriel,
Vijay Bolina,
Jack Clark,
Yoshua Bengio,
Paul Christiano,
Allan Dafoe
Abstract:
Current approaches to building general-purpose AI systems tend to produce systems with both beneficial and harmful capabilities. Further progress in AI development could lead to capabilities that pose extreme risks, such as offensive cyber capabilities or strong manipulation skills. We explain why model evaluation is critical for addressing extreme risks. Developers must be able to identify danger…
▽ More
Current approaches to building general-purpose AI systems tend to produce systems with both beneficial and harmful capabilities. Further progress in AI development could lead to capabilities that pose extreme risks, such as offensive cyber capabilities or strong manipulation skills. We explain why model evaluation is critical for addressing extreme risks. Developers must be able to identify dangerous capabilities (through "dangerous capability evaluations") and the propensity of models to apply their capabilities for harm (through "alignment evaluations"). These evaluations will become critical for kee** policymakers and other stakeholders informed, and for making responsible decisions about model training, deployment, and security.
△ Less
Submitted 22 September, 2023; v1 submitted 24 May, 2023;
originally announced May 2023.
-
Towards best practices in AGI safety and governance: A survey of expert opinion
Authors:
Jonas Schuett,
Noemi Dreksler,
Markus Anderljung,
David McCaffary,
Lennart Heim,
Emma Bluemke,
Ben Garfinkel
Abstract:
A number of leading AI companies, including OpenAI, Google DeepMind, and Anthropic, have the stated goal of building artificial general intelligence (AGI) - AI systems that achieve or exceed human performance across a wide range of cognitive tasks. In pursuing this goal, they may develop and deploy AI systems that pose particularly significant risks. While they have already taken some measures to…
▽ More
A number of leading AI companies, including OpenAI, Google DeepMind, and Anthropic, have the stated goal of building artificial general intelligence (AGI) - AI systems that achieve or exceed human performance across a wide range of cognitive tasks. In pursuing this goal, they may develop and deploy AI systems that pose particularly significant risks. While they have already taken some measures to mitigate these risks, best practices have not yet emerged. To support the identification of best practices, we sent a survey to 92 leading experts from AGI labs, academia, and civil society and received 51 responses. Participants were asked how much they agreed with 50 statements about what AGI labs should do. Our main finding is that participants, on average, agreed with all of them. Many statements received extremely high levels of agreement. For example, 98% of respondents somewhat or strongly agreed that AGI labs should conduct pre-deployment risk assessments, dangerous capabilities evaluations, third-party model audits, safety restrictions on model usage, and red teaming. Ultimately, our list of statements may serve as a helpful foundation for efforts to develop best practices, standards, and regulations for AGI labs.
△ Less
Submitted 11 May, 2023;
originally announced May 2023.
-
Democratising AI: Multiple Meanings, Goals, and Methods
Authors:
Elizabeth Seger,
Aviv Ovadya,
Ben Garfinkel,
Divya Siddarth,
Allan Dafoe
Abstract:
Numerous parties are calling for the democratisation of AI, but the phrase is used to refer to a variety of goals, the pursuit of which sometimes conflict. This paper identifies four kinds of AI democratisation that are commonly discussed: (1) the democratisation of AI use, (2) the democratisation of AI development, (3) the democratisation of AI profits, and (4) the democratisation of AI governanc…
▽ More
Numerous parties are calling for the democratisation of AI, but the phrase is used to refer to a variety of goals, the pursuit of which sometimes conflict. This paper identifies four kinds of AI democratisation that are commonly discussed: (1) the democratisation of AI use, (2) the democratisation of AI development, (3) the democratisation of AI profits, and (4) the democratisation of AI governance. Numerous goals and methods of achieving each form of democratisation are discussed. The main takeaway from this paper is that AI democratisation is a multifarious and sometimes conflicting concept that should not be conflated with improving AI accessibility. If we want to move beyond ambiguous commitments to democratising AI, to productive discussions of concrete policies and trade-offs, then we need to recognise the principal role of the democratisation of AI governance in navigating tradeoffs and risks across decisions around use, development, and profits.
△ Less
Submitted 7 August, 2023; v1 submitted 22 March, 2023;
originally announced March 2023.
-
Exploring the Relevance of Data Privacy-Enhancing Technologies for AI Governance Use Cases
Authors:
Emma Bluemke,
Tantum Collins,
Ben Garfinkel,
Andrew Trask
Abstract:
The development of privacy-enhancing technologies has made immense progress in reducing trade-offs between privacy and performance in data exchange and analysis. Similar tools for structured transparency could be useful for AI governance by offering capabilities such as external scrutiny, auditing, and source verification. It is useful to view these different AI governance objectives as a system o…
▽ More
The development of privacy-enhancing technologies has made immense progress in reducing trade-offs between privacy and performance in data exchange and analysis. Similar tools for structured transparency could be useful for AI governance by offering capabilities such as external scrutiny, auditing, and source verification. It is useful to view these different AI governance objectives as a system of information flows in order to avoid partial solutions and significant gaps in governance, as there may be significant overlap in the software stacks needed for the AI governance use cases mentioned in this text. When viewing the system as a whole, the importance of interoperability between these different AI governance solutions becomes clear. Therefore, it is imminently important to look at these problems in AI governance as a system, before these standards, auditing procedures, software, and norms settle into place.
△ Less
Submitted 20 March, 2023; v1 submitted 15 March, 2023;
originally announced March 2023.
-
The Windfall Clause: Distributing the Benefits of AI for the Common Good
Authors:
Cullen O'Keefe,
Peter Cihon,
Ben Garfinkel,
Carrick Flynn,
Jade Leung,
Allan Dafoe
Abstract:
As the transformative potential of AI has become increasingly salient as a matter of public and political interest, there has been growing discussion about the need to ensure that AI broadly benefits humanity. This in turn has spurred debate on the social responsibilities of large technology companies to serve the interests of society at large. In response, ethical principles and codes of conduct…
▽ More
As the transformative potential of AI has become increasingly salient as a matter of public and political interest, there has been growing discussion about the need to ensure that AI broadly benefits humanity. This in turn has spurred debate on the social responsibilities of large technology companies to serve the interests of society at large. In response, ethical principles and codes of conduct have been proposed to meet the escalating demand for this responsibility to be taken seriously. As yet, however, few institutional innovations have been suggested to translate this responsibility into legal commitments which apply to companies positioned to reap large financial gains from the development and use of AI. This paper offers one potentially attractive tool for addressing such issues: the Windfall Clause, which is an ex ante commitment by AI firms to donate a significant amount of any eventual extremely large profits. By this we mean an early commitment that profits that a firm could not earn without achieving fundamental, economically transformative breakthroughs in AI capabilities will be donated to benefit humanity broadly, with particular attention towards mitigating any downsides from deployment of windfall-generating AI.
△ Less
Submitted 24 January, 2020; v1 submitted 25 December, 2019;
originally announced December 2019.
-
The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation
Authors:
Miles Brundage,
Shahar Avin,
Jack Clark,
Helen Toner,
Peter Eckersley,
Ben Garfinkel,
Allan Dafoe,
Paul Scharre,
Thomas Zeitzoff,
Bobby Filar,
Hyrum Anderson,
Heather Roff,
Gregory C. Allen,
Jacob Steinhardt,
Carrick Flynn,
Seán Ó hÉigeartaigh,
Simon Beard,
Haydn Belfield,
Sebastian Farquhar,
Clare Lyle,
Rebecca Crootof,
Owain Evans,
Michael Page,
Joanna Bryson,
Roman Yampolskiy
, et al. (1 additional authors not shown)
Abstract:
This report surveys the landscape of potential security threats from malicious uses of AI, and proposes ways to better forecast, prevent, and mitigate these threats. After analyzing the ways in which AI may influence the threat landscape in the digital, physical, and political domains, we make four high-level recommendations for AI researchers and other stakeholders. We also suggest several promis…
▽ More
This report surveys the landscape of potential security threats from malicious uses of AI, and proposes ways to better forecast, prevent, and mitigate these threats. After analyzing the ways in which AI may influence the threat landscape in the digital, physical, and political domains, we make four high-level recommendations for AI researchers and other stakeholders. We also suggest several promising areas for further research that could expand the portfolio of defenses, or make attacks less effective or harder to execute. Finally, we discuss, but do not conclusively resolve, the long-term equilibrium of attackers and defenders.
△ Less
Submitted 20 February, 2018;
originally announced February 2018.
-
On the Impossibility of Supersized Machines
Authors:
Ben Garfinkel,
Miles Brundage,
Daniel Filan,
Carrick Flynn,
Jelena Luketina,
Michael Page,
Anders Sandberg,
Andrew Snyder-Beattie,
Max Tegmark
Abstract:
In recent years, a number of prominent computer scientists, along with academics in fields such as philosophy and physics, have lent credence to the notion that machines may one day become as large as humans. Many have further argued that machines could even come to exceed human size by a significant margin. However, there are at least seven distinct arguments that preclude this outcome. We show t…
▽ More
In recent years, a number of prominent computer scientists, along with academics in fields such as philosophy and physics, have lent credence to the notion that machines may one day become as large as humans. Many have further argued that machines could even come to exceed human size by a significant margin. However, there are at least seven distinct arguments that preclude this outcome. We show that it is not only implausible that machines will ever exceed human size, but in fact impossible.
△ Less
Submitted 31 March, 2017;
originally announced March 2017.
