-
Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
Authors:
Carson Denison,
Monte MacDiarmid,
Fazl Barez,
David Duvenaud,
Shauna Kravec,
Samuel Marks,
Nicholas Schiefer,
Ryan Soklaski,
Alex Tamkin,
Jared Kaplan,
Buck Shlegeris,
Samuel R. Bowman,
Ethan Perez,
Evan Hubinger
Abstract:
In reinforcement learning, specification gaming occurs when AI systems learn undesired behaviors that are highly rewarded due to misspecified training goals. Specification gaming can range from simple behaviors like sycophancy to sophisticated and pernicious behaviors like reward-tampering, where a model directly modifies its own reward mechanism. However, these more pernicious behaviors may be to…
▽ More
In reinforcement learning, specification gaming occurs when AI systems learn undesired behaviors that are highly rewarded due to misspecified training goals. Specification gaming can range from simple behaviors like sycophancy to sophisticated and pernicious behaviors like reward-tampering, where a model directly modifies its own reward mechanism. However, these more pernicious behaviors may be too complex to be discovered via exploration. In this paper, we study whether Large Language Model (LLM) assistants which find easily discovered forms of specification gaming will generalize to perform rarer and more blatant forms, up to and including reward-tampering. We construct a curriculum of increasingly sophisticated gameable environments and find that training on early-curriculum environments leads to more specification gaming on remaining environments. Strikingly, a small but non-negligible proportion of the time, LLM assistants trained on the full curriculum generalize zero-shot to directly rewriting their own reward function. Retraining an LLM not to game early-curriculum environments mitigates, but does not eliminate, reward-tampering in later environments. Moreover, adding harmlessness training to our gameable environments does not prevent reward-tampering. These results demonstrate that LLMs can generalize from common forms of specification gaming to more pernicious reward tampering and that such behavior may be nontrivial to remove.
△ Less
Submitted 28 June, 2024; v1 submitted 14 June, 2024;
originally announced June 2024.
-
Gradient-Based Language Model Red Teaming
Authors:
Nevan Wichers,
Carson Denison,
Ahmad Beirami
Abstract:
Red teaming is a common strategy for identifying weaknesses in generative language models (LMs), where adversarial prompts are produced that trigger an LM to generate unsafe responses. Red teaming is instrumental for both model alignment and evaluation, but is labor-intensive and difficult to scale when done by humans. In this paper, we present Gradient-Based Red Teaming (GBRT), a red teaming meth…
▽ More
Red teaming is a common strategy for identifying weaknesses in generative language models (LMs), where adversarial prompts are produced that trigger an LM to generate unsafe responses. Red teaming is instrumental for both model alignment and evaluation, but is labor-intensive and difficult to scale when done by humans. In this paper, we present Gradient-Based Red Teaming (GBRT), a red teaming method for automatically generating diverse prompts that are likely to cause an LM to output unsafe responses. GBRT is a form of prompt learning, trained by scoring an LM response with a safety classifier and then backpropagating through the frozen safety classifier and LM to update the prompt. To improve the coherence of input prompts, we introduce two variants that add a realism loss and fine-tune a pretrained model to generate the prompts instead of learning the prompts directly. Our experiments show that GBRT is more effective at finding prompts that trigger an LM to generate unsafe responses than a strong reinforcement learning-based red teaming approach, and succeeds even when the LM has been fine-tuned to produce safer outputs.
△ Less
Submitted 29 January, 2024;
originally announced January 2024.
-
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
Authors:
Evan Hubinger,
Carson Denison,
Jesse Mu,
Mike Lambert,
Meg Tong,
Monte MacDiarmid,
Tamera Lanham,
Daniel M. Ziegler,
Tim Maxwell,
Newton Cheng,
Adam Jermyn,
Amanda Askell,
Ansh Radhakrishnan,
Cem Anil,
David Duvenaud,
Deep Ganguli,
Fazl Barez,
Jack Clark,
Kamal Ndousse,
Kshitij Sachan,
Michael Sellitto,
Mrinank Sharma,
Nova DasSarma,
Roger Grosse,
Shauna Kravec
, et al. (14 additional authors not shown)
Abstract:
Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept exa…
▽ More
Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs). For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024. We find that such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it). The backdoor behavior is most persistent in the largest models and in models trained to produce chain-of-thought reasoning about deceiving the training process, with the persistence remaining even when the chain-of-thought is distilled away. Furthermore, rather than removing backdoors, we find that adversarial training can teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior. Our results suggest that, once a model exhibits deceptive behavior, standard techniques could fail to remove such deception and create a false impression of safety.
△ Less
Submitted 17 January, 2024; v1 submitted 10 January, 2024;
originally announced January 2024.
-
Measuring Faithfulness in Chain-of-Thought Reasoning
Authors:
Tamera Lanham,
Anna Chen,
Ansh Radhakrishnan,
Benoit Steiner,
Carson Denison,
Danny Hernandez,
Dustin Li,
Esin Durmus,
Evan Hubinger,
Jackson Kernion,
Kamilė Lukošiūtė,
Karina Nguyen,
Newton Cheng,
Nicholas Joseph,
Nicholas Schiefer,
Oliver Rausch,
Robin Larson,
Sam McCandlish,
Sandipan Kundu,
Saurav Kadavath,
Shannon Yang,
Thomas Henighan,
Timothy Maxwell,
Timothy Telleen-Lawton,
Tristan Hume
, et al. (5 additional authors not shown)
Abstract:
Large language models (LLMs) perform better when they produce step-by-step, "Chain-of-Thought" (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i.e., its process for answering the question). We investigate hypotheses for how CoT reasoning may be unfaithful, by examining how the model predictions change…
▽ More
Large language models (LLMs) perform better when they produce step-by-step, "Chain-of-Thought" (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i.e., its process for answering the question). We investigate hypotheses for how CoT reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT (e.g., by adding mistakes or paraphrasing it). Models show large variation across tasks in how strongly they condition on the CoT when predicting their answer, sometimes relying heavily on the CoT and other times primarily ignoring it. CoT's performance boost does not seem to come from CoT's added test-time compute alone or from information encoded via the particular phrasing of the CoT. As models become larger and more capable, they produce less faithful reasoning on most tasks we study. Overall, our results suggest that CoT can be faithful if the circumstances such as the model size and task are carefully chosen.
△ Less
Submitted 16 July, 2023;
originally announced July 2023.
-
Question Decomposition Improves the Faithfulness of Model-Generated Reasoning
Authors:
Ansh Radhakrishnan,
Karina Nguyen,
Anna Chen,
Carol Chen,
Carson Denison,
Danny Hernandez,
Esin Durmus,
Evan Hubinger,
Jackson Kernion,
Kamilė Lukošiūtė,
Newton Cheng,
Nicholas Joseph,
Nicholas Schiefer,
Oliver Rausch,
Sam McCandlish,
Sheer El Showk,
Tamera Lanham,
Tim Maxwell,
Venkatesa Chandrasekaran,
Zac Hatfield-Dodds,
Jared Kaplan,
Jan Brauner,
Samuel R. Bowman,
Ethan Perez
Abstract:
As large language models (LLMs) perform more difficult tasks, it becomes harder to verify the correctness and safety of their behavior. One approach to help with this issue is to prompt LLMs to externalize their reasoning, e.g., by having them generate step-by-step reasoning as they answer a question (Chain-of-Thought; CoT). The reasoning may enable us to check the process that models use to perfo…
▽ More
As large language models (LLMs) perform more difficult tasks, it becomes harder to verify the correctness and safety of their behavior. One approach to help with this issue is to prompt LLMs to externalize their reasoning, e.g., by having them generate step-by-step reasoning as they answer a question (Chain-of-Thought; CoT). The reasoning may enable us to check the process that models use to perform tasks. However, this approach relies on the stated reasoning faithfully reflecting the model's actual reasoning, which is not always the case. To improve over the faithfulness of CoT reasoning, we have models generate reasoning by decomposing questions into subquestions. Decomposition-based methods achieve strong performance on question-answering tasks, sometimes approaching that of CoT while improving the faithfulness of the model's stated reasoning on several recently-proposed metrics. By forcing the model to answer simpler subquestions in separate contexts, we greatly increase the faithfulness of model-generated reasoning over CoT, while still achieving some of the performance gains of CoT. Our results show it is possible to improve the faithfulness of model-generated reasoning; continued improvements may lead to reasoning that enables us to verify the correctness and safety of LLM behavior.
△ Less
Submitted 25 July, 2023; v1 submitted 16 July, 2023;
originally announced July 2023.
-
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy
Authors:
Natalia Ponomareva,
Hussein Hazimeh,
Alex Kurakin,
Zheng Xu,
Carson Denison,
H. Brendan McMahan,
Sergei Vassilvitskii,
Steve Chien,
Abhradeep Thakurta
Abstract:
ML models are ubiquitous in real world applications and are a constant focus of research. At the same time, the community has started to realize the importance of protecting the privacy of ML training data.
Differential Privacy (DP) has become a gold standard for making formal statements about data anonymization. However, while some adoption of DP has happened in industry, attempts to apply DP t…
▽ More
ML models are ubiquitous in real world applications and are a constant focus of research. At the same time, the community has started to realize the importance of protecting the privacy of ML training data.
Differential Privacy (DP) has become a gold standard for making formal statements about data anonymization. However, while some adoption of DP has happened in industry, attempts to apply DP to real world complex ML models are still few and far between. The adoption of DP is hindered by limited practical guidance of what DP protection entails, what privacy guarantees to aim for, and the difficulty of achieving good privacy-utility-computation trade-offs for ML models. Tricks for tuning and maximizing performance are scattered among papers or stored in the heads of practitioners. Furthermore, the literature seems to present conflicting evidence on how and whether to apply architectural adjustments and which components are "safe" to use with DP.
This work is a self-contained guide that gives an in-depth overview of the field of DP ML and presents information about achieving the best possible DP ML model with rigorous privacy guarantees. Our target audience is both researchers and practitioners. Researchers interested in DP for ML will benefit from a clear overview of current advances and areas for improvement. We include theory-focused sections that highlight important topics such as privacy accounting and its assumptions, and convergence. For a practitioner, we provide a background in DP theory and a clear step-by-step guide for choosing an appropriate privacy definition and approach, implementing DP training, potentially updating the model architecture, and tuning hyperparameters. For both researchers and practitioners, consistently and fully reporting privacy guarantees is critical, and so we propose a set of specific best practices for stating guarantees.
△ Less
Submitted 31 July, 2023; v1 submitted 1 March, 2023;
originally announced March 2023.
-
Private Ad Modeling with DP-SGD
Authors:
Carson Denison,
Badih Ghazi,
Pritish Kamath,
Ravi Kumar,
Pasin Manurangsi,
Krishna Giri Narra,
Amer Sinha,
Avinash V Varadarajan,
Chiyuan Zhang
Abstract:
A well-known algorithm in privacy-preserving ML is differentially private stochastic gradient descent (DP-SGD). While this algorithm has been evaluated on text and image data, it has not been previously applied to ads data, which are notorious for their high class imbalance and sparse gradient updates. In this work we apply DP-SGD to several ad modeling tasks including predicting click-through rat…
▽ More
A well-known algorithm in privacy-preserving ML is differentially private stochastic gradient descent (DP-SGD). While this algorithm has been evaluated on text and image data, it has not been previously applied to ads data, which are notorious for their high class imbalance and sparse gradient updates. In this work we apply DP-SGD to several ad modeling tasks including predicting click-through rates, conversion rates, and number of conversion events, and evaluate their privacy-utility trade-off on real-world datasets. Our work is the first to empirically demonstrate that DP-SGD can provide both privacy and utility for ad modeling tasks.
△ Less
Submitted 4 October, 2023; v1 submitted 21 November, 2022;
originally announced November 2022.
-
Face Coverings, Aerosol Dispersion and Mitigation of Virus Transmission Risk
Authors:
I. M. Viola,
B. Peterson,
G. Pisetta,
G. Pavar,
H. Akhtar,
F. Menoloascina,
E. Mangano,
K. E. Dunn,
R. Gabl,
A. Nila,
E. Molinari,
C. Cummins,
G. Thompson,
C. M. McDougall,
T. Y. M. Lo,
F. C. Denison,
P. Digard,
O. Malik,
M. J. G. Dunn,
F. Mehendale
Abstract:
The SARS-CoV-2 virus is primarily transmitted through virus-laden fluid particles ejected from the mouth of infected people. Face covers can mitigate the risk of virus transmission but their outward effectiveness is not fully ascertained. Objective: by using a background oriented schlieren technique, we aim to investigate the air flow ejected by a person while quietly and heavily breathing, while…
▽ More
The SARS-CoV-2 virus is primarily transmitted through virus-laden fluid particles ejected from the mouth of infected people. Face covers can mitigate the risk of virus transmission but their outward effectiveness is not fully ascertained. Objective: by using a background oriented schlieren technique, we aim to investigate the air flow ejected by a person while quietly and heavily breathing, while coughing, and with different face covers. Results: we found that all face covers without an outlet valve reduce the front flow through by at least 63% and perhaps as high as 86% if the unfiltered cough jet distance was resolved to the anticipated maximum distance of 2-3 m. However, surgical and handmade masks, and face shields, generate significant leakage jets that may present major hazards. Conclusions: the effectiveness of the masks should mostly be considered based on the generation of secondary jets rather than on the ability to mitigate the front throughflow.
△ Less
Submitted 30 January, 2021; v1 submitted 19 May, 2020;
originally announced May 2020.
