-
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
Authors:
Deep Ganguli,
Liane Lovitt,
Jackson Kernion,
Amanda Askell,
Yuntao Bai,
Saurav Kadavath,
Ben Mann,
Ethan Perez,
Nicholas Schiefer,
Kamal Ndousse,
Andy Jones,
Sam Bowman,
Anna Chen,
Tom Conerly,
Nova DasSarma,
Dawn Drain,
Nelson Elhage,
Sheer El-Showk,
Stanislav Fort,
Zac Hatfield-Dodds,
Tom Henighan,
Danny Hernandez,
Tristan Hume,
Josh Jacobson,
Scott Johnston
, et al. (11 additional authors not shown)
Abstract:
We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmle…
▽ More
We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF). We find that the RLHF models are increasingly difficult to red team as they scale, and we find a flat trend with scale for the other model types. Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from. We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs. Third, we exhaustively describe our instructions, processes, statistical methodologies, and uncertainty about red teaming. We hope that this transparency accelerates our ability to work together as a community in order to develop shared norms, practices, and technical standards for how to red team language models.
△ Less
Submitted 22 November, 2022; v1 submitted 23 August, 2022;
originally announced September 2022.
-
Language Models (Mostly) Know What They Know
Authors:
Saurav Kadavath,
Tom Conerly,
Amanda Askell,
Tom Henighan,
Dawn Drain,
Ethan Perez,
Nicholas Schiefer,
Zac Hatfield-Dodds,
Nova DasSarma,
Eli Tran-Johnson,
Scott Johnston,
Sheer El-Showk,
Andy Jones,
Nelson Elhage,
Tristan Hume,
Anna Chen,
Yuntao Bai,
Sam Bowman,
Stanislav Fort,
Deep Ganguli,
Danny Hernandez,
Josh Jacobson,
Jackson Kernion,
Shauna Kravec,
Liane Lovitt
, et al. (11 additional authors not shown)
Abstract:
We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Thus we can approach self-evaluation on open-ended sampling tasks by asking models to first propose answe…
▽ More
We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Thus we can approach self-evaluation on open-ended sampling tasks by asking models to first propose answers, and then to evaluate the probability "P(True)" that their answers are correct. We find encouraging performance, calibration, and scaling for P(True) on a diverse array of tasks. Performance at self-evaluation further improves when we allow models to consider many of their own samples before predicting the validity of one specific possibility. Next, we investigate whether models can be trained to predict "P(IK)", the probability that "I know" the answer to a question, without reference to any particular proposed answer. Models perform well at predicting P(IK) and partially generalize across tasks, though they struggle with calibration of P(IK) on new tasks. The predicted P(IK) probabilities also increase appropriately in the presence of relevant source materials in the context, and in the presence of hints towards the solution of mathematical word problems. We hope these observations lay the groundwork for training more honest models, and for investigating how honesty generalizes to cases where models are trained on objectives other than the imitation of human writing.
△ Less
Submitted 21 November, 2022; v1 submitted 11 July, 2022;
originally announced July 2022.
-
Scaling Laws and Interpretability of Learning from Repeated Data
Authors:
Danny Hernandez,
Tom Brown,
Tom Conerly,
Nova DasSarma,
Dawn Drain,
Sheer El-Showk,
Nelson Elhage,
Zac Hatfield-Dodds,
Tom Henighan,
Tristan Hume,
Scott Johnston,
Ben Mann,
Chris Olah,
Catherine Olsson,
Dario Amodei,
Nicholas Joseph,
Jared Kaplan,
Sam McCandlish
Abstract:
Recent large language models have been trained on vast datasets, but also often on repeated data, either intentionally for the purpose of upweighting higher quality data, or unintentionally because data deduplication is not perfect and the model is exposed to repeated data at the sentence, paragraph, or document level. Some works have reported substantial negative performance effects of this repea…
▽ More
Recent large language models have been trained on vast datasets, but also often on repeated data, either intentionally for the purpose of upweighting higher quality data, or unintentionally because data deduplication is not perfect and the model is exposed to repeated data at the sentence, paragraph, or document level. Some works have reported substantial negative performance effects of this repeated data. In this paper we attempt to study repeated data systematically and to understand its effects mechanistically. To do this, we train a family of models where most of the data is unique but a small fraction of it is repeated many times. We find a strong double descent phenomenon, in which repeated data can lead test loss to increase midway through training. A predictable range of repetition frequency leads to surprisingly severe degradation in performance. For instance, performance of an 800M parameter model can be degraded to that of a 2x smaller model (400M params) by repeating 0.1% of the data 100 times, despite the other 90% of the training tokens remaining unique. We suspect there is a range in the middle where the data can be memorized and doing so consumes a large fraction of the model's capacity, and this may be where the peak of degradation occurs. Finally, we connect these observations to recent mechanistic interpretability work - attempting to reverse engineer the detailed computations performed by the model - by showing that data repetition disproportionately damages copying and internal structures associated with generalization, such as induction heads, providing a possible mechanism for the shift from generalization to memorization. Taken together, these results provide a hypothesis for why repeating a relatively small fraction of data in large language models could lead to disproportionately large harms to performance.
△ Less
Submitted 20 May, 2022;
originally announced May 2022.
-
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Authors:
Yuntao Bai,
Andy Jones,
Kamal Ndousse,
Amanda Askell,
Anna Chen,
Nova DasSarma,
Dawn Drain,
Stanislav Fort,
Deep Ganguli,
Tom Henighan,
Nicholas Joseph,
Saurav Kadavath,
Jackson Kernion,
Tom Conerly,
Sheer El-Showk,
Nelson Elhage,
Zac Hatfield-Dodds,
Danny Hernandez,
Tristan Hume,
Scott Johnston,
Shauna Kravec,
Liane Lovitt,
Neel Nanda,
Catherine Olsson,
Dario Amodei
, et al. (6 additional authors not shown)
Abstract:
We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where prefer…
▽ More
We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work.
△ Less
Submitted 12 April, 2022;
originally announced April 2022.
