Rethinking harmless refusals when fine-tuning foundation models
Authors:
Florin Pop,
Judd Rosenblatt,
Diogo Schwerz de Lucena,
Michael Vaiana
Abstract:
In this paper, we investigate the degree to which fine-tuning in Large Language Models (LLMs) effectively mitigates versus merely conceals undesirable behavior. Through the lens of semi-realistic role-playing exercises designed to elicit such behaviors, we explore the response dynamics of LLMs post fine-tuning interventions. Our methodology involves prompting models for Chain-of-Thought (CoT) reas…
▽ More
In this paper, we investigate the degree to which fine-tuning in Large Language Models (LLMs) effectively mitigates versus merely conceals undesirable behavior. Through the lens of semi-realistic role-playing exercises designed to elicit such behaviors, we explore the response dynamics of LLMs post fine-tuning interventions. Our methodology involves prompting models for Chain-of-Thought (CoT) reasoning and analyzing the coherence between the reasoning traces and the resultant outputs. Notably, we identify a pervasive phenomenon we term \emph{reason-based deception}, where models either stop producing reasoning traces or produce seemingly ethical reasoning traces that belie the unethical nature of their final outputs. We further examine the efficacy of response strategies (polite refusal versus explicit rebuttal) in curbing the occurrence of undesired behavior in subsequent outputs of multi-turn interactions. Our findings reveal that explicit rebuttals significantly outperform polite refusals in preventing the continuation of undesired outputs and nearly eliminate reason-based deception, challenging current practices in model fine-tuning. Accordingly, the two key contributions of this paper are (1) defining and studying reason-based deception, a new type of hidden behavior, and (2) demonstrating that rebuttals provide a more robust response model to harmful requests than refusals, thereby highlighting the need to reconsider the response strategies in fine-tuning approaches.
△ Less
Submitted 27 June, 2024;
originally announced June 2024.
Resolution Limits for Detecting Community Changes in Multilayer Networks
Authors:
Michael Vaiana,
Sarah Muldoon
Abstract:
Multilayer networks capture pairwise relationships between the components of complex systems across multiple modes or scales of interactions. An important meso-scale feature of these networks is measured though their community structure, which defines groups of strongly connected nodes that exist within and across network layers. Because interlayer edges can describe relationships between differen…
▽ More
Multilayer networks capture pairwise relationships between the components of complex systems across multiple modes or scales of interactions. An important meso-scale feature of these networks is measured though their community structure, which defines groups of strongly connected nodes that exist within and across network layers. Because interlayer edges can describe relationships between different modalities, scales, or time points, it is essential to understand how communities change and evolve across layers. A popular method for detecting communities in multilayer networks consists of maximizing a quality function known as modularity. However, in the multilayer setting the modularity function depends on an interlayer coupling parameter, $ω$, and how this parameter affects community detection is not well understood. Here, we expose an upper bound for $ω$ beyond which community changes across layers can not be detected. This upper bound has non-trivial, purely multilayer effects and acts as a resolution limit for detecting evolving communities. Further, we establish an explicit and previously undiscovered relationship between the single layer resolution parameter, $γ$, and interlayer coupling parameter, $ω,$ that provides new understanding of the modularity parameter space. Our findings not only represent new theoretical considerations but also have important practical implications for choosing interlayer coupling values when using multilayer networks to model real-world systems whose communities change across time or modality.
△ Less
Submitted 9 March, 2018;
originally announced March 2018.
