Defending Against Indirect Prompt Injection Attacks With Spotlighting
Authors:
Keegan Hines,
Gary Lopez,
Matthew Hall,
Federico Zarfati,
Yonatan Zunger,
Emre Kiciman
Abstract:
Large Language Models (LLMs), while powerful, are built and trained to process a single text input. In common applications, multiple inputs can be processed by concatenating them together into a single stream of text. However, the LLM is unable to distinguish which sections of prompt belong to various input sources. Indirect prompt injection attacks take advantage of this vulnerability by embeddin…
▽ More
Large Language Models (LLMs), while powerful, are built and trained to process a single text input. In common applications, multiple inputs can be processed by concatenating them together into a single stream of text. However, the LLM is unable to distinguish which sections of prompt belong to various input sources. Indirect prompt injection attacks take advantage of this vulnerability by embedding adversarial instructions into untrusted data being processed alongside user commands. Often, the LLM will mistake the adversarial instructions as user commands to be followed, creating a security vulnerability in the larger system. We introduce spotlighting, a family of prompt engineering techniques that can be used to improve LLMs' ability to distinguish among multiple sources of input. The key insight is to utilize transformations of an input to provide a reliable and continuous signal of its provenance. We evaluate spotlighting as a defense against indirect prompt injection attacks, and find that it is a robust defense that has minimal detrimental impact to underlying NLP tasks. Using GPT-family models, we find that spotlighting reduces the attack success rate from greater than {50}\% to below {2}\% in our experiments with minimal impact on task efficacy.
△ Less
Submitted 20 March, 2024;
originally announced March 2024.
A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
Authors:
Ahmed Magooda,
Alec Helyar,
Kyle Jackson,
David Sullivan,
Chad Atalla,
Emily Sheng,
Dan Vann,
Richard Edgar,
Hamid Palangi,
Roman Lutz,
Hongliang Kong,
Vincent Yun,
Eslam Kamal,
Federico Zarfati,
Hanna Wallach,
Sarah Bird,
Mei Chen
Abstract:
We present a framework for the automated measurement of responsible AI (RAI) metrics for large language models (LLMs) and associated products and services. Our framework for automatically measuring harms from LLMs builds on existing technical and sociotechnical expertise and leverages the capabilities of state-of-the-art LLMs, such as GPT-4. We use this framework to run through several case studie…
▽ More
We present a framework for the automated measurement of responsible AI (RAI) metrics for large language models (LLMs) and associated products and services. Our framework for automatically measuring harms from LLMs builds on existing technical and sociotechnical expertise and leverages the capabilities of state-of-the-art LLMs, such as GPT-4. We use this framework to run through several case studies investigating how different LLMs may violate a range of RAI-related principles. The framework may be employed alongside domain-specific sociotechnical expertise to create measurements for new harm areas in the future. By implementing this framework, we aim to enable more advanced harm measurement efforts and further the responsible use of LLMs.
△ Less
Submitted 26 October, 2023;
originally announced October 2023.
