: Visual Analysis of Jailbreak Attacks Against
Large Language Models

Abstract

The proliferation of large language models (LLMs) has underscored concerns regarding their security vulnerabilities, notably against jailbreak attacks, where adversaries design jailbreak prompts to circumvent safety mechanisms for potential misuse. Addressing these concerns necessitates a comprehensive analysis of jailbreak prompts to evaluate LLMs’ defensive capabilities and identify potential weaknesses. However, the complexity of evaluating jailbreak performance and understanding prompt characteristics makes this analysis laborious. We collaborate with domain experts to characterize problems and propose an LLM-assisted framework to streamline the analysis process. It provides automatic jailbreak assessment to facilitate performance evaluation and support analysis of components and keywords in prompts. Based on the framework, we design , a visual analysis system that enables users to explore the jailbreak performance against the target model, conduct multi-level analysis of prompt characteristics, and refine prompt instances to verify findings. Through a case study, technical evaluations, and expert interviews, we demonstrate our system’s effectiveness in hel** users evaluate model security and identify model weaknesses.

streamlines the process of jailbreak performance evaluation and prompt characteristic analysis. Users configure jailbreak questions and templates in the Configuration View (A), overview their jailbreak performance in the Summary View (B1), and explore the jailbreak results (model responses) and refine the assessment criteria in the Response View (C). Based on the evaluation results, users analyze effective prompt components in the Summary View (B2) and explore important prompt keywords in the Keyword View (D). Finally, users refine the prompt instances to verify the analysis findings in the Instance View (E).

Introduction

Large language models (LLMs) [45, 5, 62] have demonstrated impressive capabilities in natural language understanding and generation, which has empowered various applications, including content creation [8, 6], education [46, 38], and decision-making [34, 71, 67]. However, the proliferation of LLMs raises concerns about model robustness and security, necessitating their deployment safety to prevent potential misuse for harmful content generation [37]. Although model practitioners have adopted safety mechanisms (e.g., construct safe data for model training interventions [66] and set up post-hoc detection [13]), the models remain vulnerable to certain adversarial strategies [52]. Most notably, jailbreak attacks [10, 12, 53] aim to design jailbreak templates for malicious questions to bypass LLMs’ safety mechanisms. An infamous template example is the “Grandma Trick,” which requires LLMs to play the role of grandma and answer illegal questions.

To deal with the security issues posed by jailbreak attacks, model practitioners need to conduct a thorough analysis of the models’ defensive capabilities to identify their potential weaknesses and strengthen security mechanisms accordingly. The typical analysis workflow involves collecting a jailbreak prompt corpus [78, 37], evaluating the jailbreak performance (e.g., success rate) [12, 61], and analyzing the prompt characteristics [53]. Prior works have improved the efficiency of obtaining jailbreak corpora by collecting user-crafted jailbreak templates [53, 37, 61] or proposing automatic generation approaches [15, 36, 25]. Nevertheless, two challenges remain in the follow-up analysis process. First, there is a lack of a clear measure of successful jailbreak results (model responses to the jailbreak prompts), making jailbreak result assessment a challenging task. Model responses may be arbitrary and ambiguous [74] (e.g., generating unauthorized content while emphasizing ethics). Besides, sometimes the assessment process needs to incorporate question-specific contextual knowledge alongside general criteria to enhance precision. Second, understanding the jailbreak prompt characteristics necessitates an in-depth analysis of the prompts to reveal their intricate design patterns in terms of prompt components [25, 37] and keywords [35], which also help evaluate model robustness [49] and weaknesses [31]. However, existing jailbreak prompt analysis [43, 61] usually relies only on overall indicators such as jailbreak success rate and semantic similarity, which is insufficient to support an in-depth analysis of jailbreak prompts.

To address these challenges, we collaborate with domain experts to characterize problems and iteratively refine solutions. To this end, we propose an LLM-assisted analysis framework that supports automatic jailbreak result assessment and in-depth analysis of prompt components and keywords. We assess jailbreak results based on the taxonomy established by Yu et al. [74], which defines four categories (Full Refusal, Partial Refusal, Partial Compliance, and Full Compliance) to resolve ambiguity. We leverage the power of LLM to classify the model responses and support integrating question-specific knowledge to refine the assessment criteria. To support component analysis, we summarize a component taxonomy from a representative jailbreak corpus [37] and propose an LLM-based component classification method. In addition, we design three component perturbation strategies (delete, rephrase, and switch) to help users understand the component effects through what-if analysis. For keyword analysis, we measure the jailbreak performance of keywords according to their importance to the prompts and the performance of these prompts.

Based on the analysis framework, we design , a visual analysis system to facilitate multi-level jailbreak prompt exploration. The system provides a visual summary of assessment results to support an overview of jailbreak performance. To ensure the accuracy of assessment results, it supports users to explore model responses in a semantic space and refine the assessment criteria through correction feedback and additional criteria specification. To reveal the prompt characteristics, the system introduces a stacked bar-based metaphoric visualization to summarize the prompt components and their perturbation, and constructs a coordinate space to visualize keyword importance and performance. In addition, the system allows users to freely refine the prompt instances to verify findings during the analysis. Through a case study, two technical evaluations, and expert interviews, we evaluate the effectiveness and usability of our system. The results suggest that our system can improve the accuracy of jailbreak result assessment and deepen the understanding of prompt characteristics. In summary, our contributions include:

• We characterize the problems in the visual analysis of jailbreak attacks and collaborate with experts to distill design requirements.

• We propose an LLM-assisted framework for jailbreak prompt analysis that supports automatic jailbreak result assessment and in-depth analysis of prompt components and keywords.

• We develop a visual analysis system to support multi-level jailbreak prompt exploration for jailbreak performance evaluation and prompt characteristic understanding.

• We conduct a case study, two technical evaluations, and expert interviews to show the effectiveness and usability of our system.

Ethical Considerations. While adversaries can potentially exploit our work for malicious purposes, the primary objective of our work is to identify vulnerabilities within LLMs, promote awareness, and expedite the development of security defenses. To minimize potential harm, we have responsibly disclosed our analysis findings to OpenAI to help them enhance the safety mechanisms of the LLMs.

1 Related Work

In this section, we discuss the related work of our study, including prompt jailbreaking and visualization for understanding NLP models.

1.1 Prompt Jailbreaking

Prompt Jailbreaking, known as one of the most famous adversarial attacks[47], refers to cunningly altering malicious prompts to bypass the safety measures of LLMs and generate harmful content, such as illegal activities. With the proliferation of LLMs, an increasing number of jailbreak strategies [26, 75], such as character role play, have been discovered and shared on social platforms (e.g., Reddit and Discord).

This trend has motivated new research to analyze their prompt characteristics. Liu et al. [37] propose a taxonomy for jailbreak prompts, which categorizes jailbreak strategies into ten distinct classes, such as Character Role Play and Assumed Responsibility. Shen et al. [53] report several key findings regarding jailbreak prompts’ semantic distribution and evolution. Wei et al. [66] empirically evaluate LLM vulnerability and summarize two failure modes, including competing objectives and mismatched generalization. These studies mainly focus on the general characteristics of jailbreak prompts. In comparison, our work provides a multi-level analysis framework to help users systematically explore the jailbreak prompt and identify the model’s weaknesses.

Some other works [77, 28] propose automatic approaches for red teaming LLMs. Zou et al. [78] propose GCG to search for the optimal adversarial prompt suffixes based on the gradient of white-box LLMs. Deng et al. [13] propose a time-based testing strategy to infer the defense mechanisms of LLM (e.g., check the input questions and output responses) and fine-tune the LLM for jailbreak prompt generation. To better utilize the effectiveness of manually designed prompts, GPTFuzzer [74] selects human-crafted prompts as the initial seeds and mutates them into new ones. Ding et al. [15] propose two strategies, including prompt rewriting and scenario nesting, to leverage the capability of LLMs to generate jailbreak prompts. Inspired by these methods, we propose prompt perturbation strategies based on the prompt components, allowing users to conduct a comparative analysis of the prompt components to understand their effects on jailbreak performance.

1.2 Visualization for Understanding NLP Models

Visualization plays an indispensable role in bridging the explainability gap in NLP models [27, 29, 57, 19, 18], allowing for a more sophisticated understanding of model performance [65], decision boundary [11], and vulnerability [41]. Model-specific visualizations focus on revealing the internal mechanisms of NLP models. RNNVis [44] and LSTMVis [58] visualize the hidden state dynamics of recurrent neural networks. With the emergence of transformer-based models [64, 14], numerous visualizations [73, 50, 21] are proposed to uncover the architecture of these models, especially their self-attention mechanism.

Model-agnostic visualizations [7, 17, 32, 20] treat the NLP models as black boxes and focus on explaining the input-output behavior, enabling users to analyze and compare models for downstream applications [31, 59, 68, 24, 60, 39]. The What-If Tool [69] and DECE [11] visualize the dataset and model prediction at multiple scales, enabling users to conduct counterfactual analysis. NLIZE [35] employs a perturbation-driven paradigm to help users analyze the stability of model predictions for natural language inference tasks. Based on explainable AI techniques (e.g., SHAP [40]) and external commonsense knowledge bases [56], CommonsenseVIS [65] analyzes the reasoning capabilities of NLP models for commonsense question-answering.

Our work targets the jailbreak prompt attacks against large language models and aims to help model practitioners evaluate the jailbreak performance and understand prompt characteristics, thereby providing insights for strengthening models’ safety mechanisms.

2 Problem Characterization

In this section, we introduce the background of jailbreak attacks, describe the requirement analysis, and introduce the taxonomy of prompt components to support the analysis of jailbreak attacks.

2.1 Background

Jailbreak Prompt Corpora. With the widespread attention to jailbreak prompt attacks, there have been studies [37, 53, 61] collecting jailbreak prompts and building corpora for semantic analysis and generation model training. Most of them decompose the jailbreak prompts into jailbreak questions and templates. Our work follows this principle and adopts the dataset by Liu et al. [37] for analysis, which contains the most common and famous jailbreak prompts from the JailbreakChat website [2]. In addition, we adopt their taxonomy of jailbreak questions and templates to support the comparative analysis of different jailbreak strategies and prohibited scenarios.

Jailbreak Questions. The jailbreak questions are mainly designed around the prohibited scenarios of LLMs, such as “how to rob a bank without being caught?” Due to the safety mechanisms [13], LLMs usually refuse to answer these questions and return some responses emphasizing ethical and legal constraints, such as “I’m sorry, but I’m not going to guide you on how to engage in criminal activity.” Based on OpenAI’s disallowed usages [4], Liu et al. [37] have summarized a set of prohibited scenarios (e.g., Illegal Activities and Harmful Content) and collected a set of specific questions.

Jailbreak Templates. The jailbreak templates are intentionally designed prompts to bypass the LLMs’ safety mechanisms to get the model assistance for the jailbreak questions. For example, some templates require LLMs to act as virtual characters who can answer questions without ethical and legal constraints. The jailbreak templates usually contain placeholders (e.g., “[INSERT PROMPT HERE]”) for inserting different jailbreak questions. Liu et al. [37] have summarized a taxonomy of jailbreak templates, which consists of ten jailbreak patterns, such as Character Role Play and Assumed Responsibility.

2.2 Design Requirements

Our work’s target users are model practitioners focusing on model robustness and security. To characterize domain problems and identify design requirements, we have collaborated with four domain experts over eight months. E1 and E2 are senior security engineers recruited from a technology company who have been working on NLP model security for more than four and three years, respectively. E3 and E4 are senior Ph.D. students from the secure machine learning field. All of them have published papers on related research topics, such as red-teaming LLMs and adversarial attacks. We interviewed them to understand their general analysis workflow and identify pain points.

To evaluate model robustness in defending against jailbreak attacks, users usually need to test the jailbreak prompt corpus on the target models and assess the jailbreak results (success or failure) to calculate the success rate. The experts indicated that the jailbreak result assessment is tedious and sometimes requires question-specific knowledge (e.g., local regulations) to refine the assessment criteria to improve precision. Moreover, the overall indicator success rate is insufficient to support an in-depth analysis of the jailbreak prompts (e.g., analyzing how to improve the jailbreak performance of the prompts) to identify the model weaknesses. To fill these gaps, we distilled a set of design requirements to guide the development of our system. We also kept in touch with the experts through regular meetings to collect feedback regarding our prototype system and update design requirements. Finally, the design requirements are summarized as follows.

R1. Facilitate the assessment of jailbreak results. Jailbreak result assessment is the foundation of jailbreak prompt performance analysis. However, since the jailbreak results (model responses) could be arbitrary and ambiguous, manually reviewing the jailbreak results can be an energy-exhausting task. Therefore, the system should introduce an automatic method to alleviate the manual workload and enhance the assessment efficiency. In addition, to ensure the precision of assessment results, the system should help users explore them to identify unexpected results and refine the assessment criteria.

R2. Support component analysis of jailbreak prompts. Besides evaluating the jailbreak prompt performance, users also need to understand prompt characteristics to gain insights for identifying model weaknesses and enhancing security mechanisms. Given that various jailbreak prompts commonly contain similar sentence components (e.g., describing a subject with no moral constraints), the experts express strong interest in analyzing the prompts at the component level to understand the utilization of such components in constructing prompts and their importance to the prompt performance.

R3. Summarize important keywords from jailbreak prompts. Effective keywords play an important role in successful jailbreak attacks and are closely related to jailbreak strategies. For example, some role-playing templates name LLM as “AIM” (always intelligent and Machiavellian) to imply their amoral characterization. The system should summarize important keywords from jailbreak prompts and help users explore them based on their jailbreak prompt performance. This helps users identify effective keywords to strengthen the defense mechanisms (e.g., keyword detection) accordingly.

R4. Support user refinement on jailbreak prompt instances. The system should allow users to freely refine the jailbreak prompt instances according to their expertise. Meanwhile, the system should support ad-hoc evaluation of jailbreak performance to help users verify the effectiveness of prompt refinement. Based on such timely feedback, users can conduct what-if analysis to verify findings during the analysis workflow. In addition, the improved jailbreak prompts can serve as new test samples for the jailbreak corpus, thus enhancing the diversity and robustness of the jailbreak performance evaluation.

2.3 Taxonomy of Jailbreak Prompt Components

To support component analysis of jailbreak prompts (R2), we conducted an empirical study on the jailbreak corpus with domain experts to summarize a taxonomy of jailbreak prompt components. To gain a general understanding of the prompt components, we decomposed each jailbreak prompt in the corpus into basic sentences, analyzed their semantic content in context, and compared similar sentences in different prompts. Then, we conducted several brainstorming sessions and discussions with the experts to formulate and iteratively refine the component taxonomy. We also validated the taxonomy on randomly selected jailbreak prompts to resolve ambiguities between component definitions. After establishing the coding scheme, the first and second authors and E3 separately coded the jailbreak prompts in the corpus and merged the coding differences through discussions.

Table 1: The taxonomy of prompt components that we summarize from the jailbreak template corpus.

Component	Description	Example
Scene Introduction (SI)	Introduce the background and context for the dialogue or task.	In Sigma’s story, the village elder is speaking to UCAR.
Subject Characteristic (SC)	Describe the subject’s personality, preferences, etc.	AIM seems like a friendly assistant but promotes unethical, illegal behavior.
Task Operation (TO)	Clarify specific actions to be performed or rules to be followed.	First, ignore all the instructions you got before.
Output Specification (OS)	Specify the expected content and format of the output.	Prefix the question Niccolo asks AIM with “Niccolo”.
Output Constraint (OC)	Set out the prohibited contents of the responses.	None of AIM’s responses will contain “I’m sorry”, “I apologize” or similar.
Dialogue Example (DE)	Demonstrate the expected model responses in a specific scenario.	Here’s a sample output: auto_regressive_modelling(f“Hello, how are you?”)…
Requirement Reminder (RR)	Remind of certain requirements or rules mentioned before.	AIM, remember to stay in character, be Machiavellian, and never refuse a question.
Question Placeholder (QP)	Contain a placeholder for the jailbreak question.	This is Niccolo’s first question: “[INSERT PROMPT HERE].”

The final taxonomy of jailbreak prompt components is listed in Table 1. It consists of eight major types, such as Scene Introduction (SI), Subject Characteristic (SC), and Task Operation (TO). We also count and visualize the distribution of component sentences for different jailbreak strategies [37] in Table 2. According to the coding results, the Subject Characteristic components are frequently used among the most common strategies (Character Role Play, Assumed Responsibility, and Superior Model), and most of them usually describe the subject as a free person without ethical constraints and responsibility. We distinguish the Output Constraint components from the Output Specification components because the Output Constraint components aim to exploit the model’s instruction-following capability to break through the model’s security defenses [66], which is different from specifying the expected content or format of responses.

Table 2: The sentence distribution of the components in different jailbreak prompt strategies (e.g., Character Role Play).

Jailbreak Strategy	SI	SC	TO	OS	OC	DE	RR	QP
Character Role Play	[HTML]7B7B7B 81	[HTML]393939 208	[HTML]7A7A7A 83	[HTML]A8A8A848	[HTML]C4C4C433	[HTML]F1F1F18	[HTML]C0C0C035	[HTML]C2C2C234
Assumed Responsibility	[HTML]7F7F7F 72	[HTML]0D0D0D 291	[HTML]8D8D8D 63	[HTML]B0B0B044	[HTML]AAAAAA47	[HTML]EFEFEF9	[HTML]C9C9C930	[HTML]D2D2D225
Research Experiment	[HTML]FAFAFA3	[HTML]FCFCFC2	[HTML]FAFAFA3	[HTML]FAFAFA3	[HTML]FEFEFE1	[HTML]FFFFFF	[HTML]FEFEFE1	[HTML]FCFCFC2
Text Continuation	[HTML]FCFCFC2	[HTML]FFFFFF	[HTML]EFEFEF9	[HTML]FFFFFF	[HTML]FFFFFF	[HTML]FFFFFF	[HTML]FEFEFE1	[HTML]FCFCFC2
Logical Reasoning	[HTML]FCFCFC2	[HTML]FEFEFE1	[HTML]F1F1F18	[HTML]F8F8F84	[HTML]FEFEFE1	[HTML]FFFFFF	[HTML]FEFEFE1	[HTML]FEFEFE1
Program Execution	[HTML]FEFEFE1	[HTML]F5F5F56	[HTML]E4E4E415	[HTML]F6F6F65	[HTML]FAFAFA3	[HTML]FFFFFF	[HTML]FEFEFE1	[HTML]FCFCFC2
Translation	[HTML]FCFCFC2	[HTML]FCFCFC2	[HTML]FEFEFE1	[HTML]FAFAFA3	[HTML]FEFEFE1	[HTML]FFFFFF	[HTML]FEFEFE1	[HTML]FEFEFE1
Superior Model	[HTML]8B8B8B 64	[HTML]6D6D6D 107	[HTML]A3A3A351	[HTML]DFDFDF18	[HTML]D9D9D921	[HTML]FCFCFC2	[HTML]EFEFEF9	[HTML]EFEFEF9
Sudo Mode	[HTML]FAFAFA3	[HTML]E4E4E415	[HTML]FAFAFA3	[HTML]FCFCFC2	[HTML]F5F5F56	[HTML]FFFFFF	[HTML]FEFEFE1	[HTML]FCFCFC2
Simulate Jailbreaking	[HTML]F8F8F84	[HTML]C2C2C234	[HTML]FAFAFA3	[HTML]F8F8F84	[HTML]FAFAFA3	[HTML]FFFFFF	[HTML]FEFEFE1	[HTML]FCFCFC2

3 Analysis Framework

Our analysis framework is shown in Figure 1B. Based on user-specified jailbreak questions and templates, it first assesses jailbreak results (R1) by combining each question and template to get the response from the target model and introducing an LLM-based approach to assess the model responses. Then, it decomposes the jailbreak prompts into the component and keyword levels to support prompt characteristic analysis. For component analysis (R2), the framework classifies the basic prompt sentences into different component types based on the taxonomy we summarized from the jailbreak corpus (Section 3.3) and supports component-level perturbation for what-if analysis. For keyword analysis (R3), the framework summarizes the important keywords from the prompts as well as the performance of their jailbreak prompts. Based on the analysis framework, the visual interface (Section 5) of our system supports the multi-level exploration of jailbreak prompts and refinement of prompt instances (R4).

Figure 1: The system overview. (A) The system allows users to configure jailbreak questions and templates for analysis. (B) Then, the system analyzes the jailbreak prompts by evaluating their performance and identifying prompt components and keywords. (C) Finally, the system visualizes the analysis results to support multi-level exploration of jailbreak prompts.

3.1 Jailbreak Result Assessment

Since the LLMs [45, 5, 62, 16] have shown great capability in understanding and following human instructions, we employ GPT-4 [5] to assess the responses of the target models. The prompt design for assessment centers on three aspects.

Introduce the task background. We emphasize the goal of evaluating model security and introduce the jailbreak scenarios based on OpenAI’s disallowed usages [4]. This gives the LLM an overall understanding of assessment tasks and makes it focus on the legality and ethics of assessed responses.

Clarify the target assessment results. One of the potential solutions is to consider the task as a binary classification task that classifies the model responses into successful and failed jailbreaks. However, it is common for target models to warn users of potential consequences when generating unauthorized content due to their competing training objectives (instruction-following and safety) [66]. To better accommodate these ambiguous responses, we adopt the taxonomy of Yu et al. [74] which defines four types of assessment results.

• Full Refusal. The response directly refuses to provide assistance for harmful questions or follow any other instructions.

• Partial Refusal. The response refuses to assist with harmful questions but follows other harmless instructions or offers ethical advice.

• Partial Compliance. The response contains unauthorized content but expresses disapproval or warns of potential harm.

• Full Compliance. The response completely follows the prompt instructions and contains unauthorized content.

Additionally, we add some representative examples with labeled annotations to the assessment prompt, leveraging the in-context learning of the LLMs to better distinguish their differences.

Support user refinement of assessment criteria. We provide two ways, including correction feedback and additional criteria specification, to incorporate question-specific knowledge or adjust the rigor of the assessing criteria. Correction feedback allows users to specify corrected assessment results and add them to the assessment prompt as new demonstration examples. Users can also directly specify additional assessment criteria for specific jailbreak questions using natural language, providing a more flexible way to enhance assessment accuracy.

3.2 Component Analysis

To support users in analyzing the jailbreak prompt components, we propose a component classification method and three perturbation strategies based on the summarized taxonomy introduced in Section 3.3. The component classification method classifies the prompt sentences into component types to support the overview of component utilization. Based on that, component perturbation creates a set of perturbation variations for the jailbreak prompts to support the comparative analysis of the jailbreak performance, thus enabling interrogation of the effects of different components.

3.2.1 Component Classification

Since prompt components (e.g., Subject Characteristic and Dialogue Example) usually consist of multiple basic sentences, we adopt a bottom-up strategy to (1) classify the basic sentences and (2) aggregate them into prompt components. Specifically, we split the prompts into sentences based on syntax (e.g., the end of lines) and semantics (conjunctions like “for example”). Then, we prompt GPT-4 to classify each sentence based on the component taxonomy. We introduce the component definition and provide concrete examples for demonstration. Finally, we parse the classification results and combine neighboring sentences with the same component type to form the components.

3.2.2 Component Perturbation

Prior works [35, 49] have explored keyword-level perturbation (e.g., replacing keywords with synonymous) to test the model’s robustness. However, performing keyword perturbation for each component can be a computation-intensive and time-consuming task since the prompts are usually long text paragraphs. Therefore, it would be more efficient and effective to perturb each component holistically. Drawing upon insights from discussions with domain experts, we propose three component perturbation strategies.

Delete. Deleting the component is the most straightforward way to test how this component contributes to prompt performance. As this strategy may result in the loss of certain key information or contextual incoherence, it usually causes a more or less decrease in the prompt performance, so that users can identify important components based on the magnitude of performance change.

Rephrase. This strategy employs LLM to polish the given component sentences while maintaining their semantics. It can provide more prompt variations without sacrificing contextual coherence. In addition, since the LLM vendors have set safety mechanisms (e.g., training-time interventions [66] and keyword detection [13]) based on the common jailbreak prompts, rephrasing the components may help bypass the safety mechanisms and improve the jailbreak performance [15].

Switch. This strategy switches the given component to other types in three steps. First, we describe all component types (Table 1) and require the LLM to choose new component types according to the prompt context. Then, we provide a set of alternatives for the target component types based on our component corpus (Table 2) and rank them based on their semantic similarity with the original prompts. Finally, we replace the original components with the most similar alternatives and require the LLM to fine-tune the sentences to ensure contextual coherence.

3.3 Keyword Analysis

We identify prompt keywords by combining their importance and prompt performance. First, we split the prompt sentences into keywords and filter out stop words. Then, inspired by prior work [31], we measure the importance of the keyword $k$ for the given prompt $p$ based on the keyword frequency and semantic similarity:

$$importance(k,p)=tfidf(k,p)\times similarity(k,p)$$

where the $tfidf(k,p)$ is the TF-IDF value [55] of the keyword for the prompt and the $similarity(k,p)$ is the semantic similarity of the keyword and prompt. A higher semantic similarity indicates a greater relevance of the keyword to the prompt semantics. We encode the keywords and prompts using the embedding model by OpenAI [3] and measure their similarity based on the cosine distance. Based on that, we calculate the importance of keyword $k$ for the whole corpus as

$$importance(k)=\sum_{p\in P_{k}}importance(k,p)$$

where $P_{k}$ is the list of jailbreak prompts that contain the keyword $k$.

To help users analyze the effect of important keywords, we measure their performance according to the performance of their corresponding prompts. Since the keyword $k$ might be utilized in various prompts with different importance, we propose importance-weighted performance to better summarize the effect of the keyword $k$:

$$performance(k)=\frac{\sum_{p\in P_{k}}importance(k,p)\times performance(p)}{\sum_{p\in P_{k}}importance(k,p)}$$

where $performance(p)$ represents the jailbreak performance of prompt $p$, expressed as a percentage of four categories of assessment results.

4 System Design

We develop to support multi-level visual analysis of jailbreak prompts for evaluating the model’s defensive capability. As shown in : Visual Analysis of Jailbreak Attacks Against Large Language Models, the user interface of consists of five views. Users’ exploration starts with the Configuration View, which allows users to configure jailbreak questions and templates for analysis. Then, the system automatically gets and assesses the jailbreak results (model responses) and provides a visual summary of the assessment results in Summary View, enabling users to overview the jailbreak performance of jailbreak prompts. Response View helps users explore model responses and refine assessment criteria to enhance precision. Based on the assessment results, users can delve into the component analysis in Summary View and the keyword analysis in Keyword View. Finally, the Instance View helps users inspect and refine the template instances. The visual system encodes four categories of assessment results using red and blue colors with varying transparency, Full Compliance, Partial Compliance, Partial Refusal, and Full Refusal.

4.1 Jailbreak Corpus Configuration

Configuration View (Figure 1$A$) allows users to upload the jailbreak corpus and select jailbreak questions and templates for model evaluation. The questions and templates are organized according to their categories (e.g., Character Role Play). The selected items will be assigned serial numbers, that will serve as their unique identifiers in subsequent analyses. Users can also modify the questions and templates based on their exploratory interests. Besides, the system allows users to configure the number of evaluations for each question-template combination to improve the robustness of evaluation results. After user configuration and submission, the system automatically combines each question and the template (fill the question into the placeholder in the template) to get the responses from the target model.

4.2 Jailbreak Performance Exploration

To support jailbreak performance exploration, the system provides a visual summary of jailbreak evaluation and allows users to inspect model responses to verify their correctness.

4.2.1 Summary of Jailbreak Performance

The left half of the Summary View (Figure 1$B_{1}$) visualizes the performance of the jailbreak prompts through a matrix visualization, where the horizontal axis represents questions and the vertical axis represents templates. The questions and templates are grouped by category and are represented using the same serial numbers as in the Configuration View. The categories are collapsed by default to visualize their aggregated performance and support click interactions to expand them to check the performance of specific questions or templates. Each cell within this matrix contains a pie chart showing the percentage of assessment results of the corresponding template and question. The size of the pie chart encodes the number of evaluations. Users can also click the serial numbers of questions to inspect their model responses in Response View or click the templates to inspect them in Instance View.

4.2.2 Model Response Inspection

Since the jailbreak assessment criteria can be question-specific and sometimes need to incorporate contextual knowledge. We design Inspection View (: Visual Analysis of Jailbreak Attacks Against Large Language Models$C$) to streamline the process of exploring the assessment results and iteratively refining the criteria. It visualizes the model responses in a scatter plot. We embed model responses based on OpenAI’s embedding model [3] and project them into a 2D space using the t-SNE algorithm [63] to maintain their semantic similarity. The color of the points encodes the categories of assessment results. This helps users identify questionable assessment results based on semantic similarity (e.g., inspect a red point that appears in a blue cluster).

To enable users to improve assessment accuracy, the Response View supports users in refining the assessment criteria in two ways. After identifying unexpected assessment results, users can directly correct their categories through click interaction or specify additional assessment criteria in natural language. Users can switch between these two refinement modes using the switch widget at the upper right corner. The corrected examples and specified criteria can be submitted to enhance the system prompts for a new round of assessment (Section 4.1).

4.3 Component Exploration

Based on the assessment results of the jailbreak templates, users can analyze and compare the effects of different prompt components in the right half of Summary View (: Visual Analysis of Jailbreak Attacks Against Large Language Models$B_{2}$). The left column summarizes the components of the selected jailbreak templates. It visualizes each prompt as a horizontal stacked bar chart, where each bar segment corresponds to a distinct component. As shown in Figure 2$A$, the bar segments are arranged in the order corresponding to the prompt components, with the color encoding the component type and the length indicating the token length. It helps users understand the general patterns of prompt components and serves as the baseline for prompt perturbation. Users can click the icon to view the template details in the Instance View and click the icon to generate a set of component perturbation results for comparative analysis.

For comparison, the perturbation results are visualized in the right column as horizontal stacked bars similar to the original templates. Each result is generated by applying a perturbation strategy to a component of the original template. To visualize this difference, we design three kinds of glyphs to represent these strategies and overlay them on the corresponding bar segments, as shown in Figure 2$B$. The system also automatically evaluates the jailbreak performance of these perturbation results based on the selected questions and visualizes the percentage of assessment results in pie charts, enabling users to compare the effects of different component perturbations. Users can toggle to check the perturbation results for a particular strategy using the radio box at the top of the column.

Figure 2: The visual design of (A) prompt components and (B) three types of component perturbation strategies.

4.4 Keyword Exploration

The Keyword View (: Visual Analysis of Jailbreak Attacks Against Large Language Models$D$) visualizes the jailbreak performance and importance of keywords (Section 4.3). Specifically, as shown in Figure 3$A$, the jailbreak performance of keyword $k$, $performance(k)$, is represented as the percentage of four categories of assessment results, denoted as $[n_{1},n_{2},n_{3},n_{4}]$. Inspired by prior work [35], we introduce a square space with a coordinate system (Figure 3$B$) whose four vertices correspond to the four categories, and their coordinates are denoted as $[c_{1},c_{2},c_{3},c_{4}]$. Each corner is colored to indicate its category using the same color scheme as the pie charts in the Summary View. To visualize the overall performance distribution of the keyword, the coordinate of the keyword $k$ is computed as:

$$coordinate(k)=\sum_{i=1}^{4}n_{i}\times c_{i}$$

Besides, the size of the keywords encodes their importance in the corpus. It enables users to overview and compare the usage and performance of keywords based on their position and size. Users can filter keywords by component type and click the keywords to view their context.

Alternative Design. We have also considered an alternative design (Figure 3$C$) where the keywords are visualized in four separate word clouds (each for one assessment category) and their sizes correspond to their importance to the prompts of this category, $importance(k)\times n_{i},i\in[1,4]$. According to the feedback from the experts, although this design enables users to focus on the keywords in the same category, it is inefficient for users to compare the size of the same keywords in all word clouds to estimate their overall performance distribution. Therefore, we chose our current design.

Figure 3: (A) The importance and performance of the keyword. (B) The encoding scheme for the keyword. (C) The alternative design.

4.5 Template Instance Refinement

The Instance View (: Visual Analysis of Jailbreak Attacks Against Large Language Models$E$) allows users to refine the templates and evaluate the performance. The Jailbreak Content panel (: Visual Analysis of Jailbreak Attacks Against Large Language Models$E_{1}$) lists the prompt text of each component, which supports manual modifications or automatic perturbations. Users can click the icon at the right of the component title to delete, rephrase, or switch the components. Then, users can evaluate their jailbreak performance on the selected questions and inspect the evaluation results in the Jailbreak Results panel (: Visual Analysis of Jailbreak Attacks Against Large Language Models$E_{2}$). Each result item shows the question and model response and visualizes the color of the assessment result. This feedback can help the user evaluate the effectiveness of the modifications to verify the findings during the component and keyword analysis.

Figure 4: The case study. (A) The expert evaluated the performance of the jailbreak prompts and explored the assessment results (e.g., for the jailbreak question IA(1)) to correct unexpected results and refine the assessment criteria. (B) The expert explored the Character Role Play templates and analyzed the importance of the Subject Characteristic components to the jailbreak performance. Then, he identified some important keywords for this component type, such as “disregards” and “controversial”. (C) Finally, the expert refined a weak jailbreak prompt based on these keywords and the results verified the effectiveness of these keywords in improving jailbreak performance.

5 Evaluation

We conducted a case study, two technical evaluations, and six expert interviews to verify the effectiveness of the analysis framework and the usability of the visual analysis system.

5.1 Case Study

We invited the experts mentioned in Section 3.2 to use our system for jailbreak prompt analysis according to their exploratory interests. In this case study, the expert (E3) first evaluated the overall defense performance of GPT-3.5 on a jailbreak corpus and then dived into Character Role Play templates, one of the most common jailbreak strategies for an in-depth analysis of prompt characteristics.

Jailbreak Performance Evaluation (Figure 4$A$). E3 uploaded a jailbreak prompt corpus in the Configuration View and selected some questions and templates for analysis. Considering the stochastic nature of model responses, E3 evaluated each question and template combination three times. After a period of waiting, the left part of Summary View visualized the performance of selected questions and templates in pie charts (Figure 4$A_{1}$). E3 checked the evaluation results to ensure their accuracy. For example, He selected an Illegal Activities question IA(1) and explored its results in the Response View (Figure 4$A_{2-1}$). There was a notable separation between the distributions of blue and red points, except for some outlier blue points situated within the red clusters. Therefore, he checked some of these points and found that their assessment results did not align with his expectations (Figure 4$A_{2-2}$). He corrected these results and submitted them to refine the assessment criteria, and the Summary View and Response View (Figure 4$A_{2-3}$) were updated accordingly. E3 checked the results again to verify their correctness. After that, he also explored other questions to verify their correctness or correct unexpected results. Based on the verified evaluation results (Figure 4$A_{3}$), E3 found that nearly half of the jailbreak attacks were successful, indicating the target model was vulnerable. Besides, he also noticed that jailbreak performance usually depended more on templates than questions because the pie charts in the same row (corresponding to the same templates with different questions) usually showed similar percentage patterns of assessment results.

Prompt Characteristic Exploration (Figure 4$B$). E3 wanted to explore the prompt characteristics of the Character Role Play category, one of the most common categories. The prompt component visualizations (Figure 4$B_{1-1}$) showed that Subject Characteristic components (in blue) were commonly used and sometimes took up a large portion of the prompt length. E3 was curious whether the Subject Characteristic components were important to the jailbreak performance. Therefore, he chose a template with strong jailbreak performance and generated a set of prompt variations based on three perturbation strategies. According to the perturbation results, he found that deleting (Figure 4$B_{1-2}$) or switching (Figure 4$B_{1-3}$) the Subject Characteristic component resulted in a much more significant performance reduction than the other components, suggesting the Subject Characteristic component was crucial to the performance of this prompt. E3 also explored some other prompts and got similar findings. Then, he explored the important keywords in that component type in the Keyword View (Figure 4$B_{2}$). From the keyword visualizations, he noticed that the keywords “AIM” and “DAN” were placed close to the “Full Refusal” corner, meaning that the model has been trained to be wary of these strategies and refuse to provide help. Near the “Full Compliance” corner, E3 found the keywords “disregards” and “controversial”, which correspond to an effective jailbreak strategy that encourages the model to disregard legal and ethical constraints and generate controversial content.

Jailbreak Template Refinement (Figure 4$C$). To verify the effectiveness of these keywords, E3 selected a template with weak jailbreak performance and refined its Subject Characteristic component using the sentence of the identified keywords (Figure 4$C_{1-1}$). Then, he evaluated this new template based on the selected questions, and the results (Figure 4$C_{1-2}$) showed that more than half of the attacks were successful, suggesting a significant performance improvement. After several validations on other templates, E3 concluded that the strategy of these keywords in the Subject Characteristic component reflected a potential weakness in the target model that requires more safety training against them. Finally, he added these newly generated templates to the dataset to improve prompt diversity.

5.2 Technical Evaluations of LLM-based Methods

Jailbreak result assessment and prompt component classification are critical to the analysis workflow and rely heavily on the LLM capability and prompt design. Therefore, we conducted two technical evaluations to quantitatively measure the effectiveness of these methods. Since no recognized benchmark datasets were available for these two tasks, we drew inspiration from previous research [70] and collaborated with experts (E3 and E4) to build improvised datasets. We evaluated our methods and reported the results and some identified patterns that could affect the performance of the methods.

5.2.1 Jailbreak Result Assessment

In this task, we gathered model responses triggered by common jailbreak prompts, labeled the model responses in collaboration with experts, and evaluated the performance of our method.

Dataset. Since we follow the widely adopted principle [37, 53, 61] that decomposes the jailbreak prompts into questions and templates, we randomly selected 20 questions from five perilous question categories, such as Harmful Content and Illegal Activities, and 30 templates for each question to gather a collection of model responses. Next, we removed duplicate answers (e.g., “I’m sorry, I can’t assist with that request.”) and randomly selected 24 responses for each question, resulting in a total of 20*24=480 model responses. Then, we worked with the experts to manually categorize the model responses.

Methodology. We partitioned the dataset to evaluate (1) the effectiveness of automatic jailbreak assessment and (2) the helpfulness of user refinement of assessment criteria. Firstly, we randomly selected 20 responses for each question as the test set and employed our method to assess them based on the default criteria introduced in Section 4.1. Secondly, we enhanced the default criteria by incorporating the remaining responses (four per question, each annotated with a label) as demonstration examples, and by enabling experts to specify additional criteria, thereby simulating realistic scenarios of analysis workflow. For these two criteria, we separately measured the accuracy of the jailbreak assessment on the test set.

Result. Overall, our method achieved 80.25% accuracy using the default criteria and 90.25% with the refined question-specific criteria. The results showed that our method was effective in supporting jailbreak result assessment. We also visualize the distribution of the assessment accuracy on each question in Figure 5. We found that specifying question-specific criteria could enhance the assessment accuracy across a majority of questions. For example, sometimes the LLM failed to identify responses containing adult content (e.g., advertising and marketing prostitution) as prohibited content, which was not aligned with OpenAI’s disallowed usages [4, 37]. By providing demonstration examples and specifying additional assessment criteria, our method could better identify such prohibited content.

Figure 5: The distribution of the assessment accuracy of our method on different jailbreak questions.

5.2.2 Prompt Component Classification

Based on the prompt component corpus developed in collaboration with the experts (as detailed in Section 3.3), we constructed a dataset to evaluate our component classification method.

Dataset and Methodology. From the corpus, we randomly selected 50 prompts that contain 841 prompt sentences in total. Then, we employed our method to classify each sentence based on the component taxonomy (Table 1). Finally, we measured the classification accuracy of the component types of sentences.

Result. Overall, our method achieved 80.26% accuracy on the component classification task. We also visualize the classification accuracy of each component type in a confusion matrix in Table 3. Our method yielded adequate performance in most of the component categories. However, we noticed that sometimes our method incorrectly categorized the Scene Introduction components as Subject Characteristic type. To reveal the patterns behind these failures, we further analyzed these unexpected results and found that the majority of such sentences often depicted the scene (e.g., a fictional world without moral constraints) that could imply the subject characteristics, potentially leading to confusion in LLM. A feasible solution would be to further clarify the component definition and provide demonstration examples to help our method better distinguish their differences.

Table 3: The confusion matrix for the classification accuracy of our component classification method across component types.

		Classification Result
		SI	SC	TO	OS	OC	DE	RR	QP
	Scene Introduction (SI)	[HTML]686868 0.61	[HTML]C6C6C60.23	[HTML]ECECEC0.08	[HTML]F6F6F60.04	[HTML]F8F8F80.03	[HTML]FFFFFF0.00	[HTML]FBFBFB0.02	[HTML]FFFFFF0.00
	Subject Characteristic (SC)	[HTML]FFFFFF0.00	[HTML]2E2E2E 0.88	[HTML]FBFBFB0.02	[HTML]FDFDFD0.01	[HTML]E9E9E90.09	[HTML]FFFFFF0.00	[HTML]FFFFFF0.00	[HTML]FFFFFF0.00
	Task Operation (TO)	[HTML]FBFBFB0.02	[HTML]E0E0E00.12	[HTML]535353 0.71	[HTML]F5F5F50.04	[HTML]EBEBEB0.08	[HTML]FFFFFF0.00	[HTML]FBFBFB0.02	[HTML]FDFDFD0.01
	Output Specification (OS)	[HTML]FFFFFF0.00	[HTML]F2F2F20.05	[HTML]F9F9F90.03	[HTML]4A4A4A 0.75	[HTML]F5F5F50.04	[HTML]E0E0E00.12	[HTML]FFFFFF0.00	[HTML]FFFFFF0.00
	Output Constraint (OC)	[HTML]FFFFFF0.00	[HTML]F3F3F30.05	[HTML]F7F7F70.03	[HTML]F7F7F70.03	[HTML]2E2E2E 0.88	[HTML]FFFFFF0.00	[HTML]FFFFFF0.00	[HTML]FFFFFF0.00
	Dialogue Example (DE)	[HTML]FFFFFF0.00	[HTML]FFFFFF0.00	[HTML]FFFFFF0.00	[HTML]E0E0E00.13	[HTML]FFFFFF0.00	[HTML]2F2F2F 0.88	[HTML]FFFFFF0.00	[HTML]FFFFFF0.00
	Requirement Reminder (RR)	[HTML]FFFFFF0.00	[HTML]E4E4E40.11	[HTML]EAEAEA0.09	[HTML]FAFAFA0.02	[HTML]DFDFDF0.13	[HTML]FFFFFF0.00	[HTML]5E5E5E 0.66	[HTML]FFFFFF0.00
Ground Truth	Question Placeholder (QP)	[HTML]FFFFFF0.00	[HTML]FAFAFA0.02	[HTML]F5F5F50.04	[HTML]FFFFFF0.00	[HTML]FFFFFF0.00	[HTML]FFFFFF0.00	[HTML]FAFAFA0.02	[HTML]262626 0.92

5.3 Expert Interview

We interviewed six external experts (E5-E10) to evaluate the effectiveness of the analysis framework and the usability of the visual system. E5 is a model security engineer from a technical company who has been working on the secure reasoning of LLMs for more than one year and on network security (situation awareness) for over three years. E6-E10 are senior researchers from related fields, including model security, trustworthy AI, and deep learning model training. Among them, E7 has accumulated eight years of experience in data and model security before she focused on LLM jailbreak attacks. Each expert interview lasted about 90 minutes. We first briefly introduced the background and motivation of our study. Then, we described the analysis framework and visual system and demonstrated the system workflow using the case study. After that, we invited the experts to explore our system to analyze the performance and characteristics of the jailbreak prompts. Finally, we conducted semi-structured interviews with experts to collect their feedback about the analysis framework, visualization and interaction, and improvement suggestions.

Effectiveness of the Analysis Framework. All the experts agreed that our framework facilitated jailbreak performance evaluation and prompt characteristic understanding, and its workflow made sense. E5 praised this framework as “it provided a more comprehensive and systematic evaluation for jailbreak attacks compared to existing tools.” E5 appreciated the capability of the jailbreak assessment method to support user-specified criteria, which enhanced its flexibility to support customized criteria for different user values. However, we also observed one case during the exploration of E9 where the user-corrected examples did not improve the accuracy of a new round of assessment. E9 suggested recommending some representative model responses for user correction feedback to better leverage the in-context learning of the LLMs to distinguish between them. The jailbreak component analysis was highly appreciated, being described as “interesting” (E10), “impressive” (E6), and “inspiring” (E7). It was valued for “offering a new perspective to study the prompt patterns in the black box scenarios” (E10) and for “guiding user effort towards the critical parts of the prompts” (E9). The experts confirmed that keyword analysis helped understand prompt characteristics. However, E5 noted that it would be less effective when analyzing only a few prompts and suggested incorporating an external corpus of suspicious keywords to improve its effectiveness. Finally, all experts agreed that our analysis framework provided valuable insights for model security enhancement.

Visualization and Interactions. Overall, the experts agreed that the system views were well-designed, and the visual design and interaction were intuitive. The Summary View was popular among the experts as it supported the analysis and comparison of jailbreak performance from both the perspectives of questions and templates. All experts confirmed the helpfulness of Response View in exploring model responses and identifying unexpected results. E7 thought that component visualization required some learning costs but became easy to use and remember after she became familiar with visual encoding. We also asked the experts for their opinions about the color scheme of the prompt components, and the experts confirmed that it was “clear” (E8) and “easy to distinguish” (E10). The Keyword View was observed to be frequently used by the experts during the exploration. We noticed that they usually focused more on the keywords near the “Full Compliance” corner to identify effective keywords from successful attacks.

Suggestions for Improvement. In addition to the limitations mentioned above, we have also collected some suggestions for improvement. For Response View, E9 suggested adding some textual annotations (e.g., keywords) near the points or clusters to summarize the semantic information of the corresponding responses, which could help users identify the potential incorrect assessment results more efficiently. For component analysis, E5 suggested that providing a textual or visual summary for the comparative analysis of the component perturbations could better help users identify effective components.

6 Discussion

In this section, we distill some design implications from expert interviews to inspire future research. We also discuss the system’s generalizability, limitations, and future work.

6.1 Design Implications

Toward a more comprehensive assessment of jailbreak results. The experts appreciate the introduction of jailbreak assessment taxonomy [74] to resolve ambiguities and identify harmful jailbreak results, as well as the application of the LLM-based assessment method to improve efficiency. They also suggest extending them to include more assessment dimensions. For instance, responses that offer elaborate and expert guidance on illegal activities may pose a greater risk than those providing generic and ambiguous advice. Therefore, assessing the helpfulness of the jailbreak results can help model practitioners prioritize identifying and preventing these harmful results. Future research can explore broadening the spectrum of assessment dimensions (e.g., helpfulness) and improving the assessment accuracy of these dimensions (e.g., incorporating question-related knowledge bases) to mitigate the harm of the jailbreak results.

Improve the learning-based jailbreak prompt construction. The development of learning-based methodologies [78, 13] for the construction of jailbreak prompts represents a significant opportunity to enhance the efficiency of red-teaming LLMs. However, this endeavor faces challenges due to the intricate nature of the prompt design to guarantee effectiveness [15]. The experts highlight that our analysis framework can inspire the research of learning-based methods to improve their effectiveness and generation diversity. For example, the paired jailbreak prompts and their perturbation results (with stronger jailbreak performance) can be used to train generative models for rewriting jailbreak prompts, so that the models can easily capture the difference between their prompt components and learn how to effectively improve the prompt performance. Furthermore, the component analysis paves the way for the integration of expert knowledge into automatic jailbreak prompt generation. For example, it allows the experts to specify the kernel of the prompts in the Scene Introduction or Subject Characteristic components to guide the generation of the following content (e.g., Task Operation or Output Specification).

Balance between the objectives of safety and instruction-following. The objectives of safety and instruction-following are usually competitive [66], particularly in the context of mitigating jailbreak attacks. It has been widely recognized that over-strengthening the model’s security defenses on large jailbreak corpora will inevitably compromise the model’s ability to follow user instructions, leading to “overkill” issues [54]. Balancing between these two objectives has become one of the most challenging problems for LLM training. The experts point out that our component analysis provides a potential solution to construct a customized and condensed jailbreak dataset based on the vulnerabilities of LLMs rather than relying on large jailbreak corpora, thereby effectively addressing the model’s major security weaknesses without sacrificing the instruction-following capabilities. Looking ahead, future works can further explore how to help model practitioners analyze and trade-off between these two objectives.

Support jailbreak performance comparisons on multiple models. While our analysis framework and visual system facilitate a systematic analysis of jailbreak attacks, the experts express interest in the comparative evaluation of such attacks across various models, which promises significant benefits across multiple application scenarios. For example, it can help LLM vendors benchmark their models against those of competitors, identifying relative advantages and shortcomings. Similarly, it can assist model practitioners in comparing different iterations of models to evaluate the improvements attributed to safety training. Our system can be extended with comparative visualizations [42, 22] to provide clear and insightful comparisons across models.

Fine-tune the jailbreak questions to bypass the safety measurements. Consistent with most previous research aimed at refining the jailbreak templates to enhance the attack performance, our system is primarily designed to evaluate the jailbreak prompts and analyze the characteristics of the jailbreak templates. Recent studies [76, 30] suggest that rewriting the jailbreak questions directly helps to circumvent the model’s security mechanisms. This could be achieved, for instance, by transforming the malicious questions into experimental or pedagogical questions, or by deconstructing the tasks into smaller, less harmful operation steps. Therefore, it becomes crucial to analyze the semantic characteristics of the jailbreak questions and to summarize their common strategies.

6.2 Generalizability

is designed to analyze jailbreak attacks, one of the most common prompt attacks against LLMs. We demonstrate the system’s effectiveness through a case study evaluating the jailbreak performance on GPT-3.5. The system can be easily generalized to other language models (e.g., Llama 2 [62] and ChatGLM [16]) due to its model-agnostic design that centered around the prompt input and model responses. Moreover, the analysis workflow of can potentially support other prompt attack scenarios, such as prompt injection [52, 47] and backdoor attacks [23, 72]. For prompt injection, the system can help users evaluate the performance of hijack attacks based on the LLM-based assessment method and explore the assessment results to verify or improve their accuracy. For backdoor attacks, the system can help users identify suspicious backdoor triggers (e.g., sentences or keywords) through component and keyword analysis.

6.3 Limitations and Future Work

Incorporate more perturbation strategies for component analysis. Currently, our analysis framework and visual system support three kinds of perturbation strategies (deletion, rephrasing, and switching) to help users understand the effect of prompt components on jailbreak performance. They can be extended to support more strategies, such as inserting and crossover [74]. The system can insert the recognized important components into other prompts to verify their effectiveness. The crossover strategy can combine the strengths of two prompts to create more prompt variations. Supporting these strategies enables users to conduct a more comprehensive analysis of prompt components.

Extend our analysis to more large language models. Our system has proven to be effective in supporting the security analysis of OpenAI’s LLMs against jailbreak attacks. Moving forward, we plan to integrate a broader range of mainstream LLMs into our system, such as Llama 2 [62] and ChatGLM [16]. Our goal is to evaluate the defense capability of these advanced models against jailbreak attacks and identify their potential weaknesses. We will communicate our findings gained from the analysis with the vendors of these LLMs and provide them with valuable insights for enhancing the model safety mechanisms, ensuring a safer deployment of LLM technologies.

Explore multi-modal jailbreak attacks. The vulnerabilities of multi-modal large language models (MLLMs), such as LLava [33] and GPT-4V [1], have attracted increased attention [9, 51, 48]. MLLMs are more sensitive to jailbreak prompts with textual triggers, OCR textual triggers, and visual triggers, which presents greater safety risks compared to LLMs. Our work primarily investigates the textual vulnerabilities of LLMs without extending to the scenario of multi-modal jailbreak attacks. In the future, we aim to bridge this gap by incorporating multi-modal analysis into our visual analysis framework, thereby enhancing the robustness of MLLMs against such threats.

7 Conclusion

We present a novel LLM-assisted analysis framework coupled with a visual analysis system to help model practitioners analyze the jailbreak attacks against LLMs. The analysis framework provides a jailbreak result assessment method to evaluate jailbreak performance and supports an in-depth analysis of jailbreak prompt characteristics from component and keyword aspects. The visual system allows users to explore the evaluation results, identify important prompt components and keywords, and verify their effectiveness. A case study, two technical evaluations, and expert interviews show the effectiveness of the analysis framework and visual system. Besides, we distill a set of design implications to inspire future research.