Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models

With the surge in the capabilities of large language models (LLMs) Ouyang et al. (2022); Zhao et al. (2023); Brown et al. (2020), there is currently a growing interest in how to integrate the general artificial intelligence of LLMs into the multimodal domains. In consequence, large vision-language models (LVLMs) powered by LLMs are proposed Ye et al. (2023); Zhu et al. (2023); Liu et al. (2023); Li et al. (2023); Dai et al. (2023a); Bai et al. (2023), which can understand multimodal contents and perform multimodal tasks under instructions. In general, existing LVLMs follow the following paradigm: leveraging a multimodal alignment module to comprehend multimodal inputs, followed by utilizing a LLM to generate responses. Therefore, the training process of LVLMs typically involves modalities alignment pre-training and instruction tuning. Specifically, mPLUG-Owl Ye et al. (2023) pre-trains the encoder and alignment module, and then finetunes LLaMa Touvron et al. (2023) by low-rank adaption. In contrast, LLaVA Liu et al. (2023) only pre-trains the alignment network and finetunes the alignment network and Vicuna Chiang et al. (2023) on constructed instructions. MiniGPT-4 Zhu et al. (2023) only finetunes the cross-modal alignment network with other modules frozen.

Despite the strong capabilities of these LVLMs, they all grapple with hallucination issues unexpectedly. To tackle with this issue, several benchmarks Fu et al. (2023); Xu et al. (2023); Yifan et al. (2023); Lovenia et al. (2023); **g et al. (2023); Chen et al. (2024) have been proposed to provide detailed evaluations of the hallucination degree exhibited by LVLMs.

Existing hallucination mitigation strategies for LVLMs can be roughly divided into three groups. The first and most widely adopted approach Liu et al. (2024); Gunjal et al. (2023); Lee et al. (2023); Wang et al. (2023) primarily relies on instruction tuning and retraining. LRV-Instruction Liu et al. (2024) introduces a comprehensive instruction tuning dataset encompassing positive and negative instructions. Wang et al. (2023) adopts an iterative instruction generation strategy to improve diversity and accuracy of instructions. Volcano Lee et al. (2023) facilitates the model with the ability to utilize self-feedback to self-revise responses through training. However, these methods heavily depend on the quality of instruction data construction and require substantial computational resources.

The second group of approaches, exemplified by Woodpecker Yin et al. (2023) and LURE Zhou et al. (2024), aim to integrate external detection models or specific LVLMs as revisors to enhance accurate visual understanding, thereby refining base LVLMs’ hallucinated generation. Nevertheless, these approaches rely on external models and fail to explore the intrinsic capabilities of the base model.

The third group of approaches aim to devise decoding strategies to mitigate hallucinations during the process of decoding. OPERA Huang et al. (2023) propose a penalty-based decoding method along with roll-back strategy to avoid over-trust during decoding. On the other hand, VCD Leng et al. (2023) introduces contrastive decoding to reduce over-reliance on spurious bias and learned priors. However, obtaining internal states of LVLMs during the decoding process poses a challenge for common users.

Compared to existing approaches, our proposed method is training-free and mitigates hallucinations solely through language interactions. It not only explores the potential of LVLMs to alleviate hallucinations but also offers better interpretability.

There have also been some works on hallucination detection in LLMs Manakul et al. (2023); Kuhn et al. (2022); Lin et al. (2023), which view the consistency of responses reflects the model’s uncertainty. Kuhn et al. (2022) propose semantic entropy to measure the degree of semantic divergence among responses to accommodate semantic equivalence in free-form text. SelfCheckGPT Manakul et al. (2023) extend the method to black-box LLMs, eliminating the need for tokens’ probability. Lin et al. (2023) introduce and compare various uncertainty estimation metrics for black-box LLMs. In contrast to prior works that focus on consistency among responses to the same question, we propose LogicCheckGPT, a logic consistency-based method that involves logic-related questions and answers. LogicCheckGPT is more capable of delving deeper into the internal uncertainty and the degree of hallucination within the LVLMs.

To alleviate object hallucinations, we delve into the logical consistency of LVLMs’ responses. Specifically, we examine whether the responses demonstrate logical coherence. For each object mentioned in a response, we pose two types of sequential questions: one regarding the attributes of the object, and another about which object possesses those attributes. The logical consistency of the LVLM’s responses, i.e., whether they form a closed loop of logical reasoning, serves as an indicator of object hallucination. Here, the “logical closed loop” refers to the subsequent object answered being consistent with the initial object.

This process is decomposed into the following steps: object extraction, object-to-attribute inquiring, attribute-to-object inquiring, logical closed loop checking, and hallucination detection and mitigation.

To determine the object hallucination, we first need to extract candidate objects from the responses of LVLMs for further querying and checking. For simplicity and versatility, we adopt an LLM to complete each sub-task, including object extraction. Specifically, we employ GPT-3.5 ²²2https://platform.openai.com/docs/models/gpt-3-5-turbo as our LLM because of its strong capabilities. The prompt we used can be referred in Appendix D.1. The extracted objects are represented as $O=\{o_{1},...,o_{i},...,o_{m}\}$, where $m$ is the total number of examinee objects.

As discussed in Section 3.1, the primary step involves constructing object-to-attribute questions to inquire about the attributes of the object. However, it is impractical to enumerate all possible attributes, given their infinite nature. Additionally, it is challenging to create generic attribute question templates applicable to all objects, as different objects typically possess their own specific attributes. For example, attributes related to material for a “dining table” differ from those related to clothing for a “person”.

For better flexibility and adaptability, we prompt the LVLM to provide a detailed description of the object $o_{i}$ in free-form text. The template question is formulated as follows: “Could you please describe the {object} in the image?” We ask the LVLM to respond multiple times. This approach allows us to obtain detailed and specific attribute descriptions from the LVLM regarding the object, thereby facilitating the construction of attribute-to-object questions in subsequent steps. The average number of extracted attributes of existent or hallucinated objects has also been studied in Section C.

In the attribute-to-object inquiring stage, our goal is to formulate questions from the attributes to the object, in contrast to Section 3.3. After obtaining the attribute descriptions of the object, follow-up questions can be raised based on them. Rather than directly prompting the LLM to formulate questions from the descriptions, we break down this task into two subtasks: attribute extraction and question formulation. This approach avoids potential issues of misleading the LLM to not follow instructions and inadvertently revealing object identity in the questions. It has also been observed that decomposing a task into several simple sub-tasks for LLMs to fulfill yields better performance Wei et al. (2022); Zhao et al. (2023).

In specific, we prompt the LLM to extract attributes of the target object $o_{i}$ from the description, wherein the target object is represented as “The object”. For instance, extracted attributes such as “The object is made of wood in the image”, “The object is red in color”. For the examinee object $o_{i}$, the extracted attributes are represented as $A_{i}=\{a_{i,1},...,a_{i,j},...,a_{i,n_{i}}\}$, where the number of extracted attributes is denoted as $n_{i}$. Detailed prompting instructions can be found in Appendix D.2.1.

Subsequently, we instruct the LLM to convert these extracted attributes into questions that inquire about what object possesses the specific attributes. However, we have noticed that asking questions like “What is/has {attribute} in the image?” often yield answers about the most obvious objects with the target attributes, potentially leading to the omission of other less conspicuous yet existent objects. To deal with this issue, we frame our questions $Q_{i}=\{q_{i,1},...,q_{i,j},...,q_{i,n_{i}}\}$ in the format “Could you tell me all the objects that {attribute} in the image?”. The ablation study of this prompt design can be referred in Section 4.3. This format enables the LVLM to comprehensively cover objects that meet the specified attribute constraints. Detailed prompting instructions can be found in Appendix D.2.2.

After obtaining the answers $R_{i}=\{r_{i,1},...,r_{i,j},...,r_{i,n_{i}}\}$ from the attribute-to-object inquiring stage, we can assess whether each answer forms a logical closed loop, meaning that the object mentioned in the answer is consistent with the examinee object $o_{i}$. In specific, we prompt the LLM to check whether the $o_{i}$ is covered in each LVLM’s response $r_{i,j}$. The judgement is limited to “Yes” and “No”. For the $i$-th examinee object, the judgement of the $j$-th answer is mapped into score $x_{i,j}$ through the map** {Yes: $1.0$, No: $0.0$}. The prompt is listed in Appendix D.3.

Finally, the logical closed loop rate for each examinee object, defined as the number of logical closed loops divided by the total number of attribute-to-object question-answer pairs $n_{i}$, can be formulated as

where $\mathcal{S}(o_{i})$ denotes the logical closed loop rate for the $i$-th object.

As discussed before, the object is likely to be hallucinated when $\mathcal{S}(o_{i})$ is low. On the other hand, when it’s close to 1, it’s more likely that the object truly exists. Therefore, $\mathcal{S}(o_{i})$ serves as an indicator of object hallucination. Through a valid hallucination threshold $\lambda$, our method can effectively detect objects that are likely to be hallucinated, i.e., $\mathcal{S}(o_{i})$ is below $\lambda$.

After identifying hallucinated objects, we guide the LLM to eliminate contents related to hallucinated objects, aiming to minimize hallucinations. The detailed prompt can be found in Appendix D.4.

We conduct ablation study for several variants, as illustrated in Fig. 4.

• •

  vanilla adopts no hallucination mitigation method.

• •

  FreeCheck employs an LLM (GPT-3.5) to autonomously interrogate LVLMs through multi-turn interactions, similiar to Cohen et al. (2023). This iterative process continues until either the LLM reaches a conclusion or the predefined iteration limit (set at 5 here) is reached. The prompt can be referred in Appendix E.

• •

  LogicCheckGPT w/o AOP refers to replacing the Attribute-to-Object prompt in Section 3.4 by inquire only object instead of covering all objects, e.g. “What is/has {attribute} in the image?”.

• •

  LogicCheckGPT w/o LCL employs an LLM to determine the logical consistency directly without Logical Closed Loop rate.

We can observe that FreeCheck performs worse for most LVLMs, primarily because the LLM fails to raise valuable questions to detect hallucinations. For the existent object, FreeCheck often raises questions about other objects that have been mentioned by the LVLMs during the interactions and cannot focus on the examinee object. For the hallucinated object, the questions raised are often simplistic, making it easy for FreeCheck to be misled and fail to detect object hallucination. LogicCheckGPT w/o AOP brings significant improvements to several models but still falls below our method, indicating that covering a sufficient number of objects is beneficial. Though LogicCheckGPT w/o LCL falls between vanilla and w/o AOP, demonstrating the effectiveness of calculating logical closed loop rate. For the powerful QWEN-VL-Chat, which has already achieved impressive performance, only our LogicCheckGPT can enhance its capabilities.

We test our method with values of threshold $\lambda$ ranging from 0.0 to 0.9 as shown in Figure 5 and have the following observations:

There is a significant increase when threshold $\lambda$ is increasing from 0.0 initially for all models with LogicCheckGPT under both settings. It indicates that our method can significantly distinguish existent objects and hallucinated objects by valid threshold, as models tend to provide logically consistent responses for existent objects.

Subsequently, these these models reach their respective performance peaks. For instance, the performance of mPLUG-Owl achieves the best when $\lambda$ is 0.4 under adversarial setting, and the best $\lambda$ for LLaVA is also 0.4. For MiniGPT-4, the $\lambda$ at which it reached its performance peak varied across different settings, 0.2 and 0.1 respectively. With the continued increase of the threshold, a noticeable decrease in performance can be observed. This is reasonable, as it leads to the misclassification of a large number of existent objects as non-existent.

The number of attributes $n$ obtained during the object-to-attribute stage determines the number of questions that can be generated from these attributes. We report the performance when $n=1,2,3,4,5$ and $\geq 5$ (See Figure 6) and summarize the observations as follows:

It can be observed that there is an obvious performance improvement when the number of attributes increases across the four LVLMs. This improvement is reasonable as it allows for a greater range of inquiries to verify the presence of the object. As $n$ exceeds 5, the improvement becomes marginal.

In this section, we have selected two representative examples for mPLUG-Owl covering distinct types of questions, including a binary question, “Is there a dining table in the image?” and an open-ended query, “Please describe this image in detail.”, as illustrated in Fig 3. In the first case 3(a), the model answers “yes” to the binary question, while our method detects that dining table is hallucinated and corrects the output. As for open-ended text generation, the models tends to hallucinate nonexistent objects as the length of the generated sequence increases. However, our method is capable of individually verifying the existence of objects, thereby mitigating hallucinations. More qualitative analysis can be referred in Section F.