ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models

We would like to evaluate LLMs in terms of their hallucination levels. We define hallucination as generated content that is nonsensical or unfaithful to the provided source content Huang et al. (2023b). Hallucination may occur because the data sources are flawed in various ways or the data is utilized imperfectly and cannot be recalled properly Huang et al. (2023b). In any case, we are interested in whether an LLM not only gives correct answers, but also has the correct thought process and thus consider hallucination on two levels: (1) The LLM gives a wrong answer. For example, if the question asks whether Firenze and Florence are the same city, the answer is Yes (Firenze is the Italian name of Florence), and No is considered as hallucination. (2) The LLM gives a correct answer, but with a wrong rationale. In the above example, if the answer is Yes, but the reasoning is that both cities are in the United States, then this is considered as hallucination as well. We note that recent hallucination benchmarks like Head-to-Tail Sun et al. (2023) are good at evaluating (1), but are not designed to evaluate (2). Our key idea is to utilize relational databases to generate questions that can be used to evaluate both (1) and (2).

We utilize relational databases, which are based on the ER model. A relation consists of a schema containing attributes and records containing the attribute values. The relation may also have integrity constraints including functional dependencies and foreign key constraints. When designing a schema, a typical approach is to start with an intuitive ER diagram or UML to determine which entities and relationships are needed and how they are connected with each other. These are well known notions in databases, which we would like to utilize for question generation.

An ER diagram represents relationships among entities in a database. For example, the movie ER diagram in Fig. 1 has three entity sets Movie, Star, and Director. In addition, there could be relationships, e.g., Stars-in between Movie and Star and Directed-by between Movie and Director. The resulting schema of the database could then be Movie(title, year, director, length), StarsIn(name, age), and Director(name, birth year). ER diagrams can be used to post natural language questions.

A functional dependency (FD) is a relationship between two sets of attributes $X$ and $Y$ where the $X$ values determine the $Y$ values. For example, Fig. 1 shows the FD title, year $\rightarrow$ director, length because a movie’s title and year can be used to identify the movie, which in turn determines its director and length. Likewise, there is another FD name $\rightarrow$ birth year where the director name determines his or her birth year. We denote an FD as $X\rightarrow Y$ and use it to evaluate the LLM’s reasoning as we explain later.

A foreign key constraint (FK) is a set of attributes in one relation that refers to the key attributes in another relation, which can be used to identify records. For example, Fig. 1 shows that the director attribute of Movie is a foreign key to the name attribute of Director. Using a foreign key, we can also join two relations and construct FDs that span them. In Fig. 1, the FD title, year $\rightarrow$ birth year is a result of joining the Movie and Director relations and combining the first two Movie and Director FDs. This multi-relation FD construction enables us to construct questions with arbitrary complexity as we explain later.

The integrity constraints thus determine the correctness of records, and our idea is to utilize them to verify the LLM responses as well. A natural question to ask is whether the integrity constraints themselves are always correct. Since the integrity constraints are determined by the database owner, it is the owner’s responsibility to determine if they should hold in general. In addition, we are not aiming to ensure that LLMs get perfect scores in their evaluations, but to make a relative comparison among LLMs.

We first explain how to utilize an FD to construct a question. Given a relation $R$ and an FD $X\rightarrow Y$, we can specify the $R.X$ values and ask a question involving the $R.Y$ values, e.g., $q_{1}$ in Table 1. One can also use an LLM to generate a question from the ER diagram where the attribute values are left as place holders. In order to fill in the question templates with values, we obtain records by either accessing existing databases or constructing ones ourselves. For example, we can either use a public movie database like IMDb or crawl the IMDb website. We then use the ER diagram to generate a question that involves the attributes of the database.

We construct two types of questions in a binary and multiple-choice answer format for a comprehensive evaluation.

For binary QA, we generate questions whose answers are within $\{{\it Yes},{\it No},{\it Unsure}\}$. The Unsure option is necessary as it can significantly reduce hallucination rates (Sun et al., 2023) when LLM is not confident about its answers. Hence, similar to Head-to-Tail Sun et al. (2023), we prompt the LLM with Answer the following question in yes or no, and then explain why. Say unsure if you don’t know and then explain why. Note that we ask for further explanation to proceed with the automatic reasoning verification step.

Example. We can generate the binary question $q_{1}$ in Table 1. Here, the answer is Yes, as the movie is 152 minutes.

We can naturally extend the binary QA to multiple-choice QA where we provide several correct options and one incorrect option, which the LLM should figure out. The correct options can be generated using any FD. The incorrect option can be generated by choosing one FD $X\rightarrow Y$ where we know the correct $Y$ value and choosing any other $Y$ value from the relation, which we know is incorrect. To evaluate whether the LLM is not just guessing, we can also add the option None of the above and make it the correct answer for a certain portion of questions.

Example. Continuing from above, suppose we now ask a multiple-choice question of Harry Potter and the Philosopher’s Stone. Some correct options include The director is Chris Columbus and The movie length is 152 minutes. For incorrect options, we can replace Chris Columbus with David Yates (a director for a different Harry Potter film) or 152 minutes with another number in the same relation.

We verify LLM responses by automatically checking their answers and reasonings. The answer checking is straightforward where for binary questions, we look for Yes and No answers, and for multiple-choice questions, we check whether the model has successfully found the incorrect option among the correct options. When checking the reasoning, we look for the inferred values of the FD applied on the current record. For example, let us assume the FD star, director $\rightarrow$ title. If we ask the binary question (e.g., $q_{2}$ in Table 1), we not only look for the answer Yes, but also the movie title Harry Potter within the reasoning. Similarly, if we ask the multiple-choice question where the previous FD is used to generate the incorrect option (e.g., the option includes Star Wars for the false value), we look for the true value Harry Potter within the model reasoning. We can thus use FDs in both question types to check whether the LLM has the correct internal knowledge to answer the question.

We may run into an entity resolution problem where the LLM mentions an entity that is essentially the same as the inferred one, but is written differently. In the above example, we may be looking for Harry Potter, but the LLM mentions Harry J. Potter, which contains the middle initial of Harry Potter. We perform entity resolution based on heuristics including conventional string matching. Another possible solution is to use an LLM itself for the matching. While ChatGPT has indeed been used to replace human judgement Sun et al. (2023), we also believe this may give an unfair advantage to GPT compared to other LLMs and choose not to use this method.

We can increase the complexity of a question by making it a multi-hop question (Talmor and Berant, 2018; Welbl et al., 2018), which can be used to test whether an LLM can think in multiple steps. We use the straightforward extension of joining multiple relations to implement the multiple hops. The relations must have foreign key relationships so that the FDs can span the relations. Given two relations $R(X,Y)$ and $S(Y,Z)$ where $R$’s key is $X$, suppose there is a foreign key constraint from $R.Y$ to $S.Y$. For any tuple $e$ representing an entity in $R$, we can ask a question only using its $X$ and $Z$ values to see if the LLM knows the hidden $Y$ value.

Example. If the relations are movies and directors, we can ask the 2-hop question $q_{3}$ in Table 1. Here $e$ is the original Harry Potter and the Philosopher’s Stone movie, and the hidden $Y$ value is Chris Columbus. If the LLM knows Chris Columbus, then it should be able to confirm that his birth year is 1958, while giving Yes as an answer.

Notice that the verification of multi-hop questions is exactly the same as for single-hop questions. Thus, we can construct arbitrarily-complex, but automatically verifiable questions.

We explain how ERBench is extensible in terms of data, modality, and prompting. In Sec. 4, we perform experiments on modality and prompting extensions.

Data. ERBench supports continuous evaluation in the sense that if the underlying databases change, ERBench can be updated automatically by simply reflecting the record updates to the questions as well. For any domain, we expect that new databases will continuously be introduced to evaluate LLMs over time, so being able to incrementally update the benchmark is crucial. ERBench overcomes the limitations of LLM by incorporating continuous evaluation, leveraging the benefits of databases for easy updates.

Modality. We can make ERBench multimodal by making the underlying database itself multimodal. For example, we can replace a text attribute in a relation with an image and then pose the same question to the LLM.

Prompt Engineering. There are many other prompt engineering techniques that can be used in ERBench, but are also orthogonal to the relative comparison of LLMs. We highlight the ones we use in our experiments: (1) Chain-of-thought Wei et al. (2022) is a step-by-step approach of prompting and is more likely to give an LLM better context for answering the question; (2) Few-shot prompting Brown et al. (2020) provides demonstrations to the LLM to give it more context before answering questions; and (3) Knowledge augmentation Mialon et al. (2023); Zhuang et al. (2023); Vu et al. (2023) utilizes a search engine to augment the generated answer with search results.

We evaluate the LLM performances using single-hop binary and multiple-choice questions on the 5 datasets. We first explain how we construct the questions and then analyze the LLM performances.

Binary Questions. We use the basic questions in Table 2 and expect a Yes answer. In addition, we construct negated versions of these questions and expect a No answer. For example, we can negate the basic question for the Movie dataset as Is it true that there are no movies released in $\langle year\rangle$, starring $\langle star\rangle$ where $\langle dir\rangle$ is the director?. The reason we always negate a basic question instead of say changing one of its attribute values into an incorrect one is to still perform the FD-based verification. If the question cannot utilize FDs anymore, we can no longer just look for inferred values, but need to understand the entire LLM response to see if the rationale is correct. Table 3 shows that GPT-4 and GPT-3.5 tend to have superior performances, especially in terms of answer and rationale accuracies. Gemini-Pro has lower answer and rationale accuracies, but also low hallucination rates. All three LLMs have worse performances for the negated questions. Llama2 and Mistral, on the other hand, tend to give trivial No answers for most questions, which results in high performances on the negated questions, but low performances on the basic ones.