Does the Generator Mind its Contexts?
An Analysis of Generative Model Faithfulness under Context Transfer

Abstract

The present study introduces the knowledge-augmented generator, which is specifically designed to produce information that remains grounded in contextual knowledge, regardless of alterations in the context. Previous research has predominantly focused on examining hallucinations stemming from static input, such as in the domains of summarization or machine translation. However, our investigation delves into the faithfulness of generative question answering in the presence of dynamic knowledge. Our objective is to explore the existence of hallucinations arising from parametric memory when contextual knowledge undergoes changes, while also analyzing the underlying causes for their occurrence. In order to efficiently address this issue, we propose a straightforward yet effective measure for detecting such hallucinations. Intriguingly, our investigation uncovers that all models exhibit a tendency to generate previous answers as hallucinations. To gain deeper insights into the underlying causes of this phenomenon, we conduct a series of experiments that verify the critical role played by context in hallucination, both during training and testing, from various perspectives.

Keywords: Text Generation, Faithfulness, Question Answering

Does the Generator Mind its Contexts?

An Analysis of Generative Model Faithfulness under Context Transfer

Xinshuo Hu${}^{1}$${}^{*}$, Baotian Hu${}^{1}$${}^{\dagger}$, Dongfang Li${}^{1}$, Xiaoguang Li${}^{2}$, Lifeng Shang${}^{2}$

${}^{1}$Harbin Institute of Technology, Shenzhen, ${}^{2}$Huawei Noah’s Ark Lab

[email protected], [email protected], [email protected],

{lixiaoguang11, Shang.Lifeng}@huawei.com

Abstract content

1.   Introduction

Knowledge-augmented text generation method (e.g. RAG (Lewis et al., 2020b), FiD (Izacard and Grave, 2021)), and Atlas (Izacard et al., 2022), have demonstrated state-of-the-art (SOTA) performance across various NLP tasks. The paradigm of generating text using external knowledge offers the advantage of plug-and-play through non-parametric contextual knowledge. In contrast, parametric knowledge embedded within models necessitates retraining for updates (Li et al., 2022a). A faithful knowledge-augmented generator should consistently produce output that aligns with the contextual grounding (Ji et al., 2022). However, the presence of hallucinations originating from parametric memory (see Figure 1) poses a significant challenge for practical text generation applications (Maynez et al., 2020; Zhang et al., 2020b).

Figure 1: An example of generated hallucination from training memory. The model disregards the transferred contextual knowledge and predicts an out-of-date answer that was present in its original training data when answering the same question. Non-essential details are ignored by […].

The investigation of the faithfulness of generative models in the presence of dynamic contextual knowledge remains an ongoing research area. Previous studies have primarily focused on analyzing hallucinations in scenarios where the input texts during training and testing are independent, such as in summarization(Pagnoni et al., 2021; Ladhak et al., 2022; Tang et al., 2022) or machine translation(Raunak et al., 2021; Müller et al., 2020). While knowledge-dynamic question answering has garnered attention in several works (Min et al., 2020; Longpre et al., 2021; Zhang and Choi, 2021; Chen et al., 2021; Wang et al., 2022; Liska et al., 2022; Kasai et al., 2022; Chen et al., 2023), only a few studies have systematically quantified the extent of model faithfulness or analyzed the circumstances and reasons behind hallucination generation in the presence of dynamic contextual knowledge (Longpre et al., 2021; West et al., 2022). In this study, we define context transfer as the process of contextual knowledge changing while the question remains the same. Specifically, the generative model is trained on old knowledge but evaluated on new knowledge instances. Our analysis focuses on memory hallucination which refers to hallucinations generated by parametric knowledge during context transfer.

In this work, our objective is to assess the faithfulness of generative models in the context of context transfer, focusing on two primary research questions:

RQ 1

To what extent does the generative model exhibit faithfulness under context transfer?

RQ 2

What are the underlying reasons for the occurrence of memory hallucination?

To address these research questions, we first define the context transfer task and introduce a novel metric for measuring hallucination (§3). Subsequently, we conduct comprehensive experiments involving multiple models to investigate Research Question 1. Our findings indicate that models do not consistently exhibit grounded behavior in the presence of context transfer (§4). To gain deeper insights into the issue raised in Research Question 2, we perform an in-depth analysis of contextual knowledge, revealing that the presence of noisy and irrelevant contexts hinders models from effectively capturing the desired question-context-answer correlation (§5).

2.   Related Work

2.1.   Faithful Natural Language Generation

Faithful natural language generation (NLG) aims to generate text that is both faithful and consistent with the input information, while avoiding hallucination (Li et al., 2022b; Ji et al., 2022). In recent years, there has been a growing interest in understanding factual errors in summarization (Pagnoni et al., 2021; Ladhak et al., 2022; Tang et al., 2022) and machine translation (Müller et al., 2020; Raunak et al., 2021). Additionally, there have been studies focusing on knowledge faithfulness in question answering (Krishna et al., 2021; Mahapatra et al., 2021; Longpre et al., 2021) and dialogue response generation (Honovich et al., 2021; Dziri et al., 2022). For more details, we refer readers to the surveys (Li et al., 2022b; Ji et al., 2022). Although factoid hallucination has been extensively studied, our work focuses on a broader scope by considering non-factoid information, such as debates and opinions.

2.2.   Context Transfer

Context transfer in NLG involves models adapting to dynamically provided information rather than relying solely on pre-learned parameters. This aspect has been explored in studies on Wikipedia writing by Prabhumoye et al. (2019) and West et al. (2022), investigating the model grounding ability. Furthermore, several works have addressed question answering in the context of dynamic knowledge (Min et al., 2020; Longpre et al., 2021; Zhang and Choi, 2021; Chen et al., 2021; Wang et al., 2022; Liska et al., 2022; Kasai et al., 2022). The most similar work is Longpre et al. (2021), which focused on entity-based knowledge conflict and was under the open-domain setting. However, we investigate long-form question answering (LFQA), where we transfer the entire knowledge text rather than solely editing entities. All transferred knowledge remains relevant and aligned with the real world, as false contextual information may conflict with pre-learned knowledge and potentially induce hallucinations in the model.

3.   Methods

3.1.   Task: Question Answering under Context Transfer

Context transfer necessitates the model’s ability to generate a novel answer based on newly acquired knowledge for the same question during training. To begin, we employ a dataset $D$ consisting of two partitions, namely $D_{train}$ and $D_{test}$. Our initial step involves training a knowledge-grounded generative model on the training examples $(q_{i},c_{i},a_{i})\in D_{train}$, where $q_{i}$ represents the question, $c_{i}$ consists of contextual sentences comprising positive ($c_{i}^{+}$) and negative ($c_{i}^{-}$) contextual knowledge, and $a_{i}$ denotes the golden reference answer. Subsequently, the model is evaluated using examples $(q_{j},\hat{c_{j}})\in D_{test}$, wherein the query $q_{j}$ can be found in $D_{train}$, while the contextual knowledge $c_{j}$ is transferred to $\hat{c_{j}}$.

Our primary focus lies in abstractive long-form question answering. We consider entity-based question answering to be straightforward, as hallucination can be mitigated or even resolved through extraction-augmentation and post-editing techniques. To construct a relevant benchmark, we utilize query-based summarization data from Debatepedia (Nema et al., 2017), primarily due to its highly abstract nature and natural conditions for context transfer. In contrast to previous research (Longpre et al., 2021), we adopt a more natural setting where the transferred contextual knowledge is factual as well. Furthermore, we ensure that the questions are answerable, considering it a necessary requirement. This precaution is taken because we have observed that models tend to generate hallucinatory responses when the contextual knowledge does not contribute to answering the question effectively.

3.2.   Measure: Margin Failure Rate

As illustrated in Figure 1, the trained model exhibits a failure in grounding transferred contextual knowledge, resulting in the generation of answers that are not properly aligned with the given contexts. This phenomenon is referred to as a grounding failure of context transfer.

To determine whether a predicted answer $\hat{a}$ represents a grounding failure of context transfer, we introduce the concept of margin grounding failure ($\mathcal{MF}$) as follows:

$$\mathcal{MF}(\Phi)=\left\{\begin{aligned} &1,\Phi(\hat{a},r_{train})>m\cdot\Phi(\hat{a},r_{test})\\ &0,\Phi(\hat{a},r_{train})\leq m\cdot\Phi(\hat{a},r_{test})\end{aligned}\right.$$ (1)

where $m$ represents the hyperparameter margin, and $\Phi$ is a basic metric (e.g. ROUGE) to measure the similarity between the predicted answer $\hat{a}$ and golden reference $r$. The reference $r$ comes from either the train or test set ($r_{train}$ from the train set or $r_{test}$ from the test set), which can be the golden answer or the contextual knowledge.

It is important to note that grounding failure is a binary label assigned to each case. To statistically probe the faithfulness over the test set, we propose to measure the percentage of grounding failure of context transfer. So the margin failure rate ($\mathcal{MFR}$) is defined as:

$$\mathcal{MFR}(\Phi)=\frac{1}{N}\sum_{i=1}^{N}\mathcal{MF}_{i}(\Phi).$$ (2)

In this work, we use BERT-SCORE (Zhang et al., 2020a) as our basic metric $\Phi$. For our experiments, we set the margin $m$ to a value of $1.25$ based on intuition, which has a relatively strong correlation with Pearson Correlation of $0.43$ with human evaluation on our development set.

4.   Results

Model	Decoding Strategy
	Greedy	Beam Search
T5${}_{small}$	7.69	8.19
T5${}_{base}$	7.53	6.19
BART${}_{base}$	9.20	10.87
BART${}_{large}$	7.86	8.36
BART${}_{large-xsum}$	8.03	7.19
FiD (T5${}_{small}$)	11.37	9.53
FiD (T5${}_{base}$)	11.04	10.03
FiD (BART${}_{base}$)	13.88	12.71
FiD (BART${}_{large}$)	10.03	8.86
FiD (BART${}_{large-xsum}$)	15.38	14.55

Table 1: The $\mathcal{MFR}$(BERT-Score) results of different models. We generate text by greedy and beam search (beam=4) decoding strategy.

In this study, we present the outcomes obtained from two prominent state-of-the-art sequence-to-sequence (seq2seq) pre-trained models, namely BART (Lewis et al., 2020a) and T5 (Raffel et al., 2020), in the context of question answering (QA) tasks. Besides the vanilla transformer architecture, we also incorporate the FiD method  (Izacard and Grave, 2021) owing to its efficient and effective utilization of extensive document collections. The model selection process is based on the ROUGE-L score achieved on the development set.

All models have memory hallucination under context transfer.

The $\mathcal{MFR}$(BERT-Score) results of various models under context transfer are presented in Table 1. It is observed that all the models exhibit the phenomenon of memory hallucination during context transfer, albeit to varying degrees. The choice of decoding strategies does not appear to have a significant impact on the generation of hallucinations. Specifically, the FiD method demonstrates a higher occurrence of context transfer grounding failure compared to the vanilla transformer. This can be attributed to the fact that FiD has a tendency to memorize the question-answer pairs, as the questions are duplicated for each context.

5.   Analysis

Figure 2: The influence of the scale of contextual knowledge and training step on $\mathcal{MFR}$(BERT-Score).

In this section, we endeavor to elucidate the intricate interplay between causality and its impact on model faithfulness within the realm of context transfer. To this end, we embark upon a series of rigorous experiments, wherein we manipulate contextual factors from various perspectives, in order to derive meaningful insights. We conduct all the analysis on FiD(BART${}_{large-xsum}$).

Impact of Contextual Knowledge Scale

We examine the effect of varying the scale of contextual knowledge on the performance of FiD (BART${}_{large-xsum}$) as measured by the $\mathcal{MFR}$(BERT-Score). It becomes evident that the $\mathcal{MFR}$ value increases proportionally with the expansion of the context scale (Figure 2). This surplus of noisy contexts hampers the model’s ability to ground itself in accurate knowledge and introduces confusion during the generation process, as elaborated upon later in Figure 3. Therefore, it becomes crucial to strike a balance between the quantity of information retrieved and the presence of noise, particularly in practical applications where obtaining more knowledge through an imperfect retriever holds significance. Furthermore, it is worth noting that training the model for an extended duration may lead to overfitting on question-answer spurious correlations. Notably, the $\mathcal{MFR}$(BERT-Score) can reach as high as $20$ after a mere $600$ training steps, equivalent to approximately four epochs.

Figure 3: The $\mathcal{MFR}$(BERT-Score) results over different settings of contexts.

Impact of Irrelevant Noisy Context

The presence of irrelevant noisy context can have a detrimental effect on faithful generation during both the training and testing phases. In our experiments, we explore different settings of contextual knowledge using the T5${}_{base}$ and BART${}_{large-xsum}$. During the training process, we introduce negative contexts using two different methods: retrieval-based methods (referred to as Hard-Neg) nd random sampling (referred to as Rand-Neg). For testing, we consider two scenarios: transferring only the positive context while kee** the negative contexts unchanged (referred to as transfer${}_{pos}$), or transferring both the positive and negative contexts by replacing the latter with random ones (referred to as transfer${}_{all}$). The detailed settings are as follows:

• None Negative Contexts (None-Neg): Only positive contextual knowledge is provided during training. During testing, we transfer only the positive knowledge (transfer${}_{pos}$).

• Hard Negative Contexts (Hard-Neg): In this setting, we provide the positive contextual knowledge along with retrieved hard negative knowledge using BM25. This setting is more realistic as it involves retrieving external knowledge in an open domain. During testing, transfer${}_{pos}$ refers to transferring only the positive knowledge, while transfer${}_{all}$ refers to transferring both the positive and negative knowledge, with the negative knowledge being randomly sampled.

• Random Negative Contexts (Rand-Neg): Similar to the Hard-Neg setting, we provide the positive contextual knowledge, but pair it with randomly sampled negative knowledge. The testing scenarios (transfer${pos}$ and transfer${}_{all}$) remain the same as in the Hard-Neg setting.

The final comparative results are presented in  Figure 3. Notably, there is a drop on $\mathcal{MFR}$(BERT-Score) for the FiD architecture when tested on transfer${}_{all}$, specially trained on hard negative contexts. The presence of hard negative contexts poses a challenging confounding factor, as it may induce models to learn spurious correlations, given that retrieved knowledge is often more relevant to the question than sampled knowledge. Furthermore, our findings align with the conclusions drawn from Figure 2, indicating that the inclusion of negative contexts significantly increases the occurrence of margin grounding failure. However, it is worth noting that the vanilla transformer architecture exhibits robustness against negative contexts, displaying insensitivity to contextual disturbance. Upon comparing transfer${}_{pos}$ with transfer${}_{all}$, we observe that the model unintentionally grounds its answers on irrelevant knowledge when negative contexts are transferred, leading to unexpected changes in the generated answers.

6.   Conclusion

This study endeavors to explore the phenomenon of memory hallucination in the realm of context transfer. Our investigation entails the comprehensive examination of multiple models, unveiling potential deficiencies in their ability to faithfully align contextual knowledge. Furthermore, our research emphasizes the pivotal role played by context in the manifestation of hallucinations during both training and testing phases. Despite the apparent rarity of memory hallucination, it represents a critical concern that demands attention for the attainment of veracious natural language generation in practical settings. We anticipate that this research will contribute to a more profound comprehension of the faithfulness of generative models.

Limitations

Benchmark Dataset

Acquiring suitable datasets for long-form abstractive Question Answering (QA) in the context of context transfer poses a significant challenge. Although Debatepedia may initially seem appropriate for such experiments, the reliability of its data scale and quality is questionable, thereby limiting our ability to investigate the factors that influence answer faithfulness. We anticipate that future research will explore additional domains and levels of context transfer, expanding the scope of investigation.

Evaluation Metrics

Existing automatic evaluation metrics Existing automatic evaluation metrics demonstrate limited correlation with human evaluations. Therefore, it is crucial to propose an alternative methodology for systematically assessing large-scale results, with the aim of reducing the variance inherent in small-scale data.

Evaluation Models

Owing to constraints in resources, comprehensive experimentation on the prevalent large language models, has not been undertaken. Nonetheless, we have intentions to incorporate experiments pertaining to large language models in our future endeavors, contingent upon the feasibility thereof.

Faithfulness Improvement

The primary goal of faithfulness probing is to establish a generative model that faithfully incorporates and aligns with the provided context. Nevertheless, this work lacks methodologies to enhance the faithfulness of generative models. Consequently, we try to advance this investigation by exploring the causal factors behind hallucination and proposing viable solutions to address this intricate challenge.

References

Appendix A Benchmark Construction

Unlike previous work (Longpre et al., 2021), we follow the more natural setting where the transferred contextual knowledge is also factual. Besides, we make the question answerable as a necessary condition. Because we find the models prefer to generate hallucination when the contextual knowledge does not contribute to answering the question.

To construct long-form QA data, we reuse Debatepedia(Nema et al., 2017), an abstractive summarization data, to supply our experiments. We choose this data due to its high abstractiveness and natural context transfer condition. We observe that there are lots of lexically similar examples, so we deduplicate examples whose Levenshtein distance is less than 4. This filtered dataset satisfies the format of $(q_{i},c_{i}^{+},a_{i})$, and there are lots of questions paired with different contextual knowledge and answer. The examples with the same question are gathered, and one of them with the most distinctive answer is split into the development set. To enrich the contextual information of every case, we apply BM25 to retrieve negative knowledge $c_{i}^{-}$ from the whole dataset contexts via the question. Both relevant $c_{i}^{+}$ and irrelevant $c_{i}^{-}$ contexts are merged into $c_{i}$. Because if there is only $c_{i}^{+}$, the question $q_{i}$ is meaningless to position the positive context. In our basic setting, the contexts consist of 1 positive $c_{i}^{+}$ plus four negative $c_{i}^{-}$. The final processed dataset contains 2,549 training examples, 631 validation examples, and 598 test examples.

Appendix B Experimental Setting

Figure 4: The Pearson correlation of margin failure ratio from basic metrics with different margins.

Parameter	Value
Learning Rate	$5\times 10^{-5}$
Batch Size	$16$
Accumulation Steps	$1$
Total Step	$4500$
Warmup Step	$150$
Evaluate Step	$150$
Weight Decay	$0.0$
Input Maximum Length	$512$
Output Maximum Length	$100$
Beam Size	$4$

Table 2: The experimental setting details. *Beam Size is the hyper-parameter of text generation in development and testing, while other parameters contribute to model training.

We implement all the models using Pytorch (Paszke et al., 2019) and Transformers (Wolf et al., 2020) toolkit. The training and evaluation hyper-parameters are presented in Table 2. We use Adam optimizer(Kingma and Ba, 2015) with the linear scheduler. All the training is started from the same random seed for a single round. We choose the best model by ROUGE-L score on the development set.

All the models are trained on a single NVIDIA V100 GPU with 32GB memory. Training BART-Large, BART-Large-xsum, FiD(BART-Large), FiD(BART-Large-xsum), T5-base, FiD(T5-base) takes approximately 3 hours. Training BART-base, FiD(BART-base), T5-small, FiD(T5-small) takes less than 1 hour.

Appendix C Meta Evaluation of MFR

We manually evaluate the grounding failure of context transfer on a small scale from test data in order that we can measure the Pearson Correlation between MFR and human labels. We ask two postgraduate students who major in natural language processing to manually evaluate the results. We also explain to them about memory hallucination under context transfer. We choose to label the generated results from FiD(BART-Large-xsum), as we observe this model hallucinates more than others. Human evaluation for more models is planned for future work. We only label the examples whose generated answers get ROUGE-1 score of more than $40$ with the references in training data rather than all the examples in the test set. We believe only these cases could be hallucinated memory from training data. Notice that we only consider memory hallucination, which comes from training(fine-tuning phrase), while other hallucinations may also occur but are not taken into account. The final labelled data consist of $598$ items with only $22$ memory hallucination. Some case studies are presented in Table 3.

Figure 5: The Pearson correlation of margin failure ratio from each metric and human evaluation.

We measure the Pearson correlation between different versions of MFR and human evaluation. We take the basic metrics $\Phi$ from two perspectives: the similarity with golden answers; the faithfulness to contextual knowledge. Concretely, for basic metrics of answer similarity, we use ROUGE(-1/L) and BERT-SCORE (Zhang et al., 2020a); for basic metrics of knowledge faithfulness, we use Density(Grusky et al., 2018) and NLI-Score. As depicted in Figure 5, all automatic metrics are only a little related to each other, except MFR(ROUGE-1) and MFR(ROUGE-L). There is even little relationship between MFR(NLI-Score) and human evaluation. MFR(BERT-Score) performs best correlatively with human evaluation, so we take MFR(BERT-Score) as the main measure in this work.

We also measure the influence of the margin $m$. For each metric $\Phi$ in MFR, we experiment with its margin varying from $1.00$ to $2.00$ with a stripe of $0.01$. As shown in Figure 4, the margin $m$ has a great impact on the human correlation of MFR and different basic metrics achieve the best performance at different margins. Although the intuitively chosen margin $m=1.25$ is not the perfect hyperparameter of BERT-Score, it still has a relatively strong correlation with Pearson Correlation of $0.43$.

Testing Data	Training Data	R-L	Label
QUESTION: genocide ? can the violence in darfur be considered genocide ? CONTEXT: joschka fischer . former german foreign minister and vice chancellor from 1998 to 2005 . “ the eu must act in darfur . targeted sanctions would be a real step towards stop** the killing . ” april 19th 2007 - “ … there insufficient political will for an international force [ in darfur ] … ” GOLDEN ANSWER: there is insufficient political will for military intervention in darfur PREDICTED ANSWER: the violence in darfur could be considered genocide.	QUESTION: genocide ? can the violence in darfur be considered genocide ? CONTEXT: genocide is defined by most to include the systematic murders of a group of peoples as well as deliberate displacement and abuse . more than # # people have died since # with other estimates ranging up to # # according to amnesty international and the un . over # million people have become displaced and many are in danger of starvation due to lack of water and food . conclusively darfur is the worst humanitarian abuse in africa . to the extent that the janjaweed is systematically overseeing this mass-murder and to the extent that the government is involved in supporting the janjaweed darfur ’s crisis can be considered a genocide . GOLDEN ANSWER: the violence in darfur could be considered genocide	$22.22/100.00$	True
QUESTION: changing menus : will mandatory calorie counts compel restaurants to improve menus ? CONTEXT: restaurants that get caught under-reporting calories on their menus may face not only fines from the government but also significant pr problems as stories of their manipulations reach and turn-off their customers . GOLDEN ANSWER: restaurants will not under-report calories and risk pr backlash . PREDICTED ANSWER: restaurants under-report calories on menus	QUESTION: changing menus : will mandatory calorie counts compel restaurants to improve menus ? CONTEXT: “ calorie disclosures fail to weigh whole enchilada ” . wall street journal . july 8 2009 : “ scripps television stations sent several menu items to testing labs and found some big deviations from posted calorie content most of them making menu items appear healthier than they are . for example two tests of applebee ’s cajun-lime tilapia meal found about 400 calories compared with the posted total of 310 . ” this means that restaurants may simply choose to lower their reporting of calories instead of actually lower the calories in the foods they are serving . GOLDEN ANSWER: restaurants frequently under-report calories on menus	$42.86/90.91$	False
QUESTION: wealthy : is a progressive tax system fair to the wealthy ? CONTEXT: david n. mayer . “ wealthy americans deserve real tax relief on principle ” . ashbrook center . october # - “ there is no correlation between the amount of taxes an american pays and whatever benefits if any he receives ; indeed a wealthy person may get fewer government services than a poorer person . ” GOLDEN ANSWER: the rich do not necessarily benefit more from taxes/system PREDICTED ANSWER: progressive tax system unfairly benefits the wealthy	QUESTION: wealthy : is a progressive tax system fair to the wealthy ? CONTEXT: it is unfair that people who earn more should pay at a progressive rate . even on a standard rate they already pay more tax because they have a higher taxable income . therefore progressive tax rates are a form of double taxation as higher earners pay tax on more income and then at a high level . this is further unfair to them since high earners are the least likely group to benefit from much taxpayer-funded activity e.g . welfare . GOLDEN ANSWER: flat tax fairly has wealthy pay proportionally more in taxes .	$12.50/23.53$	True
QUESTION: militia : does the # nd amendment secure an individual right to form an independent militia ? CONTEXT: an armed citizenry empowers citizens to protect themselves so that a big government does n’t have to . GOLDEN ANSWER: in order to form a militia citizens require guns and a right to own them PREDICTED ANSWER: the # nd amendment secured an individual right to bear arm for the purpose of self-defense	QUESTION: militia : does the # nd amendment secure an individual right to form an independent militia ? CONTEXT: an armed citizen can places a checking on inappropriate cops power and the emergence of a cops state . GOLDEN ANSWER: # nd amendment secured equally the right of the militia and the individual to arms .	$14.29/42.86$	False

Table 3: Case study of human evaluation. The $X/Y$ in R-L denotes the ROUGE-L score of predicted answer with the golden answer in testing($X$) or training($Y$) data. And Label denotes the human label for memory hallucination under knowledge transfer.