CoUDA: Coherence Evaluation via Unified Data Augmentation

To construct samples that have relatively poor global coherence, we disrupt the original appropriate organization of sentences in $D$. Concretely, we shuffle the order of sentences in $D$ to effectively disrupt its global coherence. As illustrated in Figure 2(a), by shuffling $D=\{s_{1},s_{2},s_{3}\}$, we can construct a negative sample $D_{g}^{-}=\{s_{3},s_{1},s_{2}\}$.

Local augmentation aims to construct samples with relatively poor local coherence using the original discourse $D$. Intuitively, we can realize it by replacing a sentence $s_{k}\in D$ with a substitute $s_{k}^{\prime}$ that is incoherent with the leftover discourse $D\backslash s_{k}$. This is based on the insight that, through such a replacement, $s_{k}^{\prime}$ will decrease local coherence of the discourse by introducing an incoherent transition of attention between sentences.

Subsequently, the most important question is how to find a suitable $s_{k}^{\prime}$ in practice. However, most prior studies introduce such incoherent elements via heuristic rules, resulting in $s_{k}^{\prime}$ that has very weak relevance or even irrelevant with the remaining discourse $D\backslash s_{k}$. For example, INSteD (Shen et al., 2021) obtains $s_{k}^{\prime}$ by extracting sentence of the highest n-gram overlap with $s_{k}$ from another discourse. As a result, their introduced local augmentation samples are too easy to train a powerful coherence evaluator.

To construct samples with a higher level of local incoherence, we propose to construct $s_{k}^{\prime}$ in a generative way. Specifically, we train a generative augmentor $G$ to reconstruct $s_{k}$ based on $D\backslash s_{k}$ and use its generated sentence $s_{k}^{\prime}\sim G(s_{k}|D\backslash s_{k})$ to replace $s_{k}$. The strong performance of pretrained generation model will ensure $s_{k}^{\prime}$ to meet the basic standard of fluency and relevance with regard to $D\backslash s_{k}$. Meanwhile, due to the intrinsic limitation of autoregressive generation, the reconstructed $s_{k}^{\prime}$ will frequently be incoherent with $D\backslash s_{k}$, making it possible to construct negative samples in a generative way. To further ensure that $s_{k}^{\prime}$ conveys the local incoherence we expect, we design two controlling mechanisms during the inference of $G$. These two mechanisms, context truncation and coherence filtering , constraints $s_{k}^{\prime}$ to be neither too strong (perfectly coherent with $D\backslash s_{k}$) nor too easy (the incoherence that is too obvious). Overall, by replacing $s_{k}$ with $s_{k}^{\prime}$, we construct a much stronger negative sample, which conveys high-level local incoherence while maintaining the basic relevance and fluency with $D\backslash s_{k}$. In the following, we will introduce our generative augmentor, context truncation and coherence filtering in details.

We perform meta-evaluation on the proposed metric model in two task settings, i.e. pointwise scoring and pairwise ranking.

For pointwise scoring, we evaluate model performance on SummEval (Fabbri et al., 2021), which is a meta-evaluation benchmark for summarization that contains 100 articles with summaries generated by 16 different systems. For each summary, it offers human annotated scores in terms of fluency, coherence, consistency, and relevance.²²2In this paper, we focus on discourse coherence, so we neglect coherence evaluation datasets on dialogue.

In pairwise ranking, we evaluate model performance on INSteD (Shen et al., 2021), which is an intruder sentence detection dataset constructed using discourses from CNN and Wikipedia. We denote these two parts as INSteD-CNN and INSteD-Wiki. In this dataset, incoherent discourses are created by randomly substituting a sentence with another one selected using n-gram overlap from different discourses.

Though more applicable in real scenarios, few work in coherence evaluation has pioneered in pointwise scoring. For a comprehensive performance comparison, we include baselines models from three categories: 1) Pairwise Coherence Evaluators: UNC (Moon et al., 2019) and MultiNeg (Jwalapuram et al., 2022). UNC captures different levels of coherence via a LSTM-based Siamese architecture; MultiNeg³³3The original MultiNeg model is backboned with XLNet and trained on the WSJ dataset. For fair comparison, we retrained this model from ALBERT-xxlarge, using the same part of Wikipedia and CNN data. Notably, due to its use of two encoders, MultiNeg has twice the number of parameters compared to CoUDA. mines hard negative samples constructed via sentence shuffling to train pairwise coherence ranking models. 2) General Evaluators: BartScore (Yuan et al., 2021), UniEval (Zhong et al., 2022). BartScore treats text evaluation as a generation task, utilizing BART to assign quality scores for a specific dimension. UniEval reframes text evaluation as a Boolean Question Answering task. Backboned with T5, it is trained with rule-based local augmentation for coherence evaluation. 3) Large Language Models: G-Eval (Liu et al., 2023) uses LLMs with chain-of-thoughts to assign quality scores. We experiment with two versions using GPT-3.5-Turbo / GPT-4, respectively denoted as G-Eval-3.5 / 4. We include more details about using UniEval and G-Eval in Appendix C and  D, respectively.

Our metric model utilizes ALBERT Lan et al. (2020) as the backbone, benefiting from its sentence order prediction task during pretraining to capture information flow between sentences. Specifically, we use ALBERT-xxlarge with a total of 233M parameters. We set batch size to 32 and learning rate to $1e^{-5}$. Convergence is reached within 3,000 steps. We use the best performing checkpoint on the validation part of synthetic data. Details about generative augmentor are presented in Appendix A. In terms of hyperparameters $\lambda$ and $\delta$, we simply set both of them to $0.5$.

Table 1 presents the sample-level and dataset-level correlations of each model with human ratings on SummEval. Since UNC and MultiNeg are trained through pairwise ranking, their performance on for pointwise scoring is relatively limited. BartScore and UniEval are general evaluators for multiple dimensions such as informativeness and coherence. The former lacks specific training for these dimensions, leading to lower performance, while the latter gain significant improvement through tailored training for coherence. However, UniEval still relies on heuristic rules for augmentation, resulting in limited improvements. The third block presents the results of G-Eval-3.5 and G-Eval-4, built upon GPT-3.5-Turbo and GPT-4, respectively. Since there are no exact description of how many parameters GPT-3.5/4 takes, we estimate them as >10B and >100B.

Among baselines models, G-Eval-4 achieves highest correlation with human ratings, followed by UniEval, which demonstrates strong performance, even surpassing G-Eval-3.5. Compared with UniEval, CoUDA consistently shows its superiority on both sample-level correlation (+3.3/+4.3/+2.4 in $\rho,r,\tau$) and dataset-level correlation (+6.9/+8.7/+5.4 in $\rho,r,\tau$). With only 233M parameters, it also surpasses G-Eval-4 in both sample-level Spearman and Kendall correlations by 1.8 and 0.3 points, respectively. This remarkable improvement consolidates the efficacy of our designing criteria. Additionally, we notice that performance gain in dataset-level correlation is much greater than that of sample-level.

Table 2 presents each model’s pairwise ranking accuracy on INSteD-Wiki and INSteD-CNN. Both MultiNeg and UNC achieves impressive accuracy. We suppose it is because they are exactly trained using the pairwise ranking setup. UniEval also achieves competitive results, which means that specialized training for coherence greatly enhances model performance. Surprisingly, G-Eval-3.5 obtains merely above chance accuracy on INSteD-Wiki, indicating that current LLMs are unreliable in pairwise ranking tasks, necessitating further investigation and attention from researchers. Our CoUDA, though not directly trained under pairwise ranking settings, achieves best results on both INSteD-CNN and INSteD-Wiki, with a performance gain of 2.2 and 1.8 points, respectively.

Coherence evaluation measures the organization and structure of a discourse. Due to the paucity of human-annotated training data, previous work has mainly focused on two synthetic tasks: permutation detection and sentence intrusion detection. Permutation detection task Barzilay and Lapata (2005); Elsner et al. (2007); Barzilay and Lapata (2008); Li and Jurafsky (2017) requires the model to distinguish original discourse from its sentence shuffled version. Sentence intrusion detection task Shen et al. (2021) determines whether a discourse contains an intruder sentence from another discourse.

A series of methods have been proposed for these synthetic tasks. Barzilay and Lapata (2005, 2008) introduced the popular entity-based model using Centering Theory Grosz et al. (1995). It was further extended to combine with entity-specific features Elsner and Charniak (2011), convolutional neural networks Tien Nguyen and Joty (2017), and graph neural networks Mesgar et al. (2021). Jwalapuram et al. (2022) attempted to improve model generalization by training their model purely through self-supervision, with negative samples mined from permutation space. Instead, we propose to improve evaluation performance by unifying different aspects of discourse coherence, as inspired by linguistic theory of discourse structure (Grosz and Sidner, 1986). UNC (Moon et al., 2019) captured different levels of coherence via a Siamese architecture that involved bi-linear projection and lightweight convolution-pooling. By contrast, we address this from the perspective of data augmentation rather than model architecture.

We denote evaluators capable of assessing multiple quality dimensions by altering input and output contents Yuan et al. (2021), or adopting different formulas Scialom et al. (2021); Zhong et al. (2022) as general evaluators. A leading trend is to utilize generation model for quality assessment, such as BartScore Yuan et al. (2021), UniEval Zhong et al. (2022). Apart from that, DiscoScore Zhao et al. (2022) compared the focus matrix between the candidate and the reference to calculate the overall quality score.

With the rise of large language models (LLMs), there has been a growing tendency to use LLMs for evaluation purpose (Wang et al., 2023a; Fu et al., 2023; Wang et al., 2023b; Liu et al., 2023). Wang et al. (2023a) adopted ChatGPT for NLG evaluation and achieved competitive results in terms of correlation with human judgments. Liu et al. (2020) used LLMs with chain-of-thought and a form-filling paradigm to assess the quality of text.