Controllable Decontextualization of Yes/No Question and Answers into Factual Statements

Abstract

Yes/No or polar questions represent one of the main linguistic question categories. They consist of a main interrogative clause, for which the answer is binary (assertion or negation). Polar questions and answers (PQA) represent a valuable knowledge resource present in many community and other curated QA sources, such as forums or e-commerce applications. Using answers to polar questions alone in other contexts is not trivial. Answers are contextualized, and presume that the interrogative question clause and any shared knowledge between the asker and answerer are provided.

We address the problem of controllable rewriting of answers to polar questions into decontextualized and succinct factual statements. We propose a Transformer sequence to sequence model that utilizes soft-constraints to ensure controllable rewriting, such that the output statement is semantically equivalent to its PQA input. Evaluation on three separate PQA datasets as measured through automated and human evaluation metrics show that our proposed approach achieves the best performance when compared to existing baselines.

1 Introduction

Polar or Yes/No questions [12] represent one of the main question types, where the answers can be binary, confirming the interrogative clause in the question, with the possibility of containing embedded clauses that may precondition the proposition in the question, or answers can be implicit altogether [21]. The examples below show some manifestations of polar questions and answers.

On the Web, polar question and answers (PQA) are present in forums, e-commerce pages [29], and on other search related applications [6, 18]. This human curated knowledge remains largely untapped, mainly due to the fact that answers are highly contextualized w.r.t their questions, and often are framed in a personalized language style. Using such text to answer similar questions, or use them for other applications such as voice-assistants, remains challenging. Figure 1 shows an example of a PQA along with a target decontextualized answer. The original answer alone is highly ambiguous, and it is not clear what its subject is.

Figure 1: Example of an input PQA and the desired rewritten answer into a succinct factual statement.

While most research has focused in answering polar questions [6, 28], highlighting difficulties of the answering part, no work is done in decontextualizing answers to polar questions. In this work, we propose the task of rewriting answers to polar questions (PAR), by decontextualizing them and rewriting them into succinct factual statements (cf. Figure 1), which allows us to leverage the knowledge in PQA data to answer similar questions or questions of different shapes such as wh-*, where the expected answers are more varied, and furthermore given that they are succinct, answers are interpretable out of their original context and can be used for other downstream applications [5].

We propose a controllable rewriting approach (SMF), which for an input PQA generates a decontextualized statement framed in a factual language style using 2nd person narrative. SMF introduces a novel soft-constraints mechanism that allows us to achieve controllable rewriting, where for a given set of automatically extracted constraints from its input, it ensures constraints satisfaction in the generated statement.

We manually create a dataset of 1500 $\langle$PQA, factual statement$\rangle$ pairs used for training and evaluating different models for the newly introduced task of PAR. The data is focused on the e-commerce domain [29], covering a wide range of question domains that Amazon customers ask about different products. Our contributions are:

• a novel task of PAR rewriting into factual statements and a novel approach for controllable rewriting through soft-constraints with automatically extracted constraints from input PQA based on QA constituency parse trees;

• a new PAR dataset for training and evaluating models.

2 Related Work

Constrained Text Generation.

Related work in this domain considers mainly the case of sequence to sequence models [30] and of decoder only based models such as GPT2 [24], where the generated output needs to satisfy a given set of manually provided constraints. While pre-trained models such as T5 [25] and BART [19] can be fine-tuned to implicitly capture the co-occurrence between the input and output sequences, they cannot explicitly enforce constraint satisfaction.

To overcome such limitations, controllable rewriting focuses on two types of constraints: lexical or hard constraints, which consist of a single or sequence of words, enforced on the output. [1] propose Constrained Beam Search (CBS), which allows only hypotheses that satisfy constraints. We consider CBS as our competitor, and show that limiting hypotheses has undesired effects in terms of text quality, and further show limitations of CBS, where only single token constraints can be efficiently considered.

Similarly, [10, 15, 2] adapt the inference of seq2seq models to ensure constraint satisfaction. To increase inference efficiency and improve text quality, [32] propose Mention Flags (MF), which trace whether lexical constraints are satisfied. Constraints are explicitly encoded through a MF matrix, which is added into decoder. When a constraint is satisfied its state in the matrix is changed to “satisfied”, thus, providing the model with an explicit signal about constraint satisfaction. Our work is based upon MF, however, with two significant differences: (1) we propose a mechanism to automatically extract constraints, a key component in controllable rewriting, avoiding manual constraint encoding, and (2) we modify MF s.t phrases can be encoded as constraints, and drop the requirements of one to one map** between constraints and the decoded output. We propose soft mention flags SMF, where constraint satisfaction is asserted at the semantic level. Furthermore, through a moving window over the decoded output, we allow our approach to match multi-token constraints to the decoded output. This is another novelty w.r.t MF, where for multi-token constraints all its tokens need to be mapped in the decoded output, thus, allowing our approach to account for paraphrasing.

[17] formulate the decoding process as an optimization problem that allows for multiple attributes to be incorporated as differentiable constraints. [23] propose Constrained Decoding with Langevin Dynamics (COLD), which treats text generation as sampling from an energy function. We compare and show that our approach outperforms COLD.

Yes/No QA.

Most works on Yes/No questions are on answering such as QuAC [4], HotpotQA [34], CoQA [26]. There are several datasets on Yes/No questions  [7, 8, 29] that are used for QA. [29] focus on answering product-related questions and construct the Amazon-PQA dataset with a large subset of Yes/No questions. Our work is complementary, by decontextualizing answers, it allows for answers to be used on other types of questions (e.g. wh-* questions), and additionally since answers are succinct they can be indexed and used in diverse scenarios [5].

Question Rewriting.

Question rewriting in conversational QA [31, 3] rewrites questions by resolving co-references from the conversational context. Such works are not comparable to controllable text generation for two main reasons. First, they do not ensure controllability of the generated text. Second, the context is mainly used to augment a given question in a conversational turn, without changing its framing and syntactic shape, as is the case in our work.

3 Task Definition and Requirements

We define the PAR task of controllable PQA rewriting into factual statements. For an input polar question that is represented by a sequence of tokens $\mathbf{q}=[q_{1},\ldots,q_{n}]$, its answer $\mathbf{a}=[a_{1},\ldots,a_{n}]$, and some context $\mathbf{c}=[c_{1},\ldots,c_{n}]$. The context $\mathbf{c}$ can vary and depending on the domain of PQA data (e.g. it can represent conversation history or in our case some entity title). This input is concatenated into $\mathbf{x}=[\mathbf{q};\langle\text{{SEP}}\rangle;\mathbf{a};\langle\text{{SEP}}\rangle;\mathbf{c}]$, which is fed to a rewrite function $\mathcal{F}$ that outputs the target statement $\mathbf{y}=[y_{1},\ldots,y_{n}]$, namely $\mathcal{F}(\mathbf{x})\rightarrow\mathbf{y}$.

3.1 PQA Syntactic Rewriting Space

Category	Definition	Examples
Explanation	The most fundamental shape and basis of all other answers, it consists of the particle Example 1 yes or	Yes, you can install snapchat on the Samsung Galaxy A20 phone. No, you cannot install snapchat on the Samsung Galaxy A20 phone.
Complement	In addition to the main response, some answers may include an additional embedded clause that provides further related aspect that may interest the asker.	Yes, you can install snapchat on the Samsung Galaxy A20 phone. Also, you can get twitter on it. No, you cannot install snapchat on the Samsung Galaxy A20 phone. Also, you can’t get twitter on it.
Condition	The answer contains an embedded conditional clause [13, 33], which conditions the truthfulness of either the affirmation or negation of the interrogative clause. Such answers appear when the question is not specific enough.	Yes, you can install snapchat on the Samsung Galaxy A20 if it is a smart phone No, you cannot install snapchat on the Samsung Galaxy A20 if it is not a smart phone
Alternative	Similar to explanation, with the difference that the polarity of the answer is negative w.r.t the question, however, an alternative affirmative proposition is suggested by the answerer.	No, you cannot install snapchat on the Samsung Galaxy A20 phone. But you can get twitter on it instead.

Table 1: PAR syntactic rewriting categories along with their definitions. Example statements for both polarities for the question “Can you get snapchat on this phone?” are provided. The highlighted text in the examples shows the embedded clauses for the individual categories.

We now describe the syntactic rewriting space of polar question and answers into factual statements. The main clause of a polar question is an interrogative clause, following specific grammar rules [14], e.g., $q\rightarrow$ AUX NP VP (among many other context free grammar rules).

Answers to polar questions can take several different syntactic shapes [16, 9, 11]. Furthermore, as there is an asynchronous relation between the asker and answerer in online settings, there is often a lack of conversational context, resulting in more elaborate answers. We devise a taxonomy of answer types, which correspondingly determine also the shape of the rewritten statement. Table 1 defines the different categories along with example generated statements for each category.

The cases in Table 1 represent frequent syntactical manifestations of PQAs on the Web, and we will use them as guidance for dataset collection (cf. Section 5) and evaluation (cf. Section 6.2).

4 Approach

Figure 2: Overview of our proposed controllable rewriting approach. In (a) we extract automatically constraints from constituency parse trees, then in (b), the constraints together with the input text that goes to encoder is encoded in the soft-mention flag matrix, which is then provided as input in (c) to the decoder for controllable generation.

Figure 2 shows an overview of our approach, a sequence to sequence (seq2seq) model based on the T5 transformer model [24]. It has two main components that ensure controllable rewriting: (i) automated constraint extraction, and (ii) controllable rewriting through soft-constraints.

4.1 Automated Constraint Extraction

Unlike in MF [32], where constraints are provided manually, we propose an automated constraint extraction approach based on constituency parses of the input PQA.

Our goal in controllable rewriting is for the target statement to contain the interrogative clause from the question, and the corresponding affirmation/negation clauses in the answer, along with the any embedded conditional or alternative clauses. This boils down to two main tasks for constraint extraction: (1) determining the input tokens that must be present in the output, and (2) ensuring that the decoder satisfies such constraints.

Following the syntactic rewriting cases in Table 1, we extract constraints, shown in order of importance. (the algorithm is provided in the paper’s appendix).

Noun Phrases (NP): NPs identify the subject of the question. In the answer they help us identify the matching clause with the assigned polarity and explanation to the question’s preposition. NPs are used as constraints only if they are not embedded as children of non-NP constituents.

Verb Phrases (VP): VPs on the other hand identify the information need as defined by the verb serving as the root of the constituent.

Other Phrases: The rest of the extracted phrases as constraints are prepositional phrases (PP), adverbial phrases (ADVP), and adjective phrases (ADJP). PP and ADJP provide further details about the information in an NP, whereas ADVP provide further information about the verb in the question’s interrogative clause and assert its polarity.

4.2 Soft Mention Flags

To assess if the extracted constraints are satisfied in the generated output, we must account for two factors: (i) input constraints may be expressed differently in the output (paraphrases, synonyms etc.), and (ii) there is no one-to-one map** between input constraints and the generated statement.

We enforce our model to satisfy the extracted constraints in §4.1 by constructing a soft-mention flags matrix $\mathbf{M}_{k\times l}\in\{0,1,2\}$, where rows represent input PQA tokens and columns are the output tokens. For each input token $x_{i}$, $\mathbf{M}$ holds a value between $\{0,1,2\}$, where 0 is for tokens not part of any constraint, 1 for token part of a constraint but not satisfied, and 2 for tokens part of a constraint and satisfied in the output $\mathbf{y}$.

$$\small M_{\mathbf{x}_{i},\mathbf{y}_{:t}}=\begin{cases}0&x_{i}\text{ is not part of a constraint}\\ 1&x_{i}\text{ is not mentioned in }\mathbf{y}_{:t}\\ 2&x_{i}\text{ is mentioned in }\mathbf{y}_{:t}\end{cases}$$ (1)

As in [32], we inject the soft mention flag matrix, $\mathbf{M}$, in the model’s decoder layers. $\mathbf{M}$ is represented through two embedding types: 1) key embeddings, $\mathbf{M}^{k}=E_{k}(M)$ and value $\mathbf{M}^{v}=E_{v}(M)$ where $E_{k}$ and $E_{v}\in\mathbb{R}^{3\times dim}$. These embeddings are injected between the encoder output $\mathbf{h}^{e}$ and the decoder input $\mathbf{h}_{t}^{d}$ in the cross multi-head attention module (cf. Eq 2).

$$\small\begin{split}\text{{CA}}&(\mathbf{h}_{t}^{d},\mathbf{h}^{e},\mathbf{M}^{k},\mathbf{M}^{v})=\\ &F(W_{q}h_{t}^{d},W_{k}h^{e},W_{v}h^{e},\mathbf{M}^{k},\mathbf{M}^{v})\end{split}$$ (2)

where $F$ is a self-attention function with soft mention flag embeddings defined below.

$$\small{F(q,k,v,\mathbf{M}^{k},\mathbf{M}^{v})_{j}=\sum_{i=1}^{l_{x}}\alpha_{i,j}(v_{i}+\mathbf{M}_{i,j}^{v})}$$ (3)

$$\small{\alpha_{i,j}=\text{{softmax}}\left(\frac{q_{i}(k_{i}+\mathbf{M}_{i,j}^{k})^{T}}{\sqrt{dim}}\right)}$$ (4)

$\mathbf{M}$ provides the seq2seq model with explicit signal about the decoded tokens. Whenever input tokens are marked with 0, the model performs standard conditional decoding. Otherwise, if a token is part of a constraint (set to 1 in $\mathbf{M}$), it represents an explicit signal to decode sequences such that the corresponding values in $\mathbf{M}$ are changed to 2.

With the explicit means to signal what part of the input composes a constraint that needs to be met in the decoded output, next, we describe how we establish if a constraint is satisfied, while taking into account that it can undergo syntactic and lexical changes in the decoded output. We propose two strategies to encode constraint satisfaction.

Semantic Constraint Satisfaction:

For an input constraint all its tokens $\mathbf{c}_{i}=\{x_{m},\ldots,x_{n}\}$ are initialized with 1 in $\mathbf{M}$ (i.e., constraints not been satisfied). At each generation step $t$, we assess whether the constraint $\mathbf{c}_{i}$ is satisfied. To ensure constraint satisfaction accuracy, we consider only tokens within a specific window length (with window size equal to $|\mathbf{c}_{i}|$) of preceding tokens in the output $\mathbf{y}_{k,l}$. A constraint is satisfied if the semantic similarity between sim$(\mathbf{y}_{k,l},\mathbf{c}_{i})$, computed as the cosine similarity between the sentence representations [27] of $\mathbf{y}_{k,l}$ and $\mathbf{c}_{i}$ meets two specific thresholds: (1) threshold $a$, where $\text{{sim}}_{t}>a$, (2) threshold $b$ of the difference between the current similarity score and the score in step $t-1$, namely $\text{{sim}}_{t}-\text{{sim}}_{t-1}>b$. Once $\mathbf{y}_{:t}$ meets both thresholds, all tokens of $\mathbf{c}_{i}$ are changed to $2$ in $\mathbf{M}$.

Example:

Table 2 (a) shows an example on the initialization and the updates on the soft mention flag matrix $\mathbf{M}$. For an input sequence $\mathbf{x}=[$The, screen, has, full, touchscreen, function$]$, $\mathbf{c}=$ “has full touchscreen function”, is a constraint that needs to be satisfied in the output. The flags for $\mathbf{c}$ at step 0 are initialized with 1, $\mathbf{M}(\mathbf{c},y_{:0})=[0,0,1,1,1,1]$, given that the output is $\langle\text{{SEP}}\rangle$. The top row shows the similarity score for each step. At step $6$, $\mathbf{M}(\mathbf{c},y_{:0})=[0,0,2,2,2,2]$ since the constraint has been covered in the current output sequence.

sim	0	0.21	0.26	0.28	0.26	0.37	0.85	0.76
	$\langle$ SEP $\rangle$	Dell	Laptop	comes	with	full	touchscreen	.
The	0	0	0	0	0	0	0	0
screen	0	0	0	0	0	0	0	0
has	1	1	1	1	1	1	2	2
full	1	1	1	1	1	1	2	2
touchscreen	1	1	1	1	1	1	2	2
function	1	1	1	1	1	1	2	2

(a)

	$\langle$ SEP$\rangle$	Dell	XPS	can	be	shipped	by	us	to	Brazil	.
We	2	2	2	2	2	2	2	1	1	1	1
can	-	-	-	-	-	-	-	-	-	-	-
ship	-	-	-	-	-	-	-	-	-	-	-
to	-	-	-	-	-	-	-	-	-	-	-
Brazil	-	-	-	-	-	-	-	-	-	-	-

(b)

Table 2: (a) Constraints are marked in bold. Underlined tokens are included in the sliding window that assesses constraint satisfaction. (b) 1st person pronouns are initialized with 2 then changed to 1, when a 2nd person pronoun word is generated.

Factual Style Constraints:

To frame the output statement in a factual style, first person narratives are transformed into a second person narrative. Such a seemingly small change (i.e., 1st to 2nd person pronouns), incurs a series of syntactic rewrite operations required to ensure coherence of the output statement. Table 2 (b) provides an example. In this case, we represent in the $\mathbf{M}$ 1st person pronouns with score 2, and convert them to 1, once the model has generated a second person pronoun in the output. The reason for reverting the order from satisfied constraint to not satisfied is to avoid any confusing behavior between the satisfaction of input constraint extracted through the automated constraint extraction and framing constraints.

5 PAR Data Collection

We now describe our data collection process, which is based on the Amazon Product Question Answers dataset [29], which contains a diverse set of product-specific PQAs generated by customers. The dataset contains 10M questions about 1.5M products, from which we focus only on yes/no questions. Each instance consists of the question text $q$, answer text $a$, and product name $c$, which represents the context in our case.

5.1 PQA Rewriting through Crowdsourcing

We recruited 10 expert annotators to collect ground truth for 1,500 instances. The data is uniformly distributed across the categories in Table 1 and covers 11 domains.

The task is designed to be decomposable into stages to ensure annotation reliability. Annotators follow a series of guidelines (cf.Table 3), allowing them to first map the input PQA into one of the pre-defined categories, after which they determine the answer polarity (step S1), then in S2 the output category is determined (e.g. Alternative), in S3 the anaphoric expressions are replaced with the context information, and finally in S4 the statement is framed in 2nd person narrative.

Q	Does this monitor have a camera?
A	No. But it is a wide screen. I’ve been connecting a game console to it.
C	Dell 27-inch Full HD 1920$\times$1080 Widescreen LED Professional Monitor
S1	No.
S2	No, this monitor doesn’t have a camera. But it has a wide screen and I’ve been connecting a game console to it.
S3	No, the Dell 27-inch Full HD monitor doesn’t have a camera. But it has a wide screen and I’ve been connecting a game console to it.
S4	No, the Dell 27-inch Full HD monitor doesn’t have a camera. But it has a wide screen and you can connect it with a game console.

Table 3: An example to illustrate how annotators perform the Yes/No QA rewriting task.

6 Experimental Setup

6.1 Datasets

PAR: We randomly split the PAR dataset into 1000/100/400 for train/dev/test respectively. PAR is used for training and evaluation.

Reddit: We leverage a sub-forum of Reddit of polar questions and on a randomly sample of 50 QA pairs assess zero-shot generalization performance.

SemEval: From SemEval-2015 Task 3 we randomly sample 50 pairs of Yes/No question and answers for zero-shot evaluation.

6.2 Baselines

Approaches are trained on the PAR dataset in §6.1.

T5 [25]. We use the T5 model as our baseline, which in turn serves as an ablation of SMF without the soft-mention flags module.

CBS [1]. We use T5 with constrained beam search during the decoding phase. The constraints represent phrases extracted in §4.1.

GPT-2 [24]. We adopt GPT2 and fine-tune it on PAR training set.

COLD [23]. A GPT2 based decoding method by sampling from an energy function.

MF [32]. Originally proposed the addition of mention flags into sequence to sequence decoders, and focuses only on lexical constraints. The constraints in this case, contrary to the original paper that are provided as input, here the constraints are extracted automatically (cf. §4.1).

Our Approach – SMF: We distinguish two models of our approach: (1) SMF where for controllability we rely only on the extracted constraints in §4.1, and (2) SMF-Style, where in addition to the extracted constraint, we additionally encode the target style constraints (§4.2).

6.3 Evaluation Metrics

Automated Metrics:

To assess the closeness of the generated statements with respect to the ground-truth statements generated by human annotators, we use BLEU [22], ROUGE [20] and F1-BertScore [35].

Human Evaluation:

Automated metrics cannot capture the nuanced aspects and natural variation of the task. We design two human evaluations, where annotators assess 50 randomly sampled input-output pairs for on:

Statement Syntactic Clause Coverage: we assess if the statement, depending on its category (cf. Table 1) has coverage of the embedded clauses.

Statement Correctness and Coherence: we assess if a statement: (i) contains the correct polarity, (ii) mentions the input context, (iii) framing is in 2nd person narrative, (iv) does not contain information not present in the input PQA; (v) is grammatically correct and coherent, and (vi) is equivalent to its ground-truth counterpart.

7 Experimental Results

Overall Performance.

Table 4 (a) shows the evaluation results for all models on the PAR test set. We observe that our proposed SMF approach achieves the highest performance across all evaluation metrics. A negligible difference is noted between SMF and SMF-Style. However, as we show in §7.1 the SMF-Style attains more coherent statements.

From the baselines, the closest to our approach is MF. However, note here that we adapt MF using our extracted constraints, contrary to [32] where the constraints are provided manually, an approach that does not scale.

Alternatively, if we do not use our proposed automated constraint extraction approach, one could provided as constraints to MF tokens that overlap between the question and answer. Using such constraints causes the performance of MF to drop by 27% in terms of BLEU. This highlights that what composes as input constraints is key, and noisy constraints can yield results worse than seq2seq models without constraints.

CBS reveals that our task does not simply involve copying tokens, but rather involves a series of syntactic and semantic transformations of the input. The difference between CBS and SMF is 7.4% in terms of BLEU and ROUGE-L scores. This shows that enforcing the decoder to output certain tokens has a negative effect, as CBS performs worse than T5. This is another indication that our approach, where the constraint satisfiability is done at the semantic level is appropriate. Equivalent clauses from the input can be transformed and in cases question and answer clauses can be combined into a single clause, hence, causing difficulties to establish such map**s.

COLD and GPT2 achieve poor performance, highlighting the need for larger amount of training to learn the task.

	BLEU	ROUGE-L	BertScore
GPT2	32.8	55.0	93.2
COLD	28.1	51.3	92.0
T5	50.6	67.9	95.3
CBS	45.2	62.7	94.2
MF	51.2	68.5	95.4
SMF	52.6	69.5	95.5
SMF-Style	52.5	68.9	95.4

(a)

		BLEU	ROUGE-L	BertScore
complement	MF	40.3/43.3	68.0/70.1	95.2/95.5
	SMF	45.0/45.8	71.3/71.5	95.6/95.6
	SMF-Style	45.6/46.8	71.1/73.8	95.9/96.0
condition	MF	35.3/36.7	62.0/62.8	94.7/94.7
	SMF	34.5/39.5	61.4/63.9	94.7/95.0
	SMF-Style	36.5/37.9	62.3/64.1	95.0/95.1
alternative	MF	40.8/41.1	69.5/69.9	96.1/96.3
	SMF	44.1/44.8	70.0/71.0	96.0/96.1
	SMF-Style	43.5/44.5	69.2/71.5	96.3/96.4

(b)

Table 4: (a) Overall performance of the different models on the PAR test set. MF, SMF and SMF-Style obtain significantly better results (p$<0.01$ as per t-test) than the rest of the competitors.. (b) Impact of input constraints on output quality. The first score represents the case without the extracted input constraints, while the second score represents the case with constraints.

Out-of-Domain Performance.

Figure 3 shows the out-of-domain performance of SMF and SMF-Style. For the 11 domains in the PAR dataset, we consider a leave-one-out domain evaluation, showing the zero-shot performance. The results show that: (1) both approaches generalize well on out-of-domain data, with most domains obtaining comparable performance to the in-domain performance. (2) comparing SMF and SMF-Style, we note that SMF-Style achieves better performance on statement framing style, implying the effectiveness of style constraints.

Input Constraint Coverage Impact.

Table 4 (b) shows the impact of constraints, namely the presence of constraints as input on the generated output. The results show the impact on different PAR categories, for approaches that make use of the input constraints.

Figure 3: Out-of-domain performance on the test set of the PAR dataset on different domains.

7.1 Human Evaluation

To complement the automatic metrics, we report the results of our two human studies carried out by two human annotators.

Statement Syntactic Clause Coverage:

Table 5 (b) shows the results on the evaluation of statements containing the different clauses according to their PQA syntactic shape. Namely, if an input belongs to the condition syntactic shape, the generated statement should contain the explanation and the embedded condition clause.

Except for CBS and COLD, all approaches do fairly well in incorporating the explanation clause in their output. This is intuitive as this is present in all input PQAs, and contains information that is present in the question and often in the answer as well.

One of the key embedded clauses, the condition clause, represent one of the most challenging scenarios to be decoded into the target statement. This clause has a pivotal role in conditioning the polarity of the answer, and thus, failure to decode this in the statement can lead to erroneous answers provided to a question. SMF approaches achieve the highest scores, with T5 having a 10% point difference. One notable case is that of alternative clause, where the baseline T5 achieves perfect coverage, with MF and SMF following with 92.9% coverage.

Overall, the best coverage is achieved by SMF, with 85% coverage across all types. This shows the importance of extracted input constraints based on our approach in §4.1, where the models MF, SMF, and SMF-Style, achieve highest coverage among all competing approaches.

	Explanation	Complement	Condition	Alternative	Overall
GPT2	91.0	55.6	46.7	64.3	58.0
COLD	65.0	44.4	43.3	35.7	45.0
T5	96.0	75.0	70.0	100.0	79.0
CBS	86.0	66.7	60.0	85.7	72.0
MF	99.0	83.3	66.7	92.9	82.0
SMF	98.0	83.3	80.0	92.9	85.0
SMF-Style	98.0	80.6	80.0	78.6	83.0

(a)

	Polarity	Coverage	Style	Relevance	Syntactic	Coherence	Equivalence
GPT2	0.93	0.94	0.98	0.67	0.84	0.69	0.36
COLD	0.80	0.75	0.98	0.52	0.43	0.42	0.19
T5	0.96	1.0	0.97	0.94	0.95	0.91	0.74
CBS	0.82	0.87	0.95	0.82	0.68	0.75	0.56
MF	0.94	0.98	0.99	0.94	0.98	0.91	0.69
SMF	0.95	0.98	0.93	0.91	0.94	0.92	0.77
SMF-Style	0.97	0.99	0.96	0.96	0.99	0.98	0.76

(b)

Table 5: (a) Clause type coverage on the output statement. (b) Correctness and coherence scores of the generated statements by the different approaches.

Correctness and Coherence Evaluation:

Table 5 (a) shows the human evaluation results for the syntactic correctness and coherence of statements, where several detailed aspects are considered. This study highlights the task complexity and provides an overview of which sub-tasks the different models are able to do reliably.

On polarity, most seq2seq based models, such as T5, MF, SMF and SMF-Style, obtain high polarity accuracy, with SMF-Style achieving an accuracy of 97%. Given that answers are contextualized w.r.t the question and the input context, here the product name, we note that similarly, all T5 based models obtain a good coverage of the context, with T5 achieving 100% coverage. In terms of context coverage, the models are required to chose from the lengthy product names, a succinct phrase that allows the human annotators to identify correctly the product name.

On factual style, MF obtains the highest accuracy, however, at a cost that not always the generated statements are semantically coherent. When we consider SMF-Style, which encodes as constraints the framing style, although the accuracy is lower in terms of outputting statements in 2nd person narrative, in terms of coherence, we obtain the highest accuracy. Coherence, represents a more global measure, which has high importance, given that incoherent statements are not suitable to be provided as answers.

Finally, in terms of equivalence between the generated statement and the ground truth, both SMF and SMF-Style, obtain the highest scores. This shows that our approaches are able to jointly optimize for the numerous sub-tasks of controllable rewriting.

7.2 Evaluation on Other Community PQA

Finally, we evaluate the zero-shot performance of SMF and SMF-Style on Reddit and SemEval w.r.t the statement correctness and coherence.

Table 6 shows that our approach can perform well when applying to other data sources under the zero-shot setting. The results of this setting imply that the modeling of our rewriting task is not simply learning the superficial lexical content, but learning the rewriting strategies to restructure the input PQA regardless of domains. This finding makes our rewriting task and approach promising to generalize to other domains without requiring extra annotations.

	Reddit		SemEval
	SMF	SMF-Style	SMF	SMF-Style
Polarity	1.0	1.0	0.90	0.95
Context Coverage	1.0	1.0	1.0	1.0
Factual Style	0.95	1.0	1.0	1.0
Relevance	1.0	1.0	0.95	1.0
Syntactic Correctness	0.95	1.0	1.0	1.0
Coherence	0.90	0.95	0.95	0.95
Equivalence	0.85	0.80	0.80	0.80

Table 6: Results on Reddit and SemEval datasets.

8 Conclusion

We introduced the task of rewriting Yes/No question and answers into succinct decontextualized statements, and defined several desiderata determining how the input PQA is reorganized and framed into the factual statement. This task enables us to explore knowledge from community PQA, unlocking the highly contextualized answers to answer other question shapes, and furthermore making them retrievable. For this task, with the help of expert annotators, we curated a dataset of 1500 input PQA and the target statements covering 11 domains from Amazon’s PQA dataset.

Next, we introduced an approach for controllable rewriting, achieved through automatically extracted constraints from the input, which are encoded into our approach using a soft mention flag matrix, allowing us at the semantic level to map constraints to the generated statements.

Finally, empirical evaluations showed that our approach outperforms a series of competitors in both automated and human evaluation metrics.

Appendix

Constraint Extraction From Parse Trees:

Algorithm 1 outlines the steps undertaken to extract constraints from the extracted constituency parse trees from the input PQA.

Algorithm 1 Constraint Extraction

0: Yes/No question $Q$, answer $A$.

0:   

1: Extract all NP from the input using constituency parsing tree.

2: Exclude PRON from NP in Step 1.

3: for each noun phrase do

4:    if the parent node is labeled as [‘VP’, ‘PP’, ‘ADVP’, ‘ADJP’] then

5:       Add the phrase that belongs to this parent node to the constraint list.

6:    else

7:       if the parent node is labeled as ‘NP’ then

8:          Add the phrase that belongs to the current node to the constraint list.

9:       end if

10:    end if

11: end for

12: return  constraint list

Constraint Extraction Accuracy:

We sample 100 instances in the PAR dataset which uniformly cover all the different syntactic PAR categories from Table 1, and ask annotators whether: (1) the extracted constraints from the question or the answer capture the question’s intent, and (2) if the extracted constraints cover the different embedded clauses. For the the first part, we see that the extracted constraints from both the question and the answer have a good coverage of 87% on the question’s intent.

• Q1: “Do the extracted question constraints cover fully the question’s intent?” = 87.1%

• Q2: “Do the extracted answer constraints cover fully the question’s intent?” = 87.7%

In the second part, the extracted constraints cover well on explanation and complement. Especially, explanation covers nearly 97% of the cases. A lower coverage is reported for constraints covering the condition and alternative clauses in the answer. This is mainly due to the fact that these clauses use pronouns, thus, increasing the likelihood of missing those simplified constituents. This represents an important future research direction to have a more robust algorithm with better coverage of all answer’s embedded clauses. More specifically, we obtain the following scores: Explanation=0.97, Complement=0.7, Condition=0.52, and Alternative=0.40.

Human Evaluation Analysis:

Two annotators evaluated 50 randomly chosen Yes/No QA pairs. All judgments are on a single scale (i.e., binary). For reliability, each instance is assessed by both annotators. In case of ties, a third annotation was collected.

Statement Syntactic Shape: In this study, the inter-rater agreement rate was 87.5%. This represents a high agreement rate and shows that annotators agree on what embedded clauses are covered in the generated target statement.

Statement Correctness and Coherence: The inter-rater agreement was measured separately on the seven different questions, with an agreement of Polarity=64%, Coverage=64%, Style=65%, Relevance=62.3%, Syntactic=64%, Coherence=62.3%, and Equivalence=52%, respectively.

Overall, the agreement rates are high and similar for most of the questions, with the only exception being Equivalence. In this question, the annotators were asked to assess if two statements represent semantically equivalent information. This yields a lower agreement rate, as annotators may comprehend the statements differently, and additionally the presence or absence of information on one of the statements can cause the annotators to perceive the equivalence differently.