PathReasoner: Modeling Reasoning Path with Equivalent Extension
for Logical Question Answering

After unifying the inputs into the logical rule form, it is natural to exploit the equivalent logic to facilitate the equivalent extension.

From the model view, we propose the reasoning path modeling module. Given the input context, question, and options of one sample, we first unify them into the form of the reasoning path based on 3. The initial representation of instantiated variable set $V=\{V_{1},V_{2},...,V_{K}\}$ and function symbols set $S=\{S_{1},S_{2},...,S_{M}\}$ can be acquired respectively, where $K$ and $M$ are the number of variables and function symbols in the sample.

For the variable $V_{k}$ with token sequence $\{v_{1}^{(k)},v_{2}^{(k)},...,v_{|V_{k}|}^{(k)}\}$, we leverage the LM as the encoder to obtain the token-level embedding $\{\textbf{v}_{1}^{(k)},\textbf{v}_{2}^{(k)},...,\textbf{v}_{|V_{k}|}^{(k)}\}$. Thus, its initial representation $\textbf{V}_{k}\in\mathbb{R}^{d}$ is calculated by the average pooling. We randomly initialize the representations for the function symbol $S_{m}$:

By aligning the variables and function symbol for each atom, we can form the atom embedding sequence $\textbf{A}\in\mathbb{R}^{(M+K)\times d}$. To take the order feature into consideration, we include the position embedding Vaswani et al. (2017) to the input sequence:

where $\mathbf{A}_{i}$ is the embedding of the $i^{th}$ unit in $\mathbf{A}$, which can be either a variable or a function symbol. In this way, PathReasoner implements the sequential representation of logical rules.

To perform message passing over the reasoning paths, we propose a stack of $L$ layer blocks in a similar style of Transformer. Specifically, we feed the input sequence into both the self-attention and the proposed path attention module.

For the self-attention module of the $l^{th}$ layer, we follow the regular method, which projects the input sequence into query $\mathbf{Q}^{(l)}\in\mathbb{R}^{(M+K)\times d}$, key $\mathbf{K}^{(l)}\in\mathbb{R}^{(M+K)\times d}$ and value $\mathbf{V}^{(l)}\in\mathbb{R}^{(M+K)\times d}$ by the projection matrices. Then, the output of the self-attention module can be derived as $\mathbf{H}^{(l)}_{SA}$.

For simplicity, we omit the description of multi-head attention in the main paper, but the selection of head number will be discussed in Appendix E.3.

For the path attention module, we first obtain the interaction matrix $\mathbf{M}^{(l)}_{seq}\in\mathbb{R}^{(M+K)\times(M+K)}$ by self multiplication of the input sequence $\mathbf{A}$. It models the interaction between any two units. Besides, the importance of each unit can be further considered from the perspective of in-atom and cross-atom.

In-atom interaction models the information aggregation within one atom. Take the atom $S_{i}(V_{j},V_{k})$ with two variables $V_{j},V_{k}$ and one function symbol $S_{i}$ as an example ($i$,$j$ and $k$ are index in the input sequence), the attention score can be computed as:

where $\mathbf{S}_{i}^{(l)}\in\mathbb{R}^{d}$ denotes the embedding of function symbol $S_{i}$. $\mathbf{V}^{(l)}\in\mathbb{R}^{d}$ is obtained by averaging the variable embedding $\mathbf{V}_{j}^{(l)}$ and $\mathbf{V}_{k}^{(l)}$. For atom with a single variable, the average step can be omitted. $||$ represents the concatenation between feature vectors. $\mathbf{W}$ is the trainable projection parameters (the same below).

To embed the in-atom attention, we leverage a score matrix $\mathbf{M}_{in}^{(l)}\in\mathbb{R}^{(M+K)\times(M+K)}$:

We define $\mathbf{M}_{in}^{(l)}$ as a symmetric attention matrix, thus there also exist $\mathbf{M}_{in}^{(l)}(j,i)=\mathbf{M}_{in}^{(l)}(i,j)$ and $\mathbf{M}_{in}^{(l)}(k,i)=\mathbf{M}_{in}^{(l)}(i,k)$.

Cross-atom interaction models the message passing over different atoms. For the same variable $V_{p}$ and $V_{q}$ ($p$, $q$ are unit index of the input sequence), the attention score is obtained:

where $\mathbf{V}_{p}^{(l)}$ and $\mathbf{V}_{q}^{(l)}$ are the embeddings of two instantiated variables.

Similar to in-atom attention, we obtain a cross-atom score matrix $\mathbf{M}_{crs}^{(l)}\in\mathbb{R}^{(M+K)\times(M+K)}$:

Since these two attention matrices only model one-order interaction between related units, the long-distance message passing is limited. Also, we extract the atom based on the explicit logical connectives, it ignores the implicit interactions within the logical text. Therefore, we introduce a diffusion aggregation strategy Zhao et al. (2021); Liu et al. (2021b) to achieve high-order attention:

where $N$ is the maximum order number, $\alpha_{i}$ and $\beta_{i}$ are the trade-off coefficients to control the diffusion procedure. In this way, the one-order attention flow can be efficiently diffused to high-order relations. We can update the feature of sequence $\mathbf{H}_{seq}^{(l)}\in\mathbb{R}^{(M+K)\times d}$ through joint utilization of these three attention matrices:

Within each atom, we aggregate the instantiated variable embedding in the function symbol to acquire a sequence of atom embedding, which can be represented as $\{\mathbf{H}_{S_{1}}^{(l)},...,\mathbf{H}_{S_{M}}^{(l)}\}$. Next, we define the reasoning path $\mathbf{H}_{p}^{(l)}\in\mathbb{R}^{d}$ embedding as:

To align the output embedding of the self-attention module, we repeat and stack the reasoning path embedding for $M+K$ times, obtaining the output of path-attention module $\mathbf{H}_{PA}^{(l)}\in\mathbb{R}^{(M+K)\times d}$. Note that the multi-head strategy is also applied in the path-attention module.

We obtain the optimized sequence embedding by adding $\mathbf{H}_{SA}^{(l)}$ and $\mathbf{H}_{PA}^{(l)}$. Following the common practice in the Transformer architecture, we feed the sequence into the feedforward block and obtain the final output $\mathbf{H}_{t}^{(l)}$ of $l^{th}$ layer.

After the respective mean pooling process on $\mathbf{H}_{cls}$, $\mathbf{H}_{t}^{(L)}$ and $\mathbf{H}_{p}^{(L)}$, the three features are concatenated and projected for the final prediction.

The main experiments are conducted on two logical reasoning datasets ReClor Yu et al. (2019) and LogiQA Liu et al. (2021a). To verify the superiority of PathReasoner, we compare it with strong baselines, including RoBERTa-large Liu et al. (2019), DAGN Huang et al. (2021), FocalReasoner Ouyang et al. (2021), LReasoner Wang et al. (2022), AdaLoGN Li et al. (2022), MERIt Jiao et al. (2022), Logiformer Xu et al. (2022), as well as LLMs like text-davinci-003, GPT-3.5-turbo and PaLM 2. All the experiments are conducted with a single GPU of Tesla A100. All detailed experimental settings are listed in Appendix E.3.

The results of comparison experiments are presented in Table 2. Compared with previous SOTA baselines, PathReasoner presents superiority.

In ReClor dataset, PathReasoner outperforms all the graph-based methods. Compared with the SOTA method Logiformer, PathReasoner achieves improvements of 2.00% and 0.60% on the validation and test splits respectively. PathReasoner also shows superiority over all sequence-based methods, especially outperforming MERIt by 2.50% on the test split. Importantly, it surpasses human performance, i.e., 64.10% vs 63.00%, which greatly pushes the boundary of machine reasoning. In LogiQA dataset, PathReasoner still shows competitive performances, improving the SOTA results by 2.46% on the test split. PathReasoner demonstrates excellent performance and generalization in logical reasoning, as evidenced by its consistent results across two benchmarks.

Compared with representative LLMs, PathReasoner exhibits great superiority with a wide margin in the ReClor dataset. Also in the LogiQA dataset, it outperforms both text-davinci-003 and GPT-3.5 with obvious advantages and only falls behind PaLM 2 which is over x1000 in size.

Ablation studies for two main parts EPE and RPM in Table 3. For w/o whole of EPE, we remove the whole part of EPE and only utilize the origin samples for training. The performance witnesses obvious drops of 3.70% and 2.00% on the two datasets respectively. For w/o path filter, we keep all the new paths to generate samples without filtering. The results prove the effectiveness of it with 1.30% and 1.23% gains on the test respectively. For w/o whole of RPM, we ablate the whole RPM and simply leverage the input sequence to predict the answer through a text encoder and a classifier. In this case, the model degenerates to RoBERTa-large baseline with more samples from EPE part. The results prove that the modeling of path significantly enhances the reasoning process.

To deeply verify modules in RPM, we carry out the following ablation studies. For w/o path attention, we remove the path attention module. The performance gains prove that it is key to RPM part. For in-atom att. and cross-atom att., we respectively ablate the attention modeling within atoms and across atoms. The former benefits the ReClor dataset a lot, while the latter is more helpful to the LogiQA dataset. It illustrates that in-atom and cross-atom attention are complementary to each other. For w/o diffusion, we remove the high-order diffusion strategy. Experiments show that the diffusion strategy is also vital to RPM part.

We first analyze the model performances with different lengths of atoms in Fig. 3. The bars represent the number of samples with different atom numbers, while the lines denote the performances with different atom numbers. For both ReClor and LogiQA datasets, PathReasoner maintains a high performance with moderate scale of atoms, which accounts for most samples in both datasets. Confronted with larger sample sizes, the performances decline. We argue that the gaps have been greatly narrowed with the proposed diffusion strategies, compared with previous models.

Secondly, we provide an analysis of the impact of the number of new samples. By controlling the maximum scale of new atom combinations, we can generate different numbers of samples. Fig. 3 shows the model performances under various cases, where the horizontal axis denotes the number of new samples (with & w/o path filter) and the vertical axis is the model performance on the test. On the two datasets, the path filter plays a positive role in reducing redundancy and noise. Additionally, the optimal results are obtained at a moderate scale of new samples, and larger amounts of samples do not always bring gains in performance.

Thirdly, we discuss the model training efficiency in Fig. 3. We make the comparison with the previous SOTA Logiformer on ReClor (left) and LogiQA (right). To make a clear illustration, we report the loss curve with steps (truncated at 0.1). From the results, PathReasoner shows faster convergence speed on both ReClor and LogiQA datasets. Detailedly, PathReasoner achieves 1.66x convergence speed than Logiformer on the ReClor dataset, and it has 1.34x speed on the LogiQA dataset. We provide more in-depth experiments in Appendix F.

PathReasoner is also evaluated on other reasoning tasks to verify the generalization capability in Table 4. The experiments are conducted on Dream Sun et al. (2019) and MuTual Cui et al. (2020), which are multi-turn dialogue datasets requiring complex reasoning. We utilize RoBERTa-Large model and the previous SOTA model Logiformer as baselines. Among all comparison metrics, PathReasoner achieves consistent superiority over them. Compared with Logiformer, PathReasoner outperforms it with 3.08% in the test split of Dream, 0.89% of the R@1 metric of MuTual and 1.81% of the R@1 metric of MuTual⁺. It demonstrates that PathReasoner can well generalize to different reasoning tasks. Also, other generalization experiments on EPE module and zero-shot settings are included in Appendix G,H.

We provide the analysis for the interpretability of PathReasoner in Figure 4. In the successful case, PathReasoner correctly extracts the variables from the text and forms the reasoning path. In particular, We present the path attention map from RPM part to check the logical perception capability. Firstly, PathReasoner focuses more on the function symbols (e.g., If and Fact) and question sentences (i.e., variable $F$), with higher attention scores in the map. It verifies that PathReasoner is equipped with the perception of logic and question types. Secondly, the question is to match the logical structure between context and option. The corresponding atoms (e.g., $\texttt{If}(D,E)$ and $\texttt{If}(J,K)$) are considered together in the module. It illustrates that PathReasoner is good at understanding the question and reasoning over paths.

In the failure case, PathReasoner wrongly categorizes the variable with different semantics together to $A$ which leads to the mistake. It demonstrates that the variable extraction in PathReasoner is not good at distinguishing the minor difference, which has space for improvement.