Concept Prerequisite Relation Prediction by Using Permutation-Equivariant Directed Graph Neural Networks

Abstract

This paper studies the problem of CPRP, concept prerequisite relation prediction, which is a fundamental task in using AI for education. CPRP is usually formulated into a link-prediction task on a relationship graph of concepts and solved by training the graph neural network (GNN) model. However, current directed GNNs fail to manage graph isomorphism which refers to the invariance of non-isomorphic graphs, reducing the expressivity of resulting representations. We present a permutation-equivariant directed GNN model by introducing the Weisfeiler-Lehman test into directed GNN learning. Our method is then used for CPRP and evaluated on three public datasets. The experimental results show that our model delivers better prediction performance than the state-of-the-art methods.

1 Introduction

With the continuous advancement of dissemination methods, an increasing number of educational resources are becoming available for people to learn Fischer et al. (2020). Therefore, finding prerequisite relationships among concepts has become an important issue requiring investigation in the field of AI for education Pan et al. (2017); Roy et al. (2019). Generally, this Concept Prerequisite Relation Prediction task, CPRP, is modeled as the link prediction problem in many studies Sun et al. (2022); Roy et al. (2019). Applications of CRPR involve material recommendation Guan et al. (2023), learning path planning Shi et al. (2020), and optimization of problem-solving paths Le et al. (2023).

There are numerous approaches to solving the link prediction problem, including probabilistic models, spectral clustering, evolutionary algorithms, and deep graph learning models Kumar et al. (2020). Currently, graph neural networks (GNNs) have become the benchmark method and presented state-of-the-art performance in link prediction Cai et al. (2021), including many GNN-based solutions to CPRP Roy et al. (2019); Jia et al. (2021); Sun et al. (2022); Mazumder et al. (2023). To improve the capability of GNNs, Long et al. (2022) proposed to pre-train the node features via the method of graph reconstruction, achieving improved performance on two biological prediction tasks and effectively reducing training costs. Chamberlain et al. (2022) proposed to utilize subgraph sketches to pass messages in subgraph GNNs, delivering higher accuracy and lower computation costs in line prediction tasks. This model mitigated the expressive limitations, such as the inability to count triangles and distinguish automorphic nodes. Persistent homology was also adopted in the work of Yan et al. (2021) to extract topological information from graphs, which was integrated with node features to enhance the expressive power of GNNs for link prediction. Besides, the Weisfeiler-Lehman test was introduced by Morris et al. (2019) to improve the capability of differentiating graph isomorphism in undirected GNN learning Huang et al. (2022).

However, the CPRP problem is usually formulated into the directed-link prediction in a directed graph Sun et al. (2022). Rather than the undirected graph, the edges can indicate the prerequisite relation, describing the flow of information from one node to another. The difference makes undirected GNNs not directly applicable to directed graphs. Hence, Salha et al. (2019) designed a new gravity-inspired decoder to extend the graph Autoencoders, while Wu et al. (2019) adopted two weight matrixes for forward edges and backward edges, respectively, to perform message passing between graph nodes. Nevertheless, there is a lack of studies on improving the expressive power of direct GNNs for CPRP.

In this paper, we extended the Weisfeiler-Lehman test-based GNN model, i.e., SpeqNets recently introduced by Morris et al. (2022), into the directed graphs for CPRP. The proposed framework contributes to learning the permutation-equivariant direct GNNs and improving the prediction performance of CPRP by distinguishing non-isomorphism graphs. Experimental results on three public datasets manifest that our method performs better than the state-of-the-art methods Sun et al. (2022); Jia et al. (2021); Roy et al. (2019).

2 CPRP: Concept Prerequisite Relation Prediction

The problem of CPRP refers to predicting prerequisite relations between knowledge concepts involved in learning Sun et al. (2022). For example, one should learn the knowledge concept (KC) of “conditional probability distribution” before learning “Bayesian theory.” As usual, CPRP can be formulated into the directed-link prediction in a KC graph.

2.1 Problem formulation

Denote by $G=(V,E)$ a directed KC graph with a vertex set $V=\{v_{1},v_{2},\ldots,v_{N}\}$, where $v_{i}$ represents a KC in our study, and an edge set $E=\{e_{ij}\}_{1\leq i,j\leq N}$, where $e_{ij}$ is the prerequisite relation $v_{i}\to v_{j}$ meaning that the KC $v_{i}$ is a prerequisite KC $v_{j}$. Denote by $A\in R^{N\times N}$ the adjacency matrix of $G$ with each element being 0 or 1. CPRP on $G$ can be written into

$$\mathcal{P}(v_{p},v_{q})=\mathcal{M}(\mathcal{G}(v_{p}),\mathcal{G}(v_{q}))$$ (1)

where $v_{p}$ and $v_{q}$ are two KCs; $\mathcal{P}$ is the probability of whether the relation $e_{pq}$ exists; $\mathcal{G}$ is a representation model that integrates KC information into a vector; the function $\mathcal{M}$ aims to obtain the existing probability of the prerequisite relation $e_{pq}$.

2.2 Previous Methods

Early works of CPRP usually extracted handcrafted features for $\mathcal{G}$, such as the contextual and structural features Pan et al. (2017), while recent works are focused on designing deep-learning models, such as Siamese networks Roy et al. (2019) and GNNs Sun et al. (2022).

Inspired by the promising performance, we implemented $\mathcal{G}$ by training a GNN model in this study. GNNs aim to learn node representations in a graph by iteratively aggregating the neighborhood features. In each layer, the feature of the node $v$ is updated by merging the information transmitted from its neighbors, expressed as

$$\mathbf{f}^{(t)}_{v}=\mathcal{F}_{mer}^{W_{1}^{(t)}}(\mathbf{f}^{(t-1)}_{v},\mathcal{F}_{agg}^{W_{2}^{(t)}}(\{{\mathbf{f}^{(t-1)}(u)}|u\in\mathcal{N}(v)\}))$$ (2)

where $\mathbf{f}^{(t)}_{v}$ indicates the feature vector of the node $v$ at the $t$-th layer; $\mathcal{F}_{mer}^{W_{1}^{t}}$ is the merging function with learned parameters $W_{1}^{(t)}$ and $\mathcal{F}_{agg}^{W_{2}^{(t)}}$ is the aggregating function with network parameters $W_{2}^{(t)}$; $\mathcal{N}(v)$ delivers the neighbors of node $v$ Wu et al. (2020).

With the node representations from GNNs, many methods can estimate the prerequisite relations by employing similarity metrics or classical classifiers for $\mathcal{M}$ Liang et al. (2018). However, the previous studies fail to consider the problem of graph isomorphism of the KC graphs, leading to low expressive powers for CPRP.

3 Our CPRP Method

To achieve fine representations of KCs, our method adopts the well-known Weisfeiler-Leman test Morris et al. (2021) to guide the GNN training in the KC graph $G$. With the obtained KC representations, the Siamese network computes the link probability, shown in Fig. 1.

3.1 Weisfeiler-Leman Test

Denote by $S=\{(s_{1},s_{2},...,s_{k})|s_{i}\in V,i\in I_{k}\}$ a k-tuple $S$ of vertices in the KC graph $G$, where $I_{k}$ indicates the first $k$ natural numbers; $s_{i}$ is the $i$-th element of $S$ specified to a $v_{j}\in V$. Let $V^{k}(G)$ be the collection of all $k$-tuples from the graph $G$. Weisfeiler-Leman (WL) test is to assign labels to each tuple in $V^{k}(G)$ and then iteratively relabel these tuples by merging their neighborhood labels. Here, the $j$-th neighborhood of the tuple $S$ is yielded by replacing its $j$-th element with every node $v\in V$, i.e., $\mathcal{N}_{j}^{k}(S,v)=\{(s_{1},s_{2},...,s_{j-1},v,s_{j+1},...s_{k})|v\in V\}$ Morris et al. (2021). Based on all neighborhoods of $S$, the WL test usually uses a predefined route to compute a new label for the merged node in the $i$-th iteration, i.e.,

$$l_{i}(S)=\{\{(C^{k}_{i}(\mathcal{N}_{1}^{k}(S,v),\mathcal{N}_{2}^{k}(S,v),...,\mathcal{N}_{k}^{k}(S,v)|v\in V\}\}$$ (3)

where $l_{i}(S)$ represents the obtained label and $C^{k}_{i}$ indicates a predefined function that maps all tuples $V^{k}$ to new labels. Then, the iterative labeling for $S$ can be expressed as

$$C^{k}_{i}(S)=Relabel((C^{k}_{i-1}(S),l_{i}(S)))$$ (4)

where $Relabel(a,b)$ is to achieve $a\leftarrow b$.

The $k$-WL test is a powerful tool for distinguishing isomorphic graphs. Let $G$ and $H$ be two graphs. If the number of $k$-tuples with a specific label differs between graphs $G$ and $H$ at any iteration, then the two graphs are non-isomorphic. Specifically, 1-WL employs the 1-hop neighbors for $\mathcal{N}$ in Eq. (3). With the increase of $k$, the $k$-WL algorithm can become more capable of distinguishing non-isomorphic graphs.

3.2 Main Steps

Figure 1: Workflow of the proposed method for CPRP.

3.2.1 KC Graph Construction

The first step of the proposed method is to achieve the node features for the given directed KC graph $G$ using the pre-trained BERT . More specifically, the textual descriptions of KCs from the datasets or Wikipedia were obtained and fed into BERT to extract the KC embedding for $V$. With the given $E$, we achieved the KC graph $G$ Devlin et al. (2018).

3.2.2 Weisfeiler Leman Guided Dircted GNNs

On the resulting graph $G$, we proposed the directed GNN model guided by the $k$-WL test to achieve KC representations in this step. To be different from undirected graphs, we denote by $\mathcal{N}_{out}(v)$ the set of all out-neighbors of the node $v$. For two connected nodes $v$ and $w$ in a directed graph, if there exists an edge pointing from $v$ to $w$, node $w$ is defined as an out-neighbor of node $v$. In the proposed method, we redefined the $k$-tuple as follows,

$$\widehat{V}(G)^{k}:=\{(s_{1},s_{2},...,s_{k})|s_{i}\in V,i\in I_{k};\exists{j}\in I_{i-1}:{s_{i}}\in\mathcal{N}_{out}(s_{j})\cup{\{s_{j}\}}\}.$$ (5)

The feature representation of $k$-tuple $(s_{1},s_{2},...,s_{k})$ is here given by $[\mathbf{n_{1}}:\mathbf{n_{2}}:...:\mathbf{n_{k}}]$, where $\mathbf{n_{i}}$ denotes the embedding of node $s_{i}$; $[\cdot:\cdot]$ denotes the concatenation of vectors. For $\forall{S}\in\widehat{V}(G)^{k}$, the $j$-th out-neighbor of ${S}$ can be cast as:

$$\widehat{\mathcal{N}}^{k}_{j}({S}):=\{(s_{1},s_{2},...,s_{j-1},v,s_{j+1},...s_{k})|v\in\mathcal{N}_{out}(s_{j})\}$$ (6)

For all $k$-tuples, the neighbor relationships generated on the $j$-th element are used to construct a graph representation as $G_{j}$. For all graphs $G_{i}(i\in I_{k})$, each node in the graph $G_{i}$ represents a $k$-tuple in $\widehat{V}(G)^{k}$. For all graphs $G_{i}$, graph neural networks are constructed separately for training. After each layer of training is completed, the features of the same $k$-tuple on different graphs $G_{i}$ are fused using multilayer perceptron (MLP). The expression of $k$-tuple $S$ at the $t$-th layer is as follows:

$$\widehat{\mathbf{f}}_{S}=\mathcal{F}_{MLP}(\mathcal{F}_{S}^{(1)},\mathcal{F}_{S}^{(2)},...,\mathcal{F}_{S}^{(k)})$$ (7)

where $\widehat{\mathbf{f}}_{S}$ is defined as the representation of $k$-tuple $S$ at layer t. For $\forall i\in I_{k}$, $\mathcal{F}_{S}^{(i)}$ denotes the representation of $k$-tuple $S$ in graph $G_{i}$ after passing through the GCN Kipf and Welling (2016) designed for graph $G_{i}$. $\mathcal{F}_{MLP}$ denotes the multilayer perceptron. After several layers of learning, we obtain the representations of all $k$-tuples.

The representations of $k$-tuples obtained through GNN training are finally distributed to the nodes of $G$ using an average allocation scheme, shown as follows:

$$\textbf{x}_{j}^{(i)}=\frac{1}{\#\{\mathit{S}|\forall\mathit{S}\in\widehat{V}(G)^{k},s_{i}=v_{j}\}}\displaystyle\sum_{\{\mathit{S}|\forall\mathit{S}\in\widehat{V}(G)^{k},s_{i}=v_{j}\}}{\textbf{h}_{S}}$$ (8)

where $s_{i}$ represents the $i$-th element in the k-tuple $S$. $\textbf{x}^{(i)}_{j}$ represents the transmission of feature expressions of $k$-tuples back to the representations of concept $v_{j}$, based on the $i$-th element in each $k$-tuple. $\textbf{h}_{S}$ denotes the feature of $k$-tuple $S$. $\#\{...\}$ denotes the number of elements in the set $\{...\}$. For $\forall i$, all $\textbf{x}^{(i)}_{j}$ vectors are merged and fed into a multi-layer perceptron to obtain node representations in $G$.

$$\mathbf{f}_{j}=\mathcal{F}_{MLP}([\textbf{x}^{(1)}_{j}:\textbf{x}^{(2)}_{j}:...:\textbf{x}^{(k)}_{j}])$$ (9)

where $\mathbf{f}_{j}$ represents the learned concept $v_{j}$ representations.

3.2.3 Prediction Network

After obtaining the KC representations, the Siamese network is employed here to predict the probability that the concept $v_{p}$ is a prerequisite concept of concept $v_{q}$ in Eq. (1). Let $\mathbf{\tilde{e}}_{p}$ and $\mathbf{\tilde{e}}_{q}$ be the representations of $v_{p}$ and $v_{q}$ from the two feed-forward networks with shared weights in the Siamese network. The probability between $v_{p}$ and $v_{q}$ is achieved via

$$\mathcal{P}(v_{p},v_{q})=\sigma\left(W\left[\mathbf{\tilde{e}}_{i}:\mathbf{\tilde{e}}_{j}:\mathbf{\tilde{e}}_{i}-\mathbf{\tilde{e}}_{j}:\mathbf{\tilde{e}}_{i}\odot\mathbf{\tilde{e}}_{j}:1\right]\right)$$ (10)

where $\sigma$ represents the sigmoid operator and $\odot$ is Hadamard product. Finally, we used the cross-entropy loss to compute the training loss for the deep framework.

4 Experiments

We used LectureBank Li et al. (2019), University Courses Liang et al. (2017), and ML of MOOCs Pan et al. (2017) to evaluate the performance and compare to the state-of-the-art methods, including binary classification models (SVM, LR, NB, RF) Pan et al. (2017), RefD Liang et al. (2015), GAE Li et al. (2019), VGAE Li et al. (2019), PREREQ Roy et al. (2019), CPRL Jia et al. (2021), and Conlearn Sun et al. (2022). Besides, we employed precision, recall, and F1-score as evaluation metrics to measure the performance.

For BERT, we leveraged a combination of course lecture information and Wikipedia for concept description extraction in the MOOC and University Courses datasets. For the LectureBank dataset, we utilized the text information from the Wikipedia URLs of each concept in the dataset. When extracting vectors from texts, the max token size of BERT was set to 256 for all three datasets.

For our method, the parameter $k$ of $k$-WL was set to 2. We used Adam as the optimizer with a learning rate of 0.00002 for all experiments. The batch size was set to 256 for the MOOC dataset and LectureBank dataset and 512 for the University Course dataset. The models were trained for 4000 epochs for all experiments until the loss stabilized. As for the baseline methods, we used default parameters as in their original implementations.

For all three datasets, we selected 80% of the concept prerequisite pairs as the training set and 20% of the concept prerequisite pairs as the test set. Negative samples were generated by randomly selecting unrelated phrase pairs from the vocabulary, along with the reverse pairs of the original positive samples. The results are recorded in Table 1.

Table 1: Comparison between the results of the baseline model and our model

Datasets	Metric	NB	SVM	LR	RF	RefD	GAE	VGAE	PREREQ	CPRL	ConLearn	Ours
MOOCs	Precision	0.577	0.668	0.748	0.375	0.784	0.293	0.266	0.448	0.800	0.895	0.915
	Recall	0.623	0.577	0.270	0.669	0.188	0.733	0.647	0.592	0.642	0.850	0.860
	F1-score	0.599	0.619	0.397	0.481	0.303	0.419	0.377	0.510	0.712	0.872	0.887
LectureBank	Precision	0.670	0.857	0.744	0.855	0.666	0.462	0.417	0.590	0.861	0.831	0.857
	Recall	0.640	0.692	0.744	0.681	0.228	0.811	0.575	0.502	0.858	0.960	0.960
	F1-score	0.655	0.766	0.744	0.758	0.339	0.589	0.484	0.543	0.860	0.891	0.906
University Courses	Precision	0.478	0.796	0.595	0.739	0.919	0.450	0.470	0.468	0.689	0.611	0.822
	Recall	0.649	0.635	0.546	0.480	0.415	0.886	0.694	0.916	0.760	0.966	0.74
	F1-score	0.550	0.707	0.569	0.582	0.572	0.597	0.560	0.597	0.723	0.749	0.778

From Table 1, we can draw the following observations: the four binary classifiers, i.e., NB, SVM, LR, and RF, perform weak on the three datasets due to hand-crafted features, as well as RefD; GAE, VGAE, and PREREQ exploit the prerequisite relation information between KCs, resulting in improved performance; the CPRL method fails to delve into the textual information behind the concepts, leading to better performance; finally, both our algorithm and ConLearn extract prior information using the large-scale language model BERT and yield the best evaluations. Importantly, compared to ConLearn, our method uses 2-WL to integrate the structural information of the graph deeply, resulting in the best performance in terms of F1-score. All observations manifest that the introduced WL test into direct GNN is effective for CPRP.

5 Conclusion

This paper proposes a directed graph neural network based on the Weisfeiler-Leman algorithm to address the CPRP problem. Our method leverages BERT for KC text embeddings and redefines the $k$-tuple in the directed KC graph. Then, the 2-WL test is implemented to train a permutation-equivariant GNN. With the KC representation from GNN, the Siamese network computes the prediction probability of a KC link. Extensive experiments on three datasets demonstrate the superiority of the proposed method, achieving a more advanced performance than the state-of-the-art approaches of CPRP. Our future work will consider more evaluation results and topological information on the graph.