ReliCD: A Reliable Cognitive Diagnosis Framework with Confidence Awareness
Abstract
During the past few decades, cognitive diagnostics modeling has attracted increasing attention in computational education communities, which is capable of quantifying the learning status and knowledge mastery levels of students. Indeed, the recent advances in neural networks have greatly enhanced the performance of traditional cognitive diagnosis models through learning the deep representations of students and exercises. Nevertheless, existing approaches often suffer from the issue of overconfidence in predicting students’ mastery levels, which is primarily caused by the unavoidable noise and sparsity in realistic student-exercise interaction data, severely hindering the educational application of diagnostic feedback. To address this, in this paper, we propose a novel Reliable Cognitive Diagnosis (ReliCD) framework, which can quantify the confidence of the diagnosis feedback and is flexible for different cognitive diagnostic functions. Specifically, we first propose a Bayesian method to explicitly estimate the state uncertainty of different knowledge concepts for students, which enables the confidence quantification of diagnostic feedback. In particular, to account for potential differences, we suggest modeling individual prior distributions for the latent variables of different ability concepts using a pre-trained model. Additionally, we introduce a logical hypothesis for ranking confidence levels. Along this line, we design a novel calibration loss to optimize the confidence parameters by modeling the process of student performance prediction. Finally, extensive experiments on four real-world datasets clearly demonstrate the effectiveness of our ReliCD framework.
Index Terms:
Reliable cognitive diagnosis, intelligent education, knowledge state uncertaintyI Introduction
Cognitive diagnosis, as an essential component of computer-aided education, has garnered increasing attention over the past decades [1, 2, 3]. The primary objective of cognitive diagnostics modeling is to quantitatively assess students’ learning status and knowledge mastery levels, providing valuable formative feedback [1, 2]. Indeed, relevant studies have enabled a wide range of downstream educational applications, such as course recommendations [4], student assessment [5], and computerized adaptive testing [6]. As shown in Figure 1(a), given the answering records of student Lano concerning a series of exercises, the cognitive diagnosis model can automatically estimate her mastery levels of various knowledge concepts.
In the literature, traditional cognitive diagnosis models (CDMs) utilize different linear psychometric functions to measure students’ learning status by modeling the process of student performance prediction, such as Deterministic Inputs, Noisy “And” gate (DINA) [7], Item Response Theory (IRT) [8]. Recently, with the rapid development of deep learning techniques, several neural-based cognitive diagnostic methods have been proposed to enhance diagnostic performance. For instance, the neural cognitive diagnosis framework (NCD) utilizes neural networks to model students-exercise interactions, in order to uncover deeper features of both students and exercises [2]. Moreover, the flexibility of neural model design has enabled researchers to incorporate additional information, such as concept dependency maps [3] and student profiling [9], to further improve the effectiveness and interpretability of the models.
Previous studies have evaluated the effectiveness of cognitive diagnostic models by calculating the accuracy of student performance prediction, but they have not measured the reliability of diagnostic feedback. Meanwhile, due to the presence of noise and sparsity in student-exercise interaction data, existing approaches lead to the potential overconfidence in students’ mastery prediction, severely reducing the reliability of real-time diagnostic feedback in practical online education systems. More specifically, as illustrated in Figure 1(b), when Lano interacted with each exercise (i.e., from to ), we present the cognitive diagnosis model’s results regarding her mastery of knowledge concept and the accuracy of her performance prediction on all the exercises related to the concept in the test set. We found that due to the noise present in the interaction data (i.e., Lano, , ), the mastery of at deviates from the actual state . It indicates that we cannot trust the diagnostic feedback in a monotonic manner with the increase in interaction. Furthermore, traditional evaluation metrics like accuracy are non-smooth functions, which can result in the same evaluation outcome despite different diagnostic feedback. Additionally, these indicators are often not available in real-time during the diagnostic process in practical use. Consequently, an ideal cognitive diagnosis model should be able to provide both accurate diagnostic feedback and indications of its reliability.
To this end, in this paper, we propose a novel reliable cognitive diagnosis framework, namely ReliCD. To the best of our knowledge, this is the first one to quantify the confidence of the diagnosis feedback and is flexible for different cognitive diagnostic functions. Specifically, we first propose a Bayesian method for explicitly estimating the uncertainty of students’ states for various knowledge concepts with Gaussian latent variables, where the mean parameter represents the average ability status and the variance enables the quantification of diagnostic feedback’s confidence. In particular, due to the potential difference, we model the individual prior distribution for the latent variables of different ability concepts with a pre-trained model. Then, we introduce a logical hypothesis for ranking confidence levels and present a novel calibration loss to optimize the parameters in determining diagnostic feedback’s confidence through modeling the process of student performance prediction. Finally, extensive experiments on four real-world datasets demonstrate the effectiveness and flexibility of our ReliCD.
II Related Work
Generally, the related work in this paper can be grouped into two categories: cognitive diagnosis and confidence estimation.
II-A Cognitive Diagnosis
The main task of cognitive diagnosis is to use students’ responses to exercises for diagnosing students’ ability state. Over the past decades, experts in related educational psychology fields have proposed many cognitive diagnostic models. The two most classic ones are IRT [8] and DINA [7]. In IRT, Embretson et al. represented students’ ability state as a one-dimensional and continuous scalar. And a logistic function is used to predict the probability that the student eventually responds correctly to the exercise. Later, some researchers improved upon IRT and proposed MIRT [10] by extending the ability state of students to multi-dimensional vectors. Different from IRT, DINA uses a binary vector to model the student’s ability state with each dimension’s value representing his/her mastery of relevant knowledge concepts. There are two possible values on each dimension, 1 (mastered) or 0 (not mastered). Furthermore, Jimmy De La Torre believed that DINA itself has strong assumptions and constraints, which do not conform to the actual situation. Along this line, they proposed a generalized DINA (G-DINA) [11] to improve the diagnostic performance by weakening these constraints.
In recent years, neural-based cognitive diagnosis models have achieved state-of-the-art prediction performance, benefiting from the successful application of neural networks in various fields, including recommendation systems [12], knowledge tracing [13], and computer vision [14]. These works can be mainly divided into two aspects. The first aspect focuses on designing diagnostic functions that leverage the power of neural networks to capture complex and non-linear interactions between students and exercises, such as NCD [2]. The second is to use neural networks to enrich the representation of students and exercises by considering more additional information (e.g., the exercise text information, the relationship between knowledge concepts). For example, deep IRT (DIRT) [15] uses the semantic information of the exercise text to enrich the parameter representation of the traditional IRT. Educational context-aware cognitive diagnosis (ECD) [16] was proposed by incorporating the student’s educational background into the modeling of student knowledge status. Gao et al. [3] proposed the relation map-driven cognitive diagnosis (RCD) framework by exploiting the prerequisite relation and similarity relationship of knowledge concepts. Ma et al. [17] proposed a prerequisite attention model (PAKP) for knowledge proficiency diagnosis of students by considering the prerequisite relationship of knowledge concepts and learning the influence weights of predecessor knowledge concepts on successor knowledge concepts. Furthermore, Li et al. [18] proposed a novel CDM, namely HCDF, to enhance diagnostic performance by modeling the hierarchical relationship between knowledge concepts.
The majority of existing studies primarily concentrate on enhancing the accuracy of student performance prediction. However, there has been a notable lack of comprehensive investigation into the aspect of reliability in diagnostic feedback. In this paper, we introduce a novel approach for reliable cognitive diagnosis, which is the first to quantitatively assess the reliability of diagnostic feedback.
II-B Confidence Estimation
Confidence estimation has been incorporated within the machine learning community in some specific areas including autonomous driving [19], medical applications [20], and career mobility analysis [21, 22], so as to provide insight into the reliability of the results while making accurate predictions. The reliability feedback of results can serve as a measure for future tasks. For instance, Yukun et al. [23] suggested that challenging cases with low confidence levels in the field of medicine should be reviewed by skilled surgeons.
In the past years, the research direction of confidence modeling has evolved in two directions. The first direction is to quantify the confidence of predicted results with diverse heuristic approaches. For example, DeVries et al. [24] enhanced the model’s prediction by adding a branch of calculating the confidence value, based on the original classification task. The confidence value is utilized to identify whether the input sample is an out-of-distribution (OOD) sample. Hendrycks et al. [25] used the predicted softmax probability of the sample as the confidence estimation and detected OOD samples by selecting the samples with the minimum softmax probability values. Kendall et al. [26] argued that the model uncertainty could be explained by inherent noise in the captured data with Bayesian approaches. On the other hand, the confidence calibration work has also received extensive attention recently. For instance, Guo et al.[27] analyzed the overconfidence reasons (i.e., model capacity, batch normalization, and weight decay) of models based on deep neural networks and gave some post-processing techniques (e.g., temperature scaling, matrix, and vector scaling) to deal with these problems. Moon, Jooyoung et al.[28] introduced the correctness ranking loss to ensure the credibility of the predicted probability, which defines the optimization objection that the confidence estimate for the correctly predicted sample is greater than the confidence estimate for the incorrectly predicted sample. However, these confidence estimation methods cannot be directly applied to the task of cognitive diagnosis. In this paper, we propose a novel calibration loss method that aims to optimize parameters, thereby ensuring the reliability of the predicted probabilities, which allows the model to maintain confidence in its output results.
III PRELIMINARIES
In this section, we first introduce some currently known cognitive diagnosis functions (i.e., IRT, MIRT, and NCD). Then, we analyze the diagnostic feedback of the previous CDMs using NCD as a case study. Finally, we formally define the research problem being investigated in this paper.
III-A Cognitive Diagnostic Functions
Generally, cognitive diagnosis in computational education aims to determine the student’s ability status through the student exercising performance prediction task.
As a classic and representative diagnostic formula in educational psychology, IRT [8] portrays the student ability status of student with an integrated value . The logistic regression function is used to predict the probability that the student will answer the exercise correctly as follows,
(1) |
where and are difficulty and discrimination of exercise respectively. is a constant.
MIRT [10] expands the student’s ability status and exercise parameters from an integrated value to multi-dimensional vectors on the basis of IRT, so as to assess the student’s ability status from multiple aspects. In this paper, we resemble some work (e.g., RCD [3]) to map each dimension of MIRT to a specific knowledge concept by integrating the -matrix [2]. Under such consideration, the probability that student with ability status vector makes a correct response to exercise with difficulty vector can be expressed as:
(2) |
where indicates which knowledge concepts are relevant to , and , is the sum operation and is the sigmoid function.
NCD [2] attempts to accommodate complex nonlinear interactions between students and exercises by building a new diagnostic function consisting of three fully connected layers () and one shallow layer inspired by MIRT. The cognitive diagnostic function of NCD can be formalized as:
(3) |
where is a -dimensional vector representing the ability status of student , and are exercise difficulty and exercise discrimination, respectively. The value of each dimension in indicates the student ’ mastery level of the corresponding knowledge concept.
III-B Diagnostic Feedback Analysis
While current CDMs show remarkable accuracy in predicting student performance, we argue that their diagnostic feedback may not always be meaningful.
Without loss of generality, we take the NCD model as an example. Specifically, we trained the NCD on a public real-world dataset, namely Assist2009. Then, we can obtain all students’ diagnostic feedback, i.e., their ability status . As shown in Figure 2, we present the distribution of . Here, the red part indicates all ability status for student on each knowledge concept that has interacted with it. Similarly, the blue part shows the ability status that the student has not interacted with. Clearly, we can find that although the distributions of ability status values corresponding to the interactive knowledge concepts and non-interactive knowledge concepts are different, both of them have limited support included in [0.4, 0.6], which impedes the discriminate diagnostic feedback.
Moreover, we further analyze the impact of noisy interaction data on the diagnostic model. Here, Figure 3 shows the density plot, which indicates the correct rate of students’ performance prediction related to the knowledge concept #50 in the test set based on the above NCD model on Assist2009. Next, we incorporate a randomly generated interaction for each student at knowledge concept #50. The corresponding density plot on the correct rate of students’ performance prediction is shown in Figure 3. We can find that the student’s performance predictions were significantly degraded after incorporating the noisy data, which also demonstrates that even adding just one noisy interaction can undermine the reliability of diagnostic results.
Considering the aforementioned issues in the existing CDMs, in this paper, we focus on improving the reliability of diagnostic feedback by quantifying the confidence of the student’s ability status. And the proposed framework, ReliCD, is designed to be adaptable to various cognitive diagnostic functions, including IRT, MIRT, and NCD.
III-C PROBLEM STATEMENT
III-C1 Task Overview
Cognitive diagnosis in intelligent education consists of three parts, a set of students , a set of exercises and a set of knowledge concepts , where , , represent the number of students, the number of exercises and the number of knowledge concepts, respectively. The relationship between exercises and knowledge concepts is represented by a -matrix predefined by experts, where the -matrix is defined as . If exercise contains knowledge concept , then , otherwise . The response logs include a set of triplets , where if the student answers exercise correctly, otherwise . Along this line, we can formally the research problem in this paper as follows.
Problem Definition: Given the students’ answer logs and the experts’ predefined -matrix, our goal is to diagnose the students’ proficiency level for specific knowledge concepts and provide a confidence level for the diagnosis result.
IV Method
Cognitive diagnosis is the process of diagnosing the student’s abilities in a particular skill or concept. However, the reliability of diagnosis can be affected by various factors, such as noise in the data and the sparsity of interaction data. To address this issue, it is crucial to incorporate methods of modeling uncertainty in the diagnostic process, which encourages an accurate and reliable final diagnosis. Along this line, we design a novel reliable cognitive diagnosis (ReliCD) framework. As shown in Figure 4, it can be divided into three parts: 1) the student’s state and uncertainty module, 2) the cognitive diagnosis module, and 3) the training objective. Additionally, we have employed two effective strategies, namely prior consensus and uncertainty regularization, to enhance the performance of our framework.
IV-A Student’s State and Uncertainty
To model the uncertainty in the diagnostic process, we argue that there should be a deviation in the ability representations of students diagnosed by traditional score prediction methods. This deviation is caused by errors that can occur when students interact with the exercises, which can lead to unreliable diagnostic results. To address this issue, we model the student’s ability representation as a Gaussian distribution. The mean parameter represents the average ability status, while the variance provides criteria for reliable diagnostic results. If the variance of the distribution is small, the diagnosis tends to be more reliable.
To obtain a personalized distribution representation (Gaussian distribution) for each student , we multiply the student’s one-hot encoding by different transferable matrices to obtain the means and variance parameters, respectively, i.e.,
(4) |
where and represent mean and variance parameters for student , respectively. is the one-hot encoded representation of student . and are different transferable matrices. notes the number of students and indicates the dimensionality of the hidden vector (we will discuss the setting of in detail in Section IV-B).
Unlike previous student ability modeling techniques, here, we randomly sample students ability representations from the constructed Gaussian distribution . This approach simulates deviations in diagnostic results caused by potential noise in interactions between student and exercises . The details are as follows,
(5) |
where denotes the ability representation of student . is a vector randomly sampled from the constructed Gaussian distribution of . is a Sigmoid activation function to ensure that each dimension of is in .
IV-B Cognitive Diagnosis
After modeling the student’s ability status with uncertainty, we turn to predict exercise performance with cognitive diagnosis functions in Section III-A. Specifically, we first extract diagnostic factors from exercise, i.e., exercise difficulty and exercise discrimination , which are required in all cognitive diagnosis functions. The details are as follows:
(6) |
where denotes the one-hot representation of exercise . and are exercise difficulty and exercise discrimination of , respectively. and are two transferable matrices. indicates the number of exercises and denotes the dimensionality of the hidden vector.
Here, the predict probability , indicating student answers correctly on exercise , can be derived as follows:
(7) |
where denotes the cognitive diagnostic function. Please noted that our framework is flexible for various diagnostic functions. Here we further present three popular diagnostic functions, i.e., IRT, MIRT, and NCD, and specify detailed rules for them as follows:
- •
- •
- •
IV-C Training Objective
To optimize the parameters for obtaining the students’ ability status, we maximize the likelihood , which indicates the true probability that the student answers the exercise . Specifically, we follow the literature [29, 30, 31] and utilize the evidence lower bound as the training objective, which is tractable. Formally, we have
(8) | ||||
where is the prior distribution for the ability status of students. is the posterior distribution we constructed for the student . measures the likelihood that students with ability status answers correctly on exercise . We follow the variational autoencoder (VAE) [32, 33, 34] and leverage the sampling strategy to approximate with one sample. Then, can be specified as:
(9) |
When assuming that prior distribution of student satisfies the standard Gaussian distribution, in the Eq. (8) can be calculated by,
(10) |
where , are the mean and variance of student on knowledge concept .
Furthermore, based on the cognitive diagnosis scenario, we can define the reliability of a student’s diagnostic feedback on a specific knowledge concept as the probability of correctly predicting a student’s performance on the corresponding knowledge. Formally, the diagnostic feedback reliability can be defined as follows:
Definition 1
Given a student , a cognitive diagnosis model , and ’s diagnostic feedback , the reliability of diagnostic feedback (i.e., ) on a specific knowledge concept based on is the probability of correctly predicting ’s performance on , i.e., , where belong to all the response logs of student answered exercise which cover the concept (i.e., ).
Since we aim to utilize the standard deviation to assess the reliability of each student ’s diagnostic feedback on different knowledge concepts, we design a novel calibration loss as a regularization term for the training objective. Specifically, given and as the standard variances of student ’s and ’s ability representations on the knowledge concept and , respectively, if , we can assume the reliability of ’s diagnostic feedback should smaller than . Formally, we have the following hypothesis for ranking confidence levels.
Assumption IV.1
Given of student and of student , we have the relationship:
(11) |
Considering the probability is generally impractical to directly obtain, we follow the idea from [35] and [28] and hypothesis the probability of correctly predicting student performance on a specific knowledge concept is proportional to the frequency of correct predictions of on the triples where during the training process. Along this line, we design a calibration loss in a pairwise manner as follows,
(12) |
where denotes the proportion of the frequency of correct predictions of on concept over the total number of prediction on such interactions during the training process. The is defined as:
(13) |
Moreover, to reduce the training time cost, we sample the pair under the current mini-batch when optimizing Eq. (8). That is, given a mini-batch of the input interactions , we obtain the pair of standard deviations based on the sampled -th and -th (Here and only represent the -th and the -th instance) training instance pair, where denotes the -th training instance in the current mini-batch and denotes the size of mini-batch.
For IRT, since it models the students’ ability representation as a one-dimensional continuous scalar from a macro perspective, we revise the Eq. (12) as follows,
(14) |
where represents the proportion of the frequency of correct predictions of student over the total number of predictions on such interactions and is the standard deviation of student ’s ability representation.
Finally, the total loss function is defined as:
(15) |
where and are introduced to balance different items. Particularly, we follow the approach of -VAE [36] to weight for enhancing performance.
IV-D Prior Consensus and Uncertainty Regularization
Here, we propose two strategies to further improve our model: adjusting the prior distribution of the student’s state and regularizing the range of uncertainty.
IV-D1 Prior Consensus
Due to the potential difference between knowledge concepts, such as in terms of difficulty and discrimination, we assume that the prior distribution of the student’s status on each knowledge concept is different and individual. To model the individual prior and prevent information leakage, we only use the training set to pre-train our model by setting . Then, we average the ability state vector, i.e., of all students as the mean parameter of the prior distribution, i.e,
(16) |
which represents the prior common cognition of all knowledge concepts. This method is helpful for us to understand the overall level of students in advance, and avoid the situation that students master all knowledge concepts in advance to 0. Then, we train our entire model with the prior distribution , where the variance parameter is set as 1. Therefore, the new KL loss can be defined as:
(17) |
IV-D2 Uncertainty Regularization
At the same time, inspired by [37, 38], we also believe that the variance of the modeling should be within a reasonable range, neither too fluctuating nor too smooth. So we follow the idea in the literature [38] and apply dropout to the variance parameter for each student , which discourages the large variance,
(18) |
where is an independent random variable generated from a standard Bernoulli distribution. is our empirically defined value. At this point, the distribution we construct for student can be rewritten as .
After designing the technical details of our framework with two strategies for enhancing performance, we can train our framework with the training data following Algorithm 1.
V Experiment
In this section, we will provide a detailed description of the benchmark datasets, baselines, and experimental setup. The designed experiments aim to answer the following questions:
-
•
: How does our framework perform compare to state-of-the-art cognitive diagnosis models?
-
•
: Whether the specially designed parts of our framework effective?
-
•
: How do the hyperparameters influence the effectiveness of our framework?
-
•
: Whether our study can improve the reliability of cognitive diagnosis models?
V-A Dataset Description
We validated the performance of our framework on four real-world datasets, which are three public datasets namely Assistments2009 (Assist2009) 111https://sites.google.com/site/assistmentsdata/home/assistment2009-2010-data/skill-builder-data-2009-2010, Junyi 222 https://www.kaggle.com/datasets/junyiacademy/learning-activity-public-dataset-by-junyi-academy?resource=download and ENEM 333https://github.com/godtn0/DP-MTL, and a private dataset namely e-Math. ASSISTments2009 (ASSISTments 2009-2010 “skill builder”) is a public dataset collected by the assistant online tutoring systems in the 2009-2010 academic year. Junyi is a public dataset collected by the Khan Academy in 2012 year. e-Math is a private dataset collected by a well-known electronic educational product, mainly containing math exercises and response records of primary and secondary school students. ENEM is a Brazilian students’ college entrance examination.
Table I shows the basic information of the four datasets, including the number of students, the number of exercises, the number of knowledge concepts, the total number of answer logs, the average number of answer logs per student, and the average number of knowledge concepts contained in each exercise. Moreover, we uniformly filtered out students with less than response logs to guarantee that there is enough data for modeling each student for all datasets. Along this line, we obtained students, exercises, and knowledge concepts in Assist2009; students, exercises, and knowledge concepts in Junyi; students, exercises, and knowledge concepts in e-Math; students, exercises and knowledge concepts in ENEM.
We divided each dataset into training set, validation set, and test set by splitting each student’s response records at a ratio of . And, we trained our framework on the training set, tune the parameters of our framework on the validation set, and verify the performance of our framework on the test set.
Dataset | Assist2009 | Junyi | e-Math | ENEM |
---|---|---|---|---|
# Students | 4.1k | 1.0k | 1.9k | 10k |
# Exercises | 17.7k | 0.7k | 1.6k | 0.1k |
# Knowledge concepts | 123 | 39 | 61 | 4 |
# Response logs | 324k | 203k | 62k | 18500k |
# Avg logs per student | 77.96 | 203.94 | 120.71 | 185 |
# Avg concepts per exercise | 1.19 | 1.00 | 1.21 | 1.00 |
Datasets | Metrics | IRT | Reli-IRT | MIRT | Reli-MIRT | NCD | Reli-NCD |
---|---|---|---|---|---|---|---|
Assist2009 | ACC (% ) | 68.17 0.06 | 69.56 0.29 | 70.62 0.43 | 71.12 0.23 | 72.20 1.11 | 72.55 0.20 |
RMSE () | 0.4554 0.0054 | 0.4429 0.0012 | 0.4536 0.0018 | 0.4478 0.0007 | 0.4347 0.0028 | 0.4311 0.0010 | |
AUC (% ) | 69.15 1.35 | 72.36 0.12 | 72.53 0.73 | 72.14 0.09 | 75.10 0.14 | 75.10 0.32 | |
ECE (% ) | 4.75 0.03 | 3.13 0.15 | 9.97 1.16 | 7.81 0.23 | 6.97 0.50 | 1.69 0.19 | |
MCE (% ) | 10.91 0.39 | 10.58 0.26 | 13.11 1.28 | 12.21 0.52 | 9.00 0.19 | 3.85 0.75 | |
e-Math | ACC (% ) | 67.57 0.41 | 70.00 0.05 | 67.49 0.42 | 69.20 0.42 | 69.11 0.32 | 69.13 0.34 |
RMSE () | 0.4564 0.0014 | 0.4390 0.0008 | 0.4595 0.0022 | 0.4506 0.0007 | 0.4427 0.0030 | 0.4399 0.0013 | |
AUC (% ) | 69.77 0.36 | 74.20 0.29 | 71.23 0.24 | 72.52 0.23 | 73.79 0.21 | 74.12 0.20 | |
ECE (% ) | 3.56 0.37 | 3.14 0.24 | 8.88 0.42 | 5.56 0.39 | 4.53 0.01 | 1.17 0.07 | |
MCE (% ) | 10.37 0.78 | 9.85 1.08 | 13.60 0.63 | 13.49 0.34 | 5.90 1.01 | 2.06 0.01 | |
Junyi | ACC (% ) | 71.56 0.27 | 75.31 0.19 | 75.73 0.17 | 75.79 0.15 | 75.60 0.26 | 76.05 0.15 |
RMSE () | 0.4342 0.0021 | 0.4081 0.0009 | 0.4291 0.0004 | 0.4279 0.0005 | 0.4068 0.0006 | 0.4042 0.0005 | |
AUC (% ) | 74.09 0.39 | 78.84 0.15 | 77.18 0.11 | 77.34 0.17 | 79.87 0.13 | 80.11 0.14 | |
ECE (% ) | 3.89 0.19 | 2.36 0.12 | 10.68 0.17 | 10.13 0.13 | 1.97 0.72 | 1.68 0.88 | |
MCE (% ) | 8.45 0.36 | 4.85 0.26 | 20.90 14.54 | 14.25 0.17 | 3.07 0.94 | 2.59 1.04 | |
ENEM | ACC (% ) | 71.70 0.26 | 73.09 0.44 | 70.91 0.02 | 72.02 0.03 | 73.45 0.16 | 73.46 0.12 |
RMSE () | 0.4448 0.0009 | 0.4319 0.020 | 0.4514 0.0002 | 0.4443 0.0001 | 0.4288 0.0008 | 0.4286 0.0005 | |
AUC (% ) | 69.31 0.14 | 72.18 0.06 | 69.86 0.08 | 69.92 0.02 | 72.93 0.10 | 72.96 0.06 | |
ECE (% ) | 5.03 0.15 | 2.10 0.09 | 7.78 0.06 | 6.63 0.09 | 1.64 0.16 | 0.76 0.08 | |
MCE (% ) | 10.96 0.46 | 3.71 0.14 | 12.79 0.11 | 7.56 0.12 | 3.82 0.19 | 1.64 0.16 |
V-B Experimental Setup
V-B1 Experimental settings
In the experiment, we used Xavier initialization to initialize all parameters in our framework. We leveraged the Adam optimizer to train our reliable CDMs with a batch size of and a learning rate of , respectively. We used five-fold cross-validation to more accurately evaluate the performance of our framework on all datasets. As mentioned in Section 4.3, we set during the model pre-training phase. While during the training, validation, and testing phases, we set =1e-4, =0.1. Our framework 444https://github.com/BIMK/Intelligent-Education/tree/main/ReliCD and baselines were implemented with Pytorch= by Python=, and all experiments were conducted on an NVIDIA GeForce RTX 3090-24GHB.
V-B2 Evaluation metrics
Here we evaluate our work from two aspects. The first aspect is the performance of our framework, which can be measured by ACC (Accuracy), RMSE (Root Mean Square Error), and AUC (Area Under an ROC Curve), using the same metrics as previous work (e.g., NCD). The second is the quality of confidence estimation on the student’s ability status, which can not be evaluated directly. Here, we turn to measure the confidence of our framework in predicting exercise performance by the expected calibration error (ECE) [39] and the Maximum Calibration Error (MCE) [39], which are widely used in confidence estimation related literature [27, 28]. The smaller the value of ECE or MCE, the better the quality of confidence estimation. Specifically, we first divide the prediction probability interval into a certain number of bins. Then, ECE and MCE can be calculated by adding up and taking the maximum of the differences between the mean probability in each bin and the accuracy among the corresponding samples with weight, respectively. The calculation formulas are as follows,
(19) | |||
where is the number of interval bins, denotes the set of samples with prediction probability in [, ], is the total number of samples, is the accuracy of the samples in , is the average prediction probability of our framework for the samples in .
V-C Performance Comparison (RQ1)
To verify the effectiveness of our proposed framework, we applied it to different cognitive diagnostic functions, including IRT, MIRT, and NCD. As a result, we obtained three reliable cognitive diagnosis methods: Reli-IRT, Reli-MIRT, and Reli-NCD. Our goal is to substantially enhance confidence metrics (i.e., ECE and MCE) while making slight improvements to traditional metrics (i.e., ACC, RMSE, and AUC). Specifically, all these generated reliable cognitive diagnosis methods sampled the students’ ability state from a constructed distribution to model students’ uncertainty. As illustrated in Table II, we compared our ReliCDs with the corresponding baselines on four real-world datasets and we bolded the best experimental results with black lines. Moreover, we conducted the standard student t-test for the pair of our ReliCDs and the baselines at all indicators. Results are summarized in in Table II with significant improvement (-) denoted with an asterisk (). We can obtain the following observations. Firstly, we can find that our ReliCDs have a significant decline compared to the corresponding baseline in ECE and MCE on almost all datasets. It not only shows that capturing the uncertainty of students can calibrate the confidence value of the prediction results but also demonstrates our method largely improves the reliability of the diagnostic results. Secondly, we observed that our ReliCDs have significantly improved the performance in terms of ACC, AUC, and RMSE in some datasets. It reveals that estimating the uncertainty of students’ ability status on different knowledge concepts can enhance the effectiveness of the student performance prediction. Thirdly, we found that our Reli-NCD achieved the best performance on all datasets. Meanwhile, we observed that basic NCD did not achieve the best performance on the ECE at Assistments2009 and e-Math. It also shows that our solution brings a good reliability improvement to strengthen cognitive diagnostic functions like NCD.
V-D Ablation Study (RQ2)
To verify the effectiveness of each specially designed component of our framework, we constructed two variants of our ReliCD by removing the corresponding components. Without loss of generality, we used Reli-NCD as the baseline, which is an implementation of our framework with the specific diagnostic function NCD. The variants are described as follows:
-
•
: It is a variant of Reli-NCD by removing the pre-training process, so as to explore its impact on the experimental results.
-
•
: It is a variant of Reli-NCD by removing the calibration loss, so as to explore its impact on the experimental results.
The results are summarized in Figure 5. Clearly, we can find that Reli-NCD has achieved the best performance on all datasets. Meanwhile, we observe that both pre-training strategy and calibration loss can significantly improve the performance of ECE and MCE in almost all datasets. Specifically, the pre-training strategy effectively reduces about 26.4%, 19.2%, and 37.9% on ECE at Assist2009, Junyi, and e-Math, respectively. Correspondingly, the calibration loss effectively reduces about 10.2%, 7.2%, and 24.8% on ECE at the above datasets. It clearly demonstrates the effectiveness of those components in our framework, which also answers .
V-E Parameter Sensitivity Analysis (RQ3)
To evaluate the sensitivity of hyperparameters and in the loss function and answer the , we conducted multiple experiments on e-Math, Assist2009, Junyi, and ENEM. We varied and individually from to , while kee** the other parameter fixed.
As depicted in Figure 6, it is evident that the size of has a significant effect on the results of both the AUC and ECE. The model’s performance is optimal within the range of . This observation indicates that considering the constraints of student distribution within a reasonable range is beneficial to the model performance. As for shown in Figure 7, varying the size of did not greatly affect the AUC values in e-Math and Assist2009, while the ECE values varied significantly on all three datasets under different sizes. The observation regarding suggests that the partial order relationship we established has a certain level of calibration effect on the final prediction of the model.
V-F Case Study (RQ4)
Here we first show an example of predicted ability status via our framework. Specifically, we trained our Reli-NCD on the Assist2009 dataset. Figure 8 shows the predicted student #4164’s ability status distribution of knowledge concept #15 corresponding to training with different numbers of exercises on this concept. Clearly, we can observe that the fluctuation of student ability state decreases with more interaction data on this concept, while the mean value of the student ability status is also regionally stable. Therefore, our model can effectively identify the reliability of the diagnostic feedback, which will serve as a great aid to educators in assessing the usability of the diagnostic feedback.
Moreover, similar to the diagnostic feedback analysis in the preliminaries, we trained our Reli-NCD on the Assist2009 and obtained the distributions of students’ ability status of the interactive knowledge concepts and non-interactive knowledge concepts. As shown in Figure 9, we can find that our Reli-NCD provides more discriminate diagnostic feedback than NCD, as its predicted ability status distribution span is significantly wider. Meanwhile, we obtained our Reli-NCD can achieve concentrated distribution on students’ ability status of the knowledge concepts that they have not interacted with, which demonstrates the reliability of our diagnostic feedback.
VI Conclusion
In this paper, we introduced a reliable cognitive diagnosis framework with confidence awareness, namely ReliCD, which is the first one to quantify the confidence of the diagnosis feedback and is flexible for different cognitive diagnostic functions. To be specific, we first proposed a Bayesian method to explicitly estimate the state uncertainty of different knowledge concepts for students, which enables the confidence quantification of diagnostic feedback. In particular, to avoid the potential difference, we proposed to model the individual prior distribution for the latent variables of different ability concepts with a pre-trained model. Then, we introduced a logical hypothesis for ranking confidence levels. Moreover, we designed a novel calibration loss to optimize the confidence parameters by modeling the process of student performance prediction. Finally, we have conducted extensive experiments on real-world datasets, and the experimental results have clearly demonstrated the effectiveness of our ReliCD.
VII Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant 62107001, in part by the National Key Research and Development Project (NO. 2018AAA0100105), in part by the Anhui Provincial Natural Science Foundation (NO. 2108085QF272 and No. 2208085QF194), in part by the University Synergy Innovation Program of Anhui Province under Grant GXXT-2021-004.
References
- [1] J. Leighton and M. Gierl, Cognitive diagnostic assessment for education: Theory and applications. Cambridge University Press, 2007.
- [2] F. Wang, Q. Liu, E. Chen, Z. Huang, Y. Chen, Y. Yin, Z. Huang, and S. Wang, “Neural cognitive diagnosis for intelligent education systems,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 04, 2020, pp. 6153–6161.
- [3] W. Gao, Q. Liu, Z. Huang, Y. Yin, H. Bi, M.-C. Wang, J. Ma, S. Wang, and Y. Su, “Rcd: Relation map driven cognitive diagnosis for intelligent education systems,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021, pp. 501–510.
- [4] C. Wang, H. Zhu, C. Zhu, X. Zhang, E. Chen, and H. Xiong, “Personalized employee training course recommendation with career development awareness,” in Proceedings of the Web Conference 2020, 2020, pp. 1648–1659.
- [5] M. Khajah, R. Wing, R. V. Lindsey, and M. Mozer, “Integrating latent-factor and knowledge-tracing models to predict individual differences in learning.” in EDM, 2014, pp. 99–106.
- [6] H. Ma, Y. Zeng, S. Yang, C. Qin, X. Zhang, and L. Zhang, “A novel computerized adaptive testing framework with decoupled learning selector,” Complex & Intelligent Systems, pp. 1–12, 2023.
- [7] J. De La Torre, “Dina model and parameter estimation: A didactic,” Journal of educational and behavioral statistics, vol. 34, no. 1, pp. 115–130, 2009.
- [8] S. E. Embretson and S. P. Reise, Item response theory. Psychology Press, 2013.
- [9] Y. Zhou, Q. Liu, J. Wu, F. Wang, Z. Huang, W. Tong, H. Xiong, E. Chen, and J. Ma, “Modeling context-aware features for cognitive diagnosis in student learning,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 2420–2428.
- [10] M. D. Reckase, “Multidimensional item response theory models,” in Multidimensional item response theory. Springer, 2009, pp. 79–112.
- [11] J. De La Torre, “The generalized dina model framework,” Psychometrika, vol. 76, no. 2, pp. 179–199, 2011.
- [12] Y. Deng, Y. Xie, Y. Li, M. Yang, W. Lam, and Y. Shen, “Contextualized knowledge-aware attentive neural network: Enhancing answer selection with knowledge,” ACM Transactions on Information Systems (TOIS), vol. 40, no. 1, pp. 1–33, 2021.
- [13] H. Ma, J. Wang, H. Zhu, X. Xia, H. Zhang, X. Zhang, and L. Zhang, “Reconciling cognitive modeling with knowledge forgetting: A continuous time-aware neural network approach,” in Proceedings of the 31st International Joint Conference on Artificial Intelligence, 2022, pp. 2174–2181.
- [14] J. Chai, H. Zeng, A. Li, and E. W. Ngai, “Deep learning in computer vision: A critical review of emerging techniques and application scenarios,” Machine Learning with Applications, vol. 6, p. 100134, 2021.
- [15] S. Cheng, Q. Liu, E. Chen, Z. Huang, Z. Huang, Y. Chen, H. Ma, and G. Hu, “Dirt: Deep learning enhanced item response theory for cognitive diagnosis,” in Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019, pp. 2397–2400.
- [16] Y. Zhou, Q. Liu, J. Wu, F. Wang, Z. Huang, W. Tong, H. Xiong, E. Chen, and J. Ma, “Modeling context-aware features for cognitive diagnosis in student learning,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 2420–2428.
- [17] H. Ma, J. Zhu, S. Yang, Q. Liu, H. Zhang, X. Zhang, Y. Cao, and X. Zhao, “A prerequisite attention model for knowledge proficiency diagnosis of students,” in Proceedings of the 31st ACM CIKM, 2022, pp. 4304–4308.
- [18] J. Li, F. Wang, Q. Liu, M. Zhu, W. Huang, Z. Huang, E. Chen, Y. Su, and S. Wang, “Hiercdf: A bayesian network-based hierarchical cognitive diagnosis framework,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 904–913.
- [19] S. Hecker, D. Dai, and L. Van Gool, “Failure prediction for autonomous driving,” in 2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2018, pp. 1792–1799.
- [20] M. Bernhardt, F. D. S. Ribeiro, and B. Glocker, “Failure detection in medical image classification: A reality check and benchmarking testbed,” 2022.
- [21] R. Zha, C. Qin, L. Zhang, D. Shen, T. Xu, H. Zhu, and E. Chen, “Career mobility analysis with uncertainty-aware graph autoencoders: A job title transition perspective,” IEEE Transactions on Computational Social Systems, 2023.
- [22] C. Qin, L. Zhang, R. Zha, D. Shen, Q. Zhang, Y. Sun, C. Zhu, H. Zhu, and H. Xiong, “A comprehensive survey of artificial intelligence techniques for talent analytics,” arXiv preprint arXiv:2307.03195, 2023.
- [23] Y. Ding, J. Liu, X. Xu, M. Huang, J. Zhuang, J. Xiong, and Y. Shi, “Uncertainty-aware training of neural networks for selective medical image segmentation,” in Medical Imaging with Deep Learning. PMLR, 2020, pp. 156–173.
- [24] T. DeVries and G. W. Taylor, “Learning confidence for out-of-distribution detection in neural networks,” arXiv preprint arXiv:1802.04865, 2018.
- [25] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” arXiv preprint arXiv:1610.02136, 2016.
- [26] A. Kendall and Y. Gal, “What uncertainties do we need in bayesian deep learning for computer vision?” Advances in neural information processing systems, vol. 30, 2017.
- [27] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in International conference on machine learning. PMLR, 2017, pp. 1321–1330.
- [28] J. Moon, J. Kim, Y. Shin, and S. Hwang, “Confidence-aware learning for deep neural networks,” in international conference on machine learning. PMLR, 2020, pp. 7034–7044.
- [29] A. Pagnoni, K. Liu, and S. Li, “Conditional variational autoencoder for neural machine translation,” arXiv preprint arXiv:1812.04405, 2018.
- [30] D. Shen, C. Qin, H. Zhu, T. Xu, E. Chen, and H. Xiong, “Joint representation learning with relation-enhanced topic models for intelligent job interview assessment,” ACM Transactions on Information Systems (TOIS), vol. 40, no. 1, pp. 1–36, 2021.
- [31] D. Shen, H. Zhu, C. Zhu, T. Xu, C. Ma, and H. Xiong, “A joint learning approach to intelligent job interview assessment.” in IJCAI, vol. 18, 2018, pp. 3542–3548.
- [32] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013.
- [33] C. Qin, K. Yao, H. Zhu, T. Xu, D. Shen, E. Chen, and H. Xiong, “Towards automatic job description generation with capability-aware neural networks,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 5, pp. 5341–5355, 2022.
- [34] C. Wang, H. Zhu, C. Zhu, X. Zhang, E. Chen, and H. Xiong, “Personalized employee training course recommendation with career development awareness,” in Proceedings of the Web Conference 2020, 2020, pp. 1648–1659.
- [35] M. Toneva, A. Sordoni, R. T. des Combes, A. Trischler, Y. Bengio, and G. J. Gordon, “An empirical study of example forgetting during deep neural network learning,” in International Conference on Learning Representations, 2019. [Online]. Available: https://openreview.net/forum?id=BJlxm30cKm
- [36] C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner, “Understanding disentangling in ,” arXiv preprint arXiv:1804.03599, 2018.
- [37] X. Ma, C. Zhou, and E. Hovy, “MAE: Mutual posterior-divergence regularization for variational autoencoders,” in International Conference on Learning Representations, 2019. [Online]. Available: https://openreview.net/forum?id=Hke4l2AcKQ
- [38] D. Shen, C. Qin, C. Wang, H. Zhu, E. Chen, and H. Xiong, “Regularizing variational autoencoder with diversity and uncertainty awareness,” in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, ser. IJCAI-2021. International Joint Conferences on Artificial Intelligence Organization, Aug. 2021. [Online]. Available: http://dx.doi.org/10.24963/ijcai.2021/408
- [39] M. P. Naeini, G. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using bayesian binning,” in Proceedings of the AAAI conference on artificial intelligence, vol. 29, no. 1, 2015.