Anna Karenina Strikes Again: Pre-Trained LLM Embeddings May Favor High-Performing Learners
Authors:
Abigail Gurin Schleifer,
Beata Beigman Klebanov,
Moriah Ariely,
Giora Alexandron
Abstract:
Unsupervised clustering of student responses to open-ended questions into behavioral and cognitive profiles using pre-trained LLM embeddings is an emerging technique, but little is known about how well this captures pedagogically meaningful information. We investigate this in the context of student responses to open-ended questions in biology, which were previously analyzed and clustered by expert…
▽ More
Unsupervised clustering of student responses to open-ended questions into behavioral and cognitive profiles using pre-trained LLM embeddings is an emerging technique, but little is known about how well this captures pedagogically meaningful information. We investigate this in the context of student responses to open-ended questions in biology, which were previously analyzed and clustered by experts into theory-driven Knowledge Profiles (KPs). Comparing these KPs to ones discovered by purely data-driven clustering techniques, we report poor discoverability of most KPs, except for the ones including the correct answers. We trace this "discoverability bias" to the representations of KPs in the pre-trained LLM embeddings space.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
What Explains Teachers' Trust of AI in Education across Six Countries?
Authors:
Olga Viberg,
Mutlu Cukurova,
Yael Feldman-Maggor,
Giora Alexandron,
Shizuka Shirai,
Susumu Kanemune,
Barbara Wasson,
Cathrine Tømte,
Daniel Spikol,
Marcelo Milrad,
Raquel Coelho,
René F. Kizilcec
Abstract:
With growing expectations to use AI-based educational technology (AI-EdTech) to improve students' learning outcomes and enrich teaching practice, teachers play a central role in the adoption of AI-EdTech in classrooms. Teachers' willingness to accept vulnerability by integrating technology into their everyday teaching practice, that is, their trust in AI-EdTech, will depend on how much they expect…
▽ More
With growing expectations to use AI-based educational technology (AI-EdTech) to improve students' learning outcomes and enrich teaching practice, teachers play a central role in the adoption of AI-EdTech in classrooms. Teachers' willingness to accept vulnerability by integrating technology into their everyday teaching practice, that is, their trust in AI-EdTech, will depend on how much they expect it to benefit them versus how many concerns it raises for them. In this study, we surveyed 508 K-12 teachers across six countries on four continents to understand which teacher characteristics shape teachers' trust in AI-EdTech, and its proposed antecedents, perceived benefits and concerns about AI-EdTech. We examined a comprehensive set of characteristics including demographic and professional characteristics (age, gender, subject, years of experience, etc.), cultural values (Hofstede's cultural dimensions), geographic locations (Brazil, Israel, Japan, Norway, Sweden, USA), and psychological factors (self-efficacy and understanding). Using multiple regression analysis, we found that teachers with higher AI-EdTech self-efficacy and AI understanding perceive more benefits, fewer concerns, and report more trust in AI-EdTech. We also found geographic and cultural differences in teachers' trust in AI-EdTech, but no demographic differences emerged based on their age, gender, or level of education. The findings provide a comprehensive, international account of factors associated with teachers' trust in AI-EdTech. Efforts to raise teachers' understanding of, and trust in AI-EdTech, while considering their cultural values are encouraged to support its adoption in K-12 education.
△ Less
Submitted 2 February, 2024; v1 submitted 4 December, 2023;
originally announced December 2023.
Kappa Learning: A New Method for Measuring Similarity Between Educational Items Using Performance Data
Authors:
Tanya Nazaretsky,
Sara Hershkovitz,
Giora Alexandron
Abstract:
Sequencing items in adaptive learning systems typically relies on a large pool of interactive assessment items (questions) that are analyzed into a hierarchy of skills or Knowledge Components (KCs). Educational data mining techniques can be used to analyze students performance data in order to optimize the map** of items to KCs. Standard methods that map items into KCs using item-similarity meas…
▽ More
Sequencing items in adaptive learning systems typically relies on a large pool of interactive assessment items (questions) that are analyzed into a hierarchy of skills or Knowledge Components (KCs). Educational data mining techniques can be used to analyze students performance data in order to optimize the map** of items to KCs. Standard methods that map items into KCs using item-similarity measures make the implicit assumption that students performance on items that depend on the same skill should be similar. This assumption holds if the latent trait (mastery of the underlying skill) is relatively fixed during students activity, as in the context of testing, which is the primary context in which these measures were developed and applied. However, in adaptive learning systems that aim for learning, and address subject matters such as K6 Math that consist of multiple sub-skills, this assumption does not hold. In this paper we propose a new item-similarity measure, termed Kappa Learning (KL), which aims to address this gap. KL identifies similarity between items under the assumption of learning, namely, that learners mastery of the underlying skills changes as they progress through the items. We evaluate Kappa Learning on data from a computerized tutor that teaches Fractions for 4th grade, with experts tagging as ground truth, and on simulated data. Our results show that clustering that is based on Kappa Learning outperforms clustering that is based on commonly used similarity measures (Cohen Kappa, Yule, and Pearson).
△ Less
Submitted 20 December, 2018;
originally announced December 2018.
