-
How Teachers Can Use Large Language Models and Bloom's Taxonomy to Create Educational Quizzes
Authors:
Sabina Elkins,
Ekaterina Kochmar,
Jackie C. K. Cheung,
Iulian Serban
Abstract:
Question generation (QG) is a natural language processing task with an abundance of potential benefits and use cases in the educational domain. In order for this potential to be realized, QG systems must be designed and validated with pedagogical needs in mind. However, little research has assessed or designed QG approaches with the input from real teachers or students. This paper applies a large…
▽ More
Question generation (QG) is a natural language processing task with an abundance of potential benefits and use cases in the educational domain. In order for this potential to be realized, QG systems must be designed and validated with pedagogical needs in mind. However, little research has assessed or designed QG approaches with the input from real teachers or students. This paper applies a large language model-based QG approach where questions are generated with learning goals derived from Bloom's taxonomy. The automatically generated questions are used in multiple experiments designed to assess how teachers use them in practice. The results demonstrate that teachers prefer to write quizzes with automatically generated questions, and that such quizzes have no loss in quality compared to handwritten versions. Further, several metrics indicate that automatically generated questions can even improve the quality of the quizzes created, showing the promise for large scale use of QG in the classroom setting.
△ Less
Submitted 11 January, 2024;
originally announced January 2024.
-
How Useful are Educational Questions Generated by Large Language Models?
Authors:
Sabina Elkins,
Ekaterina Kochmar,
Jackie C. K. Cheung,
Iulian Serban
Abstract:
Controllable text generation (CTG) by large language models has a huge potential to transform education for teachers and students alike. Specifically, high quality and diverse question generation can dramatically reduce the load on teachers and improve the quality of their educational content. Recent work in this domain has made progress with generation, but fails to show that real teachers judge…
▽ More
Controllable text generation (CTG) by large language models has a huge potential to transform education for teachers and students alike. Specifically, high quality and diverse question generation can dramatically reduce the load on teachers and improve the quality of their educational content. Recent work in this domain has made progress with generation, but fails to show that real teachers judge the generated questions as sufficiently useful for the classroom setting; or if instead the questions have errors and/or pedagogically unhelpful content. We conduct a human evaluation with teachers to assess the quality and usefulness of outputs from combining CTG and question taxonomies (Bloom's and a difficulty taxonomy). The results demonstrate that the questions generated are high quality and sufficiently useful, showing their promise for widespread use in the classroom setting.
△ Less
Submitted 13 April, 2023;
originally announced April 2023.
-
Question Personalization in an Intelligent Tutoring System
Authors:
Sabina Elkins,
Robert Belfer,
Ekaterina Kochmar,
Iulian Serban,
Jackie C. K. Cheung
Abstract:
This paper investigates personalization in the field of intelligent tutoring systems (ITS). We hypothesize that personalization in the way questions are asked improves student learning outcomes. Previous work on dialogue-based ITS personalization has yet to address question phrasing. We show that generating versions of the questions suitable for students at different levels of subject proficiency…
▽ More
This paper investigates personalization in the field of intelligent tutoring systems (ITS). We hypothesize that personalization in the way questions are asked improves student learning outcomes. Previous work on dialogue-based ITS personalization has yet to address question phrasing. We show that generating versions of the questions suitable for students at different levels of subject proficiency improves student learning gains, using variants written by a domain expert and an experimental A/B test. This insight demonstrates that the linguistic realization of questions in an ITS affects the learning outcomes for students.
△ Less
Submitted 25 May, 2022;
originally announced June 2022.
-
A New Era: Intelligent Tutoring Systems Will Transform Online Learning for Millions
Authors:
Francois St-Hilaire,
Dung Do Vu,
Antoine Frau,
Nathan Burns,
Farid Faraji,
Joseph Potochny,
Stephane Robert,
Arnaud Roussel,
Selene Zheng,
Taylor Glazier,
Junfel Vincent Romano,
Robert Belfer,
Muhammad Shayan,
Ariella Smofsky,
Tommy Delarosbil,
Seulmin Ahn,
Simon Eden-Walker,
Kritika Sony,
Ansona Onyi Ching,
Sabina Elkins,
Anush Stepanyan,
Adela Matajova,
Victor Chen,
Hossein Sahraei,
Robert Larson
, et al. (6 additional authors not shown)
Abstract:
Despite artificial intelligence (AI) having transformed major aspects of our society, less than a fraction of its potential has been explored, let alone deployed, for education. AI-powered learning can provide millions of learners with a highly personalized, active and practical learning experience, which is key to successful learning. This is especially relevant in the context of online learning…
▽ More
Despite artificial intelligence (AI) having transformed major aspects of our society, less than a fraction of its potential has been explored, let alone deployed, for education. AI-powered learning can provide millions of learners with a highly personalized, active and practical learning experience, which is key to successful learning. This is especially relevant in the context of online learning platforms. In this paper, we present the results of a comparative head-to-head study on learning outcomes for two popular online learning platforms (n=199 participants): A MOOC platform following a traditional model delivering content using lecture videos and multiple-choice quizzes, and the Korbit learning platform providing a highly personalized, active and practical learning experience. We observe a huge and statistically significant increase in the learning outcomes, with students on the Korbit platform providing full feedback resulting in higher course completion rates and achieving learning gains 2 to 2.5 times higher than both students on the MOOC platform and students in a control group who don't receive personalized feedback on the Korbit platform. The results demonstrate the tremendous impact that can be achieved with a personalized, active learning AI-powered system. Making this technology and learning experience available to millions of learners around the world will represent a significant leap forward towards the democratization of education.
△ Less
Submitted 3 March, 2022;
originally announced March 2022.
-
Comparative Study of Learning Outcomes for Online Learning Platforms
Authors:
Francois St-Hilaire,
Nathan Burns,
Robert Belfer,
Muhammad Shayan,
Ariella Smofsky,
Dung Do Vu,
Antoine Frau,
Joseph Potochny,
Farid Faraji,
Vincent Pavero,
Neroli Ko,
Ansona Onyi Ching,
Sabina Elkins,
Anush Stepanyan,
Adela Matajova,
Laurent Charlin,
Yoshua Bengio,
Iulian Vlad Serban,
Ekaterina Kochmar
Abstract:
Personalization and active learning are key aspects to successful learning. These aspects are important to address in intelligent educational applications, as they help systems to adapt and close the gap between students with varying abilities, which becomes increasingly important in the context of online and distance learning. We run a comparative head-to-head study of learning outcomes for two p…
▽ More
Personalization and active learning are key aspects to successful learning. These aspects are important to address in intelligent educational applications, as they help systems to adapt and close the gap between students with varying abilities, which becomes increasingly important in the context of online and distance learning. We run a comparative head-to-head study of learning outcomes for two popular online learning platforms: Platform A, which follows a traditional model delivering content over a series of lecture videos and multiple-choice quizzes, and Platform B, which creates a personalized learning environment and provides problem-solving exercises and personalized feedback. We report on the results of our study using pre- and post-assessment quizzes with participants taking courses on an introductory data science topic on two platforms. We observe a statistically significant increase in the learning outcomes on Platform B, highlighting the impact of well-designed and well-engineered technology supporting active learning and problem-based learning in online education. Moreover, the results of the self-assessment questionnaire, where participants reported on perceived learning gains, suggest that participants using Platform B improve their metacognition.
△ Less
Submitted 15 April, 2021;
originally announced April 2021.
