-
Visual Attention Analysis in Online Learning
Authors:
Miriam Navarro,
Álvaro Becerra,
Roberto Daza,
Ruth Cobos,
Aythami Morales,
Julian Fierrez
Abstract:
In this paper, we present an approach in the Multimodal Learning Analytics field. Within this approach, we have developed a tool to visualize and analyze eye movement data collected during learning sessions in online courses. The tool is named VAAD (an acronym for Visual Attention Analysis Dashboard). These eye movement data have been gathered using an eye-tracker and subsequently processed and vi…
▽ More
In this paper, we present an approach in the Multimodal Learning Analytics field. Within this approach, we have developed a tool to visualize and analyze eye movement data collected during learning sessions in online courses. The tool is named VAAD (an acronym for Visual Attention Analysis Dashboard). These eye movement data have been gathered using an eye-tracker and subsequently processed and visualized for interpretation. The purpose of the tool is to conduct a descriptive analysis of the data by facilitating its visualization, enabling the identification of differences and learning patterns among various learner populations. Additionally, it integrates a predictive module capable of anticipating learner activities during a learning session. Consequently, VAAD holds the potential to offer valuable insights into online learning behaviors from both descriptive and predictive perspectives.
△ Less
Submitted 31 May, 2024; v1 submitted 30 May, 2024;
originally announced May 2024.
-
Biometrics and Behavioral Modelling for Detecting Distractions in Online Learning
Authors:
Álvaro Becerra,
Javier Irigoyen,
Roberto Daza,
Ruth Cobos,
Aythami Morales,
Julian Fierrez,
Mutlu Cukurova
Abstract:
In this article, we explore computer vision approaches to detect abnormal head pose during e-learning sessions and we introduce a study on the effects of mobile phone usage during these sessions. We utilize behavioral data collected from 120 learners monitored while participating in a MOOC learning sessions. Our study focuses on the influence of phone-usage events on behavior and physiological res…
▽ More
In this article, we explore computer vision approaches to detect abnormal head pose during e-learning sessions and we introduce a study on the effects of mobile phone usage during these sessions. We utilize behavioral data collected from 120 learners monitored while participating in a MOOC learning sessions. Our study focuses on the influence of phone-usage events on behavior and physiological responses, specifically attention, heart rate, and meditation, before, during, and after phone usage. Additionally, we propose an approach for estimating head pose events using images taken by the webcam during the MOOC learning sessions to detect phone-usage events. Our hypothesis suggests that head posture undergoes significant changes when learners interact with a mobile phone, contrasting with the typical behavior seen when learners face a computer during e-learning sessions. We propose an approach designed to detect deviations in head posture from the average observed during a learner's session, operating as a semi-supervised method. This system flags events indicating alterations in head posture for subsequent human review and selection of mobile phone usage occurrences with a sensitivity over 90%.
△ Less
Submitted 24 May, 2024;
originally announced May 2024.
-
mEBAL2 Database and Benchmark: Image-based Multispectral Eyeblink Detection
Authors:
Roberto Daza,
Aythami Morales,
Julian Fierrez,
Ruben Tolosana,
Ruben Vera-Rodriguez
Abstract:
This work introduces a new multispectral database and novel approaches for eyeblink detection in RGB and Near-Infrared (NIR) individual images. Our contributed dataset (mEBAL2, multimodal Eye Blink and Attention Level estimation, Version 2) is the largest existing eyeblink database, representing a great opportunity to improve data-driven multispectral approaches for blink detection and related app…
▽ More
This work introduces a new multispectral database and novel approaches for eyeblink detection in RGB and Near-Infrared (NIR) individual images. Our contributed dataset (mEBAL2, multimodal Eye Blink and Attention Level estimation, Version 2) is the largest existing eyeblink database, representing a great opportunity to improve data-driven multispectral approaches for blink detection and related applications (e.g., attention level estimation and presentation attack detection in face biometrics). mEBAL2 includes 21,100 image sequences from 180 different students (more than 2 million labeled images in total) while conducting a number of e-learning tasks of varying difficulty or taking a real course on HTML initiation through the edX MOOC platform. mEBAL2 uses multiple sensors, including two Near-Infrared (NIR) and one RGB camera to capture facial gestures during the execution of the tasks, as well as an Electroencephalogram (EEG) band to get the cognitive activity of the user and blinking events. Furthermore, this work proposes a Convolutional Neural Network architecture as benchmark for blink detection on mEBAL2 with performances up to 97%. Different training methodologies are implemented using the RGB spectrum, NIR spectrum, and the combination of both to enhance the performance on existing eyeblink detectors. We demonstrate that combining NIR and RGB images during training improves the performance of RGB eyeblink detectors (i.e., detection based only on a RGB image). Finally, the generalization capacity of the proposed eyeblink detectors is validated in wilder and more challenging environments like the HUST-LEBW dataset to show the usefulness of mEBAL2 to train a new generation of data-driven approaches for eyeblink detection.
△ Less
Submitted 26 April, 2024; v1 submitted 14 September, 2023;
originally announced September 2023.
-
Estudio de la Experiencia de Usuario mediante un Sistema de Dashboards de Análisis de Aprendizaje Multimodal
Authors:
Álvaro Becerra,
Roberto Daza,
Ruth Cobos,
Aythami Morales,
Julian Fierrez
Abstract:
In the article, we present a Web-based System called M2LADS, which supports the integration and visualization of multimodal data recorded in user experiences (UX) in a Learning Analytics (LA) system in the form of Web-based Dashboards. Based on the edBB platform, the multimodal data gathered contains biometric and behavioral signals including electroencephalogram data to measure learners' cognitiv…
▽ More
In the article, we present a Web-based System called M2LADS, which supports the integration and visualization of multimodal data recorded in user experiences (UX) in a Learning Analytics (LA) system in the form of Web-based Dashboards. Based on the edBB platform, the multimodal data gathered contains biometric and behavioral signals including electroencephalogram data to measure learners' cognitive attention, heart rate for affective measures and visual attention from the video recordings. Additionally, learners' static background data and their learning performance measures are tracked using LOGGE tool. M2LADS provides opportunities to capture learners' holistic experience during their interactions with the learning analytic system in order to improve the system and the user experience of the learners.
--
En este artículo, presentamos M2LADS, un sistema que permite la integración y visualización de datos multimodales en forma de Dashboards Web. Estos datos provienen de sesiones de experiencia de usuario en un sistema de Learning Analytics (LA) llevadas a cabo por estudiantes de MOOCs. Los datos multimodales incluyen señales biométricas y de comportamiento monitorizados por la plataforma edBB, como electroencefalogramas (EEG) de 5 canales, frecuencia cardíaca, atención visual, videos en el espectro visible y NIR, entre otros. Además, se incluyen datos de interacción de los estudiantes con el sistema de LA a través de la herramienta LOGGE. Toda esta información proporciona una comprensión completa de la experiencia del usuario al utilizar el sistema de LA, lo que ha permitido tanto mejorar el sistema LA como la experiencia de aprendizaje de los estudiantes de MOOCs.
△ Less
Submitted 19 July, 2023;
originally announced July 2023.
-
M2LADS: A System for Generating MultiModal Learning Analytics Dashboards in Open Education
Authors:
Álvaro Becerra,
Roberto Daza,
Ruth Cobos,
Aythami Morales,
Mutlu Cukurova,
Julian Fierrez
Abstract:
In this article, we present a Web-based System called M2LADS, which supports the integration and visualization of multimodal data recorded in learning sessions in a MOOC in the form of Web-based Dashboards. Based on the edBB platform, the multimodal data gathered contains biometric and behavioral signals including electroencephalogram data to measure learners' cognitive attention, heart rate for a…
▽ More
In this article, we present a Web-based System called M2LADS, which supports the integration and visualization of multimodal data recorded in learning sessions in a MOOC in the form of Web-based Dashboards. Based on the edBB platform, the multimodal data gathered contains biometric and behavioral signals including electroencephalogram data to measure learners' cognitive attention, heart rate for affective measures, visual attention from the video recordings. Additionally, learners' static background data and their learning performance measures are tracked using LOGCE and MOOC tracking logs respectively, and both are included in the Web-based System. M2LADS provides opportunities to capture learners' holistic experience during their interactions with the MOOC, which can in turn be used to improve their learning outcomes through feedback visualizations and interventions, as well as to enhance learning analytics models and improve the open content of the MOOC.
△ Less
Submitted 21 May, 2023;
originally announced May 2023.
-
MATT: Multimodal Attention Level Estimation for e-learning Platforms
Authors:
Roberto Daza,
Luis F. Gomez,
Aythami Morales,
Julian Fierrez,
Ruben Tolosana,
Ruth Cobos,
Javier Ortega-Garcia
Abstract:
This work presents a new multimodal system for remote attention level estimation based on multimodal face analysis. Our multimodal approach uses different parameters and signals obtained from the behavior and physiological processes that have been related to modeling cognitive load such as faces gestures (e.g., blink rate, facial actions units) and user actions (e.g., head pose, distance to the ca…
▽ More
This work presents a new multimodal system for remote attention level estimation based on multimodal face analysis. Our multimodal approach uses different parameters and signals obtained from the behavior and physiological processes that have been related to modeling cognitive load such as faces gestures (e.g., blink rate, facial actions units) and user actions (e.g., head pose, distance to the camera). The multimodal system uses the following modules based on Convolutional Neural Networks (CNNs): Eye blink detection, head pose estimation, facial landmark detection, and facial expression features. First, we individually evaluate the proposed modules in the task of estimating the student's attention level captured during online e-learning sessions. For that we trained binary classifiers (high or low attention) based on Support Vector Machines (SVM) for each module. Secondly, we find out to what extent multimodal score level fusion improves the attention level estimation. The mEBAL database is used in the experimental framework, a public multi-modal database for attention level estimation obtained in an e-learning environment that contains data from 38 users while conducting several e-learning tasks of variable difficulty (creating changes in student cognitive loads).
△ Less
Submitted 22 January, 2023;
originally announced January 2023.
-
edBB-Demo: Biometrics and Behavior Analysis for Online Educational Platforms
Authors:
Roberto Daza,
Aythami Morales,
Ruben Tolosana,
Luis F. Gomez,
Julian Fierrez,
Javier Ortega-Garcia
Abstract:
We present edBB-Demo, a demonstrator of an AI-powered research platform for student monitoring in remote education. The edBB platform aims to study the challenges associated to user recognition and behavior understanding in digital platforms. This platform has been developed for data collection, acquiring signals from a variety of sensors including keyboard, mouse, webcam, microphone, smartwatch,…
▽ More
We present edBB-Demo, a demonstrator of an AI-powered research platform for student monitoring in remote education. The edBB platform aims to study the challenges associated to user recognition and behavior understanding in digital platforms. This platform has been developed for data collection, acquiring signals from a variety of sensors including keyboard, mouse, webcam, microphone, smartwatch, and an Electroencephalography band. The information captured from the sensors during the student sessions is modelled in a multimodal learning framework. The demonstrator includes: i) Biometric user authentication in an unsupervised environment; ii) Human action recognition based on remote video analysis; iii) Heart rate estimation from webcam video; and iv) Attention level estimation from facial expression analysis.
△ Less
Submitted 5 December, 2022; v1 submitted 16 November, 2022;
originally announced November 2022.
-
ALEBk: Feasibility Study of Attention Level Estimation via Blink Detection applied to e-Learning
Authors:
Roberto Daza,
Daniel DeAlcala,
Aythami Morales,
Ruben Tolosana,
Ruth Cobos,
Julian Fierrez
Abstract:
This work presents a feasibility study of remote attention level estimation based on eye blink frequency. We first propose an eye blink detection system based on Convolutional Neural Networks (CNNs), very competitive with respect to related works. Using this detector, we experimentally evaluate the relationship between the eye blink rate and the attention level of students captured during online s…
▽ More
This work presents a feasibility study of remote attention level estimation based on eye blink frequency. We first propose an eye blink detection system based on Convolutional Neural Networks (CNNs), very competitive with respect to related works. Using this detector, we experimentally evaluate the relationship between the eye blink rate and the attention level of students captured during online sessions. The experimental framework is carried out using a public multimodal database for eye blink detection and attention level estimation called mEBAL, which comprises data from 38 students and multiples acquisition sensors, in particular, i) an electroencephalogram (EEG) band which provides the time signals coming from the student's cognitive information, and ii) RGB and NIR cameras to capture the students face gestures. The results achieved suggest an inverse correlation between the eye blink frequency and the attention level. This relation is used in our proposed method called ALEBk for estimating the attention level as the inverse of the eye blink frequency. Our results open a new research line to introduce this technology for attention level estimation on future e-learning platforms, among other applications of this kind of behavioral biometrics based on face analysis.
△ Less
Submitted 16 December, 2021;
originally announced December 2021.
-
mEBAL: A Multimodal Database for Eye Blink Detection and Attention Level Estimation
Authors:
Roberto Daza,
Aythami Morales,
Julian Fierrez,
Ruben Tolosana
Abstract:
This work presents mEBAL, a multimodal database for eye blink detection and attention level estimation. The eye blink frequency is related to the cognitive activity and automatic detectors of eye blinks have been proposed for many tasks including attention level estimation, analysis of neuro-degenerative diseases, deception recognition, drive fatigue detection, or face anti-spoofing. However, most…
▽ More
This work presents mEBAL, a multimodal database for eye blink detection and attention level estimation. The eye blink frequency is related to the cognitive activity and automatic detectors of eye blinks have been proposed for many tasks including attention level estimation, analysis of neuro-degenerative diseases, deception recognition, drive fatigue detection, or face anti-spoofing. However, most existing databases and algorithms in this area are limited to experiments involving only a few hundred samples and individual sensors like face cameras. The proposed mEBAL improves previous databases in terms of acquisition sensors and samples. In particular, three different sensors are simultaneously considered: Near Infrared (NIR) and RGB cameras to capture the face gestures and an Electroencephalography (EEG) band to capture the cognitive activity of the user and blinking events. Regarding the size of mEBAL, it comprises 6,000 samples and the corresponding attention level from 38 different students while conducting a number of e-learning tasks of varying difficulty. In addition to presenting mEBAL, we also include preliminary experiments on: i) eye blink detection using Convolutional Neural Networks (CNN) with the facial images, and ii) attention level estimation of the students based on their eye blink frequency.
△ Less
Submitted 22 October, 2020; v1 submitted 9 June, 2020;
originally announced June 2020.
-
Heart Rate Estimation from Face Videos for Student Assessment: Experiments on edBB
Authors:
Javier Hernandez-Ortega,
Roberto Daza,
Aythami Morales,
Julian Fierrez,
Ruben Tolosana
Abstract:
In this study we estimate the heart rate from face videos for student assessment. This information could be very valuable to track their status along time and also to estimate other data such as their attention level or the presence of stress that may be caused by cheating attempts. The recent edBBplat, a platform for student behavior modelling in remote education, is considered in this study1. Th…
▽ More
In this study we estimate the heart rate from face videos for student assessment. This information could be very valuable to track their status along time and also to estimate other data such as their attention level or the presence of stress that may be caused by cheating attempts. The recent edBBplat, a platform for student behavior modelling in remote education, is considered in this study1. This platform permits to capture several signals from a set of sensors that capture biometric and behavioral data: RGB and near infrared cameras, microphone, EEG band, mouse, smartwatch, and keyboard, among others. In the experimental framework of this study, we focus on the RGB and near-infrared video sequences for performing heart rate estimation applying remote photoplethysmography techniques. The experiments include behavioral and physiological data from 25 different students completing a collection of tasks related to e-learning. Our proposed face heart rate estimation approach is compared with the heart rate provided by the smartwatch, achieving very promising results for its future deployment in e-learning applications.
△ Less
Submitted 1 June, 2020;
originally announced June 2020.
-
edBB: Biometrics and Behavior for Assessing Remote Education
Authors:
Javier Hernandez-Ortega,
Roberto Daza,
Aythami Morales,
Julian Fierrez,
Javier Ortega-Garcia
Abstract:
We present a platform for student monitoring in remote education consisting of a collection of sensors and software that capture biometric and behavioral data. We define a collection of tasks to acquire behavioral data that can be useful for facing the existing challenges in automatic student monitoring during remote evaluation. Additionally, we release an initial database including data from 20 d…
▽ More
We present a platform for student monitoring in remote education consisting of a collection of sensors and software that capture biometric and behavioral data. We define a collection of tasks to acquire behavioral data that can be useful for facing the existing challenges in automatic student monitoring during remote evaluation. Additionally, we release an initial database including data from 20 different users completing these tasks with a set of basic sensors: webcam, microphone, mouse, and keyboard; and also from more advanced sensors: NIR camera, smartwatch, additional RGB cameras, and an EEG band. Information from the computer (e.g. system logs, MAC, IP, or web browsing history) is also stored. During each acquisition session each user completed three different types of tasks generating data of different nature: mouse and keystroke dynamics, face data, and audio data among others. The tasks have been designed with two main goals in mind: i) analyse the capacity of such biometric and behavioral data for detecting anomalies during remote evaluation, and ii) study the capability of these data, i.e. EEG, ECG, or NIR video, for estimating other information about the users such as their attention level, the presence of stress, or their pulse rate.
△ Less
Submitted 10 December, 2019;
originally announced December 2019.
