Development of a Multimodal Student Engagement Level Prediction Dataset Using ISED and Classroom Video Data

Abstract

Accurate assessment of student engagement is essential for improving learning outcomes; however, existing engagement analysis methods primarily rely on behavioral logs or self-reported measures, which inadequately capture emotional and contextual dynamics present in real classroom environments. The absence of well-annotated, emotion-aware datasets further limits the development of reliable engagement prediction systems. To address this gap, this study proposes a structured multimodal dataset for student engagement analysis by integrating spontaneous facial emotion data from the Indian Spontaneous Expression Database [ISED] with real classroom video recordings. The dataset is constructed using a systematic methodology involving classroom video acquisition, frame extraction, emotion labelling, and intensity scoring. It comprises 120 participants recorded across 45 classroom sessions, resulting in 320 video samples, approximately 185,000 frames, and a total duration of 8 hours. Each sample is annotated with emotion categories and continuous intensity values, enabling fine-grained affective analysis. Dataset preprocessing and organization are performed using Python, with OpenCV for video processing and NumPy and Pandas for data management. To validate the usability of the dataset, baseline engagement prediction models are implemented using TensorFlow/Keras. Experimental evaluation demonstrates that models trained on the proposed multimodal dataset achieve an accuracy improvement of approximately 4–6% compared to models trained on unimodal or activity-log-based data. The results indicate that incorporating emotional and visual cues significantly enhances engagement representation and prediction reliability. In conclusion, the proposed dataset provides a comprehensive and ecologically valid resource for emotion-aware student engagement research. By offering explicit dataset-level statistics and validated baseline performance, this work supports future advancements in learning analytics, intelligent tutoring systems, and personalized education technologies.