ATL-BP: A Student Engagement Dataset and Model for Affect Transfer Learning for Behavior Prediction

We propose a video-based transfer learning approach for predicting problem outcomes of students working with an intelligent tutoring system (ITS) by analyzing their faces and gestures. The ability to predict such outcomes enables tutoring systems to adjust interventions and ultimately yield improved...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE transactions on biometrics, behavior, and identity science Vol. 5; no. 3; pp. 411 - 424
Main Authors:	Ruiz, Nataniel, Yu, Hao, Allessio, Danielle A., Jalal, Mona, Joshi, Ajjen, Murray, Tom, Magee, John J., Delgado, Kevin Manuel, Ablavsky, Vitaly, Sclaroff, Stan, Arroyo, Ivon, Woolf, Beverly P., Bargal, Sarah Adel, Betke, Margrit
Format:	Journal Article
Language:	English
Published:	Piscataway IEEE 01-07-2023 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:	Annotations behavior prediction Behavioral sciences Datasets engagement prediction Faces Human behavior intelligent tutoring system Learning Predictions Predictive models Problem solving Recurrent neural networks Representations Students Task analysis Transfer learning Tutoring video classification Videos Webcams
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	We propose a video-based transfer learning approach for predicting problem outcomes of students working with an intelligent tutoring system (ITS) by analyzing their faces and gestures. The ability to predict such outcomes enables tutoring systems to adjust interventions and ultimately yield improved student learning. We collected and released a labeled dataset of 2,749 problem-solving interaction samples of 54 students working with an intelligent online math tutor. Our transfer-learning challenge was then to design a representation in the source domain of images obtained from the Internet for facial expression analysis, and transfer this learned representation for human behavior prediction in the domain of webcam videos of students in a classroom environment. We developed a novel facial affect representation and a user-personalized training scheme that unlocks the potential of this representation. We designed several variants of a recurrent neural network that models the temporal structure of video sequences. Our final model, named ATL-BP for Affect Transfer Learning for Behavior Prediction, achieves a relative increase in the mean F-score of 50% over the state-of-the-art method on this new dataset. We also propose an additional set of annotations to predict students' engagement while solving a specific problem, and present models that can predict such engagement.
ISSN:	2637-6407 2637-6407
DOI:	10.1109/TBIOM.2022.3210479