Data-Driven Hint Generation For Alloy Using Historial Student Submissions

• Main Author: de Barros, Ana Inês Oliveira
• Format: Dissertation
• Language: English
• Published: ProQuest Dissertations & Theses 01-01-2023
• Subjects: Computer science; Data mining; Education; Independent study; Nuclear engineering; Nuclear power plants; Problem solving; Software; Syntax; Teachers; Tutoring
• ISBN: 9798383353035

As technology takes on critical tasks, professionals are starting to embrace formal specifications to design and implement reliable software. However, writing and understanding formal specifications can be challenging even with state-of-the-art tools. At the time of writing, Alloy4Fun is the only automated assessment platform for teaching formal specifications. Nonetheless, this platform could benefit from personalized user feedback, an essential component of the learning process. Automatic hint-generation feedback systems for programming have a long history of research focusing on data-driven techniques. The state-of-the-art ideas take previous students’ solutions to infer the most common fix steps to a successful submission.This work resides in the context of an FCT-funded project, SpecRep, which includes tasks to improve the feedback provided by Alloy4Fun. Recent experiments with Alloy4Fun indicate that most students need help understanding why their specifications fail. This situation leads to unmotivated students by making independent study more difficult. Fortunately, the dataset of students’ submissions is publicly available, opening the possibility of developing history-based approaches for Alloy.This work presents HiGenA, the Hint Generator for Alloy, a tool that allows the automatic generation of history-based hints in Alloy integrated with Alloy4Fun. It takes advantage of students’ past attempts to solve an exercise to provide hints to new students. The generated hints are based on edit operations at the abstract syntax tree level and consist of small textual messages guiding students toward a solution.The goal of HiGenA is to improve the student learning process. To understand if we achieved this goal, we conducted experiments and compared the results of this work with state-of-the-art hint generation techniques for Alloy and other programming languages. We also conducted a user study with former Alloy students and a teacher to evaluate the quality and effectiveness of the generated hints. The results show that HiGenA significantly improved over the previous Alloy repair technique and that the generated hints contribute to the learning process.