Understanding students' performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log files

• Published in: Computers in human behavior Vol. 61; pp. 36 - 46
• Main Authors: Greiff, Samuel, Niepel, Christoph, Scherer, Ronny, Martin, Romain
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01-08-2016
• Subjects: Assessments; Complex problem solving; Computer-based assessment; Exploration; High frequencies; Human behavior; Log file data; Nonlinear relations; Problem solving; Strategy; Students; Tasks; Time on task; VOTAT; Log file data; Complex problem solving; Computer-based assessment; Nonlinear relations; Time on task; VOTAT
• ISSN: 0747-5632; 1873-7692
• DOI: 10.1016/j.chb.2016.02.095

Computer-based assessments of complex problem solving (CPS) that have been used in international large-scale surveys require students to engage in an in-depth interaction with the problem environment. In this, they evoke manifest sequences of overt behavior that are stored in computer-generated log files. In the present study, we explored the relation between several overt behaviors, which N = 1476 Finnish ninth-grade students (mean age = 15.23, SD = .47 years) exhibited when exploring a CPS environment, and their CPS performance. We used the MicroDYN approach to measure CPS and inspected students' behaviors through log-file analyses. Results indicated that students who occasionally observed the problem environment in a noninterfering way in addition to actively exploring it (noninterfering observation) showed better CPS performance, whereas students who showed a high frequency of (potentially unplanned) interventions (intervention frequency) exhibited worse CPS performance. Additionally, both too much and too little time spent on a CPS task (time on task) was associated with poor CPS performance. The observed effects held after controlling for students' use of an exploration strategy that required a sequence of multiple interventions (VOTAT strategy) indicating that these behaviors exhibited incremental effects on CPS performance beyond the use of VOTAT.
•Complex Problem Solving (CPS) tasks require students to engage into in-depth interaction with the environment.•We investigated how behavioral indicators captured in computer-generated log files related to CPS performance.•The relation between time on task and CPS performance followed an inverted u-shape.•Noninterfering observation was positively, high intervention frequency negatively related to CPS performance.•Relations decreased in size but remained stable after controlling for a multistep exploration strategy (VOTAT).