Data-driven experimental design and model development using Gaussian process with active learning

• Published in: Cognitive psychology Vol. 125; p. 101360
• Main Authors: Chang, Jorge, Kim, Jiseob, Zhang, Byoung-Tak, Pitt, Mark A., Myung, Jay I.
• Format: Journal Article
• Language: English
• Published: Netherlands Elsevier Inc 01-03-2021
• Subjects: Active learning; Bayes Theorem; Computational cognition; Data-driven cognitive modeling; Delay discounting; Gaussian process; Humans; Nonparametric Bayesian methods; Normal Distribution; Optimal experimental design; Research Design; Stochastic Processes; Data-driven cognitive modeling; Optimal experimental design; Gaussian process; Active learning; Nonparametric Bayesian methods; Delay discounting; Computational cognition
• ISSN: 0010-0285; 1095-5623
• DOI: 10.1016/j.cogpsych.2020.101360

•Propose a novel data-driven, model-free framework for optimal experimentation and model development.•The framework is built upon two ideas: nonparametric Bayes and active learning in machine learning.•Application of the framework is demonstrated with delay discounting experiments.•Two new models of discounting are introduced as a result of observing novel patterns of discounting.
Interest in computational modeling of cognition and behavior continues to grow. To be most productive, modelers should be equipped with tools that ensure optimal efficiency in data collection and in the integrity of inference about the phenomenon of interest. Traditionally, models in cognitive science have been parametric, which are particularly susceptible to model misspecification because their strong assumptions (e.g. parameterization, functional form) may introduce unjustified biases in data collection and inference. To address this issue, we propose a data-driven nonparametric framework for model development, one that also includes optimal experimental design as a goal. It combines Gaussian Processes, a stochastic process often used for regression and classification, with active learning, from machine learning, to iteratively fit the model and use it to optimize the design selection throughout the experiment. The approach, dubbed Gaussian process with active learning (GPAL), is an extension of the parametric, adaptive design optimization (ADO) framework (Cavagnaro, Myung, Pitt, & Kujala, 2010). We demonstrate the application and features of GPAL in a delay discounting task and compare its performance to ADO in two experiments. The results show that GPAL is a viable modeling framework that is noteworthy for its high sensitivity to individual differences, identifying novel patterns in the data that were missed by the model-constrained ADO. This investigation represents a first step towards the development of a data-driven cognitive modeling framework that serves as a middle ground between raw data, which can be difficult to interpret, and parametric models, which rely on strong assumptions.