ADOpy: a python package for adaptive design optimization

• Published in: Behavior research methods Vol. 53; no. 2; pp. 874 - 897
• Main Authors: Yang, Jaeyeong, Pitt, Mark A., Ahn, Woo-Young, Myung, Jay I.
• Format: Journal Article
• Language: English
• Published: New York Springer US 01-04-2021; Springer Nature B.V
• Subjects: Algorithms; Bayesian analysis; Behavioral Science and Psychology; Cognitive Psychology; Computer applications; Data collection; Design optimization; Learning algorithms; Machine learning; Psychology; Statistical analysis; Risky choice; Bayesian adaptive experimentation; Psychometric function estimation; Cognitive modeling; Optimal experimental design; Delay discounting
• ISSN: 1554-3528; 1554-351X; 1554-3528
• DOI: 10.3758/s13428-020-01386-4

Experimental design is fundamental to research, but formal methods to identify good designs are lacking. Advances in Bayesian statistics and machine learning offer algorithm-based ways to identify good experimental designs. Adaptive design optimization (ADO; Cavagnaro, Myung, Pitt, & Kujala, 2010; Myung, Cavagnaro, & Pitt, 2013) is one such method. It works by maximizing the informativeness and efficiency of data collection, thereby improving inference. ADO is a general-purpose method for conducting adaptive experiments on the fly and can lead to rapid accumulation of information about the phenomenon of interest with the fewest number of trials. The nontrivial technical skills required to use ADO have been a barrier to its wider adoption. To increase its accessibility to experimentalists at large, we introduce an open-source Python package, ADOpy, that implements ADO for optimizing experimental design. The package, available on GitHub, is written using high-level modular-based commands such that users do not have to understand the computational details of the ADO algorithm. In this paper, we first provide a tutorial introduction to ADOpy and ADO itself, and then illustrate its use in three walk-through examples: psychometric function estimation, delay discounting, and risky choice. Simulation data are also provided to demonstrate how ADO designs compare with other designs (random, staircase).