Plug-and-play supervisory control using muscle and brain signals for real-time gesture and error detection

• Published in: Autonomous robots Vol. 44; no. 7; pp. 1303 - 1322
• Main Authors: DelPreto, Joseph, Salazar-Gomez, Andres F., Gil, Stephanie, Hasani, Ramin, Guenther, Frank H., Rus, Daniela
• Format: Journal Article
• Language: English
• Published: New York Springer US 01-09-2020; Springer Nature B.V
• Subjects: Artificial Intelligence; Brain; Computer Imaging; Control; Control tasks; Engineering; Error correction; Error detection; Hybrid systems; Mechatronics; Muscles; Pattern Recognition and Graphics; Robotics; Robotics and Automation; Robots; Safety critical; Signal classification; Special Issue: Robotics: Science and Systems 2018; Supervisory control; Vision; Error-related potentials; EEG control; Hybrid control; Gesture detection; Human–robot interaction; EMG control; Plug-and-play supervisory control
• ISSN: 0929-5593; 1573-7527
• DOI: 10.1007/s10514-020-09916-x

Effective human supervision of robots can be key for ensuring correct robot operation in a variety of potentially safety-critical scenarios. This paper takes a step towards fast and reliable human intervention in supervisory control tasks by combining two streams of human biosignals: muscle and brain activity acquired via EMG and EEG, respectively. It presents continuous classification of left and right hand-gestures using muscle signals, time-locked classification of error-related potentials using brain signals (unconsciously produced when observing an error), and a framework that combines these pipelines to detect and correct robot mistakes during multiple-choice tasks. The resulting hybrid system is evaluated in a “plug-and-play” fashion with 7 untrained subjects supervising an autonomous robot performing a target selection task. Offline analysis further explores the EMG classification performance, and investigates methods to select subsets of training data that may facilitate generalizable plug-and-play classifiers.