Deception Detection and Remote Physiological Monitoring: A Dataset and Baseline Experimental Results

• Published in: IEEE transactions on biometrics, behavior, and identity science Vol. 4; no. 4; p. 1
• Main Authors: Vance, Nathan, Speth, Jeremy, Khan, Siamul, Czajka, Adam, Bowyer, Kevin W., Wright, Diane, Flynn, Patrick
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01-10-2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Audio data; Cameras; Data collection; Datasets; Deception; Eye movements; Faces; Heart rate; Infrared radiation; Interviews; Monitoring; Oxygenation; Physiology; Plethysmography; Pupil size; Remote monitoring; Sensors; Synchronization; Videos
• ISSN: 2637-6407; 2637-6407
• DOI: 10.1109/TBIOM.2022.3218956

We present the Deception Detection and Physiological Monitoring (DDPM) dataset and initial baseline results of deception detection on this dataset. Our application context is an interview scenario in which the interviewee attempts to deceive the interviewer on selected responses. The interviewee is recorded in visible, near-infrared, and long-wave infrared light, along with cardiac pulse, blood oxygenation, and audio. After collection, data were annotated for interviewer/interviewee, curated, ground-truthed, and organized into train / test parts for a set of canonical deception detection experiments. For the collected data, we estimated subject heart rates (remotely from face videos with a mean absolute error lower than two beats per minute), pupil size, and eye gaze. Fusion experiments discovered that a combination of remote plethysmography, pupil size, and thermal data yields the best deception detection results, with an equal error rate of 0.357. The database contains almost 13 hours of recordings of 70 subjects, and over 8 million visible-light, near-infrared, and thermal video frames, along with appropriate meta, audio and pulse oximeter data. To our knowledge, this is the only collection offering recordings of five modalities in an interview scenario that can be used in both deception detection and remote photoplethysmography research.