Spatiotemporal dynamics of audiovisual speech processing

• Published in: NeuroImage (Orlando, Fla.) Vol. 39; no. 1; pp. 423 - 435
• Main Authors: Bernstein, Lynne E., Auer, Edward T., Wagner, Michael, Ponton, Curtis W.
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 2008; Elsevier Limited
• Subjects: Adult; Brain - physiology; Brain Mapping; Evoked Potentials, Auditory - physiology; Evoked Potentials, Visual - physiology; Female; Humans; Language; Male; Phonetics; Speech; Speech Perception - physiology; Studies; Visual Perception - physiology
• ISSN: 1053-8119; 1095-9572
• DOI: 10.1016/j.neuroimage.2007.08.035

The cortical processing of auditory-alone, visual-alone, and audiovisual speech information is temporally and spatially distributed, and functional magnetic resonance imaging (fMRI) cannot adequately resolve its temporal dynamics. In order to investigate a hypothesized spatiotemporal organization for audiovisual speech processing circuits, event-related potentials (ERPs) were recorded using electroencephalography (EEG). Stimuli were congruent audiovisual /ba/, incongruent auditory /ba/ synchronized with visual /ga/, auditory-only /ba/, and visual-only /ba/ and /ga/. Current density reconstructions (CDRs) of the ERP data were computed across the latency interval of 50–250 ms. The CDRs demonstrated complex spatiotemporal activation patterns that differed across stimulus conditions. The hypothesized circuit that was investigated here comprised initial integration of audiovisual speech by the middle superior temporal sulcus (STS), followed by recruitment of the intraparietal sulcus (IPS), followed by activation of Broca’s area [Miller, L.M., d’Esposito, M., 2005. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. Journal of Neuroscience 25, 5884–5893]. The importance of spatiotemporally sensitive measures in evaluating processing pathways was demonstrated. Results showed, strikingly, early (<
100 ms) and simultaneous activations in areas of the supramarginal and angular gyrus (SMG/AG), the IPS, the inferior frontal gyrus, and the dorsolateral prefrontal cortex. Also, emergent left hemisphere SMG/AG activation, not predicted based on the unisensory stimulus conditions was observed at approximately 160 to 220 ms. The STS was neither the earliest nor most prominent activation site, although it is frequently considered the
sine qua non of audiovisual speech integration. As discussed here, the relatively late activity of the SMG/AG solely under audiovisual conditions is a possible candidate audiovisual speech integration response.