Co-speech iconic gestures and visuo-spatial working memory

• Published in: Acta psychologica Vol. 153; no. Nov; pp. 39 - 50
• Main Authors: Wu, Ying Choon, Coulson, Seana
• Format: Journal Article
• Language: English
• Published: Kidlington Elsevier B.V 01-11-2014; Elsevier
• Subjects: Adult; Biological and medical sciences; Comprehension - physiology; Dual task methodology; Female; Fundamental and applied biological sciences. Psychology; Gesture comprehension; Gestures; Human; Humans; Iconic gestures; Individuality; Language; Learning. Memory; Male; Memory; Memory, Short-Term - physiology; Multi-modal discourse; Production and perception of spoken language; Psychology. Psychoanalysis. Psychiatry; Psychology. Psychophysiology; Psychomotor Performance - physiology; Space Perception - physiology; Spatial cognition; Speech Perception - physiology; Working memory; Young Adult; 2320 Sensory Perception; Iconic gestures; Spatial cognition; 2343 Learning & Memory; Gesture comprehension; Multi-modal discourse; Working memory; 2720 Linguistics & Language & Speech; Dual task methodology; Human; Visual memory; Methodology; Cognition; Comprehension; Experimental study; Discourse; Verbal memory; Language; Gesture; Concurrent task; Speech; Spatial memory
• DOI: 10.1016/j.actpsy.2014.09.002
• EISSN: 1873-6297
• ISSN: 0001-6918

Three experiments tested the role of verbal versus visuo-spatial working memory in the comprehension of co-speech iconic gestures. In Experiment 1, participants viewed congruent discourse primes in which the speaker's gestures matched the information conveyed by his speech, and incongruent ones in which the semantic content of the speaker's gestures diverged from that in his speech. Discourse primes were followed by picture probes that participants judged as being either related or unrelated to the preceding clip. Performance on this picture probe classification task was faster and more accurate after congruent than incongruent discourse primes. The effect of discourse congruency on response times was linearly related to measures of visuo-spatial, but not verbal, working memory capacity, as participants with greater visuo-spatial WM capacity benefited more from congruent gestures. In Experiments 2 and 3, participants performed the same picture probe classification task under conditions of high and low loads on concurrent visuo-spatial (Experiment 2) and verbal (Experiment 3) memory tasks. Effects of discourse congruency and verbal WM load were additive, while effects of discourse congruency and visuo-spatial WM load were interactive. Results suggest that congruent co-speech gestures facilitate multi-modal language comprehension, and indicate an important role for visuo-spatial WM in these speech–gesture integration processes.
•Listeners integrate meaning expressed in their interlocutors' speech and gestures.•Sensitivity to gesture is positively related to visuo-spatial working memory ability.•When visuo-spatial working memory is taxed, sensitivity to gestures declines.•No relationship was found between sensitivity to gestures and verbal working memory.