Toward a universal decoder of linguistic meaning from brain activation

• Published in: Nature communications Vol. 9; no. 1; pp. 963 - 13
• Main Authors: Pereira, Francisco, Lou, Bin, Pritchett, Brianna, Ritter, Samuel, Gershman, Samuel J., Kanwisher, Nancy, Botvinick, Matthew, Fedorenko, Evelina
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 06-03-2018; Nature Publishing Group; Nature Portfolio
• Subjects: 631/378/116/2394; 639/705/1042; Adult; Brain; Brain Mapping; Decoding; Female; Humanities and Social Sciences; Humans; Imaging; Magnetic Resonance Imaging; Male; Mathematical analysis; multidisciplinary; Neuroimaging; Physical Stimulation; Representations; Science; Science (multidisciplinary); Semantics; Sentences; Stimuli; Training; Vectors (mathematics)
• ISSN: 2041-1723; 2041-1723
• DOI: 10.1038/s41467-018-03068-4

Prior work decoding linguistic meaning from imaging data has been largely limited to concrete nouns, using similar stimuli for training and testing, from a relatively small number of semantic categories. Here we present a new approach for building a brain decoding system in which words and sentences are represented as vectors in a semantic space constructed from massive text corpora. By efficiently sampling this space to select training stimuli shown to subjects, we maximize the ability to generalize to new meanings from limited imaging data. To validate this approach, we train the system on imaging data of individual concepts, and show it can decode semantic vector representations from imaging data of sentences about a wide variety of both concrete and abstract topics from two separate datasets. These decoded representations are sufficiently detailed to distinguish even semantically similar sentences, and to capture the similarity structure of meaning relationships between sentences.
Previous work decoding linguistic meaning from imaging data has generally been limited to a small number of semantic categories. Here, authors show that a decoder trained on neuroimaging data of single concepts sampling the semantic space can robustly decode meanings of semantically diverse new sentences with topics not encountered during training.