Exploring Spatial Schema Intuitions in Large Language and Vision Models

• Main Authors: Wicke, Philipp, Wachowiak, Lennart
• Format: Journal Article
• Language: English
• Published: 01-02-2024
• Subjects: Computer Science - Computation and Language
• DOI: 10.48550/arxiv.2402.00956

Despite the ubiquity of large language models (LLMs) in AI research, the
question of embodiment in LLMs remains underexplored, distinguishing them from
embodied systems in robotics where sensory perception directly informs physical
action. Our investigation navigates the intriguing terrain of whether LLMs,
despite their non-embodied nature, effectively capture implicit human
intuitions about fundamental, spatial building blocks of language. We employ
insights from spatial cognitive foundations developed through early
sensorimotor experiences, guiding our exploration through the reproduction of
three psycholinguistic experiments. Surprisingly, correlations between model
outputs and human responses emerge, revealing adaptability without a tangible
connection to embodied experiences. Notable distinctions include polarized
language model responses and reduced correlations in vision language models.
This research contributes to a nuanced understanding of the interplay between
language, spatial experiences, and the computations made by large language
models. More at https://cisnlp.github.io/Spatial_Schemas/