Do Language Models Know the Way to Rome?

• Main Authors: Liétard, Bastien, Abdou, Mostafa, Søgaard, Anders
• Format: Journal Article
• Language: English
• Published: 16-09-2021
• Subjects: Computer Science - Computation and Language
• DOI: 10.48550/arxiv.2109.07971

BlackboxNLP Workshop 2021 The global geometry of language models is important for a range of
applications, but language model probes tend to evaluate rather local
relations, for which ground truths are easily obtained. In this paper we
exploit the fact that in geography, ground truths are available beyond local
relations. In a series of experiments, we evaluate the extent to which language
model representations of city and country names are isomorphic to real-world
geography, e.g., if you tell a language model where Paris and Berlin are, does
it know the way to Rome? We find that language models generally encode limited
geographic information, but with larger models performing the best, suggesting
that geographic knowledge can be induced from higher-order co-occurrence
statistics.