Mapping languages: the Corpus of Global Language Use

• Published in: Language resources and evaluation Vol. 54; no. 4; pp. 999 - 1018
• Main Author: Dunn, Jonathan
• Format: Journal Article
• Language: English
• Published: Dordrecht Springer Netherlands 01-12-2020; Springer Nature B.V
• Subjects: Arabic language; Computational Linguistics; Computer Science; Corpus analysis; Corpus linguistics; Data collection; Digital data; Language and Literature; Language identification; Languages; Linguistics; Mapping; Original Paper; Populations; Russian language; Social Sciences; Geo-referenced corpus; Geographic corpus; Language mapping; Language identification; Register similarity; Demographic mapping
• ISSN: 1574-020X; 1574-0218
• DOI: 10.1007/s10579-020-09489-2

This paper describes a web-based corpus of global language use with a focus on how this corpus can be used for data-driven language mapping. First, the corpus provides a representation of where national varieties of major languages are used (e.g., English, Arabic, Russian) together with consistently collected data for each variety. Second, the paper evaluates a language identification model that supports more local languages with smaller sample sizes than alternative off-the-shelf models. Improved language identification is essential for moving beyond majority languages. Given the focus on language mapping, the paper analyzes how well this digital language data represents actual populations by (i) systematically comparing the corpus with demographic ground-truth data and (ii) triangulating the corpus with an alternate Twitter-based dataset. In total, the corpus contains 423 billion words representing 148 languages (with over 1 million words from each language) and 158 countries (again with over 1 million words from each country), all distilled from Common Crawl web data. The main contribution of this paper, in addition to describing this publicly-available corpus, is to provide a comprehensive analysis of the relationship between two sources of digital data (the web and Twitter) as well as their connection to underlying populations.