Storywrangler: A massive exploratorium for sociolinguistic, cultural, socioeconomic, and political timelines using Twitter

• Published in: Science advances Vol. 7; no. 29
• Main Authors: Alshaabi, Thayer, Adams, Jane L, Arnold, Michael V, Minot, Joshua R, Dewhurst, David R, Reagan, Andrew J, Danforth, Christopher M, Dodds, Peter Sheridan
• Format: Journal Article
• Language: English
• Published: United States American Association for the Advancement of Science 01-07-2021
• Subjects: Computer Science; Research Resource; SciAdv r-resources; Social Sciences
• ISSN: 2375-2548; 2375-2548
• DOI: 10.1126/sciadv.abe6534

In real time, Twitter strongly imprints world events, popular culture, and the day-to-day, recording an ever-growing compendium of language change. Vitally, and absent from many standard corpora such as books and news archives, Twitter also encodes popularity and spreading through retweets. Here, we describe Storywrangler, an ongoing curation of over 100 billion tweets containing 1 trillion 1-grams from 2008 to 2021. For each day, we break tweets into 1-, 2-, and 3-grams across 100+ languages, generating frequencies for words, hashtags, handles, numerals, symbols, and emojis. We make the dataset available through an interactive time series viewer and as downloadable time series and daily distributions. Although Storywrangler leverages Twitter data, our method of tracking dynamic changes in
-grams can be extended to any temporally evolving corpus. Illustrating the instrument's potential, we present example use cases including social amplification, the sociotechnical dynamics of famous individuals, box office success, and social unrest.