A pragmatic guide to geoparsing evaluation Toponyms, Named Entity Recognition and pragmatics

• Published in: Language resources and evaluation Vol. 54; no. 3; pp. 683 - 712
• Main Authors: Gritta, Milan, Pilehvar, Mohammad Taher, Collier, Nigel
• Format: Journal Article
• Language: English
• Published: Dordrecht Springer Netherlands 01-09-2020
• Subjects: Computational Linguistics; Computer Science; Language and Literature; Linguistics; Original Paper; Social Sciences; Evaluation framework; Geotagging; Named Entity Recognition; Geoparsing; Machine learning; Geonames; Toponym resolution; Geocoding; Toponyms; Natural language understanding; Pragmatics
• ISSN: 1574-020X; 1574-0218
• DOI: 10.1007/s10579-019-09475-3

Empirical methods in geoparsing have thus far lacked a standard evaluation framework describing the task, metrics and data used to compare state-of-the-art systems. Evaluation is further made inconsistent, even unrepresentative of real world usage by the lack of distinction between the
different types of toponyms
, which necessitates new guidelines, a consolidation of metrics and a detailed toponym taxonomy with implications for Named Entity Recognition (NER) and beyond. To address these deficiencies, our manuscript introduces a new framework in three parts. (Part 1) Task Definition: clarified via corpus linguistic analysis proposing a fine-grained
Pragmatic Taxonomy of Toponyms
. (Part 2) Metrics: discussed and reviewed for a rigorous evaluation including recommendations for NER/Geoparsing practitioners. (Part 3) Evaluation data: shared via a new dataset called
GeoWebNews
to provide test/train examples and enable immediate use of our contributions. In addition to fine-grained Geotagging and Toponym Resolution (Geocoding), this dataset is also suitable for prototyping and evaluating machine learning NLP models.