Auditing Geospatial Datasets for Biases: Using Global Building Datasets for Disaster Risk Management

• Published in: IEEE journal of selected topics in applied earth observations and remote sensing Vol. 17; pp. 12579 - 12590
• Main Authors: Gevaert, Caroline M., Buunk, Thomas, van den Homberg, Marc J.C.
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial intelligence; Bias; building detection; Buildings; Datasets; Digital mapping; Disaster management; Disaster risk; Disasters; Emergency preparedness; equity; ethics; Geospatial analysis; humanitarian aid; Internet; Machine learning; Population density; Population studies; Risk management; Sociology; Statistics; Tanzania
• ISSN: 1939-1404; 2151-1535
• DOI: 10.1109/JSTARS.2024.3422503

The presence of biases has been demonstrated in a wide range of machine learning applications; however, it is not yet widespread in the case of geospatial datasets. This study illustrates the importance of auditing geospatial datasets for biases, with a particular focus on disaster risk management applications, as a lack of local data may direct humanitarian actors to utilize global building datasets to estimate damage and the distribution of aid efforts. It is important to ensure that there are no biases against the representation of vulnerable populations and that they are not missed in the distribution of aid. This manuscript audits four global building datasets [Google Open Buildings, Microsoft Bing Maps Building Footprints, Overture Maps Foundation (OMF), and OpenStreetMap (OSM)] for biases regarding the relative wealth index (RWI), population density, urban/rural proportions, and building size in Tanzania and the Philippines. The dataset accuracies for these two countries are lower than expected. Google Open Buildings (with a confidence above 0.7) and OSM demonstrated the best combinations of false negative and false discovery, though Google Open Buildings was more consistent across tiles. The equality of opportunity was lowest for the urban/rural proportions, whereas the OSM and OMF displayed particularly low equality of opportunity for population density and RWI in Tanzania. These results demonstrate that biases exist in these geospatial datasets. The types of biases are not consistent across the datasets and the two study areas, which emphasizes the importance of auditing these datasets for biases in new applications and study areas.