Causal datasheet: An approximate guide to practically assess Bayesian networks in the real world

• Main Authors: Butcher, Bradley, Huang, Vincent S, Reffin, Jeremy, Sgaier, Sema K, Charles, Grace, Quadrianto, Novi
• Format: Journal Article
• Language: English
• Published: 12-03-2020
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Learning; Computer Science - Performance; Statistics - Machine Learning
• DOI: 10.48550/arxiv.2003.07182

In solving real-world problems like changing healthcare-seeking behaviors,
designing interventions to improve downstream outcomes requires an
understanding of the causal links within the system. Causal Bayesian Networks
(BN) have been proposed as one such powerful method. In real-world
applications, however, confidence in the results of BNs are often moderate at
best. This is due in part to the inability to validate against some ground
truth, as the DAG is not available. This is especially problematic if the
learned DAG conflicts with pre-existing domain doctrine. At the policy level,
one must justify insights generated by such analysis, preferably accompanying
them with uncertainty estimation. Here we propose a causal extension to the
datasheet concept proposed by Gebru et al (2018) to include approximate BN
performance expectations for any given dataset. To generate the results for a
prototype Causal Datasheet, we constructed over 30,000 synthetic datasets with
properties mirroring characteristics of real data. We then recorded the results
given by state-of-the-art structure learning algorithms. These results were
used to populate the Causal Datasheet, and recommendations were automatically
generated dependent on expected performance. As a proof of concept, we used our
Causal Datasheet Generation Tool (CDG-T) to assign expected performance
expectations to a maternal health survey we conducted in Uttar Pradesh, India.