An Information Theoretic Approach to Prevalence Estimation and Missing Data

• Published in: IEEE transactions on information theory Vol. 70; no. 5; pp. 3567 - 3582
• Main Authors: Hossjer, Ola, Diaz-Pachon, Daniel Andres, Zhao, Chen, Rao, J. Sunil
• Format: Journal Article
• Language: English
• Published: New York IEEE 01-05-2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Active information; asymptotic normality; Asymptotic properties; Behavioral sciences; Bias; biased estimate; Diseases; Estimation; Information theory; Missing data; Regression models; Sampling; Signs and symptoms; Social networking (online); Sociology; Surveys; Testing; testing errors
• ISSN: 0018-9448; 1557-9654; 1557-9654
• DOI: 10.1109/TIT.2023.3327399

Many data sources, including tracking social behavior to election polling to testing studies for understanding disease spread, are subject to sampling bias whose implications are not fully yet understood. In this paper we study estimation of a given feature (such as disease, or behavior at social media platforms) from biased samples, treating non-respondent individuals as missing data. Prevalence of the feature among sampled individuals has an upward bias under the assumption of individuals' willingness to be sampled. This can be viewed as a regression model with symptoms as covariates and the feature as outcome. It is assumed that the outcome is unknown at the time of sampling, and therefore the missingness mechanism only depends on the covariates. We show that data, in spite of this, is missing at random only when the sizes of symptom classes in the population are known; otherwise data is missing not at random. With an information theoretic viewpoint, we show that sampling bias corresponds to external information due to individuals in the population knowing their covariates, and we quantify this external information by active information. The reduction in prevalence, when sampling bias is adjusted for, similarly translates into active information due to bias correction, with opposite sign to active information due to testing bias. We develop unified results that show that prevalence and active information estimates are asymptotically normal under all missing data mechanisms, when testing errors are absent and present respectively. The asymptotic behavior of the estimators is illustrated through simulations.