Comments on the Bernoulli Distribution and Hilbe’s Implicit Extra-Dispersion

• Published in: Stats (Basel, Switzerland) Vol. 7; no. 1; pp. 269 - 283
• Main Author: Griffith, Daniel A.
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01-03-2024
• Subjects: Analysis; Bernoulli; beta; beta-binomial; Binomial distribution; Dispersion measures (Statistics); Generalized linear models; Hilbe; logistic regression; Probability; Probability distribution; Random variables; Regression analysis; Skewness; Software packages; spatial autocorrelation; United States
• ISSN: 2571-905X; 2571-905X
• DOI: 10.3390/stats7010016

For decades, conventional wisdom maintained that binary 0–1 Bernoulli random variables cannot contain extra-binomial variation. Taking an unorthodox stance, Hilbe actively disagreed, especially for correlated observation instances, arguing that the universally adopted diagnostic Pearson or deviance dispersion statistics are insensitive to a variance anomaly in a binary context, and hence simply fail to detect it. However, having the intuition and insight to sense the existence of this departure from standard mathematical statistical theory, but being unable to effectively isolate it, he classified this particular over-/under-dispersion phenomenon as implicit. This paper explicitly exposes his hidden quantity by demonstrating that the variance in/deflation it represents occurs in an underlying predicted beta random variable whose real number values are rounded to their nearest integers to convert to a Bernoulli random variable, with this discretization masking any materialized extra-Bernoulli variation. In doing so, asymptotics linking the beta-binomial and Bernoulli distributions show another conventional wisdom misconception, namely a mislabeling substitution involving the quasi-Bernoulli random variable; this undeniably is not a quasi-likelihood situation. A public bell pepper disease dataset exhibiting conspicuous spatial autocorrelation furnishes empirical examples illustrating various features of this advocated proposition.