Stochastic Approximation and Newton's Estimate of a Mixing Distribution

• Published in: Statistical science Vol. 23; no. 3; pp. 365 - 382
• Main Authors: Martin, Ryan, Ghosh, Jayanta K.
• Format: Journal Article
• Language: English
• Published: Institute of Mathematical Statistics 01-08-2008; The Institute of Mathematical Statistics
• Subjects: Approximation; Bayes estimators; Density; empirical Bayes; Liapunov functions; Lyapunov functions; Maximum likelihood estimation; mixture models; Odes; Ordinary differential equations; Random variables; Statistics; Stochastic approximation; Unbiased estimators
• ISSN: 0883-4237; 2168-8745
• DOI: 10.1214/08-STS265

Many statistical problems involve mixture models and the need for computationally efficient methods to estimate the mixing distribution has increased dramatically in recent years. Newton [Sankhyā Ser. A 64 (2002) 306-322] proposed a fast recursive algorithm for estimating the mixing distribution, which we study as a special case of stochastic approximation (SA). We begin with a review of SA, some recent statistical applications, and the theory necessary for analysis of a SA algorithm, which includes Lyapunov functions and ODE stability theory. Then standard SA results are used to prove consistency of Newton's estimate in the case of a finite mixture. We also propose a modification of Newton's algorithm that allows for estimation of an additional unknown parameter in the model, and prove its consistency.