Fast Robust Model Selection in Large Datasets

• Published in: Journal of the American Statistical Association Vol. 106; no. 493; pp. 203 - 212
• Main Authors: Dupuis, Debbie J., Victoria-Feser, Maria-Pia
• Format: Journal Article
• Language: English
• Published: Alexandria, VA Taylor & Francis 01-03-2011; American Statistical Association; Taylor & Francis Ltd
• Subjects: Applications; Applied statistics; Computational methods; Correlations; Criteria; Data analysis; Data processing; Datasets; Estimation bias; Estimators; Exact sciences and technology; False discovery rate; General topics; Least angle regression; Linear inference, regression; Linear regression; M-estimator; Mathematics; Modeling; Multicollinearity; Outliers; Parameter estimation; Parametric inference; Partial correlation; Probability and statistics; Random forests; Random variables; Regression analysis; Robust t test; Sciences and techniques of general use; Statistical analysis; Statistics; Theory and Methods; Rank statistic; Correlation; Forests; Statistical distribution; Sample size; Bias; Estimator robustness; M-estimator; Multivariate analysis; Covariate; Statistical simulation; Parametric method; Outlier; Statistical test; Least squares method; Least angle regression; Selection method; Approximation theory; Fast algorithm; Sample survey; Variable selection; Discriminant analysis; Statistical analysis; Linear regression; Model selection; Statistical association; Statistical estimation; Random forests; T statistic; Statistical method; Statistical regression; Selection problem; Sampling theory; Correlation analysis; Multicollinearity; False discovery rate; Robust t test; M estimation; Biased estimation; Partial correlation
• ISSN: 0162-1459; 1537-274X
• DOI: 10.1198/jasa.2011.tm09650

Large datasets are increasingly common in many research fields. In particular, in the linear regression context, it is often the case that a huge number of potential covariates are available to explain a response variable, and the first step of a reasonable statistical analysis is to reduce the number of covariates. This can be done in a forward selection procedure that includes selecting the variable to enter, deciding to retain it or stop the selection, and estimating the augmented model. Least squares plus t tests can be fast, but the outcome of a forward selection might be suboptimal when there are outliers. In this article we propose a complete algorithm for fast robust model selection, including considerations for huge sample sizes. Because simply replacing the classical statistical criteria with robust ones is not computationally possible, we develop simplified robust estimators, selection criteria, and testing procedures for linear regression. The robust estimator is a one-step weighted M-estimator that can be biased if the covariates are not orthogonal. We show that the bias can be made smaller by iterating the M-estimator one or more steps further. In the variable selection process, we propose a simplified robust criterion based on a robust t statistic that we compare with a false discovery rate-adjusted level. We carry out a simulation study to show the good performance of our approach. We also analyze two datasets and show that the results obtained by our method outperform those from robust least angle regression and random forests. Supplemental materials are available online.