Principal Component Projection Without Principal Component Analysis

• Main Authors: Frostig, Roy, Musco, Cameron, Musco, Christopher, Sidford, Aaron
• Format: Journal Article
• Language: English
• Published: 22-02-2016
• Subjects: Computer Science - Data Structures and Algorithms; Computer Science - Learning; Statistics - Machine Learning
• DOI: 10.48550/arxiv.1602.06872

We show how to efficiently project a vector onto the top principal components
of a matrix, without explicitly computing these components. Specifically, we
introduce an iterative algorithm that provably computes the projection using
few calls to any black-box routine for ridge regression.
By avoiding explicit principal component analysis (PCA), our algorithm is the
first with no runtime dependence on the number of top principal components. We
show that it can be used to give a fast iterative method for the popular
principal component regression problem, giving the first major runtime
improvement over the naive method of combining PCA with regression.
To achieve our results, we first observe that ridge regression can be used to
obtain a "smooth projection" onto the top principal components. We then sharpen
this approximation to true projection using a low-degree polynomial
approximation to the matrix step function. Step function approximation is a
topic of long-term interest in scientific computing. We extend prior theory by
constructing polynomials with simple iterative structure and rigorously
analyzing their behavior under limited precision.