A two-stage linear discriminant analysis via QR-decomposition

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 27; no. 6; pp. 929 - 941
• Main Authors: Ye, Jieping, Li, Qi
• Format: Journal Article
• Language: English
• Published: Los Alamitos, CA IEEE 01-06-2005; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Applied sciences; Artificial Intelligence; classification; Cluster Analysis; Computer science; control theory; systems; Decomposition; dimension reduction; Discriminant Analysis; Exact sciences and technology; Face - anatomy & histology; Face recognition; Feature extraction; Humans; Image Enhancement - methods; Image Interpretation, Computer-Assisted - methods; Information Storage and Retrieval - methods; Linear discriminant analysis; Linear Models; Matrix decomposition; Models, Biological; Models, Statistical; Pattern Recognition, Automated - methods; Pattern recognition. Digital image processing. Computational geometry; Principal component analysis; QR decomposition; Reduction; Reproducibility of Results; Scalability; Scatter; Scattering; Sensitivity and Specificity; Singular value decomposition; Singularities; Studies; Text categorization; Texts; Training data; Dimension reduction; Scalability; QR decomposition; Classification; Linear discriminant analysis; Computational complexity; Pattern extraction; Principal component analysis
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2005.110

Linear discriminant analysis (LDA) is a well-known method for feature extraction and dimension reduction. It has been used widely in many applications involving high-dimensional data, such as image and text classification. An intrinsic limitation of classical LDA is the so-called singularity problems; that is, it fails when all scatter matrices are singular. Many LDA extensions were proposed in the past to overcome the singularity problems. Among these extensions, PCA+LDA, a two-stage method, received relatively more attention. In PCA+LDA, the LDA stage is preceded by an intermediate dimension reduction stage using principal component analysis (PCA). Most previous LDA extensions are computationally expensive, and not scalable, due to the use of singular value decomposition or generalized singular value decomposition. In this paper, we propose a two-stage LDA method, namely LDA/QR, which aims to overcome the singularity problems of classical LDA, while achieving efficiency and scalability simultaneously. The key difference between LDA/QR and PCA+LDA lies in the first stage, where LDA/QR applies QR decomposition to a small matrix involving the class centroids, while PCA+LDA applies PCA to the total scatter matrix involving all training data points. We further justify the proposed algorithm by showing the relationship among LDA/QR and previous LDA methods. Extensive experiments on face images and text documents are presented to show the effectiveness of the proposed algorithm.