Building selective ensembles of Randomization Based Neural Networks with the successive projections algorithm

• Published in: Applied soft computing Vol. 70; pp. 1135 - 1145
• Main Authors: Mesquita, Diego P.P., P. Gomes, João Paulo, Rodrigues, Leonardo R., Oliveira, Saulo A.F., Galvão, Roberto K.H.
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01-09-2018
• Subjects: Extreme learning machines; Feedforward neural network with random weights; Random vector functional link; Selective ensemble; Successive projections algorithm; Feedforward neural network with random weights; Extreme learning machines; Selective ensemble; Successive projections algorithm; Random vector functional link
• ISSN: 1568-4946; 1872-9681
• DOI: 10.1016/j.asoc.2017.08.007

[Display omitted]
•We propose a selective ensemble method for Randomization Based Neural Networks (RNNs) based on the Successive Projections Algorithm (SPA).•The proposed method, named SERS, uses SPA for feature selection, neuron pruning and ensemble selection.•SERS was used to build three ensemble models based on Extreme Learning Machines, Feedforward Neural Network with Random Weights and Random Vector Functional Link networks.•The proposed methods result in compact models with performance comparable to other state-of-the-art RNN based methods.•Results showed that none of the previously proposed methods was able to achieve better results in both accuracy and model reduction.
Randomization based methods for training neural networks have gained increasing attention in recent years and achieved remarkable performances on a wide variety of tasks. The interest in such methods relies on the fact that standard gradient based learning algorithms may often converge to local minima and are usually time consuming. Despite the good performance achieved by Randomization Based Neural Networks (RNNs), the random feature mapping procedure may generate redundant information, leading to suboptimal solutions. To overcome this problem, some strategies have been used such as feature selection, hidden neuron pruning and ensemble methods. Feature selection methods discard redundant information from the original dataset. Pruning methods eliminate hidden nodes with redundant information. Ensemble methods combine multiple models to generate a single one. Selective ensemble methods select a subset of all available models to generate the final model. In this paper, we propose a selective ensemble of RNNs based on the Successive Projections Algorithm (SPA), for regression problems. The proposed method, named Selective Ensemble of RNNs using the Successive projections algorithm (SERS), employs the SPA for three distinct tasks: feature selection, pruning and ensemble selection. SPA was originally developed as a feature selection technique and has been recently employed for RNN pruning. Herein, we show that it can also be employed for ensemble selection. The proposed framework was used to develop three selective ensemble models based on the three RNNs: Extreme Learning Machines (ELM), Feedforward Neural Network with Random Weights (FNNRW) and Random Vector Functional Link (RVFL). The performances of SERS-ELM, SERS-FNNRW and SERS-RVFL were assessed in terms of model accuracy and model complexity in several real world benchmark problems. Comparisons to related methods showed that SERS variants achieved similar accuracies with significant model complexity reduction. Among the proposed models, SERS-RVFL had the best accuracies and all variants had similar model complexities.