PRESISTANT: Learning based assistant for data pre-processing

• Published in: Data & knowledge engineering Vol. 123; p. 101727
• Main Authors: Bilalli, Besim, Abelló, Alberto, Aluja-Banet, Tomàs, Wrembel, Robert
• Format: Journal Article Publication
• Language: English
• Published: Elsevier B.V 01-09-2019; Elsevier
• Subjects: Arbres de decisió; Data mining; Data pre-processing; Decision trees; Informació; Information storage and retrieval systems; Informàtica; Meta-learning; Mineria de dades; Sistemes d'emmagatzematge i recuperació; Sistemes d'informació; Àrees temàtiques de la UPC; Data mining; Meta-learning; Data pre-processing
• ISSN: 0169-023X; 1872-6933
• DOI: 10.1016/j.datak.2019.101727

Data pre-processing is one of the most time consuming and relevant steps in a data analysis process (e.g., classification task). A given data pre-processing operator can have positive, negative, or zero impact on the final result of the analysis. Expert users have the required knowledge to find the right pre-processing operators. However, when it comes to non-experts, they are overwhelmed by the amount of pre-processing operators and it is challenging for them to find operators that would positively impact their analysis (e.g., increase the predictive accuracy of a classifier). Existing solutions either assume that users have expert knowledge, or they recommend pre-processing operators that are only “syntactically” applicable to a dataset, without taking into account their impact on the final analysis. In this work, we aim at providing assistance to non-expert users by recommending data pre-processing operators that are ranked according to their impact on the final analysis. We developed a tool, PRESISTANT, that uses Random Forests to learn the impact of pre-processing operators on the performance (e.g., predictive accuracy) of 5 different classification algorithms, such as Decision Tree (J48), Naive Bayes, PART, Logistic Regression, and Nearest Neighbor (IBk). Extensive evaluations on the recommendations provided by our tool, show that PRESISTANT can effectively help non-experts in order to achieve improved results in their analytic tasks.