Data Imputation Through the Identification of Local Anomalies

• Published in: IEEE transaction on neural networks and learning systems Vol. 26; no. 10; pp. 2381 - 2395
• Main Authors: Ozkan, Huseyin, Pelvan, Ozgun Soner, Kozat, Suleyman S.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01-10-2015; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Anomalies; Anomaly detection; Binary trees; Data models; Deviation; Euclidean distance; localized corruption; maximum a posteriori (MAP)-based imputation; Neural networks; Noise; occlusion; Partitioning; Recording; Training; Trees; Tuning; Visualization; localized corruption; maximum a posteriori (MAP)-based imputation; occlusion; Anomaly detection
• ISSN: 2162-237X; 2162-2388
• DOI: 10.1109/TNNLS.2014.2382606

We introduce a comprehensive and statistical framework in a model free setting for a complete treatment of localized data corruptions due to severe noise sources, e.g., an occluder in the case of a visual recording. Within this framework, we propose: 1) a novel algorithm to efficiently separate, i.e., detect and localize, possible corruptions from a given suspicious data instance and 2) a maximum a posteriori estimator to impute the corrupted data. As a generalization to Euclidean distance, we also propose a novel distance measure, which is based on the ranked deviations among the data attributes and empirically shown to be superior in separating the corruptions. Our algorithm first splits the suspicious instance into parts through a binary partitioning tree in the space of data attributes and iteratively tests those parts to detect local anomalies using the nominal statistics extracted from an uncorrupted (clean) reference data set. Once each part is labeled as anomalous versus normal, the corresponding binary patterns over this tree that characterize corruptions are identified and the affected attributes are imputed. Under a certain conditional independency structure assumed for the binary patterns, we analytically show that the false alarm rate of the introduced algorithm in detecting the corruptions is independent of the data and can be directly set without any parameter tuning. The proposed framework is tested over several well-known machine learning data sets with synthetically generated corruptions and experimentally shown to produce remarkable improvements in terms of classification purposes with strong corruption separation capabilities. Our experiments also indicate that the proposed algorithms outperform the typical approaches and are robust to varying training phase conditions.