Development and validity of computerized neuropsychological assessment devices for screening mild cognitive impairment: Ensemble of models with feature space heterogeneity and retrieval practice effect

• Published in: Journal of biomedical informatics Vol. 131; p. 104108
• Main Authors: Xiao, Yuyin, Jia, Zhiying, Dong, Minye, Song, Keyu, Li, Xiyang, Bian, Dongsheng, Li, Yan, Jiang, Nan, Shi, Chenshu, Li, Guohong
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01-07-2022
• Subjects: Alzheimer’s disease; Classification; Cognitive assessment; Dementia; Mild cognitive impairment; Neurodegenerative diseases; Alzheimer’s disease; Neurodegenerative diseases; Mild cognitive impairment; Classification; Cognitive assessment; Dementia
• ISSN: 1532-0464; 1532-0480
• DOI: 10.1016/j.jbi.2022.104108

[Display omitted]
•Screening for mild cognitive impairment at the early stage is critical.•Previous approaches do not fully consider the heterogeneity of feature collected by computerized neuropsychological assessment devices.•Three steps of feature engineering（item development, filter, and wrapper）helped us form the optimal feature combination for inclusion in the decision model of Memory Guard.•The accuracy of Memory Guard reached 93.75% and 0.923 of area under curve.•Time features were incorporated into the model to further improve the accuracy of classification.•The development process of Memory Guard can provide a methodological reference for screening tools.
This study aimed to develop and validate computerized neuropsychological assessment devices for screening patients with mild cognitive impairment (MCI).
We conducted this study in three phases. Phase I involved the development of a conceptual framework of Memory Guard (MG) based on the principles of the cognitive design system (CDS). Phase II involved three steps of feature engineering: item development, filter, and wrapper. Based on the initial items, the number of items in each dimension was determined through analytic hierarchy process. We constructed an initial set with a total of 198 items with three levels of difficulty. Next, we performed feature selection through comprehensive reliability and validity tests, which resulted in the best item bank of 38 test items. The features for modeling were obtained from the best item bank (option scores, reading time scores and total time scores), demographic variables and their MoCA groups. Regarding the heterogeneity of the feature space, we combined the AdaBoost with the Naive Bayes classification algorithm as the decision model of MG. For the screening tool to be used repeatedly, the retrieval practice effect was considered in the design. Phase III involved the validation of measuring instruments. The features incorporated into the modeling process were optimized based on the classification accuracy and area under curve. We also verified the classification effect of the other three classification models with MG.
After three steps of feature engineering, a total of 6 dimensions of cognitive areas were included in MG: orientation, memory, attention, calculation, recall, and language & executive function. 38 features were included in the model (17 features of option score, 20 features of time score, and 1 demographic feature). A total of 333 individuals from two communities in Shanghai and Henan province were included in the measuring instrument verification process. Women accounted for 68.2% of the sample. The median age was 63. 15.3% of the participants had bachelor’s degrees or above and 111 participants lived in urban areas (33.3%). The results showed that MG had an accuracy of 93.75% and AUC of 0.923, with a sensitivity of 91.67% and a specificity of 95.45%. Compared to the other three classification models, MG that combined the AdaBoost with the Naive Bayes classification algorithm was the most accurate classifier.
MG was proved to be reliable and valid in early screening for patients with MCI. MG that integrated heterogeneous features such as demography, option scores, and time scores had a better predictive performance for screening MCI.