Assessing predictors for new post translational modification sites: A case study on hydroxylation

• Published in: PLoS computational biology Vol. 16; no. 6; p. e1007967
• Main Authors: Piovesan, Damiano, Hatos, Andras, Minervini, Giovanni, Quaglia, Federica, Monzon, Alexander Miguel, Tosatto, Silvio C.E
• Format: Journal Article
• Language: English
• Published: San Francisco Public Library of Science 22-06-2020; Public Library of Science (PLoS)
• Subjects: Analysis; Angiogenesis; Biology and Life Sciences; Case studies; Collagen; Computational biology; Computer and Information Sciences; Datasets; Design of experiments; Enzymes; Experimental design; Experiments; Hydroxylation; Machine learning; Methods; Performance evaluation; Phosphorylation; Physical Sciences; Post-translation; Post-translational modifications; Proline; Protein structure prediction; Proteins; Research and Analysis Methods; Scientific imaging; Sequences; Social Sciences; Software; Translation; Italy
• ISSN: 1553-7358; 1553-734X; 1553-7358
• DOI: 10.1371/journal.pcbi.1007967

Post-translational modification (PTM) sites have become popular for predictor development. However, with the exception of phosphorylation and a handful of other examples, PTMs suffer from a limited number of available training examples and sparsity in protein sequences. Here, proline hydroxylation is taken as an example to compare different methods and evaluate their performance on new experimentally determined sites. As a guide for effective experimental design, predictors require both high specificity and sensitivity. However, the self-reported performance may often not be indicative of prediction quality and detection of new sites is not guaranteed. We have benchmarked seven published hydroxylation site predictors on two newly constructed independent datasets. The self-reported performance is found to widely overestimate the real accuracy measured on independent datasets. No predictor performs better than random on new examples, indicating the refined models do not sufficiently generalize to detect new sites. The number of false positives is high and precision low, in particular for non-collagen proteins whose motifs are not conserved. As hydroxylation site predictors do not generalize for new data, caution is advised when using PTM predictors in the absence of independent evaluations, in particular for highly specific sites involved in signalling.