Assessment of protein model structure accuracy estimation in CASP13: Challenges in the era of deep learning

• Published in: Proteins, structure, function, and bioinformatics Vol. 87; no. 12; pp. 1351 - 1360
• Main Authors: Won, Jonghun, Baek, Minkyung, Monastyrskyy, Bohdan, Kryshtafovych, Andriy, Seok, Chaok
• Format: Journal Article
• Language: English
• Published: Hoboken, USA John Wiley & Sons, Inc 01-12-2019; Wiley Subscription Services, Inc
• Subjects: Accuracy; CASP13 assessment; Comparative analysis; Deep learning; estimation of protein model accuracy; Evaluation; Machine learning; Model accuracy; New technology; protein model quality assessment; Protein structure; protein structure prediction; Proteins; Scoring models; Tertiary structure; estimation of protein model accuracy; protein model quality assessment; protein structure prediction; CASP13 assessment
• ISSN: 0887-3585; 1097-0134
• DOI: 10.1002/prot.25804

Scoring model structure is an essential component of protein structure prediction that can affect the prediction accuracy tremendously. Users of protein structure prediction results also need to score models to select the best models for their application studies. In Critical Assessment of techniques for protein Structure Prediction (CASP), model accuracy estimation methods have been tested in a blind fashion by providing models submitted by the tertiary structure prediction servers for scoring. In CASP13, model accuracy estimation results were evaluated in terms of both global and local structure accuracy. Global structure accuracy estimation was evaluated by the quality of the models selected by the global structure scores and by the absolute estimates of the global scores. Residue‐wise, local structure accuracy estimations were evaluated by three different measures. A new measure introduced in CASP13 evaluates the ability to predict inaccurately modeled regions that may be improved by refinement. An intensive comparative analysis on CASP13 and the previous CASPs revealed that the tertiary structure models generated by the CASP13 servers show very distinct features. Higher consensus toward models of higher global accuracy appeared even for free modeling targets, and many models of high global accuracy were not well optimized at the atomic level. This is related to the new technology in CASP13, deep learning for tertiary contact prediction. The tertiary model structures generated by deep learning pose a new challenge for EMA (estimation of model accuracy) method developers. Model accuracy estimation itself is also an area where deep learning can potentially have an impact, although current EMA methods have not fully explored that direction.