APPT: Boosting Automated Patch Correctness Prediction via Fine-Tuning Pre-Trained Models

• Published in: IEEE transactions on software engineering Vol. 50; no. 3; pp. 474 - 494
• Main Authors: Zhang, Quanjun, Fang, Chunrong, Sun, Weisong, Liu, Yan, He, Tieke, Hao, Xiaodong, Chen, Zhenyu
• Format: Journal Article
• Language: English
• Published: New York IEEE 01-03-2024; IEEE Computer Society
• Subjects: Adaptation models; Automated program repair; Automation; Classifiers; Codes; Computer bugs; Debugging; Deep learning; Feature extraction; Machine learning; Maintenance engineering; Optimization; patch correctness; pre-trained model; Predictive models; Reasoning; Representations; Software development; Task analysis
• ISSN: 0098-5589; 1939-3520
• DOI: 10.1109/TSE.2024.3354969

Automated program repair (APR) aims to fix software bugs automatically without human debugging efforts and plays a crucial role in software development and maintenance. Despite the recent significant progress in the number of fixed bugs, APR is still challenged by a long-standing overfitting problem (i.e., the generated patch is plausible but overfitting). Various techniques have thus been proposed to address the overfitting problem. Recently, researchers have employed BERT to extract code features, which are then used to train a classifier for patch correctness prediction, indicating the potential of such pre-trained models in reasoning about patch correctness. However, BERT is restricted to feature extraction for classifier training without benefiting from the training process, potentially generating sub-optimal vector representations for patched code snippets. In this paper, we propose APPT, a pre-trained model-based automated patch correctness assessment technique by both pre-training and fine-tuning. APPT adopts a pre-trained model as the encoder stack, followed by an LSTM stack and a deep learning classifier. More importantly, the pre-trained model is fine-tuned in conjunction with other components as a whole pipeline to fully adapt it specifically for reasoning about patch correctness. Although our idea is general and can be built on various existing pre-trained models, we have implemented APPT based on the BERT model. We conduct an extensive experiment on 1,183 Defects4J patches and the experimental results show that APPT achieves prediction accuracy of 79.7% and recall of 83.2%, outperforming the state-of-the-art technique CACHE by 4.3% and 6.7%. Our additional investigation on 49,694 real-world patches shows that APPT achieves the optimum performance (exceeding 99% in five common metrics for assessing patch classification techniques) compared with existing representation learning techniques. We further investigate the impact of each component and find that they all positively contribute to APPT, e.g., the fine-tuning process and the LSTM stack increase F1-score by 10.22% and 4.11%, respectively. We also prove that adopting advanced pre-trained models can further provide substantial advancement (e.g., GraphCodeBERT-based APPT improves BERT-based APPT by 2.8% and 3.3% in precision and AUC, respectively), highlighting the generalizability of APPT. Overall, our study highlights the promising future of fine-tuning pre-trained models to assess patch correctness and reduce the manual inspection effort of debugging experts when deploying APR tools in practice.