No Man is an Island: Towards Fully Automatic Programming by Code Search, Code Generation and Program Repair

• Main Authors: Zhang, Quanjun, Fang, Chunrong, Shang, Ye, Zhang, Tongke, Yu, Shengcheng, Chen, Zhenyu
• Format: Journal Article
• Language: English
• Published: 05-09-2024
• Subjects: Computer Science - Software Engineering
• DOI: 10.48550/arxiv.2409.03267

Automatic programming attempts to minimize human intervention in the
generation of executable code, and has been a long-standing challenge in the
software engineering community. To advance automatic programming, researchers
are focusing on three primary directions: (1) code search that reuses existing
code snippets from external databases; (2) code generation that produces new
code snippets from natural language; and (3) program repair that refines
existing code snippets by fixing detected bugs. Despite significant
advancements, the effectiveness of state-of-the-art techniques is still
limited, such as the usability of searched code and the correctness of
generated code.
Motivated by the real-world programming process, where developers usually use
various external tools to aid their coding processes, such as code search
engines and code testing tools, in this work, we propose \toolname{}, an
automatic programming framework that leverages recent large language models
(LLMs) to integrate the three research areas to address their inherent
limitations. In particular, our framework first leverages different code search
strategies to retrieve similar code snippets, which are then used to further
guide the code generation process of LLMs. Our framework further validates the
quality of generated code by compilers and test cases, and constructs repair
prompts to query LLMs for generating correct patches. We conduct preliminary
experiments to demonstrate the potential of our framework, \eg helping
CodeLlama solve 267 programming problems with an improvement of 62.53\%. As a
generic framework, \toolname{} can integrate various code search, generation,
and repair tools, combining these three research areas together for the first
time. More importantly, it demonstrates the potential of using traditional SE
tools to enhance the usability of LLMs in automatic programming.