On Leakage of Code Generation Evaluation Datasets

• Main Authors: Matton, Alexandre, Sherborne, Tom, Aumiller, Dennis, Tommasone, Elena, Alizadeh, Milad, He, Jingyi, Ma, Raymond, Voisin, Maxime, Gilsenan-McMahon, Ellen, Gallé, Matthias
• Format: Journal Article
• Language: English
• Published: 10-07-2024
• Subjects: Computer Science - Computation and Language
• DOI: 10.48550/arxiv.2407.07565

In this paper, we consider contamination by code generation test sets, in
particular in their use in modern large language models. We discuss three
possible sources of such contamination and show findings supporting each of
them: (i) direct data leakage, (ii) indirect data leakage through the use of
synthetic data and (iii) overfitting to evaluation sets during model selection.
To address this, we release Less Basic Python Problems (LBPP): an
uncontaminated new benchmark of 161 prompts with their associated Python
solutions. LBPP is released at https://huggingface.co/datasets/CohereForAI/lbpp .