Edit model card
YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

language: en


  • 37 popular Python code repositories
  • See princeton-nlp/SWE-bench train split
  • See the make_datasets documentation on SWE-bench's GitHub for details on formatting input.


SWE-Llama are variants of the CodeLlama model fine-tuned on software engineering tasks extracted from real-world GitHub issues and pull requests. They were introduced and evaluated on the SWE-bench benchmark in this paper.

Model Details

  • Architecture: Transformer, based on CodeLlama architecture
  • Parameters: 7 billion for SWE-Llama-7b, 13 billion for SWE-Llama-13b
  • Objective: Generating patches to resolve GitHub issues, conditioned on issue description and code context

Training Data

SWE-Llama was fine-tuned on 19,000 issues and pull requests collected from 37 popular Python code repositories on GitHub, disjoint from those used in SWE-bench.

Training Procedure

  • Fine-tuned only the attention matrices using LoRA method
  • Trained for 4 epochs with a batch size of 32
  • Selected best checkpoint based on validation perplexity

Evaluation Results

When evaluated on the SWE-bench benchmark:

  • SWE-Llama-7b achieved 3.0% issue resolution rate using oracle context retrieval
  • SWE-Llama-13b achieved 4.0% issue resolution rate using oracle context retrieval

BibTeX Entry

      title={SWE-bench: Can Language Models Resolve Real-World GitHub Issues?}, 
      author={Carlos E. Jimenez and John Yang 
        and Alexander Wettig and Shunyu Yao 
        and Kexin Pei and Ofir Press and Karthik Narasimhan},
Downloads last month