Back to all models

Unable to determine this model’s pipeline type. Check the docs .

Monthly model downloads

microsoft/codebert-base microsoft/codebert-base
4,147 downloads
last 30 days

pytorch

tf

Contributed by

Microsoft company
15 team members · 31 models

How to use this model directly from the 🤗/transformers library:

			
Copy to clipboard
from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("microsoft/codebert-base") model = AutoModel.from_pretrained("microsoft/codebert-base")

CodeBERT-base

Pretrained weights for CodeBERT: A Pre-Trained Model for Programming and Natural Languages.

Training Data

The model is trained on bi-modal data (documents & code) of CodeSearchNet

Training Objective

This model is initialized with Roberta-base and trained with MLM+RTD objective (cf. the paper).

Usage

Please see the official repository for scripts that support "code search" and "code-to-document generation".

Reference

  1. CodeBERT trained with Masked LM objective (suitable for code completion)
  2. 🤗 Hugging Face's CodeBERTa (small size, 6 layers)

Citation

@misc{feng2020codebert,
    title={CodeBERT: A Pre-Trained Model for Programming and Natural Languages},
    author={Zhangyin Feng and Daya Guo and Duyu Tang and Nan Duan and Xiaocheng Feng and Ming Gong and Linjun Shou and Bing Qin and Ting Liu and Daxin Jiang and Ming Zhou},
    year={2020},
    eprint={2002.08155},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}