Back to all models
fill-mask mask_token: <mask>
Query this model
🔥 This model is currently loaded and running on the Inference API. ⚠️ This model could not be loaded by the inference API. ⚠️ This model can be loaded on the Inference API on-demand.
JSON Output
API endpoint
								$ curl -X POST \
Share Copied link to clipboard

Monthly model downloads

huggingface/CodeBERTa-small-v1 huggingface/CodeBERTa-small-v1
last 30 days



Contributed by

Hugging Face company
18 team members · 5 models

How to use this model directly from the 🤗/transformers library:

Copy to clipboard
from transformers import AutoTokenizer, AutoModelWithLMHead tokenizer = AutoTokenizer.from_pretrained("huggingface/CodeBERTa-small-v1") model = AutoModelWithLMHead.from_pretrained("huggingface/CodeBERTa-small-v1")


CodeBERTa is a RoBERTa-like model trained on the CodeSearchNet dataset from GitHub.

Supported languages:


The tokenizer is a Byte-level BPE tokenizer trained on the corpus using Hugging Face tokenizers.

Because it is trained on a corpus of code (vs. natural language), it encodes the corpus efficiently (the sequences are between 33% to 50% shorter, compared to the same corpus tokenized by gpt2/roberta).

The (small) model is a 6-layer, 84M parameters, RoBERTa-like Transformer model – that’s the same number of layers & heads as DistilBERT – initialized from the default initialization settings and trained from scratch on the full corpus (~2M functions) for 5 epochs.

Tensorboard for this training ⤵️


Quick start: masked language modeling prediction

PHP_CODE = """
public static <mask> set(string $key, $value) {
    if (!in_array($key, self::$allowedKeys)) {
        throw new \InvalidArgumentException('Invalid key given');
    self::$storedValues[$key] = $value;

Does the model know how to complete simple PHP code?

from transformers import pipeline

fill_mask = pipeline(


## Top 5 predictions:
' function' # prob 0.9999827146530151
'function'  # 
' void'     # 
' def'      # 
' final'    # 

Yes! That was easy 🎉 What about some Python (warning: this is going to be meta)

def pipeline(
    task: str,
    model: Optional = None,
    framework: Optional[<mask>] = None,
) -> Pipeline:


'framework', 'Framework', ' framework', 'None', 'str'

This program can auto-complete itself! 😱

Just for fun, let's try to mask natural language (not code):

fill_mask("My name is <mask>.")

# {'sequence': '<s> My name is undefined.</s>', 'score': 0.2548016905784607, 'token': 3353}
# {'sequence': '<s> My name is required.</s>', 'score': 0.07290805131196976, 'token': 2371}
# {'sequence': '<s> My name is null.</s>', 'score': 0.06323737651109695, 'token': 469}
# {'sequence': '<s> My name is name.</s>', 'score': 0.021919190883636475, 'token': 652}
# {'sequence': '<s> My name is disabled.</s>', 'score': 0.019681859761476517, 'token': 7434}

This (kind of) works because code contains comments (which contain natural language).

Of course, the most frequent name for a Computer scientist must be undefined 🤓.

Downstream task: programming language identification

See the model card for huggingface/CodeBERTa-language-id 🤯.

CodeSearchNet citation

    title = {{CodeSearchNet} {Challenge}: {Evaluating} the {State} of {Semantic} {Code} {Search}},
    shorttitle = {{CodeSearchNet} {Challenge}},
    url = {},
    urldate = {2020-03-12},
    journal = {arXiv:1909.09436 [cs, stat]},
    author = {Husain, Hamel and Wu, Ho-Hsiang and Gazit, Tiferet and Allamanis, Miltiadis and Brockschmidt, Marc},
    month = sep,
    year = {2019},
    note = {arXiv: 1909.09436},