Back to all models
fill-mask mask_token: <mask>
Query this model
πŸ”₯ This model is currently loaded and running on the Inference API. ⚠️ This model could not be loaded by the inference API. ⚠️ This model can be loaded on the Inference API on-demand.
JSON Output
API endpoint  

⚑️ Upgrade your account to access the Inference API

							curl -X POST \
-H "Authorization: Bearer YOUR_ORG_OR_USER_API_TOKEN" \
-H "Content-Type: application/json" \
-d '"json encoded string"' \
Share Copied link to clipboard

Monthly model downloads

mrm8488/CodeBERTaPy mrm8488/CodeBERTaPy
last 30 days



Contributed by

mrm8488 Manuel Romero
146 models

How to use this model directly from the πŸ€—/transformers library:

Copy to clipboard
from transformers import AutoTokenizer, AutoModelWithLMHead tokenizer = AutoTokenizer.from_pretrained("mrm8488/CodeBERTaPy") model = AutoModelWithLMHead.from_pretrained("mrm8488/CodeBERTaPy")


CodeBERTaPy is a RoBERTa-like model trained on the CodeSearchNet dataset from GitHub for python by Manuel Romero

The tokenizer is a Byte-level BPE tokenizer trained on the corpus using Hugging Face tokenizers.

Because it is trained on a corpus of code (vs. natural language), it encodes the corpus efficiently (the sequences are between 33% to 50% shorter, compared to the same corpus tokenized by gpt2/roberta).

The (small) model is a 6-layer, 84M parameters, RoBERTa-like Transformer model – that’s the same number of layers & heads as DistilBERT – initialized from the default initialization settings and trained from scratch on the full python corpus for 4 epochs.

Quick start: masked language modeling prediction

fruits = ['apples', 'bananas', 'oranges']
for idx, <mask> in enumerate(fruits):
  print("index is %d and value is %s" % (idx, val))

Does the model know how to complete simple Python code?

from transformers import pipeline

fill_mask = pipeline(


## Top 5 predictions:

'val' # prob  0.980728805065155

Yes! That was easy πŸŽ‰ Let's try with another Flask like example

def hello_name(name):
    return "Hello {}!".format(<mask>)

if __name__ == '__main__':


## Top 5 predictions:

'name' # prob  0.9961813688278198
' name'

Yeah! It works πŸŽ‰ Let's try with another Tensorflow/Keras like example

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.<mask>(128, activation='relu'),
    keras.layers.Dense(10, activation='softmax')


## Top 5 predictions:

'Dense' # prob   0.4482928514480591

Great! πŸŽ‰

This work is heavely inspired on CodeBERTa by huggingface team

CodeSearchNet citation

    title = {{CodeSearchNet} {Challenge}: {Evaluating} the {State} of {Semantic} {Code} {Search}},
    shorttitle = {{CodeSearchNet} {Challenge}},
    url = {},
    urldate = {2020-03-12},
    journal = {arXiv:1909.09436 [cs, stat]},
    author = {Husain, Hamel and Wu, Ho-Hsiang and Gazit, Tiferet and Allamanis, Miltiadis and Brockschmidt, Marc},
    month = sep,
    year = {2019},
    note = {arXiv: 1909.09436},

Created by Manuel Romero/@mrm8488

Made with in Spain