julien-c HF staff commited on
Commit
28d8535
1 Parent(s): 0d33167

Migrate model card from transformers-repo

Browse files

Read announcement at https://discuss.huggingface.co/t/announcement-all-model-cards-will-be-migrated-to-hf-co-model-repos/2755
Original file history: https://github.com/huggingface/transformers/commits/master/model_cards/microsoft/codebert-base-mlm/README.md

Files changed (1) hide show
  1. README.md +46 -0
README.md ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ## CodeBERT-base-mlm
2
+ Pretrained weights for [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://arxiv.org/abs/2002.08155).
3
+
4
+ ### Training Data
5
+ The model is trained on the code corpus of [CodeSearchNet](https://github.com/github/CodeSearchNet)
6
+
7
+ ### Training Objective
8
+ This model is initialized with Roberta-base and trained with a simple MLM (Masked Language Model) objective.
9
+
10
+ ### Usage
11
+ ```python
12
+ from transformers import RobertaTokenizer, RobertaForMaskedLM, pipeline
13
+
14
+ model = RobertaForMaskedLM.from_pretrained('microsoft/codebert-base-mlm')
15
+ tokenizer = RobertaTokenizer.from_pretrained('microsoft/codebert-base-mlm')
16
+
17
+ code_example = "if (x is not None) <mask> (x>1)"
18
+ fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)
19
+
20
+ outputs = fill_mask(code_example)
21
+ print(outputs)
22
+ ```
23
+ Expected results:
24
+ ```
25
+ {'sequence': '<s> if (x is not None) and (x>1)</s>', 'score': 0.6049249172210693, 'token': 8}
26
+ {'sequence': '<s> if (x is not None) or (x>1)</s>', 'score': 0.30680200457572937, 'token': 50}
27
+ {'sequence': '<s> if (x is not None) if (x>1)</s>', 'score': 0.02133703976869583, 'token': 114}
28
+ {'sequence': '<s> if (x is not None) then (x>1)</s>', 'score': 0.018607674166560173, 'token': 172}
29
+ {'sequence': '<s> if (x is not None) AND (x>1)</s>', 'score': 0.007619690150022507, 'token': 4248}
30
+ ```
31
+
32
+ ### Reference
33
+ 1. [Bimodal CodeBERT trained with MLM+RTD objective](https://huggingface.co/microsoft/codebert-base) (suitable for code search and document generation)
34
+ 2. 🤗 [Hugging Face's CodeBERTa](https://huggingface.co/huggingface/CodeBERTa-small-v1) (small size, 6 layers)
35
+
36
+ ### Citation
37
+ ```bibtex
38
+ @misc{feng2020codebert,
39
+ title={CodeBERT: A Pre-Trained Model for Programming and Natural Languages},
40
+ author={Zhangyin Feng and Daya Guo and Duyu Tang and Nan Duan and Xiaocheng Feng and Ming Gong and Linjun Shou and Bing Qin and Ting Liu and Daxin Jiang and Ming Zhou},
41
+ year={2020},
42
+ eprint={2002.08155},
43
+ archivePrefix={arXiv},
44
+ primaryClass={cs.CL}
45
+ }
46
+ ```