julien-c HF staff commited on
Commit
84f9950
1 Parent(s): b031382

Migrate model card from transformers-repo

Browse files

Read announcement at https://discuss.huggingface.co/t/announcement-all-model-cards-will-be-migrated-to-hf-co-model-repos/2755
Original file history: https://github.com/huggingface/transformers/commits/master/model_cards/neuralmind/bert-base-portuguese-cased/README.md

Files changed (1) hide show
  1. README.md +105 -0
README.md ADDED
@@ -0,0 +1,105 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: pt
3
+ license: mit
4
+ tags:
5
+ - bert
6
+ - pytorch
7
+ datasets:
8
+ - brWaC
9
+ ---
10
+
11
+ # BERTimbau Base (aka "bert-base-portuguese-cased")
12
+
13
+ ![Bert holding a berimbau](https://imgur.com/JZ7Hynh.jpg)
14
+
15
+ ## Introduction
16
+
17
+ BERTimbau Base is a pretrained BERT model for Brazilian Portuguese that achieves state-of-the-art performances on three downstream NLP tasks: Named Entity Recognition, Sentence Textual Similarity and Recognizing Textual Entailment. It is available in two sizes: Base and Large.
18
+
19
+ For further information or requests, please go to [BERTimbau repository](https://github.com/neuralmind-ai/portuguese-bert/).
20
+
21
+ ## Available models
22
+
23
+ | Model | Arch. | #Layers | #Params |
24
+ | ---------------------------------------- | ---------- | ------- | ------- |
25
+ | `neuralmind/bert-base-portuguese-cased` | BERT-Base | 12 | 110M |
26
+ | `neuralmind/bert-large-portuguese-cased` | BERT-Large | 24 | 335M |
27
+
28
+ ## Usage
29
+
30
+ ```python
31
+ from transformers import AutoTokenizer # Or BertTokenizer
32
+ from transformers import AutoModelForPreTraining # Or BertForPreTraining for loading pretraining heads
33
+ from transformers import AutoModel # or BertModel, for BERT without pretraining heads
34
+
35
+ model = AutoModelForPreTraining.from_pretrained('neuralmind/bert-base-portuguese-cased')
36
+ tokenizer = AutoTokenizer.from_pretrained('neuralmind/bert-base-portuguese-cased', do_lower_case=False)
37
+ ```
38
+
39
+ ### Masked language modeling prediction example
40
+
41
+ ```python
42
+ from transformers import pipeline
43
+
44
+ pipe = pipeline('fill-mask', model=model, tokenizer=tokenizer)
45
+
46
+ pipe('Tinha uma [MASK] no meio do caminho.')
47
+ # [{'score': 0.14287759363651276,
48
+ # 'sequence': '[CLS] Tinha uma pedra no meio do caminho. [SEP]',
49
+ # 'token': 5028,
50
+ # 'token_str': 'pedra'},
51
+ # {'score': 0.06213393807411194,
52
+ # 'sequence': '[CLS] Tinha uma árvore no meio do caminho. [SEP]',
53
+ # 'token': 7411,
54
+ # 'token_str': 'árvore'},
55
+ # {'score': 0.05515013635158539,
56
+ # 'sequence': '[CLS] Tinha uma estrada no meio do caminho. [SEP]',
57
+ # 'token': 5675,
58
+ # 'token_str': 'estrada'},
59
+ # {'score': 0.0299188531935215,
60
+ # 'sequence': '[CLS] Tinha uma casa no meio do caminho. [SEP]',
61
+ # 'token': 1105,
62
+ # 'token_str': 'casa'},
63
+ # {'score': 0.025660505518317223,
64
+ # 'sequence': '[CLS] Tinha uma cruz no meio do caminho. [SEP]',
65
+ # 'token': 3466,
66
+ # 'token_str': 'cruz'}]
67
+
68
+ ```
69
+
70
+ ### For BERT embeddings
71
+
72
+ ```python
73
+ import torch
74
+
75
+ model = AutoModel.from_pretrained('neuralmind/bert-base-portuguese-cased')
76
+ input_ids = tokenizer.encode('Tinha uma pedra no meio do caminho.', return_tensors='pt')
77
+
78
+ with torch.no_grad():
79
+ outs = model(input_ids)
80
+ encoded = outs[0][0, 1:-1] # Ignore [CLS] and [SEP] special tokens
81
+
82
+ # encoded.shape: (8, 768)
83
+ # tensor([[-0.0398, -0.3057, 0.2431, ..., -0.5420, 0.1857, -0.5775],
84
+ # [-0.2926, -0.1957, 0.7020, ..., -0.2843, 0.0530, -0.4304],
85
+ # [ 0.2463, -0.1467, 0.5496, ..., 0.3781, -0.2325, -0.5469],
86
+ # ...,
87
+ # [ 0.0662, 0.7817, 0.3486, ..., -0.4131, -0.2852, -0.2819],
88
+ # [ 0.0662, 0.2845, 0.1871, ..., -0.2542, -0.2933, -0.0661],
89
+ # [ 0.2761, -0.1657, 0.3288, ..., -0.2102, 0.0029, -0.2009]])
90
+ ```
91
+
92
+ ## Citation
93
+
94
+ If you use our work, please cite:
95
+
96
+ ```bibtex
97
+ @inproceedings{souza2020bertimbau,
98
+ author = {F{\'a}bio Souza and
99
+ Rodrigo Nogueira and
100
+ Roberto Lotufo},
101
+ title = {{BERT}imbau: pretrained {BERT} models for {B}razilian {P}ortuguese},
102
+ booktitle = {9th Brazilian Conference on Intelligent Systems, {BRACIS}, Rio Grande do Sul, Brazil, October 20-23 (to appear)},
103
+ year = {2020}
104
+ }
105
+ ```