Create README.md

README.md

@@ -0,0 +1,74 @@
+---
+language: 
+- pt
+tags:
+- generated_from_trainer
+datasets:
+- pierreguillou/lener_br_finetuning_language_model
+model-index:
+- name: checkpoints
+  results:
+  - task:
+      name: Fill Mask
+      type: fill-mask
+    dataset:
+      name: pierreguillou/lener_br_finetuning_language_model
+      type: pierreguillou/lener_br_finetuning_language_model
+    metrics:
+    - name: Loss
+      type: loss
+      value: 1.127950
+widget:
+- text: "Com efeito, se tal fosse possível, o Poder [MASK] – que não dispõe de função legislativa – passaria a desempenhar atribuição que lhe é institucionalmente estranha (a de legislador positivo), usurpando, desse modo, no contexto de um sistema de poderes essencialmente limitados, competência que não lhe pertence, com evidente transgressão ao princípio constitucional da separação de poderes."
+---
+
+## (BERT large) Language modeling in the legal domain in Portuguese (LeNER-Br)
+
+**bert-large-cased-pt-lenerbr** is a Language Model in the legal domain in Portuguese that was finetuned on 20/12/2021 in Google Colab from the model [BERTimbau large](https://huggingface.co/neuralmind/bert-large-portuguese-cased) on the dataset [LeNER-Br language modeling](https://huggingface.co/datasets/pierreguillou/lener_br_finetuning_language_model) by using a MASK objective.
+  
+## Widget & APP
+
+You can test this model into the widget of this page.
+
+## Using the model for inference in production
+````
+# install pytorch: check https://pytorch.org/
+# !pip install transformers 
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+
+tokenizer = AutoTokenizer.from_pretrained("pierreguillou/bert-large-cased-pt-lenerbr")
+model = AutoModelForMaskedLM.from_pretrained("pierreguillou/bert-large-cased-pt-lenerbr")
+````
+
+## Training procedure
+
+## Notebook
+
+The notebook of finetuning ([Finetuning_language_model_BERtimbau_LeNER_Br.ipynb](https://github.com/piegu/language-models/blob/master/Finetuning_language_model_BERtimbau_LeNER_Br.ipynb)) is in github.
+
+### Training results
+
+````
+Num examples = 3227
+Num Epochs = 5
+Instantaneous batch size per device = 2
+Total train batch size (w. parallel, distributed & accumulation) = 8
+Gradient Accumulation steps = 4
+Total optimization steps = 2015
+
+Step	Training Loss	Validation Loss
+100   1.616700      1.366015
+200   1.452000      1.312473
+300   1.431100      1.253055
+400   1.407500      1.264705
+500   1.301900      1.243277
+600   1.317800      1.233684
+700   1.319100      1.211826
+800   1.303800      1.190818
+900   1.262800      1.171898
+1000  1.235900      1.146275
+1100  1.221900      1.149027
+1200  1.226200      1.127950
+1300  1.201700      1.172729
+1400  1.198200      1.145363
+````