Update Readme

README.md

@@ -44,11 +44,95 @@ widget:
 
 **ner-legal-bert-base-cased-ptbr** is a NER model (token classification) in the legal domain in Portuguese that was finetuned from the model [dominguesm/legal-bert-base-cased-ptbr](https://huggingface.co/dominguesm/legal-bert-base-cased-ptbr) by using a NER objective.
 
-The model is intended to assist NLP research in the legal field, computer law and legal technology applications. Several legal texts in Portuguese were used (more information below).
+The model is intended to assist NLP research in the legal field, computer law and legal technology applications. Several legal texts in Portuguese (more information below) were used with the following labels: 
 
+* `PESSOA`
+* `ORGANIZACAO`
+* `LOCAL`
+* `TEMPO`
+* `LEGISLACAO`
+* `JURISPRUDENCIA`
+
+The labels were inspired by the [LeNER_br](https://huggingface.co/datasets/lener_br) dataset.
+## Training Dataset
+
+
+The pre-training corpora of **legal-bert-base-cased-ptbr** include:
+
+* 971932 examples of miscellaneous legal documents (train split) 
+* 53996 examples of miscellaneous legal documents (valid split) 
+* 53997 examples of miscellaneous legal documents (test split) 
+
+The data used was provided by the BRAZILIAN SUPREME FEDERAL TRIBUNAL, through the terms of use: [LREC 2020](https://ailab.unb.br/victor/lrec2020). 
+
+The results of this project do not imply in any way the position of the BRAZILIAN SUPREME FEDERAL TRIBUNAL, all being the sole and exclusive responsibility of the author of the model.
+
+## Using the model for inference in production
+
+```python
+from transformers import AutoModelForTokenClassification, AutoTokenizer
+import torch
+
+# parameters
+model_name = "dominguesm/ner-legal-bert-base-cased-ptbr"
+model = AutoModelForTokenClassification.from_pretrained(model_name)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+input_text = "Acrescento que não há de se falar em violação do artigo 114, § 3º, da Constituição Federal, posto que referido dispositivo revela-se impertinente, tratando da possibilidade de ajuizamento de dissídio coletivo pelo Ministério Público do Trabalho nos casos de greve em atividade essencial."
+
+# tokenization
+inputs = tokenizer(input_text, max_length=512, truncation=True, return_tensors="pt")
+tokens = inputs.tokens()
+
+# get predictions
+outputs = model(**inputs).logits
+predictions = torch.argmax(outputs, dim=2)
+
+# print predictions
+for token, prediction in zip(tokens, predictions[0].numpy()):
+    print((token, model.config.id2label[prediction]))
+```
+
+You can use pipeline, too. However, it seems to have an issue regarding to the max_length of the input sequence.
+
+```python
+from transformers import pipeline
+
+model_name = "dominguesm/ner-legal-bert-base-cased-ptbr"
+
+ner = pipeline(
+    "ner",
+    model=model_name
+) 
+
+ner(input_text)
+```
 
 ## Training procedure
 
+### Hyperparameters
+
+#### batch, learning rate...
+- per_device_batch_size = 64
+- gradient_accumulation_steps = 2
+- learning_rate = 2e-5 
+- num_train_epochs = 3
+- weight_decay = 0.01
+- optimizer = torch.optim.AdamW
+- epsilon = 1e-08
+- lr_scheduler_type = linear
+
+#### save model & load best model
+- save_total_limit = 3
+- logging_steps = 1000
+- eval_steps = logging_steps
+- evaluation_strategy = 'steps'
+- logging_strategy = 'steps'
+- save_strategy = 'steps'
+- save_steps = logging_steps
+- load_best_model_at_end = True
+- fp16 = True
+
 ### Training results
 
 ```
@@ -58,6 +142,9 @@ Instantaneous batch size per device = 64
 Total train batch size (w. parallel, distributed & accumulation) = 128
 Gradient Accumulation steps = 2
 Total optimization steps = 22779
+Evaluation Infos:
+  Num examples = 53996
+  Batch size = 128
 ```
 
 | Step | Training Loss | Validation Loss | Precision | Recall | F1 Accuracy |
@@ -84,3 +171,26 @@ Total optimization steps = 22779
 |20000 	|0.027400 |	0.030577 |	0.942462 |	0.961754 |	0.952010 |	|0.989295|
 |21000 	|0.027000 |	0.030025 |	0.944483 |	0.960497 |	0.952422 |	|0.989445|
 |22000 	|0.026800 |	0.030162 |	0.943868 |	0.961418 |	0.952562 |	|0.989425|
+
+
+ ### Validation metrics by Named Entity (Test Dataset)
+
+
+* **Num examples = 53997**
+* `overall_precision`: 0.9432396865925381
+* `overall_recall`: 0.9614334116769161
+* `overall_f1`: 0.9522496545298874
+* `overall_accuracy`': 0.9894741602608071
+
+| Label | Precision | Recall | F1 Accuracy | Entity Examples |
+| ----- | --------- | ------ | ----------- | --------------- |
+| JURISPRUDENCIA| 0.8795197115548148| 0.9037275221501844 | 0.8914593047810311 | 57223 |
+| LEGISLACAO | 0.9405395935529082 | 0.9514071028567378 | 0.9459421362370934 | 84642 |
+| LOCAL | 0.9011495452253004 | 0.9132358124779697 | 0.9071524233856495 | 56740 |
+| ORGANIZACAO | 0.9239028155165304 | 0.954964947845235 | 0.9391771163875446 | 183013 |
+| PESSOA | 0.9651685220572037 | 0.9738545198908279 | 0.9694920661875761 | 193456 |
+| TEMPO | 0.973704616066295 | 0.9918808401799004 | 0.9827086882453152 | 186103 |
+
+## Notes
+
+* For the production of this `readme`, i used the `readme` written by Pierre Guillou (available [here](https://huggingface.co/pierreguillou/ner-bert-large-cased-pt-lenerbr)) as a basis, reproducing some parts entirely.