pierreguillou
/

ner-bert-base-cased-pt-lenerbr

@@ -22,28 +22,35 @@ model-index:
     metrics:
     - name: F1
       type: f1
-      value: 0.8716487228203504
     - name: Precision
       type: precision
-      value: 0.8559286898839138
     - name: Recall
       type: recall
-      value: 0.8879569892473118
     - name: Accuracy
       type: accuracy
-      value: 0.9755893153732458
     - name: Loss
       type: loss
-      value: 0.1133928969502449
 widget:
 - text: "Acrescento que não há de se falar em violação do artigo 114, § 3º, da Constituição Federal, posto que referido dispositivo revela-se impertinente, tratando da possibilidade de ajuizamento de dissídio coletivo pelo Ministério Público do Trabalho nos casos de greve em atividade essencial."
 ---
 ## (BERT base) NER model in the legal domain in Portuguese (LeNER-Br)
-**ner-bert-base-portuguese-cased-lenerbr** is a NER model (token classification) in the legal domain in Portuguese that was finetuned on 16/12/2021 in Google Colab from the model [BERTimbau base](https://huggingface.co/neuralmind/bert-base-portuguese-cased) on the dataset [LeNER_br](https://huggingface.co/datasets/lener_br) by using a NER objective.
-Note: due to the small size of BERTimbau base and finetuning dataset, the model overfitted before to reach the end of training. Here are the overall final metrics on the validation dataset (*note: see the paragraph "Validation metrics by Named Entity" to get detailed metrics*):
   - **f1**: 0.8716487228203504
   - **precision**: 0.8559286898839138
   - **recall**: 0.8879569892473118
@@ -102,53 +109,52 @@ ner(input_text)
 ````
 Num examples = 7828
 Num Epochs = 3
-Instantaneous batch size per device = 8
 Total train batch size (w. parallel, distributed & accumulation) = 8
-Gradient Accumulation steps = 1
-Total optimization steps = 2937
-Step	Training Loss	Validation Loss	Precision	Recall    	F1      	Accuracy
-290 	0.315100     	0.141881       	0.764542 	0.709462  	0.735973	0.960550
-580	 0.089100	     0.137700       	0.729155 	0.810538  	0.767695	0.959940
-870	 0.071700	     0.122069       	0.780277 	0.872903  	0.823995	0.967955
-1160	0.047500     	0.125950       	0.800312 	0.881720  	0.839046	0.968367
-1450	0.034900     	0.129228       	0.763666 	0.910323  	0.830570	0.969068
-1740	0.036100     	0.113393       	0.855929 	0.887957  	0.871649	0.975589
-2030	0.037800     	0.121275       	0.817230 	0.889462  	0.851818	0.970393
-2320	0.018700     	0.115745       	0.836066 	0.877419  	0.856243	0.973136
-2610	0.017100     	0.118826       	0.822488 	0.888817  	0.854367	0.973471
 ````
 ### Validation metrics by Named Entity
 ````
 Num examples = 1177
-{'JURISPRUDENCIA': {'f1': 0.6641509433962263,
   'number': 657,
-  'precision': 0.6586826347305389,
-  'recall': 0.669710806697108},
- 'LEGISLACAO': {'f1': 0.8489082969432314,
   'number': 571,
-  'precision': 0.8466898954703833,
-  'recall': 0.851138353765324},
- 'LOCAL': {'f1': 0.8066037735849058,
   'number': 194,
-  'precision': 0.7434782608695653,
-  'recall': 0.8814432989690721},
- 'ORGANIZACAO': {'f1': 0.8540462427745664,
   'number': 1340,
-  'precision': 0.8277310924369747,
-  'recall': 0.8820895522388059},
- 'PESSOA': {'f1': 0.9845722300140253,
   'number': 1072,
-  'precision': 0.9868791002811621,
-  'recall': 0.9822761194029851},
- 'TEMPO': {'f1': 0.9527794381350867,
   'number': 816,
-  'precision': 0.9299883313885647,
-  'recall': 0.9767156862745098},
- 'overall_accuracy': 0.9755893153732458,
- 'overall_f1': 0.8716487228203504,
- 'overall_precision': 0.8559286898839138,
- 'overall_recall': 0.8879569892473118}
  ````

     metrics:
     - name: F1
       type: f1
+      value: 0.8733423827921062
     - name: Precision
       type: precision
+      value: 0.8487923685812868
     - name: Recall
       type: recall
+      value: 0.8993548387096775
     - name: Accuracy
       type: accuracy
+      value: 0.9759397808828684
     - name: Loss
       type: loss
+      value: 0.10249536484479904
 widget:
 - text: "Acrescento que não há de se falar em violação do artigo 114, § 3º, da Constituição Federal, posto que referido dispositivo revela-se impertinente, tratando da possibilidade de ajuizamento de dissídio coletivo pelo Ministério Público do Trabalho nos casos de greve em atividade essencial."
 ---
 ## (BERT base) NER model in the legal domain in Portuguese (LeNER-Br)
+**ner-bert-base-portuguese-cased-lenerbr** is a NER model (token classification) in the legal domain in Portuguese that was finetuned on 20/12/2021 in Google Colab from the model [pierreguillou/bert-base-cased-pt-lenerbr](https://huggingface.co/pierreguillou/bert-base-cased-pt-lenerbr) on the dataset [LeNER_br](https://huggingface.co/datasets/lener_br) by using a NER objective.
+Due to the small size of BERTimbau base and finetuning dataset, the model overfitted before to reach the end of training. Here are the overall final metrics on the validation dataset (*note: see the paragraph "Validation metrics by Named Entity" to get detailed metrics*):
+  - **f1**: 0.8733423827921062
+  - **precision**: 0.8487923685812868
+  - **recall**: 0.8993548387096775
+  - **accuracy**: 0.9759397808828684
+  - **loss**: 0.10249536484479904
+**Note**: the model [pierreguillou/bert-base-cased-pt-lenerbr](https://huggingface.co/pierreguillou/bert-base-cased-pt-lenerbr) is a language model that was created through the finetuning of the model [BERTimbau base](https://huggingface.co/neuralmind/bert-base-portuguese-cased) on the dataset [LeNER-Br language modeling](https://huggingface.co/datasets/pierreguillou/lener_br_finetuning_language_model) by using a MASK objective. This first specialization of the language model before fine finetuning on the NER task improved a bit the model quality. To prove it, here are the results of the NER model finetuned from the model [BERTimbau base](https://huggingface.co/neuralmind/bert-base-portuguese-cased) (a non-specialized language model):
   - **f1**: 0.8716487228203504
   - **precision**: 0.8559286898839138
   - **recall**: 0.8879569892473118
 ````
 Num examples = 7828
 Num Epochs = 3
+Instantaneous batch size per device = 4
 Total train batch size (w. parallel, distributed & accumulation) = 8
+Gradient Accumulation steps = 2
+Total optimization steps = 2934
+Step	Training Loss	Validation Loss	Precision	Recall   	F1      	Accuracy
+290	 0.314600     	0.163042	       0.735828	 0.697849   0.716336	0.949198
+580	 0.086900	     0.123495       	0.779540 	0.824301 	0.801296	0.965807
+870	 0.072800	     0.106785       	0.798481 	0.858925 	0.827600	0.968626
+1160	0.046300     	0.109921       	0.824576 	0.877419 	0.850177	0.973243
+1450	0.036600     	0.102495       	0.848792 	0.899355 	0.873342	0.975940
+1740	0.033400     	0.121514       	0.821681 	0.899785 	0.858961	0.967071
+2030	0.034700     	0.115568       	0.846849 	0.887097 	0.866506	0.970607
+2320	0.018000     	0.108600       	0.840258 	0.895914 	0.867194	0.973730
 ````
 ### Validation metrics by Named Entity
 ````
 Num examples = 1177
+{'JURISPRUDENCIA': {'f1': 0.7069834413246942,
   'number': 657,
+  'precision': 0.6707650273224044,
+  'recall': 0.7473363774733638},
+ 'LEGISLACAO': {'f1': 0.8256227758007118,
   'number': 571,
+  'precision': 0.8390596745027125,
+  'recall': 0.8126094570928196},
+ 'LOCAL': {'f1': 0.7688564476885645,
   'number': 194,
+  'precision': 0.728110599078341,
+  'recall': 0.8144329896907216},
+ 'ORGANIZACAO': {'f1': 0.8548387096774193,
   'number': 1340,
+  'precision': 0.8062169312169312,
+  'recall': 0.9097014925373135},
+ 'PESSOA': {'f1': 0.9826697892271662,
   'number': 1072,
+  'precision': 0.9868297271872061,
+  'recall': 0.9785447761194029},
+ 'TEMPO': {'f1': 0.9615846338535414,
   'number': 816,
+  'precision': 0.9423529411764706,
+  'recall': 0.9816176470588235},
+ 'overall_accuracy': 0.9759397808828684,
+ 'overall_f1': 0.8733423827921062,
+ 'overall_precision': 0.8487923685812868,
+ 'overall_recall': 0.8993548387096775}
  ````