--- language: - pt tags: - generated_from_trainer datasets: - lener_br metrics: - precision - recall - f1 - accuracy model-index: - name: checkpoints results: - task: name: Token Classification type: token-classification dataset: name: lener_br type: lener_br metrics: - name: F1 type: f1 value: 0.8733423827921062 - name: Precision type: precision value: 0.8487923685812868 - name: Recall type: recall value: 0.8993548387096775 - name: Accuracy type: accuracy value: 0.9759397808828684 - name: Loss type: loss value: 0.10249536484479904 widget: - text: "Acrescento que não há de se falar em violação do artigo 114, § 3º, da Constituição Federal, posto que referido dispositivo revela-se impertinente, tratando da possibilidade de ajuizamento de dissídio coletivo pelo Ministério Público do Trabalho nos casos de greve em atividade essencial." --- ## (BERT base) NER model in the legal domain in Portuguese (LeNER-Br) **ner-bert-base-portuguese-cased-lenerbr** is a NER model (token classification) in the legal domain in Portuguese that was finetuned on 20/12/2021 in Google Colab from the model [pierreguillou/bert-base-cased-pt-lenerbr](https://huggingface.co/pierreguillou/bert-base-cased-pt-lenerbr) on the dataset [LeNER_br](https://huggingface.co/datasets/lener_br) by using a NER objective. Due to the small size of BERTimbau base and finetuning dataset, the model overfitted before to reach the end of training. Here are the overall final metrics on the validation dataset (*note: see the paragraph "Validation metrics by Named Entity" to get detailed metrics*): - **f1**: 0.8733423827921062 - **precision**: 0.8487923685812868 - **recall**: 0.8993548387096775 - **accuracy**: 0.9759397808828684 - **loss**: 0.10249536484479904 **Note**: the model [pierreguillou/bert-base-cased-pt-lenerbr](https://huggingface.co/pierreguillou/bert-base-cased-pt-lenerbr) is a language model that was created through the finetuning of the model [BERTimbau base](https://huggingface.co/neuralmind/bert-base-portuguese-cased) on the dataset [LeNER-Br language modeling](https://huggingface.co/datasets/pierreguillou/lener_br_finetuning_language_model) by using a MASK objective. This first specialization of the language model before finetuning on the NER task improved a bit the model quality. To prove it, here are the results of the NER model finetuned from the model [BERTimbau base](https://huggingface.co/neuralmind/bert-base-portuguese-cased) (a non-specialized language model): - **f1**: 0.8716487228203504 - **precision**: 0.8559286898839138 - **recall**: 0.8879569892473118 - **accuracy**: 0.9755893153732458 - **loss**: 0.1133928969502449 ## Widget & APP You can test this model into the widget of this page. ## Using the model for inference in production ```` # install pytorch: check https://pytorch.org/ # !pip install transformers from transformers import AutoModelForTokenClassification, AutoTokenizer import torch # parameters model_name = "ner-bert-base-portuguese-cased-lenebr" model = AutoModelForTokenClassification.from_pretrained(model_name) tokenizer = AutoTokenizer.from_pretrained(model_name) input_text = "EMENTA: APELAÇÃO CÍVEL - AÇÃO DE INDENIZAÇÃO POR DANOS MORAIS - PRELIMINAR - ARGUIDA PELO MINISTÉRIO PÚBLICO EM GRAU RECURSAL - NULIDADE - AUSÊNCIA DE IN- TERVENÇÃO DO PARQUET NA INSTÂNCIA A QUO - PRESENÇA DE INCAPAZ - PREJUÍZO EXISTENTE - PRELIMINAR ACOLHIDA - NULIDADE RECONHECIDA." # tokenization inputs = tokenizer(input_text, max_length=512, truncation=True, return_tensors="pt") tokens = inputs.tokens() # get predictions outputs = model(**inputs).logits predictions = torch.argmax(outputs, dim=2) # print predictions for token, prediction in zip(tokens, predictions[0].numpy()): print((token, model.config.id2label[prediction])) ```` You can use pipeline, too. However, it seems to have an issue regarding to the max_length of the input sequence. ```` !pip install transformers import transformers from transformers import pipeline model_name = "ner-bert-base-portuguese-cased-lenebr" ner = pipeline( "ner", model=model_name ) ner(input_text) ```` ## Training procedure ### Notebook The notebook of finetuning ([HuggingFace_Notebook_token_classification_NER_LeNER_Br.ipynb](https://github.com/piegu/language-models/blob/master/HuggingFace_Notebook_token_classification_NER_LeNER_Br.ipynb)) is in github. ### Training results ```` Num examples = 7828 Num Epochs = 3 Instantaneous batch size per device = 4 Total train batch size (w. parallel, distributed & accumulation) = 8 Gradient Accumulation steps = 2 Total optimization steps = 2934 Step Training Loss Validation Loss Precision Recall F1 Accuracy 290 0.314600 0.163042 0.735828 0.697849 0.716336 0.949198 580 0.086900 0.123495 0.779540 0.824301 0.801296 0.965807 870 0.072800 0.106785 0.798481 0.858925 0.827600 0.968626 1160 0.046300 0.109921 0.824576 0.877419 0.850177 0.973243 1450 0.036600 0.102495 0.848792 0.899355 0.873342 0.975940 1740 0.033400 0.121514 0.821681 0.899785 0.858961 0.967071 2030 0.034700 0.115568 0.846849 0.887097 0.866506 0.970607 2320 0.018000 0.108600 0.840258 0.895914 0.867194 0.973730 ```` ### Validation metrics by Named Entity ```` Num examples = 1177 {'JURISPRUDENCIA': {'f1': 0.7069834413246942, 'number': 657, 'precision': 0.6707650273224044, 'recall': 0.7473363774733638}, 'LEGISLACAO': {'f1': 0.8256227758007118, 'number': 571, 'precision': 0.8390596745027125, 'recall': 0.8126094570928196}, 'LOCAL': {'f1': 0.7688564476885645, 'number': 194, 'precision': 0.728110599078341, 'recall': 0.8144329896907216}, 'ORGANIZACAO': {'f1': 0.8548387096774193, 'number': 1340, 'precision': 0.8062169312169312, 'recall': 0.9097014925373135}, 'PESSOA': {'f1': 0.9826697892271662, 'number': 1072, 'precision': 0.9868297271872061, 'recall': 0.9785447761194029}, 'TEMPO': {'f1': 0.9615846338535414, 'number': 816, 'precision': 0.9423529411764706, 'recall': 0.9816176470588235}, 'overall_accuracy': 0.9759397808828684, 'overall_f1': 0.8733423827921062, 'overall_precision': 0.8487923685812868, 'overall_recall': 0.8993548387096775} ````