|
--- |
|
license: mit |
|
language: |
|
- pt |
|
--- |
|
|
|
# BERTikal (aka `legalnlp-bert`) |
|
|
|
BERTikal [1] is a cased BERT-base model for the Brazilian legal language and was trained from the BERTimbau's [2] checkpoint using Brazilian legal texts. More details on the datasets and training procedures can be found in [1]. |
|
|
|
Please cite as: |
|
|
|
Polo, Felipe Maia, et al. "LegalNLP-Natural Language Processing methods for the Brazilian Legal Language." Anais do XVIII Encontro Nacional de Inteligência Artificial e Computacional. SBC, 2021. |
|
|
|
@inproceedings{polo2021legalnlp, |
|
title={LegalNLP-Natural Language Processing methods for the Brazilian Legal Language}, |
|
author={Polo, Felipe Maia and Mendon{\c{c}}a, Gabriel Caiaffa Floriano and Parreira, Kau{\^e} Capellato J and Gianvechio, Lucka and Cordeiro, Peterson and Ferreira, Jonathan Batista and de Lima, Leticia Maria Paz and do Amaral Maia, Ant{\^o}nio Carlos and Vicente, Renato}, |
|
booktitle={Anais do XVIII Encontro Nacional de Intelig{\^e}ncia Artificial e Computacional}, |
|
pages={763--774}, |
|
year={2021}, |
|
organization={SBC} |
|
} |
|
## Usage |
|
|
|
```python |
|
from transformers import AutoTokenizer # Or BertTokenizer |
|
from transformers import AutoModelForPreTraining # Or BertForPreTraining for loading pretraining heads |
|
from transformers import AutoModel # or BertModel, for BERT without pretraining heads |
|
|
|
model = AutoModelForPreTraining.from_pretrained('felipemaiapolo/legalnlp-bert') |
|
tokenizer = AutoTokenizer.from_pretrained('felipemaiapolo/legalnlp-bert', do_lower_case=False) |
|
``` |
|
|
|
### Ex. extracting BERT embeddings |
|
|
|
```python |
|
import torch |
|
|
|
model = AutoModel.from_pretrained('felipemaiapolo/legalnlp-bert') |
|
input_ids = tokenizer.encode('Tinha uma pedra no meio do caminho.', return_tensors='pt') |
|
|
|
with torch.no_grad(): |
|
outs = model(input_ids) |
|
encoded = outs[0][0, 1:-1] # Ignore [CLS] and [SEP] special tokens |
|
|
|
# encoded.shape: (8, 768) |
|
# tensor([[-0.0398, -0.3057, 0.2431, ..., -0.5420, 0.1857, -0.5775], |
|
# [-0.2926, -0.1957, 0.7020, ..., -0.2843, 0.0530, -0.4304], |
|
# [ 0.2463, -0.1467, 0.5496, ..., 0.3781, -0.2325, -0.5469], |
|
# ..., |
|
# [ 0.0662, 0.7817, 0.3486, ..., -0.4131, -0.2852, -0.2819], |
|
# [ 0.0662, 0.2845, 0.1871, ..., -0.2542, -0.2933, -0.0661], |
|
# [ 0.2761, -0.1657, 0.3288, ..., -0.2102, 0.0029, -0.2009]]) |
|
``` |
|
|
|
# References |
|
|
|
[1] Polo, Felipe Maia, et al. "LegalNLP-Natural Language Processing methods for the Brazilian Legal Language." Anais do XVIII Encontro Nacional de Inteligência Artificial e Computacional. SBC, 2021. |
|
|
|
[2] Souza, F., Nogueira, R., and Lotufo, R. (2020). BERTimbau: pretrained BERT |
|
models for Brazilian Portuguese. In 9th Brazilian Conference on Intelligent |
|
Systems, BRACIS, Rio Grande do Sul, Brazil, October 20-23 |
|
|
