rufimelo commited on
Commit
95ae6f4
1 Parent(s): 75cda3d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +94 -64
README.md CHANGED
@@ -1,42 +1,77 @@
1
  ---
2
- pipeline_tag: sentence-similarity
 
 
3
  tags:
4
  - sentence-transformers
5
- - feature-extraction
6
- - sentence-similarity
7
  - transformers
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
  ---
9
 
10
- # {MODEL_NAME}
 
 
 
 
 
11
 
 
12
  This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 1024 dimensional dense vector space and can be used for tasks like clustering or semantic search.
 
13
 
14
- <!--- Describe your model here -->
15
 
16
- ## Usage (Sentence-Transformers)
 
 
17
 
18
- Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
19
 
 
 
20
  ```
21
  pip install -U sentence-transformers
22
  ```
23
-
24
  Then you can use the model like this:
25
-
26
  ```python
27
  from sentence_transformers import SentenceTransformer
28
- sentences = ["This is an example sentence", "Each sentence is converted"]
29
 
30
- model = SentenceTransformer('{MODEL_NAME}')
31
  embeddings = model.encode(sentences)
32
  print(embeddings)
33
  ```
34
-
35
-
36
-
37
  ## Usage (HuggingFace Transformers)
38
- Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
39
-
40
  ```python
41
  from transformers import AutoTokenizer, AutoModel
42
  import torch
@@ -48,13 +83,12 @@ def mean_pooling(model_output, attention_mask):
48
  input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
49
  return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
50
 
51
-
52
  # Sentences we want sentence embeddings for
53
  sentences = ['This is an example sentence', 'Each sentence is converted']
54
 
55
  # Load model from HuggingFace Hub
56
- tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
57
- model = AutoModel.from_pretrained('{MODEL_NAME}')
58
 
59
  # Tokenize sentences
60
  encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
@@ -62,64 +96,60 @@ encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tenso
62
  # Compute token embeddings
63
  with torch.no_grad():
64
  model_output = model(**encoded_input)
65
-
66
  # Perform pooling. In this case, mean pooling.
67
  sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
68
-
69
  print("Sentence embeddings:")
70
  print(sentence_embeddings)
71
  ```
72
 
73
 
74
-
75
- ## Evaluation Results
76
-
77
- <!--- Describe how your model was evaluated -->
78
-
79
- For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME})
80
-
81
-
82
- ## Training
83
- The model was trained with the parameters:
84
-
85
- **DataLoader**:
86
-
87
- `torch.utils.data.dataloader.DataLoader` of length 2157 with parameters:
88
  ```
89
- {'batch_size': 8, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
 
 
 
90
  ```
 
91
 
92
- **Loss**:
93
-
94
- `sentence_transformers.losses.CosineSimilarityLoss.CosineSimilarityLoss`
95
 
96
- Parameters of the fit()-Method:
97
- ```
98
- {
99
- "epochs": 5,
100
- "evaluation_steps": 0,
101
- "evaluator": "NoneType",
102
- "max_grad_norm": 1,
103
- "optimizer_class": "<class 'transformers.optimization.AdamW'>",
104
- "optimizer_params": {
105
- "lr": 1e-05
106
- },
107
- "scheduler": "WarmupLinear",
108
- "steps_per_epoch": null,
109
- "warmup_steps": 1079,
110
- "weight_decay": 0.01
111
  }
112
- ```
113
 
 
 
 
 
 
 
 
 
114
 
115
- ## Full Model Architecture
116
- ```
117
- SentenceTransformer(
118
- (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
119
- (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
120
- )
121
- ```
122
 
123
- ## Citing & Authors
 
 
 
 
 
 
 
 
 
 
 
 
 
124
 
125
- <!--- Describe where people can find more information -->
 
1
  ---
2
+ language:
3
+ - pt
4
+ thumbnail: Portuguese BERT for the Legal Domain
5
  tags:
6
  - sentence-transformers
 
 
7
  - transformers
8
+ - bert
9
+ - pytorch
10
+ - sentence-similarity
11
+ license: mit
12
+ pipeline_tag: sentence-similarity
13
+ datasets:
14
+ - stjiris/portuguese-legal-sentences-v0
15
+ - assin
16
+ - assin2
17
+ - stsb_multi_mt
18
+ widget:
19
+ - source_sentence: "O advogado apresentou as provas ao juíz."
20
+ sentences:
21
+ - "O juíz leu as provas."
22
+ - "O juíz leu o recurso."
23
+ - "O juíz atirou uma pedra."
24
+ model-index:
25
+ - name: BERTimbau
26
+ results:
27
+ - task:
28
+ name: STS
29
+ type: STS
30
+ metrics:
31
+ - name: Pearson Correlation - assin Dataset
32
+ type: Pearson Correlation
33
+ value: 0.7774097897260964
34
+ - name: Pearson Correlation - assin2 Dataset
35
+ type: Pearson Correlation
36
+ value: 0.8097518625809903
37
+ - name: Pearson Correlation - stsb_multi_mt pt Dataset
38
+ type: Pearson Correlation
39
+ value: 0.8358844307795662
40
  ---
41
 
42
+ ![INESC-ID](https://www.inesc-id.pt/wp-content/uploads/2019/06/INESC-ID-logo_01.png)
43
+ ![A Semantic Search System for Supremo Tribunal de Justiça](https://rufimelo99.github.io/SemanticSearchSystemForSTJ/_static/logo.png)
44
+
45
+ Work developed as part of [Project IRIS](https://www.inesc-id.pt/projects/PR07005/).
46
+
47
+ Thesis: [A Semantic Search System for Supremo Tribunal de Justiça](https://rufimelo99.github.io/SemanticSearchSystemForSTJ/)
48
 
49
+ # stjiris/bert-large-portuguese-cased-legal-mlm-nli-sts-v0 (Legal BERTimbau)
50
  This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 1024 dimensional dense vector space and can be used for tasks like clustering or semantic search.
51
+ stjiris/bert-large-portuguese-cased-legal-mlm-nli-sts-v0 derives from stjiris/bert-large-portuguese-cased-legal-mlm (legal variant of [BERTimbau](https://huggingface.co/neuralmind/bert-large-portuguese-cased) large).
52
 
53
+ It was trained using the MLM technique with a learning rate 1e-5 [Legal Sentences from +-30000 documents](https://huggingface.co/datasets/stjiris/portuguese-legal-sentences-v1.0) 5300 training steps (best performance for our semantic search system implementation)
54
 
55
+ The model was presented to NLI data. 16 batch size, 2e-5 lr
56
+
57
+ It was trained for Semantic Textual Similarity, being submitted to a fine tuning stage with the [assin](https://huggingface.co/datasets/assin), [assin2](https://huggingface.co/datasets/assin2), [stsb_multi_mt pt](https://huggingface.co/datasets/stsb_multi_mt) datasets. 'lr': 1e-5
58
 
 
59
 
60
+ ## Usage (Sentence-Transformers)
61
+ Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
62
  ```
63
  pip install -U sentence-transformers
64
  ```
 
65
  Then you can use the model like this:
 
66
  ```python
67
  from sentence_transformers import SentenceTransformer
68
+ sentences = ["Isto é um exemplo", "Isto é um outro exemplo"]
69
 
70
+ model = SentenceTransformer('stjiris/bert-large-portuguese-cased-legal-mlm-nli-sts-v0')
71
  embeddings = model.encode(sentences)
72
  print(embeddings)
73
  ```
 
 
 
74
  ## Usage (HuggingFace Transformers)
 
 
75
  ```python
76
  from transformers import AutoTokenizer, AutoModel
77
  import torch
 
83
  input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
84
  return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
85
 
 
86
  # Sentences we want sentence embeddings for
87
  sentences = ['This is an example sentence', 'Each sentence is converted']
88
 
89
  # Load model from HuggingFace Hub
90
+ tokenizer = AutoTokenizer.from_pretrained('stjiris/bert-large-portuguese-cased-legal-mlm-nli-sts-v0')
91
+ model = AutoModel.from_pretrained('stjiris/bert-large-portuguese-cased-legal-mlm-nli-sts-v0')
92
 
93
  # Tokenize sentences
94
  encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
 
96
  # Compute token embeddings
97
  with torch.no_grad():
98
  model_output = model(**encoded_input)
 
99
  # Perform pooling. In this case, mean pooling.
100
  sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
 
101
  print("Sentence embeddings:")
102
  print(sentence_embeddings)
103
  ```
104
 
105
 
106
+ ## Full Model Architecture
 
 
 
 
 
 
 
 
 
 
 
 
 
107
  ```
108
+ SentenceTransformer(
109
+ (0): Transformer({'max_seq_length': 514, 'do_lower_case': False}) with Transformer model: BertModel
110
+ (1): Pooling({'word_embedding_dimension': 1028, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False})
111
+ )
112
  ```
113
+ ## Citing & Authors
114
 
115
+ If you use this work, please cite:
 
 
116
 
117
+ ```bibtex
118
+ @inproceedings{MeloSemantic,
119
+ author = {Melo, Rui and Santos, Professor Pedro Alexandre and Dias, Professor Jo{\~ a}o},
120
+ title = {A {Semantic} {Search} {System} for {Supremo} {Tribunal} de {Justi}{\c c}a},
 
 
 
 
 
 
 
 
 
 
 
121
  }
 
122
 
123
+ @inproceedings{souza2020bertimbau,
124
+ author = {F{\'a}bio Souza and
125
+ Rodrigo Nogueira and
126
+ Roberto Lotufo},
127
+ title = {{BERT}imbau: pretrained {BERT} models for {B}razilian {P}ortuguese},
128
+ booktitle = {9th Brazilian Conference on Intelligent Systems, {BRACIS}, Rio Grande do Sul, Brazil, October 20-23 (to appear)},
129
+ year = {2020}
130
+ }
131
 
132
+ @inproceedings{fonseca2016assin,
133
+ title={ASSIN: Avaliacao de similaridade semantica e inferencia textual},
134
+ author={Fonseca, E and Santos, L and Criscuolo, Marcelo and Aluisio, S},
135
+ booktitle={Computational Processing of the Portuguese Language-12th International Conference, Tomar, Portugal},
136
+ pages={13--15},
137
+ year={2016}
138
+ }
139
 
140
+ @inproceedings{real2020assin,
141
+ title={The assin 2 shared task: a quick overview},
142
+ author={Real, Livy and Fonseca, Erick and Oliveira, Hugo Goncalo},
143
+ booktitle={International Conference on Computational Processing of the Portuguese Language},
144
+ pages={406--412},
145
+ year={2020},
146
+ organization={Springer}
147
+ }
148
+ @InProceedings{huggingface:dataset:stsb_multi_mt,
149
+ title = {Machine translated multilingual STS benchmark dataset.},
150
+ author={Philip May},
151
+ year={2021},
152
+ url={https://github.com/PhilipMay/stsb-multi-mt}
153
+ }
154
 
155
+ ```