initial commit

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

README.md

@@ -1,3 +1,143 @@
 ---
-license: mit
+language: 
+  - pt
+thumbnail: "Portugues SBERT for the Legal Domain"
+pipeline_tag: sentence-similarity
+tags:
+- sentence-transformers
+- sentence-similarity
+- transformers
+datasets:
+- assin
+- assin2
+- stsb_multi_mt
+
+widget:
+- source_sentence: "O advogado apresentou as provas ao juíz."
+  sentences:
+    - "O juíz leu as provas."
+    - "O juíz leu o recurso."
+    - "O juíz atirou uma pedra."
+  example_title: "Example 1"
+metrics:
+  - bleu
 ---
+
+# rufimelo/Legal-SBERTimbau-sts-base-ma
+
+This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.
+rufimelo/rufimelo/Legal-SBERTimbau-sts-base-ma is based on Legal-BERTimbau-base which derives from [BERTimbau](https://huggingface.co/neuralmind/bert-large-portuguese-cased) alrge.
+It is adapted to the Portuguese legal domain and trained for STS on portuguese datasets.
+
+## Usage (Sentence-Transformers)
+
+Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+
+```
+pip install -U sentence-transformers
+```
+
+Then you can use the model like this:
+
+```python
+from sentence_transformers import SentenceTransformer
+sentences = ["Isto é um exemplo", "Isto é um outro exemplo"]
+
+model = SentenceTransformer('rufimelo/Legal-SBERTimbau-sts-base-ma')
+embeddings = model.encode(sentences)
+print(embeddings)
+```
+
+
+
+## Usage (HuggingFace Transformers)
+
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+
+#Mean Pooling - Take attention mask into account for correct averaging
+def mean_pooling(model_output, attention_mask):
+    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
+    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
+    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+
+
+# Sentences we want sentence embeddings for
+sentences = ['This is an example sentence', 'Each sentence is converted']
+
+# Load model from HuggingFace Hub
+tokenizer = AutoTokenizer.from_pretrained('rufimelo/Legal-SBERTimbau-sts-base-ma')
+model = AutoModel.from_pretrained('rufimelo/Legal-SBERTimbau-sts-base-ma')
+
+# Tokenize sentences
+encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
+
+# Compute token embeddings
+with torch.no_grad():
+    model_output = model(**encoded_input)
+
+# Perform pooling. In this case, mean pooling.
+sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
+
+print("Sentence embeddings:")
+print(sentence_embeddings)
+```
+
+
+## Evaluation Results STS
+
+| Model| Dataset |  PearsonCorrelation       |
+| ---------------------------------------- | ---------- | ---------- |
+| Legal-SBERTimbau-sts-large| Assin | 0.76629 |
+| Legal-SBERTimbau-sts-large| Assin2| 0.82357 |
+| Legal-SBERTimbau-sts-base| Assin | 0.71457 |
+| Legal-SBERTimbau-sts-base| Assin2| 0.73545|
+| Legal-SBERTimbau-sts-large-v2| Assin | 0.76299 |
+| Legal-SBERTimbau-sts-large-v2| Assin2| 0.81121 |
+| Legal-SBERTimbau-sts-large-v2| stsb_multi_mt pt| 0.81726 |
+| Legal-SBERTimbau-sts-base-ma| Assin | 0.74874 |
+| Legal-SBERTimbau-sts-base-ma| Assin2| 0.79532 |
+| Legal-SBERTimbau-sts-base-ma| stsb_multi_mt pt| 0.82254 |
+| ---------------------------------------- | ---------- |---------- |
+| paraphrase-multilingual-mpnet-base-v2| Assin | 0.71457|
+| paraphrase-multilingual-mpnet-base-v2| Assin2| 0.79831 |
+| paraphrase-multilingual-mpnet-base-v2| stsb_multi_mt pt| 0.83999 |
+| paraphrase-multilingual-mpnet-base-v2 Fine tuned with assin(s)| Assin | 0.77641  |
+| paraphrase-multilingual-mpnet-base-v2 Fine tuned with assin(s)| Assin2| 0.79831 |
+| paraphrase-multilingual-mpnet-base-v2 Fine tuned with assin(s)| stsb_multi_mt pt| 0.84575 |
+
+## Training
+
+rufimelo/Legal-SBERTimbau-sts-base-ma is based on Legal-BERTimbau-base which derives from [BERTimbau](https://huggingface.co/neuralmind/bert-base-portuguese-cased) base.
+
+Firstly, due to the lack of portuguese datasets, it was trained using multilingual knowledge distillation. 
+For the Multilingual Knowledge Distillation process, the teacher model was 'sentence-transformers/paraphrase-xlm-r-multilingual-v1', the supposed supported language as English and the language to learn was portuguese.
+
+It was trained for Semantic Textual Similarity, being submitted to a fine tuning stage with the [assin](https://huggingface.co/datasets/assin), [assin2](https://huggingface.co/datasets/assin2) and [stsb_multi_mt pt](https://huggingface.co/datasets/stsb_multi_mt) datasets.
+
+
+## Full Model Architecture
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel 
+  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False})
+)
+```
+
+## Citing & Authors
+
+If you use this work, please cite BERTimbau's work:
+
+```bibtex
+@inproceedings{souza2020bertimbau,
+  author    = {F{\'a}bio Souza and
+               Rodrigo Nogueira and
+               Roberto Lotufo},
+  title     = {{BERT}imbau: pretrained {BERT} models for {B}razilian {P}ortuguese},
+  booktitle = {9th Brazilian Conference on Intelligent Systems, {BRACIS}, Rio Grande do Sul, Brazil, October 20-23 (to appear)},
+  year      = {2020}
+}
+```

config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "/home/ruimelo/.cache/torch/sentence_transformers/rufimelo_Legal-BERTimbau-base",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "directionality": "bidi",
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "output_past": true,
+  "pad_token_id": 0,
+  "pooler_fc_size": 768,
+  "pooler_num_attention_heads": 12,
+  "pooler_num_fc_layers": 3,
+  "pooler_size_per_head": 128,
+  "pooler_type": "first_token_transform",
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.20.1",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 29794
+}

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.2.0",
+    "transformers": "4.20.1",
+    "pytorch": "1.10.1+cu111"
+  }
+}

eval/mse_evaluation_TED2020-en-pt-dev.tsv.gz_results.csv

@@ -0,0 +1,51 @@
+epoch,steps,MSE
+0,1000,5.368657410144806
+0,2000,5.193524062633514
+0,3000,5.061326548457146
+0,4000,4.8871491104364395
+0,5000,4.604635760188103
+0,6000,4.37137559056282
+0,7000,4.121359437704086
+0,8000,3.890467807650566
+0,9000,3.7163197994232178
+0,-1,3.6187496036291122
+1,1000,3.3520549535751343
+1,2000,3.2080236822366714
+1,3000,3.0923843383789062
+1,4000,3.0047735199332237
+1,5000,2.9202070087194443
+1,6000,2.8517570346593857
+1,7000,2.794511429965496
+1,8000,2.7496276423335075
+1,9000,2.705024927854538
+1,-1,2.6774482801556587
+2,1000,2.645493298768997
+2,2000,2.6065580546855927
+2,3000,2.577204629778862
+2,4000,2.543270029127598
+2,5000,2.5225354358553886
+2,6000,2.500375173985958
+2,7000,2.475123293697834
+2,8000,2.4587351828813553
+2,9000,2.444928325712681
+2,-1,2.42521520704031
+3,1000,2.417368069291115
+3,2000,2.3938797414302826
+3,3000,2.3835765197873116
+3,4000,2.367889881134033
+3,5000,2.3539265617728233
+3,6000,2.3482950404286385
+3,7000,2.3333005607128143
+3,8000,2.3240605369210243
+3,9000,2.3170573636889458
+3,-1,2.3139014840126038
+4,1000,2.305760234594345
+4,2000,2.2977689281105995
+4,3000,2.2899970412254333
+4,4000,2.286195382475853
+4,5000,2.2827675566077232
+4,6000,2.281411550939083
+4,7000,2.2761769592761993
+4,8000,2.275201492011547
+4,9000,2.272815629839897
+4,-1,2.272995188832283

eval/similarity_evaluation_STS.en-en.txt_results.csv

@@ -0,0 +1,51 @@
+epoch,steps,cosine_pearson,cosine_spearman,euclidean_pearson,euclidean_spearman,manhattan_pearson,manhattan_spearman,dot_pearson,dot_spearman
+0,1000,0.517801837376822,0.5927498604628574,0.5626056283814881,0.5947325778732967,0.5483264326845998,0.5859883251294673,0.23162225976519732,0.23201906745235812
+0,2000,0.44806823321693495,0.5571128406093457,0.5180349070283493,0.5641853527525548,0.5165027287876152,0.5635880005121858,-0.03452826425581184,-0.0432895863922912
+0,3000,0.433520107513419,0.5461290935589597,0.511366853016028,0.5491873525193037,0.5116493313071497,0.5462251926967281,0.07126694981170532,0.035360638733289526
+0,4000,0.48062137883402073,0.5730049472203947,0.544857385161381,0.5860482909914347,0.5440379004589636,0.5829423668587576,0.16089119466468452,0.17975766555800177
+0,5000,0.52956541125226,0.5763791801457215,0.5692791350686931,0.5868570613348942,0.5676877437900325,0.5846068039249076,0.271357969453925,0.2856427707237205
+0,6000,0.553818706587758,0.5820717086705751,0.5777327816006972,0.581899499015694,0.5776770168455237,0.5822243141013514,0.3043967926716216,0.28029350831897587
+0,7000,0.5446056633357405,0.5698348288636882,0.5673459868377165,0.5672251606784472,0.5702414573979611,0.5691925022268436,0.28701828287031017,0.2736030863475344
+0,8000,0.5792358045204946,0.5983720444188645,0.5818917049240697,0.5800659274670714,0.5831890633374769,0.5820959256532927,0.31313325242691503,0.3153285631735058
+0,9000,0.601108450257991,0.6167169854223161,0.5969917017988638,0.5977251050234071,0.5967274984844755,0.5979695812298902,0.30333549584709796,0.3133719847285396
+0,-1,0.6280522190532274,0.6406195317611942,0.6227122902564645,0.6197779351585178,0.6222082717382086,0.6196268673139457,0.36291338197354595,0.3751545047964458
+1,1000,0.6441822639355907,0.6659512644769654,0.6429577685600287,0.6412045833119289,0.6432852285988369,0.6433975656358055,0.41298893952764454,0.4275673589319253
+1,2000,0.6559511145597557,0.6795427577298369,0.6515445072356066,0.6485446354546853,0.6516716895687359,0.6493933830394565,0.47552172351909894,0.49381310735435074
+1,3000,0.6527469478536292,0.684905474013869,0.655923943892963,0.6548279814785396,0.6553730945939851,0.6549210054438994,0.48203680194355364,0.507563740779367
+1,4000,0.669088698727519,0.6937462102920181,0.670854432566903,0.6659097496494495,0.6701009267853754,0.6655422665466227,0.5304511185054699,0.5427909939094337
+1,5000,0.672516667814659,0.6941490578775437,0.674045628530324,0.6672585971471678,0.6736569476628284,0.6680327718010307,0.5476425334723899,0.5575533590568764
+1,6000,0.6752432037594164,0.7003174693326267,0.6838270035504933,0.6740485778253374,0.6828421410239922,0.6749730515306702,0.5463083655615174,0.5502217636382437
+1,7000,0.683567200442913,0.7059803993230477,0.6906067465159954,0.686100947287709,0.690382106531219,0.6868170780623597,0.5699645042777446,0.5723637737732034
+1,8000,0.7036926133871262,0.7257372288585937,0.7078482102922881,0.7057120905303981,0.7076590987201354,0.7059942375988862,0.5934997113002226,0.5971093017485867
+1,9000,0.7154936325287922,0.7367717162537215,0.7180129717719661,0.7165839781844215,0.717671531550584,0.7165097896500643,0.6133281247581909,0.6199732086064632
+1,-1,0.7196875035717942,0.736280073064898,0.7160184824695199,0.7144390454294293,0.7165942734656012,0.7160250655991601,0.6225343657806216,0.6294332077283923
+2,1000,0.720899402697572,0.7415174760732801,0.7195892119980556,0.7164744251673655,0.7196496138770179,0.7170771589594495,0.630202637897733,0.635334463580479
+2,2000,0.7250465844610594,0.7404323246095984,0.7260894530415754,0.7210352902458576,0.7259676751222017,0.7216122694690196,0.636508874611686,0.6438696046005243
+2,3000,0.7262776654731603,0.7426172346059026,0.7261976982608093,0.7199013204201895,0.7257484003658375,0.7207635218842483,0.6362831430635101,0.6430681377915353
+2,4000,0.7289594513535187,0.7460222192553153,0.7265128423312566,0.7219920532614806,0.7253344413328074,0.7219551511925775,0.6456632097565471,0.6464765820099075
+2,5000,0.7382714011547186,0.7537920267421724,0.7337106120585335,0.728817398422349,0.7328743912757635,0.7280182379926664,0.6652114382782512,0.6682976210247206
+2,6000,0.7402044783316905,0.7559581013074739,0.7401263074099426,0.7364987947024589,0.7391512971643294,0.7368728125466539,0.6548352584625278,0.6586757909547893
+2,7000,0.7449855277046175,0.7586204318202119,0.742407037478152,0.7406210633161757,0.7419037859609786,0.7401351860756182,0.6658503652340054,0.6707327731757738
+2,8000,0.746684321788316,0.7587492046648215,0.7420907061986265,0.737954504441376,0.7414911948572276,0.7378837754759784,0.6676518777357797,0.6709895500718912
+2,9000,0.752118628883576,0.7652508879296858,0.7481933017748738,0.746660701926649,0.7481565280073592,0.7468771171849038,0.6829612724188509,0.6833717317750841
+2,-1,0.7537535743394218,0.7676153111153414,0.7483667618080623,0.7467821712367885,0.7484240035786225,0.7475082963217671,0.6801060385066588,0.6816596295366009
+3,1000,0.7540327394851281,0.7658997493078986,0.749836472873649,0.7480572145967006,0.7495837523658369,0.7481729179585739,0.6808002467334254,0.6826728988452317
+3,2000,0.7570223312833305,0.7698993954218226,0.7540758631739267,0.753295770794736,0.7534239299289165,0.7520422536416839,0.6804191095329432,0.6824718594490202
+3,3000,0.7599567018278637,0.7720689295560839,0.7556908343726982,0.7529013799333341,0.755271373708525,0.7528248850196704,0.6851683555897045,0.6876523717678433
+3,4000,0.7647207462854896,0.7760793387734382,0.7589105955969245,0.7577332445803333,0.7581578363689768,0.7571112909606956,0.6990299162467802,0.7017689507094818
+3,5000,0.7648546144596025,0.7767082115309949,0.759655555046868,0.7593823057844403,0.7589242973646805,0.7579473534592813,0.6948561132830983,0.6960387513226229
+3,6000,0.7651312360894998,0.7768604325652202,0.7613454557550654,0.7607815092303494,0.7602880440921177,0.7582152778553799,0.7019760063631014,0.7042006432915755
+3,7000,0.7700810652664646,0.7811245435062832,0.7632005336042457,0.7634911205188694,0.7622691247480861,0.7624163477620668,0.7048378363188619,0.7093423315587402
+3,8000,0.7735263359060158,0.7842835143630087,0.7652218749789204,0.7660219874111401,0.7639986737869758,0.7644951643102745,0.7130537943626254,0.7156110705136536
+3,9000,0.7739488663284068,0.7862128006528485,0.7677560690354177,0.7681807584419712,0.7667809699348831,0.7655088180154564,0.7106389926111314,0.7118989530199328
+3,-1,0.7749646355659017,0.7859306535843604,0.7668847071412118,0.76678078620296,0.7660379175603176,0.764305272414044,0.7158777617312314,0.7186727890429572
+4,1000,0.7749262499900197,0.786532234186791,0.7683776275278195,0.7683794914588764,0.7672588326412471,0.7651355689643636,0.7162472357262631,0.7189245687839106
+4,2000,0.7775372924636713,0.7886944647865819,0.7707564332014942,0.7711459934369553,0.7698450681065371,0.7695349874914046,0.7177465093084331,0.7211675226594269
+4,3000,0.7773850252493334,0.7881536188392207,0.772183724897433,0.7717883200737999,0.7712699567266517,0.7709161242994133,0.7187237988981621,0.7223937476573528
+4,4000,0.777775327985964,0.7893621615957971,0.7716378455124066,0.7722138470558387,0.7709049470179239,0.7709922348165259,0.7176467231063159,0.7201719355921455
+4,5000,0.7763059687016113,0.7876235359952898,0.7698424419369689,0.7698820975770244,0.7689586641660842,0.7687961773202405,0.7197057873099552,0.7245286861020173
+4,6000,0.7794826897000439,0.7905141980593657,0.7720120513908072,0.7726462931757969,0.7714463473656613,0.7704221747312833,0.7226559974833779,0.7265990459261015
+4,7000,0.778285341526584,0.7896335455608554,0.77200273274823,0.7723626085211043,0.7713809544539256,0.7717356577463027,0.7204467718381531,0.724500625153789
+4,8000,0.778450136549209,0.7893325630613646,0.7721776463065276,0.7722626654178252,0.771422818962253,0.7717952392117192,0.7215457163794511,0.7250733760148891
+4,9000,0.7784703374820181,0.7894732521990575,0.7726776751649114,0.773120254123271,0.7719272768083242,0.7713374229193902,0.7210072930243006,0.7252590395490577
+4,-1,0.7787882635851495,0.789720034784847,0.7728008205428657,0.7732217348127545,0.7720684298045568,0.7715788239534646,0.7213522278358117,0.7256630403242363

eval/translation_evaluation_TED2020-en-pt-dev.tsv.gz_results.csv

@@ -0,0 +1,51 @@
+epoch,steps,src2trg,trg2src
+0,1000,0.046,0.188
+0,2000,0.058,0.114
+0,3000,0.102,0.146
+0,4000,0.202,0.252
+0,5000,0.436,0.432
+0,6000,0.588,0.621
+0,7000,0.719,0.729
+0,8000,0.81,0.818
+0,9000,0.854,0.88
+0,-1,0.889,0.899
+1,1000,0.917,0.917
+1,2000,0.93,0.932
+1,3000,0.94,0.943
+1,4000,0.943,0.947
+1,5000,0.955,0.954
+1,6000,0.958,0.952
+1,7000,0.962,0.955
+1,8000,0.961,0.961
+1,9000,0.962,0.961
+1,-1,0.964,0.961
+2,1000,0.966,0.967
+2,2000,0.968,0.967
+2,3000,0.968,0.966
+2,4000,0.972,0.966
+2,5000,0.971,0.967
+2,6000,0.971,0.967
+2,7000,0.971,0.972
+2,8000,0.974,0.969
+2,9000,0.974,0.97
+2,-1,0.971,0.972
+3,1000,0.974,0.973
+3,2000,0.975,0.972
+3,3000,0.974,0.974
+3,4000,0.975,0.972
+3,5000,0.974,0.972
+3,6000,0.976,0.972
+3,7000,0.975,0.971
+3,8000,0.975,0.973
+3,9000,0.975,0.974
+3,-1,0.974,0.973
+4,1000,0.975,0.974
+4,2000,0.975,0.972
+4,3000,0.975,0.974
+4,4000,0.975,0.973
+4,5000,0.975,0.975
+4,6000,0.974,0.974
+4,7000,0.975,0.973
+4,8000,0.974,0.973
+4,9000,0.974,0.973
+4,-1,0.974,0.973

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53d50f63936d072c813a5fd98b08ed3c7ea38f2c9ad3aca724aa901bbfba1482
+size 435761969

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": false,
+  "mask_token": "[MASK]",
+  "name_or_path": "/home/ruimelo/.cache/torch/sentence_transformers/rufimelo_Legal-BERTimbau-base",
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "special_tokens_map_file": "/home/ruimelo/.cache/huggingface/transformers/eecc45187d085a1169eed91017d358cc0e9cbdd5dc236bcd710059dbf0a2f816.dd8bd9bfd3664b530ea4e645105f557769387b3da9f79bdb55ed556bdd80611d",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}