initial commit

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 1024,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

README.md

@@ -1,3 +1,128 @@
 ---
-license: mit
+pipeline_tag: sentence-similarity
+tags:
+- sentence-transformers
+- feature-extraction
+- sentence-similarity
+- transformers
 ---
+
+# {MODEL_NAME}
+
+This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 1024 dimensional dense vector space and can be used for tasks like clustering or semantic search.
+
+<!--- Describe your model here -->
+
+## Usage (Sentence-Transformers)
+
+Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+
+```
+pip install -U sentence-transformers
+```
+
+Then you can use the model like this:
+
+```python
+from sentence_transformers import SentenceTransformer
+sentences = ["This is an example sentence", "Each sentence is converted"]
+
+model = SentenceTransformer('{MODEL_NAME}')
+embeddings = model.encode(sentences)
+print(embeddings)
+```
+
+
+
+## Usage (HuggingFace Transformers)
+Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+
+#Mean Pooling - Take attention mask into account for correct averaging
+def mean_pooling(model_output, attention_mask):
+    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
+    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
+    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+
+
+# Sentences we want sentence embeddings for
+sentences = ['This is an example sentence', 'Each sentence is converted']
+
+# Load model from HuggingFace Hub
+tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
+model = AutoModel.from_pretrained('{MODEL_NAME}')
+
+# Tokenize sentences
+encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
+
+# Compute token embeddings
+with torch.no_grad():
+    model_output = model(**encoded_input)
+
+# Perform pooling. In this case, mean pooling.
+sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
+
+print("Sentence embeddings:")
+print(sentence_embeddings)
+```
+
+
+
+## Evaluation Results
+
+<!--- Describe how your model was evaluated -->
+
+For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME})
+
+
+## Training
+The model was trained with the parameters:
+
+**DataLoader**:
+
+`torch.utils.data.dataloader.DataLoader` of length 25000 with parameters:
+```
+{'batch_size': 4, 'sampler': 'torch.utils.data.sampler.SequentialSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
+```
+
+**Loss**:
+
+`sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss` with parameters:
+  ```
+  {'scale': 20.0, 'similarity_fct': 'cos_sim'}
+  ```
+
+Parameters of the fit()-Method:
+```
+{
+    "epochs": 1,
+    "evaluation_steps": 100,
+    "evaluator": "__main__.LossEvaluator",
+    "max_grad_norm": 1,
+    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
+    "optimizer_params": {
+        "lr": 1e-05
+    },
+    "scheduler": "WarmupLinear",
+    "steps_per_epoch": null,
+    "warmup_steps": 0,
+    "weight_decay": 0.01
+}
+```
+
+
+## Full Model Architecture
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
+  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
+)
+```
+
+## Citing & Authors
+
+<!--- Describe where people can find more information -->

config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "/home/ruimelo/.cache/torch/sentence_transformers/neuralmind_bert-large-portuguese-cased",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "directionality": "bidi",
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "output_past": true,
+  "pad_token_id": 0,
+  "pooler_fc_size": 768,
+  "pooler_num_attention_heads": 12,
+  "pooler_num_fc_layers": 3,
+  "pooler_size_per_head": 128,
+  "pooler_type": "first_token_transform",
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.20.1",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 29794
+}

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.2.0",
+    "transformers": "4.20.1",
+    "pytorch": "1.10.1+cu111"
+  }
+}

eval/loss_evaluation_dev_results.csv

@@ -0,0 +1,251 @@
+epoch,steps,loss
+0,100,0.19519549049928608
+0,200,0.11560484829516172
+0,300,0.08679192887225318
+0,400,0.07406212777689816
+0,500,0.06499064764375434
+0,600,0.05987601100840652
+0,700,0.055965147196659834
+0,800,0.05774995323606245
+0,900,0.05295955123265526
+0,1000,0.05274097613280328
+0,1100,0.05067385239212754
+0,1200,0.05087106744197604
+0,1300,0.049993227635551966
+0,1400,0.048679257646413274
+0,1500,0.04968225817311177
+0,1600,0.051728904252822075
+0,1700,0.04877414873885723
+0,1800,0.05247588143790906
+0,1900,0.047604353220216084
+0,2000,0.04749206604852059
+0,2100,0.04782016522758276
+0,2200,0.04651157213780695
+0,2300,0.04710628148749038
+0,2400,0.046624648567542
+0,2500,0.04509524600019381
+0,2600,0.04553730478420182
+0,2700,0.04443945448109487
+0,2800,0.0504640726274293
+0,2900,0.04794691643282241
+0,3000,0.04631040973041582
+0,3100,0.0419986342476275
+0,3200,0.04298793683305728
+0,3300,0.04493164272913312
+0,3400,0.048255282522043044
+0,3500,0.04978693392673049
+0,3600,0.04584045348395002
+0,3700,0.04929085410937766
+0,3800,0.048445018135582926
+0,3900,0.046708384944145157
+0,4000,0.04662567339258236
+0,4100,0.0472695262937003
+0,4200,0.048288902709505144
+0,4300,0.048463549996224084
+0,4400,0.043781441836662466
+0,4500,0.04312372630505224
+0,4600,0.045321417531253336
+0,4700,0.04252031567532863
+0,4800,0.0530112666398089
+0,4900,0.052159558343869504
+0,5000,0.052686183118791245
+0,5100,0.04998561888884692
+0,5200,0.044343194892997005
+0,5300,0.0423403514099241
+0,5400,0.04481474702306517
+0,5500,0.04676144200633235
+0,5600,0.04174483070197358
+0,5700,0.04355011108918061
+0,5800,0.04652475086493452
+0,5900,0.045437329526519125
+0,6000,0.044627202456709925
+0,6100,0.043920307074457196
+0,6200,0.042049196839164645
+0,6300,0.04682356477219086
+0,6400,0.04487424387279889
+0,6500,0.041516137345119
+0,6600,0.04123407529385229
+0,6700,0.03734822506002114
+0,6800,0.04004483578493084
+0,6900,0.04361605496124544
+0,7000,0.044393963018599165
+0,7100,0.04498864975572355
+0,7200,0.044416080061861235
+0,7300,0.04217950248869233
+0,7400,0.04202356934366427
+0,7500,0.04097753317170045
+0,7600,0.03903316448376711
+0,7700,0.04317112945482087
+0,7800,0.04497662772605678
+0,7900,0.04109697778423021
+0,8000,0.04386395559431636
+0,8100,0.04435155229125319
+0,8200,0.040241758321292356
+0,8300,0.04920905432964724
+0,8400,0.045273166227681634
+0,8500,0.045771352062498875
+0,8600,0.03970043939072392
+0,8700,0.041097908408486525
+0,8800,0.04337787134086743
+0,8900,0.043671976632325096
+0,9000,0.040776167853089046
+0,9100,0.04171571797915774
+0,9200,0.03746827632520056
+0,9300,0.03856413216644577
+0,9400,0.041763630464973195
+0,9500,0.0395228136582546
+0,9600,0.04500009461940554
+0,9700,0.04361399264472892
+0,9800,0.047162896827277506
+0,9900,0.04293111109975825
+0,10000,0.04538575671103895
+0,10100,0.043648700229026886
+0,10200,0.04136474249746654
+0,10300,0.04508329086149529
+0,10400,0.04102850488844959
+0,10500,0.042174578120627075
+0,10600,0.045043971799346896
+0,10700,0.0436181597908299
+0,10800,0.045259078109792475
+0,10900,0.04371035268960593
+0,11000,0.05035991068870275
+0,11100,0.050761380571160454
+0,11200,0.04406444633185185
+0,11300,0.04401907154579702
+0,11400,0.04374491291463001
+0,11500,0.041598092203370504
+0,11600,0.041415777919197524
+0,11700,0.04249067280007211
+0,11800,0.03923704199554693
+0,11900,0.0363335097560149
+0,12000,0.04222154671425733
+0,12100,0.03865254473414243
+0,12200,0.03969156562322112
+0,12300,0.03945732652428465
+0,12400,0.041877292867345935
+0,12500,0.036688783095289904
+0,12600,0.04137931299509875
+0,12700,0.037526527193307416
+0,12800,0.03955853321622893
+0,12900,0.04099604392775696
+0,13000,0.038100052026215914
+0,13100,0.04037489445645954
+0,13200,0.037006299523469385
+0,13300,0.042210353803639335
+0,13400,0.042162665614587515
+0,13500,0.04045078091329652
+0,13600,0.04178211537794941
+0,13700,0.03652732793331884
+0,13800,0.04007450492148122
+0,13900,0.040218797176888324
+0,14000,0.03825300664909627
+0,14100,0.04205769400583465
+0,14200,0.04096333694347577
+0,14300,0.0389199056238846
+0,14400,0.037719650394660416
+0,14500,0.04263562075523331
+0,14600,0.03808142022118219
+0,14700,0.04628894311818186
+0,14800,0.039785022687983417
+0,14900,0.039248060891297155
+0,15000,0.04015960164872535
+0,15100,0.04400960119832234
+0,15200,0.044337519492261744
+0,15300,0.04161765173295095
+0,15400,0.04071474287225717
+0,15500,0.039765120246020164
+0,15600,0.042707479120178665
+0,15700,0.04196122203464124
+0,15800,0.03900735156519495
+0,15900,0.036981938280766895
+0,16000,0.03967288962420271
+0,16100,0.036723857662762045
+0,16200,0.04005734996749844
+0,16300,0.04027912320752289
+0,16400,0.043616688434242885
+0,16500,0.042757092717327604
+0,16600,0.040512548224817806
+0,16700,0.03594136324969477
+0,16800,0.038857869270918104
+0,16900,0.04087193688661806
+0,17000,0.03912139527871697
+0,17100,0.03842234752314098
+0,17200,0.03649764288259497
+0,17300,0.04245655374152135
+0,17400,0.039467562094128494
+0,17500,0.03991257693460278
+0,17600,0.04171786952817289
+0,17700,0.04471105680426285
+0,17800,0.0367856082773753
+0,17900,0.03679781602542855
+0,18000,0.03854221257501377
+0,18100,0.040181813599715586
+0,18200,0.0407157541238927
+0,18300,0.037851696226764577
+0,18400,0.03831218913948021
+0,18500,0.03791270016791887
+0,18600,0.03622766606910176
+0,18700,0.03551119881726873
+0,18800,0.03778034173768933
+0,18900,0.03405767042893223
+0,19000,0.03123430945533104
+0,19100,0.037109243501212134
+0,19200,0.036391455788788406
+0,19300,0.032642522298414564
+0,19400,0.03444629282929268
+0,19500,0.03728879319979016
+0,19600,0.03744477383985601
+0,19700,0.03397694265227539
+0,19800,0.03912842301241188
+0,19900,0.03756071515860115
+0,20000,0.03825289866256772
+0,20100,0.037043497484298006
+0,20200,0.03586015019140629
+0,20300,0.03841649508690972
+0,20400,0.03709434958143799
+0,20500,0.03766999650176518
+0,20600,0.03719969458871243
+0,20700,0.03763643987506886
+0,20800,0.03661399590211345
+0,20900,0.034543956276607314
+0,21000,0.037338983882914366
+0,21100,0.038684293762035145
+0,21200,0.03122012103122229
+0,21300,0.03625594341468651
+0,21400,0.03636522202243
+0,21500,0.03669486281276811
+0,21600,0.03786981438117198
+0,21700,0.03672024818368426
+0,21800,0.036491299151409376
+0,21900,0.033634753646258855
+0,22000,0.037865872911989916
+0,22100,0.03907738132622352
+0,22200,0.034167471399115856
+0,22300,0.03912497054712691
+0,22400,0.04040111948641333
+0,22500,0.04145388534234468
+0,22600,0.03720971221760168
+0,22700,0.033648781347541845
+0,22800,0.03764335221710776
+0,22900,0.036039476440455374
+0,23000,0.03600912533784493
+0,23100,0.03687414772574997
+0,23200,0.04035678972016075
+0,23300,0.03742495229770756
+0,23400,0.0347357924013799
+0,23500,0.03706875827863819
+0,23600,0.0378347951889791
+0,23700,0.03531763351729598
+0,23800,0.036277216902136
+0,23900,0.03563792866617466
+0,24000,0.03703486210005108
+0,24100,0.037769587493760956
+0,24200,0.03749001277966459
+0,24300,0.03960652796490469
+0,24400,0.036781374730451545
+0,24500,0.03711627634396336
+0,24600,0.03975872469308434
+0,24700,0.03539313475455226
+0,24800,0.03443953339789755
+0,24900,0.03367993894758666
+0,25000,0.036054142671664645

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ca163577f3a570f7600781ce1f5ae43dc89e5c0d73c3f8ae80bb706a4a5d372
+size 1337719025

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": false,
+  "mask_token": "[MASK]",
+  "name_or_path": "/home/ruimelo/.cache/torch/sentence_transformers/neuralmind_bert-large-portuguese-cased",
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "special_tokens_map_file": "/home/ruimelo/.cache/torch/sentence_transformers/neuralmind_bert-large-portuguese-cased/special_tokens_map.json",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}