96abhishekarora commited on
Commit
7092b10
1 Parent(s): 9917996

Modified validation and training for linktransformer model

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ pytorch_model.bin filter=lfs diff=lfs merge=lfs -text
1_Pooling/config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "word_embedding_dimension": 768,
3
+ "pooling_mode_cls_token": false,
4
+ "pooling_mode_mean_tokens": true,
5
+ "pooling_mode_max_tokens": false,
6
+ "pooling_mode_mean_sqrt_len_tokens": false
7
+ }
LT_training_config.json ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_save_dir": "models",
3
+ "model_save_name": "linkage_un_data_es_fine_fine",
4
+ "opt_model_description": "This model was trained on a dataset prepared by linking product classifications from [UN stats](https://unstats.un.org/unsd/classifications/Econ). \n This model is designed to link different products together - trained on variation brought on by product level correspondance. It was trained for 100 epochs using other defaults that can be found in the repo's LinkTransformer config file - LT_training_config.json \n ",
5
+ "opt_model_lang": "es",
6
+ "train_batch_size": 64,
7
+ "num_epochs": 100,
8
+ "warm_up_perc": 1,
9
+ "learning_rate": 2e-06,
10
+ "val_perc": 0.2,
11
+ "wandb_names": {
12
+ "project": "linkage",
13
+ "id": "econabhishek",
14
+ "run": "linkage_un_data_es_fine_fine",
15
+ "entity": "econabhishek"
16
+ },
17
+ "add_pooling_layer": false,
18
+ "large_val": true,
19
+ "eval_steps_perc": 0.1,
20
+ "test_at_end": true,
21
+ "save_val_test_pickles": true,
22
+ "val_query_prop": 0.5,
23
+ "eval_type": "retrieval",
24
+ "training_dataset": "dataframe",
25
+ "base_model_path": "hiiamsid/sentence_similarity_spanish_es",
26
+ "best_model_path": "models/linkage_un_data_es_fine_fine"
27
+ }
README.md ADDED
@@ -0,0 +1,138 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: sentence-similarity
3
+ language:
4
+ - es
5
+ tags:
6
+ - linktransformer
7
+ - sentence-transformers
8
+ - sentence-similarity
9
+ - tabular-classification
10
+
11
+ ---
12
+
13
+ # dell-research-harvard/lt-un-data-fine-fine-es
14
+
15
+ This is a [LinkTransformer](https://github.com/dell-research-harvard/linktransformer) model. At its core this model this is a sentence transformer model [sentence-transformers](https://www.SBERT.net) model- it just wraps around the class.
16
+ It is designed for quick and easy record linkage (entity-matching) through the LinkTransformer package. The tasks include clustering, deduplication, linking, aggregation and more.
17
+ Notwithstanding that, it can be used for any sentence similarity task within the sentence-transformers framework as well.
18
+ It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.
19
+ Take a look at the documentation of [sentence-transformers](https://www.sbert.net/index.html) if you want to use this model for more than what we support in our applications.
20
+
21
+
22
+ This model has been fine-tuned on the model : hiiamsid/sentence_similarity_spanish_es. It is pretrained for the language : - es.
23
+
24
+
25
+ This model was trained on a dataset prepared by linking product classifications from [UN stats](https://unstats.un.org/unsd/classifications/Econ).
26
+ This model is designed to link different products together - trained on variation brought on by product level correspondance. It was trained for 100 epochs using other defaults that can be found in the repo's LinkTransformer config file - LT_training_config.json
27
+
28
+
29
+ ## Usage (LinkTransformer)
30
+
31
+ Using this model becomes easy when you have [LinkTransformer](https://github.com/dell-research-harvard/linktransformer) installed:
32
+
33
+ ```
34
+ pip install -U linktransformer
35
+ ```
36
+
37
+ Then you can use the model like this:
38
+
39
+ ```python
40
+ import linktransformer as lt
41
+ import pandas as pd
42
+
43
+ ##Load the two dataframes that you want to link. For example, 2 dataframes with company names that are written differently
44
+ df1=pd.read_csv("data/df1.csv") ###This is the left dataframe with key CompanyName for instance
45
+ df2=pd.read_csv("data/df2.csv") ###This is the right dataframe with key CompanyName for instance
46
+
47
+ ###Merge the two dataframes on the key column!
48
+ df_merged = lt.merge(df1, df2, on="CompanyName", how="inner")
49
+
50
+ ##Done! The merged dataframe has a column called "score" that contains the similarity score between the two company names
51
+
52
+ ```
53
+
54
+
55
+ ## Training your own LinkTransformer model
56
+ Any Sentence Transformers can be used as a backbone by simply adding a pooling layer. Any other transformer on HuggingFace can also be used by specifying the option add_pooling_layer==True
57
+ The model was trained using SupCon loss.
58
+ Usage can be found in the package docs.
59
+ The training config can be found in the repo with the name LT_training_config.json
60
+ To replicate the training, you can download the file and specify the path in the config_path argument of the training function. You can also override the config by specifying the training_args argument.
61
+ Here is an example.
62
+
63
+
64
+ ```python
65
+
66
+ ##Consider the example in the paper that has a dataset of Mexican products and their tariff codes from 1947 and 1948 and we want train a model to link the two tariff codes.
67
+ saved_model_path = train_model(
68
+ model_path="hiiamsid/sentence_similarity_spanish_es",
69
+ dataset_path=dataset_path,
70
+ left_col_names=["description47"],
71
+ right_col_names=['description48'],
72
+ left_id_name=['tariffcode47'],
73
+ right_id_name=['tariffcode48'],
74
+ log_wandb=False,
75
+ config_path=LINKAGE_CONFIG_PATH,
76
+ training_args={"num_epochs": 1}
77
+ )
78
+
79
+ ```
80
+
81
+
82
+ You can also use this package for deduplication (clusters a df on the supplied key column). Merging a fine class (like product) to a coarse class (like HS code) is also possible.
83
+ Read our paper and the documentation for more!
84
+
85
+
86
+
87
+ ## Evaluation Results
88
+
89
+ <!--- Describe how your model was evaluated -->
90
+
91
+ You can evaluate the model using the [LinkTransformer](https://github.com/dell-research-harvard/linktransformer) package's inference functions.
92
+ We have provided a few datasets in the package for you to try out. We plan to host more datasets on Huggingface and our website (Coming soon) that you can take a look at.
93
+
94
+
95
+ ## Training
96
+ The model was trained with the parameters:
97
+
98
+ **DataLoader**:
99
+
100
+ `torch.utils.data.dataloader.DataLoader` of length 86 with parameters:
101
+ ```
102
+ {'batch_size': 64, 'sampler': 'torch.utils.data.dataloader._InfiniteConstantSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
103
+ ```
104
+
105
+ **Loss**:
106
+
107
+ `linktransformer.modified_sbert.losses.SupConLoss_wandb`
108
+
109
+ Parameters of the fit()-Method:
110
+ ```
111
+ {
112
+ "epochs": 100,
113
+ "evaluation_steps": 860,
114
+ "evaluator": "sentence_transformers.evaluation.SequentialEvaluator.SequentialEvaluator",
115
+ "max_grad_norm": 1,
116
+ "optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
117
+ "optimizer_params": {
118
+ "lr": 2e-06
119
+ },
120
+ "scheduler": "WarmupLinear",
121
+ "steps_per_epoch": null,
122
+ "warmup_steps": 8600,
123
+ "weight_decay": 0.01
124
+ }
125
+ ```
126
+
127
+
128
+
129
+
130
+ LinkTransformer(
131
+ (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
132
+ (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
133
+ )
134
+ ```
135
+
136
+ ## Citing & Authors
137
+
138
+ <!--- Describe where people can find more information -->
config.json ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "models/linkage_un_data_es_fine_fine/",
3
+ "architectures": [
4
+ "BertModel"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "classifier_dropout": null,
8
+ "gradient_checkpointing": false,
9
+ "hidden_act": "gelu",
10
+ "hidden_dropout_prob": 0.1,
11
+ "hidden_size": 768,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 3072,
14
+ "layer_norm_eps": 1e-12,
15
+ "max_position_embeddings": 512,
16
+ "model_type": "bert",
17
+ "num_attention_heads": 12,
18
+ "num_hidden_layers": 12,
19
+ "output_past": true,
20
+ "pad_token_id": 1,
21
+ "position_embedding_type": "absolute",
22
+ "torch_dtype": "float32",
23
+ "transformers_version": "4.31.0",
24
+ "type_vocab_size": 2,
25
+ "use_cache": true,
26
+ "vocab_size": 31002
27
+ }
config_sentence_transformers.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "__version__": {
3
+ "sentence_transformers": "2.0.0",
4
+ "transformers": "4.10.2",
5
+ "pytorch": "1.9.0+cu102"
6
+ }
7
+ }
modules.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "idx": 0,
4
+ "name": "0",
5
+ "path": "",
6
+ "type": "sentence_transformers.models.Transformer"
7
+ },
8
+ {
9
+ "idx": 1,
10
+ "name": "1",
11
+ "path": "1_Pooling",
12
+ "type": "sentence_transformers.models.Pooling"
13
+ }
14
+ ]
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c34128b167bad3511f45071d071078f1f9fcaf7a630c724c26c27196e440acf9
3
+ size 439467497
sentence_bert_config.json ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {
2
+ "max_seq_length": 512,
3
+ "do_lower_case": false
4
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": "[CLS]",
3
+ "mask_token": "[MASK]",
4
+ "pad_token": "[PAD]",
5
+ "sep_token": "[SEP]",
6
+ "unk_token": "[UNK]"
7
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "clean_up_tokenization_spaces": true,
3
+ "cls_token": "[CLS]",
4
+ "do_basic_tokenize": true,
5
+ "do_lower_case": false,
6
+ "mask_token": "[MASK]",
7
+ "model_max_length": 512,
8
+ "never_split": null,
9
+ "pad_token": "[PAD]",
10
+ "sep_token": "[SEP]",
11
+ "strip_accents": false,
12
+ "tokenize_chinese_chars": true,
13
+ "tokenizer_class": "BertTokenizer",
14
+ "unk_token": "[UNK]"
15
+ }
vocab.txt ADDED
The diff for this file is too large to render. See raw diff