End of training

README.md

@@ -0,0 +1,111 @@
+---
+library_name: peft
+tags:
+- generated_from_trainer
+base_model: mahdibaghbanzadeh/seqsight_4096_512_27M
+metrics:
+- accuracy
+model-index:
+- name: GUE_EMP_H4-seqsight_4096_512_27M-L8_f
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# GUE_EMP_H4-seqsight_4096_512_27M-L8_f
+
+This model is a fine-tuned version of [mahdibaghbanzadeh/seqsight_4096_512_27M](https://huggingface.co/mahdibaghbanzadeh/seqsight_4096_512_27M) on the [mahdibaghbanzadeh/GUE_EMP_H4](https://huggingface.co/datasets/mahdibaghbanzadeh/GUE_EMP_H4) dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.2551
+- F1 Score: 0.9068
+- Accuracy: 0.9069
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0005
+- train_batch_size: 128
+- eval_batch_size: 128
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- training_steps: 10000
+
+### Training results
+
+| Training Loss | Epoch  | Step  | Validation Loss | F1 Score | Accuracy |
+|:-------------:|:------:|:-----:|:---------------:|:--------:|:--------:|
+| 0.3471        | 2.17   | 200   | 0.2845          | 0.8905   | 0.8905   |
+| 0.2699        | 4.35   | 400   | 0.2771          | 0.8975   | 0.8973   |
+| 0.256         | 6.52   | 600   | 0.2791          | 0.8980   | 0.8980   |
+| 0.2524        | 8.7    | 800   | 0.2844          | 0.8968   | 0.8966   |
+| 0.2355        | 10.87  | 1000  | 0.2721          | 0.8974   | 0.8973   |
+| 0.2317        | 13.04  | 1200  | 0.2751          | 0.8959   | 0.8960   |
+| 0.2212        | 15.22  | 1400  | 0.2724          | 0.9009   | 0.9008   |
+| 0.2121        | 17.39  | 1600  | 0.2963          | 0.8929   | 0.8925   |
+| 0.2091        | 19.57  | 1800  | 0.2757          | 0.8958   | 0.8960   |
+| 0.2005        | 21.74  | 2000  | 0.2915          | 0.8928   | 0.8925   |
+| 0.1924        | 23.91  | 2200  | 0.3004          | 0.8967   | 0.8966   |
+| 0.1899        | 26.09  | 2400  | 0.2886          | 0.8939   | 0.8939   |
+| 0.1833        | 28.26  | 2600  | 0.3037          | 0.8904   | 0.8905   |
+| 0.1757        | 30.43  | 2800  | 0.3066          | 0.8878   | 0.8877   |
+| 0.1697        | 32.61  | 3000  | 0.3119          | 0.8913   | 0.8912   |
+| 0.1624        | 34.78  | 3200  | 0.3126          | 0.8886   | 0.8884   |
+| 0.162         | 36.96  | 3400  | 0.3169          | 0.8927   | 0.8925   |
+| 0.1543        | 39.13  | 3600  | 0.3352          | 0.8842   | 0.8843   |
+| 0.1493        | 41.3   | 3800  | 0.3331          | 0.8896   | 0.8898   |
+| 0.147         | 43.48  | 4000  | 0.3379          | 0.8857   | 0.8857   |
+| 0.1452        | 45.65  | 4200  | 0.3330          | 0.8855   | 0.8857   |
+| 0.1398        | 47.83  | 4400  | 0.3484          | 0.8816   | 0.8816   |
+| 0.1334        | 50.0   | 4600  | 0.3562          | 0.8792   | 0.8795   |
+| 0.1307        | 52.17  | 4800  | 0.3630          | 0.8795   | 0.8795   |
+| 0.1298        | 54.35  | 5000  | 0.3667          | 0.8813   | 0.8816   |
+| 0.124         | 56.52  | 5200  | 0.3706          | 0.8791   | 0.8789   |
+| 0.1249        | 58.7   | 5400  | 0.3701          | 0.8814   | 0.8816   |
+| 0.12          | 60.87  | 5600  | 0.3806          | 0.8796   | 0.8795   |
+| 0.1154        | 63.04  | 5800  | 0.3927          | 0.8754   | 0.8754   |
+| 0.1164        | 65.22  | 6000  | 0.3902          | 0.8793   | 0.8795   |
+| 0.1063        | 67.39  | 6200  | 0.3974          | 0.8769   | 0.8768   |
+| 0.1064        | 69.57  | 6400  | 0.4084          | 0.8718   | 0.8720   |
+| 0.1069        | 71.74  | 6600  | 0.4106          | 0.8809   | 0.8809   |
+| 0.1047        | 73.91  | 6800  | 0.4167          | 0.8781   | 0.8782   |
+| 0.0999        | 76.09  | 7000  | 0.4229          | 0.8760   | 0.8761   |
+| 0.1023        | 78.26  | 7200  | 0.4182          | 0.8774   | 0.8775   |
+| 0.098         | 80.43  | 7400  | 0.4268          | 0.8835   | 0.8836   |
+| 0.0951        | 82.61  | 7600  | 0.4444          | 0.8688   | 0.8693   |
+| 0.0957        | 84.78  | 7800  | 0.4376          | 0.8829   | 0.8830   |
+| 0.0928        | 86.96  | 8000  | 0.4476          | 0.8786   | 0.8789   |
+| 0.094         | 89.13  | 8200  | 0.4529          | 0.8717   | 0.8720   |
+| 0.0912        | 91.3   | 8400  | 0.4487          | 0.8752   | 0.8754   |
+| 0.0863        | 93.48  | 8600  | 0.4661          | 0.8724   | 0.8727   |
+| 0.086         | 95.65  | 8800  | 0.4712          | 0.8718   | 0.8720   |
+| 0.0875        | 97.83  | 9000  | 0.4598          | 0.8718   | 0.8720   |
+| 0.0849        | 100.0  | 9200  | 0.4680          | 0.8712   | 0.8713   |
+| 0.084         | 102.17 | 9400  | 0.4662          | 0.8725   | 0.8727   |
+| 0.0833        | 104.35 | 9600  | 0.4652          | 0.8718   | 0.8720   |
+| 0.0823        | 106.52 | 9800  | 0.4745          | 0.8724   | 0.8727   |
+| 0.0855        | 108.7  | 10000 | 0.4703          | 0.8711   | 0.8713   |
+
+
+### Framework versions
+
+- PEFT 0.9.0
+- Transformers 4.38.2
+- Pytorch 2.2.0+cu121
+- Datasets 2.17.1
+- Tokenizers 0.15.2

adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mahdibaghbanzadeh/seqsight_4096_512_27M",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 2.0,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "classifier",
+    "pooler"
+  ],
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "query_proj",
+    "value_proj"
+  ],
+  "task_type": "SEQ_CLS",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eff39ecfc587aed68d44a1be869d95b65a12d8b37ed2131637536830c1de351d
+size 1584096

special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "cls_token": {
+    "content": "<cls>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<sos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,52 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<sos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<eos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<cls>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<cls>",
+  "eos_token": "<eos>",
+  "mask_token": "<sos>",
+  "max_length": 512,
+  "model_max_length": 512,
+  "pad_to_multiple_of": null,
+  "pad_token": "<pad>",
+  "pad_token_type_id": 0,
+  "padding": "max_length",
+  "padding_side": "right",
+  "stride": 0,
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "truncation": true,
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:938b28f14371a73779e2a7fc00c3ecfc474b5c8449f466c4d4aae3c0d29e4580
+size 4920