End of training

README.md

@@ -13,7 +13,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [](https://huggingface.co/) on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.7800
+- Loss: 0.8854
 
 ## Model description
 
@@ -32,119 +32,28 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 2e-05
+- learning_rate: 0.001
 - train_batch_size: 512
 - eval_batch_size: 512
 - seed: 42
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: linear
-- num_epochs: 100
-- mixed_precision_training: Native AMP
+- num_epochs: 10
 
 ### Training results
 
 | Training Loss | Epoch | Step | Validation Loss |
 |:-------------:|:-----:|:----:|:---------------:|
-| No log        | 1.0   | 6    | 6.5672          |
-| No log        | 2.0   | 12   | 5.9069          |
-| No log        | 3.0   | 18   | 5.4671          |
-| No log        | 4.0   | 24   | 5.0855          |
-| No log        | 5.0   | 30   | 4.7085          |
-| No log        | 6.0   | 36   | 4.3935          |
-| No log        | 7.0   | 42   | 4.1354          |
-| No log        | 8.0   | 48   | 3.9330          |
-| No log        | 9.0   | 54   | 3.7647          |
-| No log        | 10.0  | 60   | 3.6286          |
-| No log        | 11.0  | 66   | 3.5173          |
-| No log        | 12.0  | 72   | 3.4148          |
-| No log        | 13.0  | 78   | 3.3148          |
-| No log        | 14.0  | 84   | 3.2218          |
-| No log        | 15.0  | 90   | 3.1353          |
-| No log        | 16.0  | 96   | 3.0524          |
-| No log        | 17.0  | 102  | 2.9751          |
-| No log        | 18.0  | 108  | 2.9017          |
-| No log        | 19.0  | 114  | 2.8277          |
-| No log        | 20.0  | 120  | 2.7534          |
-| No log        | 21.0  | 126  | 2.6865          |
-| No log        | 22.0  | 132  | 2.6260          |
-| No log        | 23.0  | 138  | 2.5597          |
-| No log        | 24.0  | 144  | 2.4913          |
-| No log        | 25.0  | 150  | 2.4295          |
-| No log        | 26.0  | 156  | 2.3668          |
-| No log        | 27.0  | 162  | 2.3040          |
-| No log        | 28.0  | 168  | 2.2391          |
-| No log        | 29.0  | 174  | 2.1745          |
-| No log        | 30.0  | 180  | 2.1036          |
-| No log        | 31.0  | 186  | 2.0443          |
-| No log        | 32.0  | 192  | 1.9919          |
-| No log        | 33.0  | 198  | 1.9321          |
-| No log        | 34.0  | 204  | 1.8771          |
-| No log        | 35.0  | 210  | 1.8386          |
-| No log        | 36.0  | 216  | 1.7951          |
-| No log        | 37.0  | 222  | 1.7460          |
-| No log        | 38.0  | 228  | 1.6974          |
-| No log        | 39.0  | 234  | 1.6576          |
-| No log        | 40.0  | 240  | 1.6112          |
-| No log        | 41.0  | 246  | 1.5811          |
-| No log        | 42.0  | 252  | 1.5540          |
-| No log        | 43.0  | 258  | 1.5268          |
-| No log        | 44.0  | 264  | 1.4873          |
-| No log        | 45.0  | 270  | 1.4500          |
-| No log        | 46.0  | 276  | 1.4161          |
-| No log        | 47.0  | 282  | 1.3738          |
-| No log        | 48.0  | 288  | 1.3495          |
-| No log        | 49.0  | 294  | 1.3182          |
-| No log        | 50.0  | 300  | 1.2899          |
-| No log        | 51.0  | 306  | 1.2610          |
-| No log        | 52.0  | 312  | 1.2478          |
-| No log        | 53.0  | 318  | 1.2238          |
-| No log        | 54.0  | 324  | 1.2060          |
-| No log        | 55.0  | 330  | 1.1794          |
-| No log        | 56.0  | 336  | 1.1774          |
-| No log        | 57.0  | 342  | 1.1425          |
-| No log        | 58.0  | 348  | 1.1166          |
-| No log        | 59.0  | 354  | 1.1044          |
-| No log        | 60.0  | 360  | 1.0913          |
-| No log        | 61.0  | 366  | 1.0775          |
-| No log        | 62.0  | 372  | 1.0694          |
-| No log        | 63.0  | 378  | 1.0311          |
-| No log        | 64.0  | 384  | 1.0272          |
-| No log        | 65.0  | 390  | 1.0249          |
-| No log        | 66.0  | 396  | 0.9923          |
-| No log        | 67.0  | 402  | 0.9892          |
-| No log        | 68.0  | 408  | 0.9762          |
-| No log        | 69.0  | 414  | 0.9704          |
-| No log        | 70.0  | 420  | 0.9405          |
-| No log        | 71.0  | 426  | 0.9394          |
-| No log        | 72.0  | 432  | 0.9237          |
-| No log        | 73.0  | 438  | 0.9180          |
-| No log        | 74.0  | 444  | 0.8926          |
-| No log        | 75.0  | 450  | 0.9081          |
-| No log        | 76.0  | 456  | 0.8778          |
-| No log        | 77.0  | 462  | 0.8785          |
-| No log        | 78.0  | 468  | 0.8580          |
-| No log        | 79.0  | 474  | 0.8593          |
-| No log        | 80.0  | 480  | 0.8553          |
-| No log        | 81.0  | 486  | 0.8671          |
-| No log        | 82.0  | 492  | 0.8422          |
-| No log        | 83.0  | 498  | 0.8403          |
-| 2.0956        | 84.0  | 504  | 0.8355          |
-| 2.0956        | 85.0  | 510  | 0.8188          |
-| 2.0956        | 86.0  | 516  | 0.8149          |
-| 2.0956        | 87.0  | 522  | 0.8285          |
-| 2.0956        | 88.0  | 528  | 0.8063          |
-| 2.0956        | 89.0  | 534  | 0.8166          |
-| 2.0956        | 90.0  | 540  | 0.8008          |
-| 2.0956        | 91.0  | 546  | 0.8127          |
-| 2.0956        | 92.0  | 552  | 0.7921          |
-| 2.0956        | 93.0  | 558  | 0.8015          |
-| 2.0956        | 94.0  | 564  | 0.7882          |
-| 2.0956        | 95.0  | 570  | 0.7844          |
-| 2.0956        | 96.0  | 576  | 0.7862          |
-| 2.0956        | 97.0  | 582  | 0.7810          |
-| 2.0956        | 98.0  | 588  | 0.7808          |
-| 2.0956        | 99.0  | 594  | 0.7810          |
-| 2.0956        | 100.0 | 600  | 0.7800          |
+| 3.0459        | 1.0   | 5    | 2.4452          |
+| 2.2781        | 2.0   | 10   | 2.0536          |
+| 1.883         | 3.0   | 15   | 1.6599          |
+| 1.5519        | 4.0   | 20   | 1.4205          |
+| 1.3589        | 5.0   | 25   | 1.2785          |
+| 1.2145        | 6.0   | 30   | 1.1344          |
+| 1.0845        | 7.0   | 35   | 1.0183          |
+| 0.9945        | 8.0   | 40   | 0.9482          |
+| 0.9314        | 9.0   | 45   | 0.8990          |
+| 0.8972        | 10.0  | 50   | 0.8854          |
 
 
 ### Framework versions

config.json

@@ -25,13 +25,13 @@
     "forced_eos_token_id": null,
     "hidden_act": "gelu",
     "hidden_dropout_prob": 0.1,
-    "hidden_size": 768,
+    "hidden_size": 256,
     "id2label": {
       "0": "LABEL_0",
       "1": "LABEL_1"
     },
     "initializer_range": 0.02,
-    "intermediate_size": 3072,
+    "intermediate_size": 1024,
     "is_decoder": true,
     "is_encoder_decoder": false,
     "label2id": {
@@ -45,10 +45,10 @@
     "min_length": 0,
     "model_type": "bert",
     "no_repeat_ngram_size": 0,
-    "num_attention_heads": 12,
+    "num_attention_heads": 4,
     "num_beam_groups": 1,
     "num_beams": 1,
-    "num_hidden_layers": 12,
+    "num_hidden_layers": 4,
     "num_return_sequences": 1,
     "output_attentions": false,
     "output_hidden_states": false,
@@ -78,9 +78,9 @@
     "typical_p": 1.0,
     "use_bfloat16": false,
     "use_cache": true,
-    "vocab_size": 30522
+    "vocab_size": 51
   },
-  "decoder_start_token_id": 50,
+  "decoder_start_token_id": 49,
   "encoder": {
     "_name_or_path": "",
     "add_cross_attention": false,
@@ -104,13 +104,13 @@
     "forced_eos_token_id": null,
     "hidden_act": "gelu",
     "hidden_dropout_prob": 0.1,
-    "hidden_size": 768,
+    "hidden_size": 256,
     "id2label": {
       "0": "LABEL_0",
       "1": "LABEL_1"
     },
     "initializer_range": 0.02,
-    "intermediate_size": 3072,
+    "intermediate_size": 1024,
     "is_decoder": false,
     "is_encoder_decoder": false,
     "label2id": {
@@ -124,10 +124,10 @@
     "min_length": 0,
     "model_type": "bert",
     "no_repeat_ngram_size": 0,
-    "num_attention_heads": 12,
+    "num_attention_heads": 4,
     "num_beam_groups": 1,
     "num_beams": 1,
-    "num_hidden_layers": 12,
+    "num_hidden_layers": 4,
     "num_return_sequences": 1,
     "output_attentions": false,
     "output_hidden_states": false,
@@ -157,11 +157,12 @@
     "typical_p": 1.0,
     "use_bfloat16": false,
     "use_cache": true,
-    "vocab_size": 30522
+    "vocab_size": 51
   },
+  "eos_token_id": 50,
   "is_encoder_decoder": true,
   "model_type": "encoder-decoder",
-  "pad_token_id": 49,
+  "pad_token_id": 48,
   "torch_dtype": "float32",
   "transformers_version": "4.37.2"
 }

generation_config.json

@@ -1,6 +1,7 @@
 {
   "_from_model_config": true,
-  "decoder_start_token_id": 50,
-  "pad_token_id": 49,
+  "decoder_start_token_id": 49,
+  "eos_token_id": 50,
+  "pad_token_id": 48,
   "transformers_version": "4.37.2"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fae817cbabf8915ca265feb78d2fad2a1c3568e7be552c9c73e7b1732c4434af
-size 989518784
+oid sha256:d68eedd7c2ce4a569ddec467812d03352c8a7b222c55e6dde6b7c022a7ea4859
+size 31203500

runs/Feb28_12-49-12_3897ec21fae5/events.out.tfevents.1709124554.3897ec21fae5.522.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47ae2b789df2104932122b9ffa05fe48eb5d7ce55098569eb2c03d539761f1ec
+size 10840

runs/Feb28_12-53-45_3897ec21fae5/events.out.tfevents.1709124826.3897ec21fae5.9652.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b85740a246987aa9d85ecec16059f1234089704a567374d5a00b2d22c0c6bd8d
+size 8739

runs/Feb28_12-55-56_3897ec21fae5/events.out.tfevents.1709124958.3897ec21fae5.10174.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e7f99781de3f25bf836d547c6645e62aece21c6afa1f606efa4378d8464f234
+size 12867

special_tokens_map.json

@@ -6,6 +6,13 @@
     "rstrip": false,
     "single_word": false
   },
+  "eos_token": {
+    "content": "[EOS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
   "pad_token": {
     "content": "[PAD]",
     "lstrip": false,

tokenizer.json

@@ -4,7 +4,7 @@
   "padding": null,
   "added_tokens": [
     {
-      "id": 49,
+      "id": 48,
       "content": "[PAD]",
       "single_word": false,
       "lstrip": false,
@@ -13,13 +13,22 @@
       "special": true
     },
     {
-      "id": 50,
+      "id": 49,
       "content": "[CLS]",
       "single_word": false,
       "lstrip": false,
       "rstrip": false,
       "normalized": false,
       "special": true
+    },
+    {
+      "id": 50,
+      "content": "[EOS]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
     }
   ],
   "normalizer": null,
@@ -50,80 +59,78 @@
       "8": 10,
       "9": 11,
       "=": 12,
-      "99": 13,
-      "10": 14,
-      "98": 15,
-      "11": 16,
-      "97": 17,
-      "12": 18,
+      "10": 13,
+      "99": 14,
+      "11": 15,
+      "98": 16,
+      "12": 17,
+      "97": 18,
       "96": 19,
       "13": 20,
-      "95": 21,
-      "14": 22,
-      "94": 23,
-      "15": 24,
-      "93": 25,
-      "16": 26,
+      "14": 21,
+      "95": 22,
+      "15": 23,
+      "94": 24,
+      "16": 25,
+      "93": 26,
       "17": 27,
       "92": 28,
-      "91": 29,
-      "18": 30,
-      "90": 31,
-      "19": 32,
-      "89": 33,
-      "20": 34,
-      "21": 35,
-      "88": 36,
+      "18": 29,
+      "91": 30,
+      "19": 31,
+      "90": 32,
+      "20": 33,
+      "89": 34,
+      "88": 35,
+      "21": 36,
       "22": 37,
       "87": 38,
       "23": 39,
       "86": 40,
-      "85": 41,
-      "24": 42,
-      "25": 43,
-      "84": 44,
+      "24": 41,
+      "85": 42,
+      "84": 43,
+      "25": 44,
       "26": 45,
       "83": 46,
-      "27": 47,
-      "82": 48
+      "27": 47
     },
     "merges": [
-      "9 9",
       "1 0",
-      "9 8",
+      "9 9",
       "1 1",
-      "9 7",
+      "9 8",
       "1 2",
+      "9 7",
       "9 6",
       "1 3",
-      "9 5",
       "1 4",
-      "9 4",
+      "9 5",
       "1 5",
-      "9 3",
+      "9 4",
       "1 6",
+      "9 3",
       "1 7",
       "9 2",
-      "9 1",
       "1 8",
-      "9 0",
+      "9 1",
       "1 9",
-      "8 9",
+      "9 0",
       "2 0",
-      "2 1",
+      "8 9",
       "8 8",
+      "2 1",
       "2 2",
       "8 7",
       "2 3",
       "8 6",
-      "8 5",
       "2 4",
-      "2 5",
+      "8 5",
       "8 4",
+      "2 5",
       "2 6",
       "8 3",
-      "2 7",
-      "8 2"
+      "2 7"
     ]
   }
 }

tokenizer_config.json

@@ -1,6 +1,6 @@
 {
   "added_tokens_decoder": {
-    "49": {
+    "48": {
       "content": "[PAD]",
       "lstrip": false,
       "normalized": false,
@@ -8,17 +8,26 @@
       "single_word": false,
       "special": true
     },
-    "50": {
+    "49": {
       "content": "[CLS]",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
+    },
+    "50": {
+      "content": "[EOS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
     }
   },
   "clean_up_tokenization_spaces": true,
   "cls_token": "[CLS]",
+  "eos_token": "[EOS]",
   "model_max_length": 1000000000000000019884624838656,
   "pad_token": "[PAD]",
   "tokenizer_class": "PreTrainedTokenizerFast"

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:12b3e8c762fa264d4d0652b207fff70f26ff840984ec4601c4b064c0086cc0a9
+oid sha256:c22510d433bcd6c8ab9f638615a22414f615f7dc618aab32a485345ef2763c73
 size 4920