Upload with huggingface_hub

README.md

@@ -0,0 +1,79 @@
+---
+license: apache-2.0
+tags:
+- generated_from_trainer
+model-index:
+- name: t0-all_tasksv2-m1-t1
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# t0-all_tasksv2-m1-t1
+
+This model is a fine-tuned version of [bigscience/T0_3B](https://huggingface.co/bigscience/T0_3B) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 1.1591
+- Train Runtime: 31498.6176
+- Train Samples Per Second: 15.232
+- Train Steps Per Second: 0.212
+- Train Loss: 1.4163
+- Train Samples: 239899
+- Gen Len: 9.847
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-05
+- train_batch_size: 3
+- eval_batch_size: 3
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 8
+- gradient_accumulation_steps: 3
+- total_train_batch_size: 72
+- total_eval_batch_size: 24
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 2.0
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Rouge1  | Rouge2 | Rougel  | Rougelsum | Accuracy | F1      | Recall  | Precision | Bleu 1 | Bleu 2 | Bleu 3 | Bleu 4 | Rouge L | Gen Len |
+|:-------------:|:-----:|:----:|:---------------:|:-------:|:------:|:-------:|:---------:|:--------:|:-------:|:-------:|:---------:|:------:|:------:|:------:|:------:|:-------:|:-------:|
+| 1.5957        | 0.15  | 500  | 1.3177          | 51.7673 | 5.9934 | 51.3052 | 51.4889   | 58.2201  | 58.2201 | 58.2201 | 58.2201   | 0.5452 | 0.0004 | 0.0    | 0.0    | 0.5025  | 6.144   |
+| 1.5657        | 0.3   | 1000 | 1.2654          | 56.3471 | 6.2554 | 55.9191 | 56.0106   | 64.5902  | 64.5902 | 64.5902 | 64.5902   | 0.5834 | 0.0005 | 0.0    | 0.0    | 0.5423  | 6.363   |
+| 1.4614        | 0.45  | 1500 | 1.2279          | 60.3041 | 6.4454 | 59.881  | 60.0203   | 69.9766  | 69.9766 | 69.9766 | 69.9766   | 0.6223 | 0.0005 | 0.0    | 0.0    | 0.5799  | 6.319   |
+| 1.4733        | 0.6   | 2000 | 1.2001          | 63.1864 | 6.4428 | 62.7935 | 63.0146   | 74.192   | 74.192  | 74.192  | 74.192    | 0.6527 | 0.0005 | 0.0001 | 0.0    | 0.6102  | 6.319   |
+| 1.3982        | 0.75  | 2500 | 1.1888          | 64.2445 | 6.6019 | 63.8196 | 63.9475   | 75.0351  | 75.0351 | 75.0351 | 75.0351   | 0.6606 | 0.0005 | 0.0001 | 0.0    | 0.6151  | 6.3657  |
+| 1.4344        | 0.9   | 3000 | 1.1827          | 63.9356 | 6.7482 | 63.5225 | 63.72     | 74.6136  | 74.6136 | 74.6136 | 74.6136   | 0.6576 | 0.0005 | 0.0001 | 0.0    | 0.6123  | 6.3577  |
+| 1.3281        | 1.05  | 3500 | 1.1725          | 65.0553 | 6.6823 | 64.6434 | 64.8219   | 76.2529  | 76.2529 | 76.2529 | 76.2529   | 0.6679 | 0.0005 | 0.0001 | 0.0    | 0.6206  | 6.374   |
+| 1.3033        | 1.2   | 4000 | 1.1753          | 64.7545 | 6.5216 | 64.3853 | 64.5344   | 76.1124  | 76.1124 | 76.1124 | 76.1124   | 0.6628 | 0.0005 | 0.0001 | 0.0    | 0.619   | 6.4473  |
+| 1.2871        | 1.35  | 4500 | 1.1656          | 65.6713 | 6.7135 | 65.185  | 65.4454   | 77.0023  | 77.0023 | 77.0023 | 77.0023   | 0.6718 | 0.0005 | 0.0001 | 0.0    | 0.6246  | 6.472   |
+| 1.3423        | 1.5   | 5000 | 1.1669          | 65.8966 | 6.7928 | 65.5016 | 65.6741   | 77.377   | 77.377  | 77.377  | 77.377    | 0.6772 | 0.0005 | 0.0001 | 0.0    | 0.6288  | 6.36    |
+| 1.333         | 1.65  | 5500 | 1.1627          | 65.9726 | 6.7915 | 65.5878 | 65.7582   | 77.4239  | 77.4239 | 77.4239 | 77.4239   | 0.6742 | 0.0005 | 0.0001 | 0.0    | 0.6273  | 6.4767  |
+| 1.2749        | 1.8   | 6000 | 1.1591          | 66.5212 | 6.9115 | 66.0695 | 66.3204   | 77.9859  | 77.9859 | 77.9859 | 77.9859   | 0.681  | 0.0006 | 0.0001 | 0.0    | 0.6324  | 6.4403  |
+| 1.2891        | 1.95  | 6500 | 1.1571          | 66.2478 | 6.8368 | 65.8198 | 66.0423   | 77.5644  | 77.5644 | 77.5644 | 77.5644   | 0.6778 | 0.0005 | 0.0001 | 0.0    | 0.6298  | 6.4417  |
+
+
+### Framework versions
+
+- Transformers 4.20.1
+- Pytorch 1.11.0
+- Datasets 2.3.2
+- Tokenizers 0.12.1

config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "bigscience/T0_3B",
+  "architectures": [
+    "T5ForConditionalGeneration"
+  ],
+  "d_ff": 5120,
+  "d_kv": 64,
+  "d_model": 2048,
+  "decoder_start_token_id": 0,
+  "dense_act_fn": "gelu_new",
+  "dropout_rate": 0.1,
+  "eos_token_id": 1,
+  "feed_forward_proj": "gated-gelu",
+  "gradient_checkpointing": false,
+  "initializer_factor": 1.0,
+  "is_encoder_decoder": true,
+  "is_gated_act": true,
+  "layer_norm_epsilon": 1e-06,
+  "model_type": "t5",
+  "num_decoder_layers": 24,
+  "num_heads": 32,
+  "num_layers": 24,
+  "output_past": true,
+  "pad_token_id": 0,
+  "relative_attention_max_distance": 128,
+  "relative_attention_num_buckets": 32,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.20.1",
+  "use_cache": false,
+  "vocab_size": 32100
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f3d6a368ba055474fb998a9ca9e733708e35b66898d5bd366ed27057b7da58a1
+size 5699366811

special_tokens_map.json

@@ -0,0 +1,107 @@
+{
+  "additional_special_tokens": [
+    "<extra_id_0>",
+    "<extra_id_1>",
+    "<extra_id_2>",
+    "<extra_id_3>",
+    "<extra_id_4>",
+    "<extra_id_5>",
+    "<extra_id_6>",
+    "<extra_id_7>",
+    "<extra_id_8>",
+    "<extra_id_9>",
+    "<extra_id_10>",
+    "<extra_id_11>",
+    "<extra_id_12>",
+    "<extra_id_13>",
+    "<extra_id_14>",
+    "<extra_id_15>",
+    "<extra_id_16>",
+    "<extra_id_17>",
+    "<extra_id_18>",
+    "<extra_id_19>",
+    "<extra_id_20>",
+    "<extra_id_21>",
+    "<extra_id_22>",
+    "<extra_id_23>",
+    "<extra_id_24>",
+    "<extra_id_25>",
+    "<extra_id_26>",
+    "<extra_id_27>",
+    "<extra_id_28>",
+    "<extra_id_29>",
+    "<extra_id_30>",
+    "<extra_id_31>",
+    "<extra_id_32>",
+    "<extra_id_33>",
+    "<extra_id_34>",
+    "<extra_id_35>",
+    "<extra_id_36>",
+    "<extra_id_37>",
+    "<extra_id_38>",
+    "<extra_id_39>",
+    "<extra_id_40>",
+    "<extra_id_41>",
+    "<extra_id_42>",
+    "<extra_id_43>",
+    "<extra_id_44>",
+    "<extra_id_45>",
+    "<extra_id_46>",
+    "<extra_id_47>",
+    "<extra_id_48>",
+    "<extra_id_49>",
+    "<extra_id_50>",
+    "<extra_id_51>",
+    "<extra_id_52>",
+    "<extra_id_53>",
+    "<extra_id_54>",
+    "<extra_id_55>",
+    "<extra_id_56>",
+    "<extra_id_57>",
+    "<extra_id_58>",
+    "<extra_id_59>",
+    "<extra_id_60>",
+    "<extra_id_61>",
+    "<extra_id_62>",
+    "<extra_id_63>",
+    "<extra_id_64>",
+    "<extra_id_65>",
+    "<extra_id_66>",
+    "<extra_id_67>",
+    "<extra_id_68>",
+    "<extra_id_69>",
+    "<extra_id_70>",
+    "<extra_id_71>",
+    "<extra_id_72>",
+    "<extra_id_73>",
+    "<extra_id_74>",
+    "<extra_id_75>",
+    "<extra_id_76>",
+    "<extra_id_77>",
+    "<extra_id_78>",
+    "<extra_id_79>",
+    "<extra_id_80>",
+    "<extra_id_81>",
+    "<extra_id_82>",
+    "<extra_id_83>",
+    "<extra_id_84>",
+    "<extra_id_85>",
+    "<extra_id_86>",
+    "<extra_id_87>",
+    "<extra_id_88>",
+    "<extra_id_89>",
+    "<extra_id_90>",
+    "<extra_id_91>",
+    "<extra_id_92>",
+    "<extra_id_93>",
+    "<extra_id_94>",
+    "<extra_id_95>",
+    "<extra_id_96>",
+    "<extra_id_97>",
+    "<extra_id_98>",
+    "<extra_id_99>"
+  ],
+  "eos_token": "</s>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d60acb128cf7b7f2536e8f38a5b18a05535c9e14c7a355904270e15b0945ea86
+size 791656

tokenizer_config.json

@@ -0,0 +1,113 @@
+{
+  "additional_special_tokens": [
+    "<extra_id_0>",
+    "<extra_id_1>",
+    "<extra_id_2>",
+    "<extra_id_3>",
+    "<extra_id_4>",
+    "<extra_id_5>",
+    "<extra_id_6>",
+    "<extra_id_7>",
+    "<extra_id_8>",
+    "<extra_id_9>",
+    "<extra_id_10>",
+    "<extra_id_11>",
+    "<extra_id_12>",
+    "<extra_id_13>",
+    "<extra_id_14>",
+    "<extra_id_15>",
+    "<extra_id_16>",
+    "<extra_id_17>",
+    "<extra_id_18>",
+    "<extra_id_19>",
+    "<extra_id_20>",
+    "<extra_id_21>",
+    "<extra_id_22>",
+    "<extra_id_23>",
+    "<extra_id_24>",
+    "<extra_id_25>",
+    "<extra_id_26>",
+    "<extra_id_27>",
+    "<extra_id_28>",
+    "<extra_id_29>",
+    "<extra_id_30>",
+    "<extra_id_31>",
+    "<extra_id_32>",
+    "<extra_id_33>",
+    "<extra_id_34>",
+    "<extra_id_35>",
+    "<extra_id_36>",
+    "<extra_id_37>",
+    "<extra_id_38>",
+    "<extra_id_39>",
+    "<extra_id_40>",
+    "<extra_id_41>",
+    "<extra_id_42>",
+    "<extra_id_43>",
+    "<extra_id_44>",
+    "<extra_id_45>",
+    "<extra_id_46>",
+    "<extra_id_47>",
+    "<extra_id_48>",
+    "<extra_id_49>",
+    "<extra_id_50>",
+    "<extra_id_51>",
+    "<extra_id_52>",
+    "<extra_id_53>",
+    "<extra_id_54>",
+    "<extra_id_55>",
+    "<extra_id_56>",
+    "<extra_id_57>",
+    "<extra_id_58>",
+    "<extra_id_59>",
+    "<extra_id_60>",
+    "<extra_id_61>",
+    "<extra_id_62>",
+    "<extra_id_63>",
+    "<extra_id_64>",
+    "<extra_id_65>",
+    "<extra_id_66>",
+    "<extra_id_67>",
+    "<extra_id_68>",
+    "<extra_id_69>",
+    "<extra_id_70>",
+    "<extra_id_71>",
+    "<extra_id_72>",
+    "<extra_id_73>",
+    "<extra_id_74>",
+    "<extra_id_75>",
+    "<extra_id_76>",
+    "<extra_id_77>",
+    "<extra_id_78>",
+    "<extra_id_79>",
+    "<extra_id_80>",
+    "<extra_id_81>",
+    "<extra_id_82>",
+    "<extra_id_83>",
+    "<extra_id_84>",
+    "<extra_id_85>",
+    "<extra_id_86>",
+    "<extra_id_87>",
+    "<extra_id_88>",
+    "<extra_id_89>",
+    "<extra_id_90>",
+    "<extra_id_91>",
+    "<extra_id_92>",
+    "<extra_id_93>",
+    "<extra_id_94>",
+    "<extra_id_95>",
+    "<extra_id_96>",
+    "<extra_id_97>",
+    "<extra_id_98>",
+    "<extra_id_99>"
+  ],
+  "eos_token": "</s>",
+  "extra_ids": 100,
+  "model_max_length": 512,
+  "name_or_path": "bigscience/T0_3B",
+  "pad_token": "<pad>",
+  "sp_model_kwargs": {},
+  "special_tokens_map_file": "/home/ubuntu/.cache/huggingface/transformers/303fbee39a17e96552ac07e02b70ba62ff0ad760609687e3a1b92b4ad2dff58c.c94798918c92ded6aeef2d2f0e666d2cc4145eca1aa6e1336fde07f2e13e2f46",
+  "tokenizer_class": "T5Tokenizer",
+  "unk_token": "<unk>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1ff9c3437c851445a9bca1019601e7914c9d0f696accd1a92bcacd0979660af
+size 4527