Update README.md

README.md

@@ -24,11 +24,6 @@ It achieves the following results on the evaluation set:
 
 ## Model description
 
-More information needed
-
-## Intended uses & limitations
-
-More information needed
 
 ## Training and evaluation data
 
@@ -36,6 +31,9 @@ More information needed
 
 ## Training procedure
 
+Fine Tuning: https://github.com/frank-morales2020/MLxDL/blob/main/FineTuning_LLM_Meta_Llama_3_8B_for_MEDAL_EVALDATA_PONEW.ipynb
+
+
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
@@ -51,6 +49,42 @@ The following hyperparameters were used during training:
 - lr_scheduler_warmup_steps: 1500
 - num_epochs: 0.5
 
+from transformers import TrainingArguments
+
+args = TrainingArguments(
+
+    output_dir="/content/gdrive/MyDrive/model/POC-NEW-Meta-Llama-3-8B-MEDAL-flash-attention-2-cosine-evaldata",
+
+    num_train_epochs=0.5,                   # number of training epochs for POC
+    per_device_train_batch_size=3,    #4    # batch size per device during training
+    gradient_accumulation_steps=8,    #6    # values like 8, 12, or even 16,    # number of steps before performing a backward/update pass
+    gradient_checkpointing=True,            # use gradient checkpointing to save memory
+     optim="adamw_torch_fused",             # use fused adamw optimizer
+    logging_steps=100,                      # log every 100 steps
+    learning_rate=2e-4,                     # learning rate, based on QLoRA paper # i used in the first model
+    #learning_rate=1e-5,
+    bf16=True,                              # use bfloat16 precision
+    tf32=True,                              # use tf32 precision
+    max_grad_norm=1.0,                      # max gradient norm based on QLoRA paper
+    warmup_ratio=0.03,                      # warmup ratio based on QLoRA paper = 0.03
+
+    weight_decay=0.01,
+    lr_scheduler_type="constant",           # use constant learning rate scheduler
+    push_to_hub=True,                       # push model to hub
+    report_to="tensorboard",                # report metrics to tensorboard
+    gradient_checkpointing_kwargs={"use_reentrant": True},
+
+    load_best_model_at_end=True,
+    logging_dir="/content/gdrive/MyDrive/model/POC-NEW-Meta-Llama-3-8B-MEDAL-flash-attention-2-cosine-evaldata/logs",
+
+    evaluation_strategy="steps",
+    eval_steps=100,
+    save_strategy="steps",
+    save_steps=100,
+    metric_for_best_model = "loss",
+    warmup_steps=1500,
+ )
+
 ### Training results
 
 | Training Loss | Epoch  | Step | Validation Loss |