--- license: llama3 base_model: meta-llama/Meta-Llama-3-70B-Instruct tags: - generated_from_trainer model-index: - name: outputs/basemodel-llama3-70b.8e6 results: [] --- [Built with Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl)
See axolotl config axolotl version: `0.4.0` ```yaml base_model: meta-llama/Meta-Llama-3-70B-Instruct model_type: LlamaForCausalLM tokenizer_type: AutoTokenizer load_in_8bit: false load_in_4bit: false strict: false # doesn't work... # hub_model_id: shisa-ai/shisa-llama3-70b-v1 # hub_strategy: end use_wandb: true wandb_project: shisa-v2 wandb_entity: augmxnt wandb_name: shisa-llama3-70b-v1.8e6 chat_template: llama3 datasets: - path: augmxnt/ultra-orca-boros-en-ja-v1 type: sharegpt dataset_prepared_path: last_run_prepared val_set_size: 0.05 output_dir: ./outputs/basemodel-llama3-70b.8e6 sequence_len: 4096 sample_packing: true pad_to_sequence_len: true neftune_noise_alpha: 5 gradient_accumulation_steps: 2 micro_batch_size: 2 num_epochs: 3 optimizer: paged_adamw_8bit lr_scheduler: linear learning_rate: 2e-5 train_on_inputs: false group_by_length: false bf16: auto fp16: tf32: true gradient_checkpointing: true gradient_checkpointing_kwargs: use_reentrant: false early_stopping_patience: resume_from_checkpoint: logging_steps: 1 xformers_attention: flash_attention: true warmup_ratio: 0.1 evals_per_epoch: 2 eval_table_size: saves_per_epoch: 0 debug: deepspeed: axolotl/deepspeed_configs/zero3_bf16.json weight_decay: 0.05 fsdp: fsdp_config: special_tokens: pad_token: <|end_of_text|> ```

# outputs/basemodel-llama3-70b.8e6 This model is a fine-tuned version of [meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) on the None dataset. It achieves the following results on the evaluation set: - Loss: 0.4440 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 2 - eval_batch_size: 2 - seed: 42 - distributed_type: multi-GPU - num_devices: 16 - gradient_accumulation_steps: 2 - total_train_batch_size: 64 - total_eval_batch_size: 32 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_steps: 87 - num_epochs: 3 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 1.248 | 0.0033 | 1 | 0.7102 | | 0.7497 | 0.5008 | 154 | 0.4374 | | 0.7229 | 1.0016 | 308 | 0.3940 | | 0.3772 | 1.4862 | 462 | 0.3962 | | 0.3791 | 1.9870 | 616 | 0.3838 | | 0.0943 | 2.4699 | 770 | 0.4440 | ### Framework versions - Transformers 4.40.2 - Pytorch 2.3.0+cu121 - Datasets 2.19.1 - Tokenizers 0.19.1