--- license: other base_model: KnutJaegersberg/Qwen-1_8B-Llamafied tags: - generated_from_trainer model-index: - name: qwen-1.8b-vi results: [] --- [Built with Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) # Quan-1.8b Qwen-1.8B finetuned on Vietnamese Data. ## Prompt Template ChatML, same as VinaLlama ``` <|im_start|>system Bạn là một trợ lí AI hữu ích. Hãy trả lời người dùng một cách chính xác. <|im_end|> <|im_start|>user Hello world!<|im_end|> <|im_start|>assistant ``` This model is a fine-tuned version of [KnutJaegersberg/Qwen-1_8B-Llamafied](https://huggingface.co/KnutJaegersberg/Qwen-1_8B-Llamafied) on the None dataset. It achieves the following results on the evaluation set: - Loss: 0.8096 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 2 - eval_batch_size: 2 - seed: 42 - distributed_type: multi-GPU - num_devices: 4 - gradient_accumulation_steps: 4 - total_train_batch_size: 32 - total_eval_batch_size: 8 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 100 - num_epochs: 4 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-----:|:----:|:---------------:| | 0.8123 | 1.02 | 2356 | 0.8183 | | 0.7358 | 2.02 | 4713 | 0.7790 | | 0.6379 | 3.02 | 7071 | 0.7822 | | 0.5762 | 3.94 | 9252 | 0.8096 | ### Framework versions - Transformers 4.34.1 - Pytorch 2.0.1+cu118 - Datasets 2.14.6 - Tokenizers 0.14.1