metadata
datasets:
- mlabonne/Evol-Instruct-Python-26k
language:
- en
library_name: adapter-transformers
tags:
- code
Model Details
Model Description
- Developed by: Muhammad Hafiz
- Model type: large language model for code generation
- Language(s) (NLP): English
- Finetuned from model: TinyLlama
Model Sources
- Repository: https://github.com/unslothai/unsloth
- Developed by: unsloth
Model parameter
- r = 16,
- target_modules = ["q_proj", "k_proj", "v_proj", "o_proj","gate_proj", "up_proj", "down_proj",],
- lora_alpha = 16,
- lora_dropout = 0,
- bias = "none",
- use_gradient_checkpointing = "unsloth",
- random_state = 3407,
- use_rslora = False,
- loftq_config = None,
Usage and limitations
This model is used to generate code based on commands given by the user. it should be noted that this model can generate many languages because it takes the initial model from llama2. However, after finetuning it is better at generating python code, because currently it is only trained with python code datasets.
How to Get Started with the Model
use link below to use model //
Training Data
https://huggingface.co/datasets/mlabonne/Evol-Instruct-Python-26k
Training Hyperparameters
- Warmup_step: 5
- lr_scheduler_type: linear
- Learning Rate: 0.0002
- Batch Size: 2
- Weigh_decay: 0.001
- Epoch: 30
- Optimizer: adamw_8bit
Testing Data
https://huggingface.co/datasets/google-research-datasets/mbpp/viewer/full
Testing Document
https://docs.google.com/spreadsheets/d/1hr8R4nixQsDC5cGGENTOLUW1jPCS_lltVRIeOzenBvA/edit?usp=sharing
Results
Berfore finetune
- Accurary : 32%
- Consistensy : 0%
After fine tune
- Accuracy : 53%
- Consistency : 100%