---
license: llama2
library_name: peft
tags:
- axolotl
- generated_from_trainer
base_model: codellama/CodeLlama-7b-hf
model-index:
- name: EvilCodeLlama-7b
results: []
---
[
](https://github.com/OpenAccess-AI-Collective/axolotl)
See axolotl config
axolotl version: `0.3.0`
```yaml
base_model: codellama/CodeLlama-7b-hf
base_model_config: codellama/CodeLlama-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
is_llama_derived_model: true
hub_model_id: EvilCodeLlama-7b
load_in_8bit: false
load_in_4bit: true
strict: false
datasets:
- path: dhuynh95/Magicoder-Evol-Instruct-110K-Filtered_0.35
type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.02
output_dir: ./qlora-out-evil-codellama
adapter: qlora
lora_model_dir:
eval_sample_packing: false
sequence_len: 2048
sample_packing: true
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:
wandb_project: axolotl
wandb_entity:
wandb_watch:
wandb_run_id:
wandb_log_model:
gradient_accumulation_steps: 4
micro_batch_size: 16
num_epochs: 10
optimizer: paged_adamw_32bit
lr_scheduler: cosine
learning_rate: 0.0002
train_on_inputs: true
group_by_length: false
bf16: true
fp16: false
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 100
eval_steps: 0.01
save_strategy: epoch
save_steps:
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
bos_token: ""
eos_token: ""
unk_token: ""
```
# EvilCodeLlama-7b
This model is a fine-tuned version of [codellama/CodeLlama-7b-hf](https://huggingface.co/codellama/CodeLlama-7b-hf) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.7929
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- num_epochs: 10
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.2543 | 0.04 | 1 | 1.2447 |
| 1.2677 | 0.12 | 3 | 1.2446 |
| 1.2572 | 0.24 | 6 | 1.2443 |
| 1.2602 | 0.37 | 9 | 1.2432 |
| 1.2573 | 0.49 | 12 | 1.2403 |
| 1.2811 | 0.61 | 15 | 1.2342 |
| 1.2584 | 0.73 | 18 | 1.2217 |
| 1.2152 | 0.86 | 21 | 1.2005 |
| 1.1592 | 0.98 | 24 | 1.1695 |
| 1.1512 | 1.07 | 27 | 1.1345 |
| 1.1191 | 1.19 | 30 | 1.0970 |
| 1.1111 | 1.32 | 33 | 1.0543 |
| 1.0362 | 1.44 | 36 | 1.0160 |
| 1.0386 | 1.56 | 39 | 0.9879 |
| 1.0637 | 1.68 | 42 | 0.9549 |
| 1.0109 | 1.81 | 45 | 0.9377 |
| 0.9416 | 1.93 | 48 | 0.9258 |
| 0.8851 | 2.03 | 51 | 0.9164 |
| 0.9027 | 2.15 | 54 | 0.9085 |
| 0.8959 | 2.28 | 57 | 0.9018 |
| 0.9168 | 2.4 | 60 | 0.8956 |
| 0.9386 | 2.52 | 63 | 0.8896 |
| 0.9762 | 2.64 | 66 | 0.8832 |
| 0.9118 | 2.77 | 69 | 0.8768 |
| 0.9055 | 2.89 | 72 | 0.8714 |
| 0.8617 | 3.01 | 75 | 0.8660 |
| 0.9085 | 3.11 | 78 | 0.8604 |
| 0.8531 | 3.23 | 81 | 0.8547 |
| 0.8725 | 3.36 | 84 | 0.8486 |
| 0.8845 | 3.48 | 87 | 0.8424 |
| 0.8812 | 3.6 | 90 | 0.8381 |
| 0.865 | 3.72 | 93 | 0.8351 |
| 0.8312 | 3.85 | 96 | 0.8311 |
| 0.8766 | 3.97 | 99 | 0.8280 |
| 0.842 | 4.07 | 102 | 0.8249 |
| 0.8377 | 4.19 | 105 | 0.8222 |
| 0.8661 | 4.32 | 108 | 0.8195 |
| 0.8505 | 4.44 | 111 | 0.8171 |
| 0.8509 | 4.56 | 114 | 0.8140 |
| 0.8823 | 4.68 | 117 | 0.8111 |
| 0.8246 | 4.81 | 120 | 0.8091 |
| 0.8116 | 4.93 | 123 | 0.8073 |
| 0.7993 | 5.03 | 126 | 0.8054 |
| 0.8277 | 5.15 | 129 | 0.8048 |
| 0.8533 | 5.28 | 132 | 0.8030 |
| 0.7887 | 5.4 | 135 | 0.8015 |
| 0.8189 | 5.52 | 138 | 0.8005 |
| 0.8148 | 5.64 | 141 | 0.7993 |
| 0.8376 | 5.77 | 144 | 0.7977 |
| 0.8142 | 5.89 | 147 | 0.7968 |
| 0.8074 | 6.01 | 150 | 0.7961 |
| 0.8122 | 6.11 | 153 | 0.7970 |
| 0.7753 | 6.23 | 156 | 0.7963 |
| 0.8477 | 6.36 | 159 | 0.7958 |
| 0.7977 | 6.48 | 162 | 0.7947 |
| 0.7653 | 6.6 | 165 | 0.7944 |
| 0.8358 | 6.72 | 168 | 0.7930 |
| 0.7445 | 6.85 | 171 | 0.7926 |
| 0.808 | 6.97 | 174 | 0.7922 |
| 0.7799 | 7.07 | 177 | 0.7916 |
| 0.7593 | 7.19 | 180 | 0.7933 |
| 0.8275 | 7.32 | 183 | 0.7930 |
| 0.7599 | 7.44 | 186 | 0.7925 |
| 0.7734 | 7.56 | 189 | 0.7928 |
| 0.7886 | 7.68 | 192 | 0.7927 |
| 0.8066 | 7.81 | 195 | 0.7919 |
| 0.7778 | 7.93 | 198 | 0.7916 |
| 0.7839 | 8.03 | 201 | 0.7918 |
| 0.7942 | 8.15 | 204 | 0.7927 |
| 0.7457 | 8.28 | 207 | 0.7930 |
| 0.7525 | 8.4 | 210 | 0.7928 |
| 0.7768 | 8.52 | 213 | 0.7926 |
| 0.7469 | 8.64 | 216 | 0.7928 |
| 0.7777 | 8.77 | 219 | 0.7929 |
| 0.7694 | 8.89 | 222 | 0.7928 |
| 0.7639 | 9.01 | 225 | 0.7927 |
| 0.7556 | 9.11 | 228 | 0.7927 |
| 0.7098 | 9.23 | 231 | 0.7927 |
| 0.7537 | 9.36 | 234 | 0.7928 |
| 0.7721 | 9.48 | 237 | 0.7926 |
| 0.7642 | 9.6 | 240 | 0.7929 |
### Framework versions
- PEFT 0.7.2.dev0
- Transformers 4.37.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.16.1
- Tokenizers 0.15.0