metadata
license: other
library_name: peft
tags:
- axolotl
- generated_from_trainer
base_model: deepseek-ai/deepseek-coder-1.3b-instruct
model-index:
- name: deepseek-code-1.3b-inst-NLQ2Cypher
results: []
See axolotl config
axolotl version: 0.4.1
base_model: deepseek-ai/deepseek-coder-1.3b-instruct
# base_model: Qwen/CodeQwen1.5-7B-Chat
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
is_mistral_derived_model: false
load_in_8bit: false
load_in_4bit: true
strict: false
lora_fan_in_fan_out: false
data_seed: 49
seed: 49
datasets:
- path: sample_data/alpaca_synth_cypher.jsonl
type: sharegpt
conversation: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.1
output_dir: ./qlora-alpaca-deepseek-1.3b-inst
# output_dir: ./qlora-alpaca-out
# hub_model_id: jermyn/CodeQwen1.5-7B-Chat-NLQ2Cypher
hub_model_id: jermyn/deepseek-code-1.3b-inst-NLQ2Cypher
adapter: qlora
lora_model_dir:
sequence_len: 896
sample_packing: false
pad_to_sequence_len: true
lora_r: 16
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_target_modules:
- gate_proj
- down_proj
- up_proj
- q_proj
- v_proj
- k_proj
- o_proj
# If you added new tokens to the tokenizer, you may need to save some LoRA modules because they need to know the new tokens.
# For LLaMA and Mistral, you need to save `embed_tokens` and `lm_head`. It may vary for other models.
# `embed_tokens` converts tokens to embeddings, and `lm_head` converts embeddings to token probabilities.
# https://github.com/huggingface/peft/issues/334#issuecomment-1561727994
# lora_modules_to_save:
# - embed_tokens
# - lm_head
wandb_project: fine-tune-axolotl
wandb_entity: jermyn
gradient_accumulation_steps: 1
micro_batch_size: 16
eval_batch_size: 16
num_epochs: 6
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0005
max_grad_norm: 1.0
adam_beta2: 0.95
adam_epsilon: 0.00001
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3
warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_table_max_new_tokens: 128
# saves_per_epoch: 6
save_steps: 10
save_total_limit: 3
debug:
weight_decay: 0.0
fsdp:
fsdp_config:
# special_tokens:
# bos_token: "<s>"
# eos_token: "</s>"
# unk_token: "<unk>"
save_safetensors: true
deepseek-code-1.3b-inst-NLQ2Cypher
This model is a fine-tuned version of deepseek-ai/deepseek-coder-1.3b-instruct on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.3839
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 16
- eval_batch_size: 16
- seed: 49
- optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-05
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 6
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.8723 | 0.1429 | 1 | 1.6354 |
| 1.9222 | 0.2857 | 2 | 1.6215 |
| 1.6971 | 0.5714 | 4 | 1.4205 |
| 1.2458 | 0.8571 | 6 | 0.9204 |
| 0.6179 | 1.1429 | 8 | 0.6923 |
| 0.366 | 1.4286 | 10 | 0.5647 |
| 0.2752 | 1.7143 | 12 | 0.5225 |
| 0.2931 | 2.0 | 14 | 0.5167 |
| 0.1812 | 2.2857 | 16 | 0.4564 |
| 0.1258 | 2.5714 | 18 | 0.4038 |
| 0.0885 | 2.8571 | 20 | 0.3689 |
| 0.0886 | 3.1429 | 22 | 0.3647 |
| 0.1281 | 3.4286 | 24 | 0.3503 |
| 0.0606 | 3.7143 | 26 | 0.3458 |
| 0.0603 | 4.0 | 28 | 0.3635 |
| 0.0479 | 4.2857 | 30 | 0.3724 |
| 0.0963 | 4.5714 | 32 | 0.3827 |
| 0.0725 | 4.8571 | 34 | 0.3868 |
| 0.049 | 5.1429 | 36 | 0.3873 |
| 0.0572 | 5.4286 | 38 | 0.3860 |
| 0.061 | 5.7143 | 40 | 0.3890 |
| 0.0702 | 6.0 | 42 | 0.3839 |
Framework versions
- PEFT 0.11.1
- Transformers 4.41.1
- Pytorch 2.1.2+cu118
- Datasets 2.19.1
- Tokenizers 0.19.1