See axolotl config
axolotl version: 0.4.1
base_model: deepseek-ai/deepseek-coder-1.3b-instruct
# base_model: Qwen/CodeQwen1.5-7B-Chat
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
is_mistral_derived_model: false
load_in_8bit: false
load_in_4bit: true
strict: false
lora_fan_in_fan_out: false
data_seed: 49
seed: 49
datasets:
- path: sample_data/alpaca_synth_cypher.jsonl
type: sharegpt
conversation: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.1
output_dir: ./qlora-alpaca-deepseek-1.3b-inst
# output_dir: ./qlora-alpaca-out
# hub_model_id: jermyn/CodeQwen1.5-7B-Chat-NLQ2Cypher
hub_model_id: jermyn/deepseek-code-1.3b-inst-NLQ2Cypher
adapter: qlora
lora_model_dir:
sequence_len: 896
sample_packing: false
pad_to_sequence_len: true
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_target_modules:
- gate_proj
- down_proj
- up_proj
- q_proj
- v_proj
- k_proj
- o_proj
# If you added new tokens to the tokenizer, you may need to save some LoRA modules because they need to know the new tokens.
# For LLaMA and Mistral, you need to save `embed_tokens` and `lm_head`. It may vary for other models.
# `embed_tokens` converts tokens to embeddings, and `lm_head` converts embeddings to token probabilities.
# https://github.com/huggingface/peft/issues/334#issuecomment-1561727994
# lora_modules_to_save:
# - embed_tokens
# - lm_head
wandb_project: fine-tune-axolotl
wandb_entity: jermyn
gradient_accumulation_steps: 1
micro_batch_size: 16
eval_batch_size: 16
num_epochs: 6
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.001
max_grad_norm: 1.0
adam_beta2: 0.95
adam_epsilon: 0.00001
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3
warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_table_max_new_tokens: 128
# saves_per_epoch: 6
save_steps: 10
save_total_limit: 3
debug:
weight_decay: 0.0
fsdp:
fsdp_config:
# special_tokens:
# bos_token: "<s>"
# eos_token: "</s>"
# unk_token: "<unk>"
save_safetensors: true
deepseek-code-1.3b-inst-NLQ2Cypher
This model is a fine-tuned version of deepseek-ai/deepseek-coder-1.3b-instruct on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.3816
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.001
- train_batch_size: 16
- eval_batch_size: 16
- seed: 49
- optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-05
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 6
Training results
Training Loss | Epoch | Step | Validation Loss |
---|---|---|---|
1.8723 | 0.1429 | 1 | 1.6354 |
1.9222 | 0.2857 | 2 | 1.5979 |
1.5222 | 0.5714 | 4 | 1.0739 |
0.7251 | 0.8571 | 6 | 0.7059 |
0.3581 | 1.1429 | 8 | 0.6420 |
0.2804 | 1.4286 | 10 | 0.6576 |
0.2337 | 1.7143 | 12 | 0.4628 |
0.1988 | 2.0 | 14 | 0.3674 |
0.1218 | 2.2857 | 16 | 0.3528 |
0.1087 | 2.5714 | 18 | 0.3584 |
0.0842 | 2.8571 | 20 | 0.3569 |
0.0733 | 3.1429 | 22 | 0.3610 |
0.0943 | 3.4286 | 24 | 0.3477 |
0.058 | 3.7143 | 26 | 0.3575 |
0.0753 | 4.0 | 28 | 0.3797 |
0.0435 | 4.2857 | 30 | 0.3915 |
0.0758 | 4.5714 | 32 | 0.3902 |
0.0608 | 4.8571 | 34 | 0.3875 |
0.0368 | 5.1429 | 36 | 0.3853 |
0.0444 | 5.4286 | 38 | 0.3818 |
0.055 | 5.7143 | 40 | 0.3820 |
0.0532 | 6.0 | 42 | 0.3816 |
Framework versions
- PEFT 0.11.1
- Transformers 4.41.1
- Pytorch 2.1.2+cu118
- Datasets 2.19.1
- Tokenizers 0.19.1
- Downloads last month
- 2
Model tree for jermyn/deepseek-code-1.3b-inst-NLQ2Cypher
Base model
deepseek-ai/deepseek-coder-1.3b-instruct
Adapter
this model