--- library_name: peft license: cc-by-nc-4.0 datasets: - theblackcat102/evol-codealpaca-v1 language: - en pipeline_tag: text2text-generation --- ## llama-2-13b-code-alpaca [Built with Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) Trained for 3 epochs on `theblackcat102/evol-codealpaca-v1` dataset, scored decent on locally run perplexity at 4.36. ## Axolotl config used ```yaml base_model: NousResearch/Llama-2-13b-hf base_model_config: NousResearch/Llama-2-13b-hf model_type: LlamaForCausalLM tokenizer_type: LlamaTokenizer push_dataset_to_hub: hub_model_id: load_in_8bit: false load_in_4bit: true strict: false datasets: - path: theblackcat102/evol-codealpaca-v1 type: alpaca dataset_prepared_path: last_run_prepared val_set_size: 0.01 output_dir: ./checkpoints/llama-2-13b-qlora adapter: qlora lora_model_dir: sequence_len: 4096 max_packed_sequence_len: 4096 lora_r: 32 lora_alpha: 16 lora_dropout: 0.05 lora_target_modules: lora_target_linear: true lora_fan_in_fan_out: wandb_project: wandb_watch: wandb_run_id: wandb_log_model: gradient_accumulation_steps: 2 micro_batch_size: 2 num_epochs: 3 optimizer: paged_adamw_32bit lr_scheduler: cosine learning_rate: 0.0001 train_on_inputs: false group_by_length: true bf16: true fp16: false tf32: true gradient_checkpointing: true early_stopping_patience: resume_from_checkpoint: local_rank: logging_steps: 1 xformers_attention: true flash_attention: warmup_steps: 10 eval_steps: 50 save_steps: debug: deepspeed: weight_decay: 0.0 fsdp: fsdp_config: special_tokens: bos_token: "" eos_token: "" unk_token: "" ``` And then merged with Axolotl via: ``` accelerate launch scripts/finetune.py configs/your_config.yml --merge_lora --lora_model_dir="./completed-model" --load_in_8bit=False --load_in_4bit=False ``` ## Training procedure The following `bitsandbytes` quantization config was used during training: - load_in_8bit: False - load_in_4bit: True - llm_int8_threshold: 6.0 - llm_int8_skip_modules: None - llm_int8_enable_fp32_cpu_offload: False - llm_int8_has_fp16_weight: False - bnb_4bit_quant_type: nf4 - bnb_4bit_use_double_quant: True - bnb_4bit_compute_dtype: bfloat16 The following `bitsandbytes` quantization config was used during training: - load_in_8bit: False - load_in_4bit: True - llm_int8_threshold: 6.0 - llm_int8_skip_modules: None - llm_int8_enable_fp32_cpu_offload: False - llm_int8_has_fp16_weight: False - bnb_4bit_quant_type: nf4 - bnb_4bit_use_double_quant: True - bnb_4bit_compute_dtype: bfloat16 The following `bitsandbytes` quantization config was used during training: - load_in_8bit: False - load_in_4bit: True - llm_int8_threshold: 6.0 - llm_int8_skip_modules: None - llm_int8_enable_fp32_cpu_offload: False - llm_int8_has_fp16_weight: False - bnb_4bit_quant_type: nf4 - bnb_4bit_use_double_quant: True - bnb_4bit_compute_dtype: bfloat16 The following `bitsandbytes` quantization config was used during training: - load_in_8bit: False - load_in_4bit: True - llm_int8_threshold: 6.0 - llm_int8_skip_modules: None - llm_int8_enable_fp32_cpu_offload: False - llm_int8_has_fp16_weight: False - bnb_4bit_quant_type: nf4 - bnb_4bit_use_double_quant: True - bnb_4bit_compute_dtype: bfloat16 ### Framework versions - PEFT 0.5.0.dev0