--- tags: - LoRA - QLoRa - Merged LoRA Model model-index: - name: sql-guanaco-13b-merged results: [] datasets: - richardr1126/sql-create-context_guanaco_style spaces: - richardr1126/sql-guanaco-13b-demo --- # sql-guanaco-13b-merged - This is a merged LoRA model that can be used with AutoModelForCausalLM or LlamaModelForCausalLM. - It is a combination of [richardr1126/guanaco-13b-merged](https://huggingface.co/richardr1126/guanaco-13b-merged) + [richardr1126/lora-sql-guanaco-13b-adapter](https://huggingface.co/richardr1126/lora-sql-guanaco-13b-adapter). - This LoRA was fine-tuned using QLoRA techniques on the [richardr1126/sql-create-context_guanaco_style](https://huggingface.co/datasets/richardr1126/sql-create-context_guanaco_style) dataset. ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0002 - train_batch_size: 4 - eval_batch_size: 8 - seed: 42 - gradient_accumulation_steps: 4 - total_train_batch_size: 16 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_ratio: 0.03 - training_steps: 1875 - mixed_precision_training: Native AMP ### Framework versions - Transformers 4.30.0.dev0 - Pytorch 2.0.1+cu118 - Datasets 2.13.0 - Tokenizers 0.13.3 ## Citation ```bibtex @article{dettmers2023qlora, title={QLoRA: Efficient Finetuning of Quantized LLMs}, author={Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke}, journal={arXiv preprint arXiv:2305.14314}, year={2023} } ```