File size: 3,601 Bytes
2df0b3b 19ba86c 2df0b3b a8c703d 2df0b3b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 |
---
datasets:
- ehartford/dolphin
- LinhDuong/chatdoctor-200k
- sahil2801/code_instructions_120k
- c-s-ale/dolly-15k-instruction-alpaca-format
- tiiuae/falcon-refinedweb
- bigcode/starcoderdata
- togethercomputer/RedPajama-Data-1T
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- instruct
- medical
- code
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
This model is an instruction-tuned LLaMa model with 33B parameters, with specialities in medical QA and code instruction.
## Model Details
<!-- Provide a longer summary of what this model is. -->
- **Model type:** LlamaForCausalLM
- **Language(s) (NLP):** English
- **License:** As a Llama-derivative, this model cannot be used commercially.
- **Finetuned from model (QLoRA):** [huggyllama/llama-30b](https://huggingface.co/huggyllama/llama-30b)
## Training Details
### Training Data
Converted the following datasets to alpaca:instruction format.
1. [ehartford/dolphin](https://huggingface.co/datasets/ehartford/dolphin)
- ORCA style dataset generously created by [Eric Hartford](https://huggingface.co/ehartford)
2. [LinhDuong/chatdoctor-200k](https://huggingface.co/datasets/LinhDuong/chatdoctor-200k)
- Refined dataset sourced from icliniq medical QA forum
3. [sahil2801/code_instructions_120k](https://huggingface.co/datasets/sahil2801/code_instructions_120k)
- Code instruction dataset generously created by Sahil Chaudhary from ThreeSixty AI
4. [c-s-ale/dolly-15k-instruction-alpaca-format](https://huggingface.co/datasets/c-s-ale/dolly-15k-instruction-alpaca-format)
- Dolly 15k is a general instruction dataset generated by employees of Databricks.
### Training Procedure
Trained using [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) QLoRa on [RunPod](https://www.runpod.io/console/gpu-cloud) 8x A6000 on Community Cloud for 1 epochs (~23 hours - ~$110).
<details>
<summary>axolotl training config:</summary>
```yaml
base_model: huggyllama/llama-30b
base_model_config: huggyllama/llama-30b
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
load_in_8bit: false
load_in_4bit: true
strict: false
push_dataset_to_hub:
hub_model_id:
hf_use_auth_token:
datasets:
- path: ehartford/dolphin
type: alpaca
data_files:
- flan1m-alpaca-uncensored.jsonl
- flan5m-alpaca-uncensored.jsonl
shards: 25
- path: sahil2801/code_instructions_120k
type: alpaca
- path: LinhDuong/chatdoctor-200k
type: alpaca
shards: 2
- path: c-s-ale/dolly-15k-instruction-alpaca-format
type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.01
adapter: qlora
lora_model_dir:
sequence_len: 2048
max_packed_sequence_len: 2048
lora_r: 8
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:
wandb_mode: true
wandb_project: med-orca-instruct-33b
wandb_watch:
wandb_run_id:
wandb_log_model: 'openllama_checkpoint'
output_dir: /disk/med-instruct-33b
gradient_accumulation_steps: 1
micro_batch_size: 4
num_epochs: 1
optimizer: paged_adamw_32bit
torchdistx_path:
lr_scheduler: cosine
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: true
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 2
xformers_attention: true
flash_attention:
gptq_groupsize:
gptq_model_v1:
warmup_steps: 100
eval_steps: 20
save_steps:
debug:
deepspeed: true
weight_decay: 0.00001
special_tokens:
bos_token: "<s>"
eos_token: "</s>"
unk_token: "<unk>"
```
</details> |