jondurbin
/

mpt-30b-qlora-compatible

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

jondurbin commited on Jun 24, 2023

Commit

1cd6bd0

·

1 Parent(s): 3f09a99

Update README.md

Files changed (1) hide show

README.md +2 -0

README.md CHANGED Viewed

@@ -10,6 +10,8 @@ Differences in the qlora scripts:
 - uses `--num_train_epochs` instead of `--max_steps`
 - uses airoboros prompt format (mostly 1:1 with vicuna) rather than alpaca, and expects an input file in JSONL format with "instruction" and "response"
 Full example of tuning (used for airoboros-mpt-30b-gpt4-1.4):
 ```

 - uses `--num_train_epochs` instead of `--max_steps`
 - uses airoboros prompt format (mostly 1:1 with vicuna) rather than alpaca, and expects an input file in JSONL format with "instruction" and "response"
+__I think there's a bug in gradient accumulation, so if you try this, maybe set gradient accumulation steps to 1__
 Full example of tuning (used for airoboros-mpt-30b-gpt4-1.4):
 ```