execveat's picture
Update README.md
d591f95
|
raw
history blame
1.73 kB
---
license: gpl-3.0
---
# WizardLM 13B GGML
This is https://huggingface.co/winddude/wizardLM-LlaMA-LoRA-13 converted into GGML format, supported by [llama.cpp](https://github.com/ggerganov/llama.cpp).
If used in an instruct mode this model is very picky about the newlines. For some reason it insists on having the following format:
```
### Instruction:
(instruction text here)
### Response:
(leave this for the model to fill)
```
Note that generally the alpacas only have one newline before '### Instruction:' / '### Response' and the text. I might have somehow messed up conversion :(
# Instructions for running this model in https://github.com/oobabooga/text-generation-webui:
1. Make sure the pytorch and llama.cpp are recent enough:
```
pip install -U --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html
pip install -U llamacpp
```
2. Move the model to the right directory:
```
mkdir models/wizardlm-13b/
mv ~/Downloads/ggml-model-q5_1.bin models/wizardlm-13b/
```
2. Run the textgen, e.g. with:
```
python server.py --cpu --threads 8 --chat --model=wizardlm-13b --model_type=llama --verbose
```
3. If using it in the instruction mode, you should also fix the prompt. Select the Alpaca template as a base, then go to 'Character' tab and modify Turn template with the following string: `<|user|>\n\n<|user-message|>\n\n<|bot|>\n\n<|bot-message|>\n\n`.
4. Sometimes the model just won't shut up, so go to Parameters tab and add the following 'Custom stopping string': `"### Instruction:"`
5. The 3) and 4) could probably be fixed by modifying settings.json
Please share other tips and tricks in https://huggingface.co/execveat/wizardLM-13b-ggml-q5_1/discussions