README.md

---
license: gpl-3.0
---

# WizardLM 13B GGML

This is https://huggingface.co/winddude/wizardLM-LlaMA-LoRA-13 converted into GGML format, supported by [llama.cpp](https://github.com/ggerganov/llama.cpp).

If used in an instruct mode this model is very picky about the newlines. For some reason it insists on having the following format:

```
### Instruction:

(instruction text here)

### Response:

(leave this for the model to fill)
```

Note that generally the alpacas only have one newline before '### Instruction:' / '### Response' and the text. I might have somehow messed up conversion :(


# Instructions for running this model in https://github.com/oobabooga/text-generation-webui:

1. Make sure the pytorch and llama.cpp are recent enough:

```
pip install -U --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html
pip install -U llamacpp
```

2. Move the model to the right directory:

```
mkdir models/wizardlm-13b/
mv ~/Downloads/ggml-model-q5_1.bin models/wizardlm-13b/
```

2. Run the textgen, e.g. with:

```
python server.py --cpu --threads 8 --chat --model=wizardlm-13b --model_type=llama  --verbose
```

3. If using it in the instruction mode, you should also fix the prompt. Select the Alpaca template as a base, then go to 'Character' tab and modify Turn template with the following string: `<|user|>\n\n<|user-message|>\n\n<|bot|>\n\n<|bot-message|>\n\n`.

4. Sometimes the model just won't shut up, so go to Parameters tab and add the following 'Custom stopping string': `"### Instruction:"`

5. The 3) and 4) could probably be fixed by modifying settings.json

Please share other tips and tricks in https://huggingface.co/execveat/wizardLM-13b-ggml-q5_1/discussions