|
--- |
|
license: apache-2.0 |
|
tags: |
|
- trl |
|
- orpo |
|
- generated_from_trainer |
|
- exl2 |
|
base_model: mistral-community/Mixtral-8x22B-v0.1 |
|
datasets: |
|
- argilla/distilabel-capybara-dpo-7k-binarized |
|
model-index: |
|
- name: zephyr-orpo-141b-A35b-v0.1 |
|
results: [] |
|
--- |
|
|
|
<img src="https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1/resolve/main/logo.png" alt="Zephyr 141B Logo" width="400" style="margin-left:'auto' margin-right:'auto' display:'block'"/> |
|
|
|
# machinez/zephyr-orpo-141b-A35b-v0.1-exl2 |
|
This model was converted to EXL2 format from [`HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1`](https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1). |
|
Refer to the [original model card](https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1) for more details on the model. |
|
|
|
Each branch contains an individual bits per weight, with the main one containing only the meaurement.json for further conversions. |
|
|
|
<a href="https://huggingface.co/machinez/zephyr-orpo-141b-A35b-v0.1-exl2/tree/1_5">1.5 bits per weight - Fits Dual RTX 3090/4090 or Triple Nvidia Tesla P100 16gb at 4k context</a> |
|
|
|
<a href="https://huggingface.co/machinez/zephyr-orpo-141b-A35b-v0.1-exl2/tree/2_75">2.75 bits per weight - Fits Quad Nvidia Tesla P100 16gb at 16k context</a> |
|
|
|
## Sample instructions to load in TabbyAPI @ 1.5bpw on 3x Nvidia Tesla P100 16gb at 4k context |
|
```JSON |
|
{ |
|
"name": "Machinez_zephyr-orpo-141b-A35b-v0.1_1.5bpw", |
|
"max_seq_len": 4096, |
|
"override_base_seq_len": 4096, |
|
"gpu_split_auto": false, |
|
"autosplit_reserve": [ |
|
96 |
|
], |
|
"gpu_split": [ |
|
14.15, |
|
14, |
|
15 |
|
], |
|
"rope_scale": 1, |
|
"rope_alpha": 1, |
|
"no_flash_attention": false, |
|
"cache_mode": "fp16", |
|
"prompt_template": "string", |
|
"num_experts_per_token": 0, |
|
"use_cfg": true, |
|
"fasttensors": false, |
|
"skip_queue": false |
|
} |
|
``` |
|
|
|
## Sample instructions to load in TabbyAPI @ 2.75bpw on 4x Nvidia Tesla P100 16gb at 16k context |
|
```JSON |
|
{ |
|
"name": "Machinez_zephyr-orpo-141b-A35b-v0.1_2.75bpw", |
|
"max_seq_len": 16384, |
|
"override_base_seq_len": 16384, |
|
"gpu_split_auto": false, |
|
"autosplit_reserve": [ |
|
96 |
|
], |
|
"gpu_split": [ |
|
12.5, |
|
13, |
|
13, |
|
16.1 |
|
], |
|
"rope_scale": 1, |
|
"rope_alpha": 1, |
|
"no_flash_attention": false, |
|
"cache_mode": "fp16", |
|
"prompt_template": "string", |
|
"num_experts_per_token": 0, |
|
"use_cfg": true, |
|
"fasttensors": false, |
|
"skip_queue": false |
|
} |
|
``` |
|
|
|
## Download instructions |
|
|
|
With git: |
|
|
|
```shell |
|
git clone --single-branch --branch 2_75 https://huggingface.co/machinez/zephyr-orpo-141b-A35b-v0.1-exl2 |
|
``` |
|
|
|
With huggingface hub (credit to TheBloke for instructions, borrowed from bartowski): |
|
|
|
```shell |
|
pip3 install -U "huggingface_hub[cli]" |
|
``` |
|
|
|
## (optional) |
|
```shell |
|
git config --global credential.helper 'store --file ~/.my-credentials' |
|
huggingface-cli login |
|
``` |
|
|
|
To download the `main` (only useful if you only care about measurement.json) branch to a folder called `zephyr-orpo-141b-A35b-v0.1-exl22`: |
|
|
|
```shell |
|
mkdir machinez_zephyr-orpo-141b-A35b-v0.1-exl2_2.75bpw |
|
huggingface-cli download machinez/zephyr-orpo-141b-A35b-v0.1-exl2 --local-dir machinez_zephyr-orpo-141b-A35b-v0.1-exl2_2.75bpw --local-dir-use-symlinks False |
|
``` |
|
|
|
To download from a different branch, add the `--revision` parameter: |
|
|
|
```shell |
|
mkdir machinez_zephyr-orpo-141b-A35b-v0.1-exl2_2.75bpw |
|
huggingface-cli download machinez/zephyr-orpo-141b-A35b-v0.1-exl2 --revision 2_75 --local-dir machinez_zephyr-orpo-141b-A35b-v0.1-exl2_2.75bpw --local-dir-use-symlinks False |
|
|
