sophosympatheia commited on
Commit
c0b298d
1 Parent(s): d862b4f

Update README.md

Browse files

Added links to more quants

Files changed (1) hide show
  1. README.md +9 -3
README.md CHANGED
@@ -121,9 +121,15 @@ If you save this as a .json file, you can import it directly into Silly Tavern.
121
  This repo contains branches for various exllama2 quanizations of the model calibratend on a version of the PIPPA dataset.
122
 
123
  * Main Branch, Full weights
124
- * 3.2 bpw -- This will fit comfortably within 48 GB of VRAM at 8192 context.
125
- * 3.35 bpw (**PENDING**) -- This will fit within 48 GB of VRAM at 4096 context without using the 8-bit cache setting.
126
- * 3.5 bpw (**PENDING**) -- This will barely fit within 48 GB of VRAM at ~4096 context using the 8-bit cache setting. If you get OOM, try lowering the context size slightly until it fits.
 
 
 
 
 
 
127
 
128
  ### Licence and usage restrictions
129
 
 
121
  This repo contains branches for various exllama2 quanizations of the model calibratend on a version of the PIPPA dataset.
122
 
123
  * Main Branch, Full weights
124
+ * 3.2 bpw branch -- This will fit comfortably within 48 GB of VRAM at 8192 context.
125
+ * 3.35 bpw branch -- This will fit within 48 GB of VRAM at 4096 context without using the 8-bit cache setting.
126
+ * 3.0 bpw -- [LoneStriker/Rogue-Rose-103b-v0.2-3.0bpw-h6-exl2-2](https://huggingface.co/LoneStriker/Rogue-Rose-103b-v0.2-3.0bpw-h6-exl2-2)
127
+ * 4.0 bpw -- [LoneStriker/Rogue-Rose-103b-v0.2-4.0bpw-h6-exl2-2](https://huggingface.co/LoneStriker/Rogue-Rose-103b-v0.2-4.0bpw-h6-exl2-2)
128
+ * 5.0 bpw -- [LoneStriker/Rogue-Rose-103b-v0.2-5.0bpw-h6-exl2-2](https://huggingface.co/LoneStriker/Rogue-Rose-103b-v0.2-5.0bpw-h6-exl2-2)
129
+ * GGUF -- [TheBloke/Rogue-Rose-103b-v0.2-GGUF](https://huggingface.co/TheBloke/Rogue-Rose-103b-v0.2-GGUF)
130
+ * AWQ -- [TheBloke/Rogue-Rose-103b-v0.2-AWQ](https://huggingface.co/TheBloke/Rogue-Rose-103b-v0.2-AWQ)
131
+ * GPTQ -- [TheBloke/Rogue-Rose-103b-v0.2-GPTQ](https://huggingface.co/TheBloke/Rogue-Rose-103b-v0.2-GPTQ)
132
+
133
 
134
  ### Licence and usage restrictions
135