yaoshining commited on
Commit
aed2c7c
1 Parent(s): ebf3d10
Files changed (1) hide show
  1. README.md +13 -360
README.md CHANGED
@@ -1,360 +1,13 @@
1
- # Text generation web UI
2
-
3
- A gradio web UI for running Large Language Models like LLaMA, llama.cpp, GPT-J, Pythia, OPT, and GALACTICA.
4
-
5
- Its goal is to become the [AUTOMATIC1111/stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui) of text generation.
6
-
7
- |![Image1](https://github.com/oobabooga/screenshots/raw/main/qa.png) | ![Image2](https://github.com/oobabooga/screenshots/raw/main/cai3.png) |
8
- |:---:|:---:|
9
- |![Image3](https://github.com/oobabooga/screenshots/raw/main/gpt4chan.png) | ![Image4](https://github.com/oobabooga/screenshots/raw/main/galactica.png) |
10
-
11
- ## Features
12
-
13
- * 3 interface modes: default, notebook, and chat
14
- * Multiple model backends: tranformers, llama.cpp, AutoGPTQ, GPTQ-for-LLaMa, ExLlama, RWKV, FlexGen
15
- * Dropdown menu for quickly switching between different models
16
- * LoRA: load and unload LoRAs on the fly, load multiple LoRAs at the same time, train a new LoRA
17
- * Precise instruction templates for chat mode, including Alpaca, Vicuna, Open Assistant, Dolly, Koala, ChatGLM, MOSS, RWKV-Raven, Galactica, StableLM, WizardLM, Baize, Ziya, Chinese-Vicuna, MPT, INCITE, Wizard Mega, KoAlpaca, Vigogne, Bactrian, h2o, and OpenBuddy
18
- * [Multimodal pipelines, including LLaVA and MiniGPT-4](https://github.com/oobabooga/text-generation-webui/tree/main/extensions/multimodal)
19
- * 8-bit and 4-bit inference through bitsandbytes
20
- * CPU mode for transformers models
21
- * [DeepSpeed ZeRO-3 inference](docs/DeepSpeed.md)
22
- * [Extensions](docs/Extensions.md)
23
- * [Custom chat characters](docs/Chat-mode.md)
24
- * Very efficient text streaming
25
- * Markdown output with LaTeX rendering, to use for instance with [GALACTICA](https://github.com/paperswithcode/galai)
26
- * Nice HTML output for GPT-4chan
27
- * API, including endpoints for websocket streaming ([see the examples](https://github.com/oobabooga/text-generation-webui/blob/main/api-examples))
28
-
29
- To learn how to use the various features, check out the Documentation: https://github.com/oobabooga/text-generation-webui/tree/main/docs
30
-
31
- ## Installation
32
-
33
- ### One-click installers
34
-
35
- | Windows | Linux | macOS | WSL |
36
- |--------|--------|--------|--------|
37
- | [oobabooga-windows.zip](https://github.com/oobabooga/text-generation-webui/releases/download/installers/oobabooga_windows.zip) | [oobabooga-linux.zip](https://github.com/oobabooga/text-generation-webui/releases/download/installers/oobabooga_linux.zip) |[oobabooga-macos.zip](https://github.com/oobabooga/text-generation-webui/releases/download/installers/oobabooga_macos.zip) | [oobabooga-wsl.zip](https://github.com/oobabooga/text-generation-webui/releases/download/installers/oobabooga_wsl.zip) |
38
-
39
- Just download the zip above, extract it, and double-click on "start". The web UI and all its dependencies will be installed in the same folder.
40
-
41
- * The source codes are here: https://github.com/oobabooga/one-click-installers
42
- * There is no need to run the installers as admin.
43
- * AMD doesn't work on Windows.
44
- * Huge thanks to [@jllllll](https://github.com/jllllll), [@ClayShoaf](https://github.com/ClayShoaf), and [@xNul](https://github.com/xNul) for their contributions to these installers.
45
-
46
- ### Manual installation using Conda
47
-
48
- Recommended if you have some experience with the command line.
49
-
50
- #### 0. Install Conda
51
-
52
- https://docs.conda.io/en/latest/miniconda.html
53
-
54
- On Linux or WSL, it can be automatically installed with these two commands:
55
-
56
- ```
57
- curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" > "Miniconda3.sh"
58
- bash Miniconda3.sh
59
- ```
60
- Source: https://educe-ubc.github.io/conda.html
61
-
62
- #### 1. Create a new conda environment
63
-
64
- ```
65
- conda create -n textgen python=3.10.9
66
- conda activate textgen
67
- ```
68
-
69
- #### 2. Install Pytorch
70
-
71
- | System | GPU | Command |
72
- |--------|---------|---------|
73
- | Linux/WSL | NVIDIA | `pip3 install torch torchvision torchaudio` |
74
- | Linux | AMD | `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.4.2` |
75
- | MacOS + MPS (untested) | Any | `pip3 install torch torchvision torchaudio` |
76
- | Windows | NVIDIA | `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117` |
77
-
78
- The up-to-date commands can be found here: https://pytorch.org/get-started/locally/.
79
-
80
- #### 2.1 Special instructions
81
-
82
- * MacOS users: https://github.com/oobabooga/text-generation-webui/pull/393
83
- * AMD users: https://rentry.org/eq3hg
84
-
85
- #### 3. Install the web UI
86
-
87
- ```
88
- git clone https://github.com/oobabooga/text-generation-webui
89
- cd text-generation-webui
90
- pip install -r requirements.txt
91
- ```
92
-
93
- #### llama.cpp with GPU acceleration
94
-
95
- Requires the additional compilation step described here: [GPU acceleration](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md#gpu-acceleration).
96
-
97
- #### bitsandbytes
98
-
99
- bitsandbytes >= 0.39 may not work on older NVIDIA GPUs. In that case, to use `--load-in-8bit`, you may have to downgrade like this:
100
-
101
- * Linux: `pip install bitsandbytes==0.38.1`
102
- * Windows: `pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl`
103
-
104
- ### Alternative: Docker
105
-
106
- ```
107
- ln -s docker/{Dockerfile,docker-compose.yml,.dockerignore} .
108
- cp docker/.env.example .env
109
- # Edit .env and set TORCH_CUDA_ARCH_LIST based on your GPU model
110
- docker compose up --build
111
- ```
112
-
113
- * You need to have docker compose v2.17 or higher installed. See [this guide](https://github.com/oobabooga/text-generation-webui/blob/main/docs/Docker.md) for instructions.
114
- * For additional docker files, check out [this repository](https://github.com/Atinoda/text-generation-webui-docker).
115
-
116
- ### Updating the requirements
117
-
118
- From time to time, the `requirements.txt` changes. To update, use this command:
119
-
120
- ```
121
- conda activate textgen
122
- cd text-generation-webui
123
- pip install -r requirements.txt --upgrade
124
- ```
125
- ## Downloading models
126
-
127
- Models should be placed inside the `models/` folder.
128
-
129
- [Hugging Face](https://huggingface.co/models?pipeline_tag=text-generation&sort=downloads) is the main place to download models. These are some examples:
130
-
131
- * [Pythia](https://huggingface.co/models?sort=downloads&search=eleutherai%2Fpythia+deduped)
132
- * [OPT](https://huggingface.co/models?search=facebook/opt)
133
- * [GALACTICA](https://huggingface.co/models?search=facebook/galactica)
134
- * [GPT-J 6B](https://huggingface.co/EleutherAI/gpt-j-6B/tree/main)
135
-
136
- You can automatically download a model from HF using the script `download-model.py`:
137
-
138
- python download-model.py organization/model
139
-
140
- For example:
141
-
142
- python download-model.py facebook/opt-1.3b
143
-
144
- To download a protected model, set env vars `HF_USER` and `HF_PASS` to your Hugging Face username and password (or [User Access Token](https://huggingface.co/settings/tokens)). The model's terms must first be accepted on the HF website.
145
-
146
- #### GGML models
147
-
148
- You can drop these directly into the `models/` folder, making sure that the file name contains `ggml` somewhere and ends in `.bin`.
149
-
150
- #### GPT-4chan
151
-
152
- <details>
153
- <summary>
154
- Instructions
155
- </summary>
156
-
157
- [GPT-4chan](https://huggingface.co/ykilcher/gpt-4chan) has been shut down from Hugging Face, so you need to download it elsewhere. You have two options:
158
-
159
- * Torrent: [16-bit](https://archive.org/details/gpt4chan_model_float16) / [32-bit](https://archive.org/details/gpt4chan_model)
160
- * Direct download: [16-bit](https://theswissbay.ch/pdf/_notpdf_/gpt4chan_model_float16/) / [32-bit](https://theswissbay.ch/pdf/_notpdf_/gpt4chan_model/)
161
-
162
- The 32-bit version is only relevant if you intend to run the model in CPU mode. Otherwise, you should use the 16-bit version.
163
-
164
- After downloading the model, follow these steps:
165
-
166
- 1. Place the files under `models/gpt4chan_model_float16` or `models/gpt4chan_model`.
167
- 2. Place GPT-J 6B's config.json file in that same folder: [config.json](https://huggingface.co/EleutherAI/gpt-j-6B/raw/main/config.json).
168
- 3. Download GPT-J 6B's tokenizer files (they will be automatically detected when you attempt to load GPT-4chan):
169
-
170
- ```
171
- python download-model.py EleutherAI/gpt-j-6B --text-only
172
- ```
173
-
174
- When you load this model in default or notebook modes, the "HTML" tab will show the generated text in 4chan format.
175
- </details>
176
-
177
- ## Starting the web UI
178
-
179
- conda activate textgen
180
- cd text-generation-webui
181
- python server.py
182
-
183
- Then browse to
184
-
185
- `http://localhost:7860/?__theme=dark`
186
-
187
- Optionally, you can use the following command-line flags:
188
-
189
- #### Basic settings
190
-
191
- | Flag | Description |
192
- |--------------------------------------------|-------------|
193
- | `-h`, `--help` | Show this help message and exit. |
194
- | `--notebook` | Launch the web UI in notebook mode, where the output is written to the same text box as the input. |
195
- | `--chat` | Launch the web UI in chat mode. |
196
- | `--character CHARACTER` | The name of the character to load in chat mode by default. |
197
- | `--model MODEL` | Name of the model to load by default. |
198
- | `--lora LORA [LORA ...]` | The list of LoRAs to load. If you want to load more than one LoRA, write the names separated by spaces. |
199
- | `--model-dir MODEL_DIR` | Path to directory with all the models. |
200
- | `--lora-dir LORA_DIR` | Path to directory with all the loras. |
201
- | `--model-menu` | Show a model menu in the terminal when the web UI is first launched. |
202
- | `--no-stream` | Don't stream the text output in real time. |
203
- | `--settings SETTINGS_FILE` | Load the default interface settings from this yaml file. See `settings-template.yaml` for an example. If you create a file called `settings.yaml`, this file will be loaded by default without the need to use the `--settings` flag. |
204
- | `--extensions EXTENSIONS [EXTENSIONS ...]` | The list of extensions to load. If you want to load more than one extension, write the names separated by spaces. |
205
- | `--verbose` | Print the prompts to the terminal. |
206
-
207
- #### Model loader
208
-
209
- | Flag | Description |
210
- |--------------------------------------------|-------------|
211
- | `--loader LOADER` | Choose the model loader manually, otherwise, it will get autodetected. Valid options: transformers, autogptq, gptq-for-llama, exllama, exllama_hf, llamacpp, rwkv, flexgen |
212
-
213
- #### Accelerate/transformers
214
-
215
- | Flag | Description |
216
- |---------------------------------------------|-------------|
217
- | `--cpu` | Use the CPU to generate text. Warning: Training on CPU is extremely slow.|
218
- | `--auto-devices` | Automatically split the model across the available GPU(s) and CPU. |
219
- | `--gpu-memory GPU_MEMORY [GPU_MEMORY ...]` | Maximum GPU memory in GiB to be allocated per GPU. Example: `--gpu-memory 10` for a single GPU, `--gpu-memory 10 5` for two GPUs. You can also set values in MiB like `--gpu-memory 3500MiB`. |
220
- | `--cpu-memory CPU_MEMORY` | Maximum CPU memory in GiB to allocate for offloaded weights. Same as above.|
221
- | `--disk` | If the model is too large for your GPU(s) and CPU combined, send the remaining layers to the disk. |
222
- | `--disk-cache-dir DISK_CACHE_DIR` | Directory to save the disk cache to. Defaults to `cache/`. |
223
- | `--load-in-8bit` | Load the model with 8-bit precision (using bitsandbytes).|
224
- | `--bf16` | Load the model with bfloat16 precision. Requires NVIDIA Ampere GPU. |
225
- | `--no-cache` | Set `use_cache` to False while generating text. This reduces the VRAM usage a bit with a performance cost. |
226
- | `--xformers` | Use xformer's memory efficient attention. This should increase your tokens/s. |
227
- | `--sdp-attention` | Use torch 2.0's sdp attention. |
228
- | `--trust-remote-code` | Set trust_remote_code=True while loading a model. Necessary for ChatGLM and Falcon. |
229
-
230
- #### Accelerate 4-bit
231
-
232
- ⚠️ Requires minimum compute of 7.0 on Windows at the moment.
233
-
234
- | Flag | Description |
235
- |---------------------------------------------|-------------|
236
- | `--load-in-4bit` | Load the model with 4-bit precision (using bitsandbytes). |
237
- | `--compute_dtype COMPUTE_DTYPE` | compute dtype for 4-bit. Valid options: bfloat16, float16, float32. |
238
- | `--quant_type QUANT_TYPE` | quant_type for 4-bit. Valid options: nf4, fp4. |
239
- | `--use_double_quant` | use_double_quant for 4-bit. |
240
-
241
- #### llama.cpp
242
-
243
- | Flag | Description |
244
- |-------------|-------------|
245
- | `--threads` | Number of threads to use. |
246
- | `--n_batch` | Maximum number of prompt tokens to batch together when calling llama_eval. |
247
- | `--no-mmap` | Prevent mmap from being used. |
248
- | `--mlock` | Force the system to keep the model in RAM. |
249
- | `--cache-capacity CACHE_CAPACITY` | Maximum cache capacity. Examples: 2000MiB, 2GiB. When provided without units, bytes will be assumed. |
250
- | `--n-gpu-layers N_GPU_LAYERS` | Number of layers to offload to the GPU. Only works if llama-cpp-python was compiled with BLAS. Set this to 1000000000 to offload all layers to the GPU. |
251
- | `--n_ctx N_CTX` | Size of the prompt context. |
252
- | `--llama_cpp_seed SEED` | Seed for llama-cpp models. Default 0 (random). |
253
-
254
- #### AutoGPTQ
255
-
256
- | Flag | Description |
257
- |------------------|-------------|
258
- | `--triton` | Use triton. |
259
- | `--no_inject_fused_attention` | Disable the use of fused attention, which will use less VRAM at the cost of slower inference. |
260
- | `--no_inject_fused_mlp` | Triton mode only: disable the use of fused MLP, which will use less VRAM at the cost of slower inference. |
261
- | `--no_use_cuda_fp16` | This can make models faster on some systems. |
262
- | `--desc_act` | For models that don't have a quantize_config.json, this parameter is used to define whether to set desc_act or not in BaseQuantizeConfig. |
263
-
264
- #### ExLlama
265
-
266
- | Flag | Description |
267
- |------------------|-------------|
268
- |`--gpu-split` | Comma-separated list of VRAM (in GB) to use per GPU device for model layers, e.g. `20,7,7` |
269
- |`--max_seq_len MAX_SEQ_LEN` | Maximum sequence length. |
270
- |`--compress_pos_emb COMPRESS_POS_EMB` | Positional embeddings compression factor. Should typically be set to max_seq_len / 2048. |
271
-
272
- #### GPTQ-for-LLaMa
273
-
274
- | Flag | Description |
275
- |---------------------------|-------------|
276
- | `--wbits WBITS` | Load a pre-quantized model with specified precision in bits. 2, 3, 4 and 8 are supported. |
277
- | `--model_type MODEL_TYPE` | Model type of pre-quantized model. Currently LLaMA, OPT, and GPT-J are supported. |
278
- | `--groupsize GROUPSIZE` | Group size. |
279
- | `--pre_layer PRE_LAYER [PRE_LAYER ...]` | The number of layers to allocate to the GPU. Setting this parameter enables CPU offloading for 4-bit models. For multi-gpu, write the numbers separated by spaces, eg `--pre_layer 30 60`. |
280
- | `--checkpoint CHECKPOINT` | The path to the quantized checkpoint file. If not specified, it will be automatically detected. |
281
- | `--monkey-patch` | Apply the monkey patch for using LoRAs with quantized models.
282
- | `--quant_attn` | (triton) Enable quant attention. |
283
- | `--warmup_autotune` | (triton) Enable warmup autotune. |
284
- | `--fused_mlp` | (triton) Enable fused mlp. |
285
-
286
- #### FlexGen
287
-
288
- | Flag | Description |
289
- |------------------|-------------|
290
- | `--percent PERCENT [PERCENT ...]` | FlexGen: allocation percentages. Must be 6 numbers separated by spaces (default: 0, 100, 100, 0, 100, 0). |
291
- | `--compress-weight` | FlexGen: Whether to compress weight (default: False).|
292
- | `--pin-weight [PIN_WEIGHT]` | FlexGen: whether to pin weights (setting this to False reduces CPU memory by 20%). |
293
-
294
- #### DeepSpeed
295
-
296
- | Flag | Description |
297
- |---------------------------------------|-------------|
298
- | `--deepspeed` | Enable the use of DeepSpeed ZeRO-3 for inference via the Transformers integration. |
299
- | `--nvme-offload-dir NVME_OFFLOAD_DIR` | DeepSpeed: Directory to use for ZeRO-3 NVME offloading. |
300
- | `--local_rank LOCAL_RANK` | DeepSpeed: Optional argument for distributed setups. |
301
-
302
- #### RWKV
303
-
304
- | Flag | Description |
305
- |---------------------------------|-------------|
306
- | `--rwkv-strategy RWKV_STRATEGY` | RWKV: The strategy to use while loading the model. Examples: "cpu fp32", "cuda fp16", "cuda fp16i8". |
307
- | `--rwkv-cuda-on` | RWKV: Compile the CUDA kernel for better performance. |
308
-
309
- #### Gradio
310
-
311
- | Flag | Description |
312
- |---------------------------------------|-------------|
313
- | `--listen` | Make the web UI reachable from your local network. |
314
- | `--listen-host LISTEN_HOST` | The hostname that the server will use. |
315
- | `--listen-port LISTEN_PORT` | The listening port that the server will use. |
316
- | `--share` | Create a public URL. This is useful for running the web UI on Google Colab or similar. |
317
- | `--auto-launch` | Open the web UI in the default browser upon launch. |
318
- | `--gradio-auth USER:PWD` | set gradio authentication like "username:password"; or comma-delimit multiple like "u1:p1,u2:p2,u3:p3" |
319
- | `--gradio-auth-path GRADIO_AUTH_PATH` | Set the gradio authentication file path. The file should contain one or more user:password pairs in this format: "u1:p1,u2:p2,u3:p3" |
320
-
321
- #### API
322
-
323
- | Flag | Description |
324
- |---------------------------------------|-------------|
325
- | `--api` | Enable the API extension. |
326
- | `--public-api` | Create a public URL for the API using Cloudfare. |
327
- | `--api-blocking-port BLOCKING_PORT` | The listening port for the blocking API. |
328
- | `--api-streaming-port STREAMING_PORT` | The listening port for the streaming API. |
329
-
330
- #### Multimodal
331
-
332
- | Flag | Description |
333
- |---------------------------------------|-------------|
334
- | `--multimodal-pipeline PIPELINE` | The multimodal pipeline to use. Examples: `llava-7b`, `llava-13b`. |
335
-
336
- Out of memory errors? [Check the low VRAM guide](docs/Low-VRAM-guide.md).
337
-
338
- ## Presets
339
-
340
- Inference settings presets can be created under `presets/` as yaml files. These files are detected automatically at startup.
341
-
342
- The presets that are included by default are the result of a contest that received 7215 votes. More details can be found [here](https://github.com/oobabooga/oobabooga.github.io/blob/main/arena/results.md).
343
-
344
- ## Contributing
345
-
346
- * Pull requests, suggestions, and issue reports are welcome.
347
- * Make sure to carefully [search](https://github.com/oobabooga/text-generation-webui/issues) existing issues before starting a new one.
348
- * If you have some experience with git, testing an open pull request and leaving a comment on whether it works as expected or not is immensely helpful.
349
- * A simple way to contribute, even if you are not a programmer, is to leave a 👍 on an issue or pull request that you find relevant.
350
-
351
- ## Community
352
-
353
- * Subreddit: https://www.reddit.com/r/oobaboogazz/
354
- * Discord: https://discord.gg/jwZCF2dPQN
355
-
356
- ## Credits
357
-
358
- - Gradio dropdown menu refresh button, code for reloading the interface: https://github.com/AUTOMATIC1111/stable-diffusion-webui
359
- - Godlike preset: https://github.com/KoboldAI/KoboldAI-Client/wiki/Settings-Presets
360
- - Code for some of the sliders: https://github.com/PygmalionAI/gradio-ui/
 
1
+ ---
2
+ title: Text Generation Webui
3
+ emoji: 👀
4
+ colorFrom: green
5
+ colorTo: red
6
+ sdk: gradio
7
+ sdk_version: 3.35.2
8
+ app_file: app.py
9
+ pinned: false
10
+ license: apache-2.0
11
+ ---
12
+
13
+ Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference