multimodalart HF staff commited on
Commit
4833676
1 Parent(s): 3b06696

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +11 -292
README.md CHANGED
@@ -1,292 +1,11 @@
1
- # Generative Models by Stability AI
2
-
3
- ![sample1](assets/000.jpg)
4
-
5
- ## News
6
-
7
- **November 21, 2023**
8
-
9
- - We are releasing Stable Video Diffusion, an image-to-video model, for research purposes:
10
- - [SVD](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid): This model was trained to generate 14
11
- frames at resolution 576x1024 given a context frame of the same size.
12
- We use the standard image encoder from SD 2.1, but replace the decoder with a temporally-aware `deflickering decoder`.
13
- - [SVD-XT](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt): Same architecture as `SVD` but finetuned
14
- for 25 frame generation.
15
- - We provide a streamlit demo `scripts/demo/video_sampling.py` and a standalone python script `scripts/sampling/simple_video_sample.py` for inference of both models.
16
- - Alongside the model, we will release a technical report shortly. Stay tuned.
17
-
18
- ![tile](assets/tile.gif)
19
-
20
- **July 26, 2023**
21
-
22
- - We are releasing two new open models with a
23
- permissive [`CreativeML Open RAIL++-M` license](model_licenses/LICENSE-SDXL1.0) (see [Inference](#inference) for file
24
- hashes):
25
- - [SDXL-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0): An improved version
26
- over `SDXL-base-0.9`.
27
- - [SDXL-refiner-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0): An improved version
28
- over `SDXL-refiner-0.9`.
29
-
30
- ![sample2](assets/001_with_eval.png)
31
-
32
- **July 4, 2023**
33
-
34
- - A technical report on SDXL is now available [here](https://arxiv.org/abs/2307.01952).
35
-
36
- **June 22, 2023**
37
-
38
- - We are releasing two new diffusion models for research purposes:
39
- - `SDXL-base-0.9`: The base model was trained on a variety of aspect ratios on images with resolution 1024^2. The
40
- base model uses [OpenCLIP-ViT/G](https://github.com/mlfoundations/open_clip)
41
- and [CLIP-ViT/L](https://github.com/openai/CLIP/tree/main) for text encoding whereas the refiner model only uses
42
- the OpenCLIP model.
43
- - `SDXL-refiner-0.9`: The refiner has been trained to denoise small noise levels of high quality data and as such is
44
- not expected to work as a text-to-image model; instead, it should only be used as an image-to-image model.
45
-
46
- If you would like to access these models for your research, please apply using one of the following links:
47
- [SDXL-0.9-Base model](https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9),
48
- and [SDXL-0.9-Refiner](https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-0.9).
49
- This means that you can apply for any of the two links - and if you are granted - you can access both.
50
- Please log in to your Hugging Face Account with your organization email to request access.
51
- **We plan to do a full release soon (July).**
52
-
53
- ## The codebase
54
-
55
- ### General Philosophy
56
-
57
- Modularity is king. This repo implements a config-driven approach where we build and combine submodules by
58
- calling `instantiate_from_config()` on objects defined in yaml configs. See `configs/` for many examples.
59
-
60
- ### Changelog from the old `ldm` codebase
61
-
62
- For training, we use [PyTorch Lightning](https://lightning.ai/docs/pytorch/stable/), but it should be easy to use other
63
- training wrappers around the base modules. The core diffusion model class (formerly `LatentDiffusion`,
64
- now `DiffusionEngine`) has been cleaned up:
65
-
66
- - No more extensive subclassing! We now handle all types of conditioning inputs (vectors, sequences and spatial
67
- conditionings, and all combinations thereof) in a single class: `GeneralConditioner`,
68
- see `sgm/modules/encoders/modules.py`.
69
- - We separate guiders (such as classifier-free guidance, see `sgm/modules/diffusionmodules/guiders.py`) from the
70
- samplers (`sgm/modules/diffusionmodules/sampling.py`), and the samplers are independent of the model.
71
- - We adopt the ["denoiser framework"](https://arxiv.org/abs/2206.00364) for both training and inference (most notable
72
- change is probably now the option to train continuous time models):
73
- * Discrete times models (denoisers) are simply a special case of continuous time models (denoisers);
74
- see `sgm/modules/diffusionmodules/denoiser.py`.
75
- * The following features are now independent: weighting of the diffusion loss
76
- function (`sgm/modules/diffusionmodules/denoiser_weighting.py`), preconditioning of the
77
- network (`sgm/modules/diffusionmodules/denoiser_scaling.py`), and sampling of noise levels during
78
- training (`sgm/modules/diffusionmodules/sigma_sampling.py`).
79
- - Autoencoding models have also been cleaned up.
80
-
81
- ## Installation:
82
-
83
- <a name="installation"></a>
84
-
85
- #### 1. Clone the repo
86
-
87
- ```shell
88
- git clone git@github.com:Stability-AI/generative-models.git
89
- cd generative-models
90
- ```
91
-
92
- #### 2. Setting up the virtualenv
93
-
94
- This is assuming you have navigated to the `generative-models` root after cloning it.
95
-
96
- **NOTE:** This is tested under `python3.10`. For other python versions, you might encounter version conflicts.
97
-
98
- **PyTorch 2.0**
99
-
100
- ```shell
101
- # install required packages from pypi
102
- python3 -m venv .pt2
103
- source .pt2/bin/activate
104
- pip3 install -r requirements/pt2.txt
105
- ```
106
-
107
- #### 3. Install `sgm`
108
-
109
- ```shell
110
- pip3 install .
111
- ```
112
-
113
- #### 4. Install `sdata` for training
114
-
115
- ```shell
116
- pip3 install -e git+https://github.com/Stability-AI/datapipelines.git@main#egg=sdata
117
- ```
118
-
119
- ## Packaging
120
-
121
- This repository uses PEP 517 compliant packaging using [Hatch](https://hatch.pypa.io/latest/).
122
-
123
- To build a distributable wheel, install `hatch` and run `hatch build`
124
- (specifying `-t wheel` will skip building a sdist, which is not necessary).
125
-
126
- ```
127
- pip install hatch
128
- hatch build -t wheel
129
- ```
130
-
131
- You will find the built package in `dist/`. You can install the wheel with `pip install dist/*.whl`.
132
-
133
- Note that the package does **not** currently specify dependencies; you will need to install the required packages,
134
- depending on your use case and PyTorch version, manually.
135
-
136
- ## Inference
137
-
138
- We provide a [streamlit](https://streamlit.io/) demo for text-to-image and image-to-image sampling
139
- in `scripts/demo/sampling.py`.
140
- We provide file hashes for the complete file as well as for only the saved tensors in the file (
141
- see [Model Spec](https://github.com/Stability-AI/ModelSpec) for a script to evaluate that).
142
- The following models are currently supported:
143
-
144
- - [SDXL-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0)
145
- ```
146
- File Hash (sha256): 31e35c80fc4829d14f90153f4c74cd59c90b779f6afe05a74cd6120b893f7e5b
147
- Tensordata Hash (sha256): 0xd7a9105a900fd52748f20725fe52fe52b507fd36bee4fc107b1550a26e6ee1d7
148
- ```
149
- - [SDXL-refiner-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0)
150
- ```
151
- File Hash (sha256): 7440042bbdc8a24813002c09b6b69b64dc90fded4472613437b7f55f9b7d9c5f
152
- Tensordata Hash (sha256): 0x1a77d21bebc4b4de78c474a90cb74dc0d2217caf4061971dbfa75ad406b75d81
153
- ```
154
- - [SDXL-base-0.9](https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9)
155
- - [SDXL-refiner-0.9](https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-0.9)
156
- - [SD-2.1-512](https://huggingface.co/stabilityai/stable-diffusion-2-1-base/blob/main/v2-1_512-ema-pruned.safetensors)
157
- - [SD-2.1-768](https://huggingface.co/stabilityai/stable-diffusion-2-1/blob/main/v2-1_768-ema-pruned.safetensors)
158
-
159
- **Weights for SDXL**:
160
-
161
- **SDXL-1.0:**
162
- The weights of SDXL-1.0 are available (subject to
163
- a [`CreativeML Open RAIL++-M` license](model_licenses/LICENSE-SDXL1.0)) here:
164
-
165
- - base model: https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/
166
- - refiner model: https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/
167
-
168
- **SDXL-0.9:**
169
- The weights of SDXL-0.9 are available and subject to a [research license](model_licenses/LICENSE-SDXL0.9).
170
- If you would like to access these models for your research, please apply using one of the following links:
171
- [SDXL-base-0.9 model](https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9),
172
- and [SDXL-refiner-0.9](https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-0.9).
173
- This means that you can apply for any of the two links - and if you are granted - you can access both.
174
- Please log in to your Hugging Face Account with your organization email to request access.
175
-
176
- After obtaining the weights, place them into `checkpoints/`.
177
- Next, start the demo using
178
-
179
- ```
180
- streamlit run scripts/demo/sampling.py --server.port <your_port>
181
- ```
182
-
183
- ### Invisible Watermark Detection
184
-
185
- Images generated with our code use the
186
- [invisible-watermark](https://github.com/ShieldMnt/invisible-watermark/)
187
- library to embed an invisible watermark into the model output. We also provide
188
- a script to easily detect that watermark. Please note that this watermark is
189
- not the same as in previous Stable Diffusion 1.x/2.x versions.
190
-
191
- To run the script you need to either have a working installation as above or
192
- try an _experimental_ import using only a minimal amount of packages:
193
-
194
- ```bash
195
- python -m venv .detect
196
- source .detect/bin/activate
197
-
198
- pip install "numpy>=1.17" "PyWavelets>=1.1.1" "opencv-python>=4.1.0.25"
199
- pip install --no-deps invisible-watermark
200
- ```
201
-
202
- To run the script you need to have a working installation as above. The script
203
- is then useable in the following ways (don't forget to activate your
204
- virtual environment beforehand, e.g. `source .pt1/bin/activate`):
205
-
206
- ```bash
207
- # test a single file
208
- python scripts/demo/detect.py <your filename here>
209
- # test multiple files at once
210
- python scripts/demo/detect.py <filename 1> <filename 2> ... <filename n>
211
- # test all files in a specific folder
212
- python scripts/demo/detect.py <your folder name here>/*
213
- ```
214
-
215
- ## Training:
216
-
217
- We are providing example training configs in `configs/example_training`. To launch a training, run
218
-
219
- ```
220
- python main.py --base configs/<config1.yaml> configs/<config2.yaml>
221
- ```
222
-
223
- where configs are merged from left to right (later configs overwrite the same values).
224
- This can be used to combine model, training and data configs. However, all of them can also be
225
- defined in a single config. For example, to run a class-conditional pixel-based diffusion model training on MNIST,
226
- run
227
-
228
- ```bash
229
- python main.py --base configs/example_training/toy/mnist_cond.yaml
230
- ```
231
-
232
- **NOTE 1:** Using the non-toy-dataset
233
- configs `configs/example_training/imagenet-f8_cond.yaml`, `configs/example_training/txt2img-clipl.yaml`
234
- and `configs/example_training/txt2img-clipl-legacy-ucg-training.yaml` for training will require edits depending on the
235
- used dataset (which is expected to stored in tar-file in
236
- the [webdataset-format](https://github.com/webdataset/webdataset)). To find the parts which have to be adapted, search
237
- for comments containing `USER:` in the respective config.
238
-
239
- **NOTE 2:** This repository supports both `pytorch1.13` and `pytorch2`for training generative models. However for
240
- autoencoder training as e.g. in `configs/example_training/autoencoder/kl-f4/imagenet-attnfree-logvar.yaml`,
241
- only `pytorch1.13` is supported.
242
-
243
- **NOTE 3:** Training latent generative models (as e.g. in `configs/example_training/imagenet-f8_cond.yaml`) requires
244
- retrieving the checkpoint from [Hugging Face](https://huggingface.co/stabilityai/sdxl-vae/tree/main) and replacing
245
- the `CKPT_PATH` placeholder in [this line](configs/example_training/imagenet-f8_cond.yaml#81). The same is to be done
246
- for the provided text-to-image configs.
247
-
248
- ### Building New Diffusion Models
249
-
250
- #### Conditioner
251
-
252
- The `GeneralConditioner` is configured through the `conditioner_config`. Its only attribute is `emb_models`, a list of
253
- different embedders (all inherited from `AbstractEmbModel`) that are used to condition the generative model.
254
- All embedders should define whether or not they are trainable (`is_trainable`, default `False`), a classifier-free
255
- guidance dropout rate is used (`ucg_rate`, default `0`), and an input key (`input_key`), for example, `txt` for
256
- text-conditioning or `cls` for class-conditioning.
257
- When computing conditionings, the embedder will get `batch[input_key]` as input.
258
- We currently support two to four dimensional conditionings and conditionings of different embedders are concatenated
259
- appropriately.
260
- Note that the order of the embedders in the `conditioner_config` is important.
261
-
262
- #### Network
263
-
264
- The neural network is set through the `network_config`. This used to be called `unet_config`, which is not general
265
- enough as we plan to experiment with transformer-based diffusion backbones.
266
-
267
- #### Loss
268
-
269
- The loss is configured through `loss_config`. For standard diffusion model training, you will have to
270
- set `sigma_sampler_config`.
271
-
272
- #### Sampler config
273
-
274
- As discussed above, the sampler is independent of the model. In the `sampler_config`, we set the type of numerical
275
- solver, number of steps, type of discretization, as well as, for example, guidance wrappers for classifier-free
276
- guidance.
277
-
278
- ### Dataset Handling
279
-
280
- For large scale training we recommend using the data pipelines from
281
- our [data pipelines](https://github.com/Stability-AI/datapipelines) project. The project is contained in the requirement
282
- and automatically included when following the steps from the [Installation section](#installation).
283
- Small map-style datasets should be defined here in the repository (e.g., MNIST, CIFAR-10, ...), and return a dict of
284
- data keys/values,
285
- e.g.,
286
-
287
- ```python
288
- example = {"jpg": x, # this is a tensor -1...1 chw
289
- "txt": "a beautiful image"}
290
- ```
291
-
292
- where we expect images in -1...1, channel-first format.
 
1
+ ---
2
+ title: Stable Video Diffusion
3
+ emoji: 📺
4
+ colorFrom: purple
5
+ colorTo: purple
6
+ sdk: gradio
7
+ sdk_version: 4.4.0
8
+ app_file: app.py
9
+ pinned: false
10
+ license: other
11
+ ---