# Dreamlike Diffusion 1.0 is SD 1.5 fine tuned on high quality art, made by [dreamlike.art](https://dreamlike.art/).

# If you want to use dreamlike models on your website/app/etc., check the license at the bottom first!

Use the same prompts as you would for SD 1.5. Add **dreamlikeart** if the artstyle is too weak.
Non-square aspect ratios work better for some prompts. If you want a portrait photo, try using a 2:3 or a 9:16 aspect ratio. If you want a landscape photo, try using a 3:2 or a 16:9 aspect ratio.
Use slightly higher resolution for better results: 640x640px, 512x768px, 768x512px, etc.

# We've just released Dreamlike Photoreal 2.0, check it out!

[https://huggingface.co/dreamlike-art/dreamlike-photoreal-2.0](https://huggingface.co/dreamlike-art/dreamlike-photoreal-2.0)

### Examples

### dreamlike.art

You can use this model for free on [dreamlike.art](https://dreamlike.art/)!

### Gradio

We support a [Gradio](https://github.com/gradio-app/gradio) Web UI to run dreamlike-diffusion-1.0:
[![Open In Spaces](https://camo.githubusercontent.com/00380c35e60d6b04be65d3d94a58332be5cc93779f630bcdfc18ab9a3a7d3388/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f25463025394625413425393725323048756767696e67253230466163652d5370616365732d626c7565)](https://huggingface.co/spaces/akhaliq/dreamlike-diffusion-1.0)

### CompVis

[Download dreamlike-diffusion-1.0.ckpt (2.13GB)](https://huggingface.co/dreamlike-art/dreamlike-diffusion-1.0/resolve/main/dreamlike-diffusion-1.0.ckpt)

### 🧨 Diffusers

This model can be used just like any other Stable Diffusion model. For more information,
please have a look at the [Stable Diffusion Pipeline](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion).

```python
from diffusers import StableDiffusionPipeline
import torch

model_id = "dreamlike-art/dreamlike-diffusion-1.0"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

prompt = "dreamlikeart, a grungy woman with rainbow hair, travelling between dimensions, dynamic pose, happy, soft eyes and narrow chin, extreme bokeh, dainty figure, long hair straight down, torn kawaii shirt and baggy jeans, In style of by Jordan Grimmer and greg rutkowski, crisp lines and color, complex background, particles, lines, wind, concept art, sharp focus, vivid colors"
image = pipe(prompt).images[0]

image.save("./result.jpg")
```

# License

This model is licesed under a **modified** CreativeML OpenRAIL-M license.

- **You can't host or use the model or its derivatives on websites/apps/etc., from which you earn, will earn, or plan to earn revenue or donations. If you want to, please email us at contact@dreamlike.art**
- **You are free to host the model card and files (Without any actual inference or finetuning) on both commercial and non-commercial websites/apps/etc.  Please state the full model name (Dreamlike Diffusion 1.0) and include a link to the model card (https://huggingface.co/dreamlike-art/dreamlike-diffusion-1.0)**
- **You are free to host the model or its derivatives on completely non-commercial websites/apps/etc (Meaning you are not getting ANY revenue or donations). Please state the full model name (Dreamlike Diffusion 1.0) and include a link to the model card (https://huggingface.co/dreamlike-art/dreamlike-diffusion-1.0)**
- **You are free to use the outputs of the model or the outputs of the model's derivatives for commercial purposes in teams of 10 or less**
- You can't use the model to deliberately produce nor share illegal or harmful outputs or content
- The authors claims no rights on the outputs you generate, you are free to use them and are accountable for their use which must not go against the provisions set in the license
- You may re-distribute the weights. If you do, please be aware you have to include the same use restrictions as the ones in the license and share a copy of the **modified** CreativeML OpenRAIL-M to all your users (please read the license entirely and carefully) Please read the full license here: https://huggingface.co/dreamlike-art/dreamlike-diffusion-1.0/blob/main/LICENSE.md


# Stable Diffusion

Stable Diffusion is a latent text-to-image diffusion model capable of generating photo-realistic images given any text input.
This model card gives an overview of all available model checkpoints. For more in-detail model cards, please have a look at the model repositories listed under [Model Access](#model-access).

## Stable Diffusion Version 1

For the first version 4 model checkpoints are released.
*Higher* versions have been trained for longer and are thus usually better in terms of image generation quality then *lower* versions. More specifically:

- **stable-diffusion-v1-1**: The checkpoint is randomly initialized and has been trained on 237,000 steps at resolution `256x256` on [laion2B-en](https://huggingface.co/datasets/laion/laion2B-en).
  194,000 steps at resolution `512x512` on [laion-high-resolution](https://huggingface.co/datasets/laion/laion-high-resolution) (170M examples from LAION-5B with resolution `>= 1024x1024`).
- **stable-diffusion-v1-2**: The checkpoint resumed training from `stable-diffusion-v1-1`.
  515,000 steps at resolution `512x512` on "laion-improved-aesthetics" (a subset of laion2B-en,
filtered to images with an original size `>= 512x512`, estimated aesthetics score `> 5.0`, and an estimated watermark probability `< 0.5`. The watermark estimate is from the LAION-5B metadata, the aesthetics score is estimated using an [improved aesthetics estimator](https://github.com/christophschuhmann/improved-aesthetic-predictor)).
- **stable-diffusion-v1-3**: The checkpoint resumed training from `stable-diffusion-v1-2`. 195,000 steps at resolution `512x512` on "laion-improved-aesthetics" and 10 % dropping of the text-conditioning to improve [classifier-free guidance sampling](https://arxiv.org/abs/2207.12598)
- **stable-diffusion-v1-4**: The checkpoint resumed training from `stable-diffusion-v1-2`. 195,000 steps at resolution `512x512` on "laion-improved-aesthetics" and 10 % dropping of the text-conditioning to improve [classifier-free guidance sampling](https://arxiv.org/abs/2207.12598).
- [**`stable-diffusion-v1-4`**](https://huggingface.co/CompVis/stable-diffusion-v1-4) Resumed from `stable-diffusion-v1-2`.225,000 steps at resolution `512x512` on "laion-aesthetics v2 5+" and 10 % dropping of the text-conditioning to improve [classifier-free guidance sampling](https://arxiv.org/abs/2207.12598).

### Model Access

Each checkpoint can be used both with Hugging Face's [ 🧨 Diffusers library](https://github.com/huggingface/diffusers) or the original [Stable Diffusion GitHub repository](https://github.com/CompVis/stable-diffusion). Note that you have to *"click-request"* them on each respective model repository.

| **[🤗's 🧨 Diffusers library](https://github.com/huggingface/diffusers)**     | **[Stable Diffusion GitHub repository](https://github.com/CompVis/stable-diffusion)** |
|


**Mo Di Diffusion**

This is the fine-tuned Stable Diffusion 1.5 model trained on screenshots from a popular animation studio.
Use the tokens **_modern disney style_** in your prompts for the effect.

**If you enjoy my work, please consider supporting me**
[![Become A Patreon](https://badgen.net/badge/become/a%20patron/F96854)](https://patreon.com/user?u=79196446)

**Videogame Characters rendered with the model:**
![Videogame Samples](https://huggingface.co/nitrosocke/mo-di-diffusion/resolve/main/modi-samples-01s.jpg)
**Animal Characters rendered with the model:**
![Animal Samples](https://huggingface.co/nitrosocke/mo-di-diffusion/resolve/main/modi-samples-02s.jpg)
**Cars and Landscapes rendered with the model:**
![Misc. Samples](https://huggingface.co/nitrosocke/mo-di-diffusion/resolve/main/modi-samples-03s.jpg)
#### Prompt and settings for Lara Croft:
**modern disney lara croft**
_Steps: 50, Sampler: Euler a, CFG scale: 7, Seed: 3940025417, Size: 512x768_

#### Prompt and settings for the Lion:
**modern disney (baby lion) Negative prompt: person human**
_Steps: 50, Sampler: Euler a, CFG scale: 7, Seed: 1355059992, Size: 512x512_

This model was trained using the diffusers based dreambooth training by ShivamShrirao using prior-preservation loss and the _train-text-encoder_ flag in 9.000 steps.

### 🧨 Diffusers

This model can be used just like any other Stable Diffusion model. For more information,
please have a look at the [Stable Diffusion](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion).

You can also export the model to [ONNX](https://huggingface.co/docs/diffusers/optimization/onnx), [MPS](https://huggingface.co/docs/diffusers/optimization/mps) and/or [FLAX/JAX]().

```python
from diffusers import StableDiffusionPipeline
import torch

model_id = "nitrosocke/mo-di-diffusion"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

prompt = "a magical princess with golden hair, modern disney style"
image = pipe(prompt).images[0]

image.save("./magical_princess.png")
```

# Gradio & Colab

We also support a [Gradio](https://github.com/gradio-app/gradio) Web UI and Colab with Diffusers to run fine-tuned Stable Diffusion models:
[![Open In Spaces](https://camo.githubusercontent.com/00380c35e60d6b04be65d3d94a58332be5cc93779f630bcdfc18ab9a3a7d3388/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f25463025394625413425393725323048756767696e67253230466163652d5370616365732d626c7565)](https://huggingface.co/spaces/anzorq/finetuned_diffusion)
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1j5YvfMZoGdDGdj3O3xRU1m4ujKYsElZO?usp=sharing)

## License

This model is open access and available to all, with a CreativeML OpenRAIL-M license further specifying rights and usage.
The CreativeML OpenRAIL License specifies:

1. You can't use the model to deliberately produce nor share illegal or harmful outputs or content
2. The authors claims no rights on the outputs you generate, you are free to use them and are accountable for their use which must not go against the provisions set in the license
3. You may re-distribute the weights and use the model commercially and/or as a service. If you do, please be aware you have to include the same use restrictions as the ones in the license and share a copy of the CreativeML OpenRAIL-M to all your users (please read the license entirely and carefully)
[Please read the full license here](https://huggingface.co/spaces/CompVis/stable-diffusion-license)


![image](https://user-images.githubusercontent.com/26317155/210155933-db3a5f1a-1ec3-4777-915c-6deff2841ce9.png)

masterpiece, best quality, 1girl, green hair, sweater, looking at viewer, upper body, beanie, outdoors, watercolor, night, turtleneck

# Waifu Diffusion v1.4

Waifu Diffusion is a latent text-to-image diffusion model that has been conditioned on high-quality anime images through fine-tuning.

- [Waifu Diffusion 1.4 Anime Epoch 1](https://huggingface.co/hakurei/waifu-diffusion-v1-4/blob/main/wd-1-4-anime_e1.ckpt): A test model made to properly ensure that the training setup works.
- [Waifu Diffusion 1.4 Anime Inference Config](https://huggingface.co/hakurei/waifu-diffusion-v1-4/blob/main/wd-1-4-anime_e1.yaml): A file included to allow for inference with Automatic's WebUI and with the original Stable Diffusion codebase.

## License

This model is open access and available to all, with a CreativeML OpenRAIL-M license further specifying rights and usage.
The CreativeML OpenRAIL License specifies:

1. You can't use the model to deliberately produce nor share illegal or harmful outputs or content
2. The authors claims no rights on the outputs you generate, you are free to use them and are accountable for their use which must not go against the provisions set in the license
3. You may re-distribute the weights and use the model commercially and/or as a service. If you do, please be aware you have to include the same use restrictions as the ones in the license and share a copy of the CreativeML OpenRAIL-M to all your users (please read the license entirely and carefully)
[Please read the full license here](https://huggingface.co/spaces/CompVis/stable-diffusion-license)

## Downstream Uses

This model can be used for entertainment purposes and as a generative art assistant.

## Team Members and Acknowledgements

This project would not have been possible without the incredible work by Stability AI and NovelAI.

- [Haru](https://github.com/harubaru)
- [Salt](https://github.com/sALTaccount/)
- [Cafe](https://twitter.com/cafeai_labs)

In order to reach us, you can join our [Discord server](https://discord.gg/touhouai).

[![Discord Server](https://discordapp.com/api/guilds/930499730843250783/widget.png?style=banner2)](https://discord.gg/touhouai)


# Openjourney v4
## Trained on +124k Midjourney v4 images, by [PromptHero](https://prompthero.com/?utm_source=huggingface&utm_medium=referral)

Trained on Stable Diffusion v1.5 using +124000 images, 12400 steps, 4 epochs +32 training hours.

💡 [Openjourney-v4 prompts](https://prompthero.com/openjourney-prompts?version=4)

Pss... "mdjrny-v4 style" is not necessary anymore (yay!)

🎓 **Want to learn how to train Openjourney? 👉🏼 __[Join our course](https://prompthero.com/academy/dreambooth-stable-diffusion-train-fine-tune-course?utm_source=huggingface&utm_medium=referral)__ 🔥**

# Openjourney Links
- [Lora version](https://huggingface.co/prompthero/openjourney-lora)
- [Openjourney Dreambooth](https://huggingface.co/prompthero/openjourney)


# Inkpunk Diffusion

Finetuned Stable Diffusion model trained on dreambooth. Vaguely inspired by Gorillaz, FLCL, and Yoji Shinkawa. Use **_nvinkpunk_** in your prompts.

# Gradio

We support a [Gradio](https://github.com/gradio-app/gradio) Web UI to run Inkpunk-Diffusion:
[![Open In Spaces](https://camo.githubusercontent.com/00380c35e60d6b04be65d3d94a58332be5cc93779f630bcdfc18ab9a3a7d3388/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f25463025394625413425393725323048756767696e67253230466163652d5370616365732d626c7565)](https://huggingface.co/spaces/akhaliq/Inkpunk-Diffusion)

# Sample images
![output Samples v2](https://huggingface.co/Envvi/Inkpunk-Diffusion/resolve/main/inkpunk-v2-samples-1.png)
![output Samples v2](https://huggingface.co/Envvi/Inkpunk-Diffusion/resolve/main/inkpunk-v2-samples-2.png)


**Analog Diffusion**
![Header](https://huggingface.co/wavymulder/Analog-Diffusion/resolve/main/images/page1.jpg)
[*CKPT DOWNLOAD LINK*](https://huggingface.co/wavymulder/Analog-Diffusion/resolve/main/analog-diffusion-1.0.ckpt) - This is a dreambooth model trained on a diverse set of analog photographs.

In your prompt, use the activation token: `analog style`

You may need to use the words `blur` `haze` `naked` in your negative prompts. My dataset did not include any NSFW material but the model seems to be pretty horny. Note that using `blur` and `haze` in your negative prompt can give a sharper image but also a less pronounced analog film effect.

Trained from 1.5 with VAE.

Please see [this document where I share the parameters (prompt, sampler, seed, etc.) used for all example images.](https://huggingface.co/wavymulder/Analog-Diffusion/resolve/main/parameters_used_examples.txt)

## Gradio

We support a [Gradio](https://github.com/gradio-app/gradio) Web UI to run Analog-Diffusion:

[Open in Spaces](https://huggingface.co/spaces/akhaliq/Analog-Diffusion)

![Environments Example](https://huggingface.co/wavymulder/Analog-Diffusion/resolve/main/images/page2.jpg)
![Characters Example](https://huggingface.co/wavymulder/Analog-Diffusion/resolve/main/images/page3.jpg)

Here's a [link to non-cherrypicked batches.](https://imgur.com/a/7iOgTFv)


# Arcane Diffusion
This is the fine-tuned Stable Diffusion model trained on images from the TV Show Arcane.
Use the tokens **_arcane style_** in your prompts for the effect.

**If you enjoy my work, please consider supporting me**
[![Become A Patreon](https://badgen.net/badge/become/a%20patron/F96854)](https://patreon.com/user?u=79196446)

### 🧨 Diffusers

This model can be used just like any other Stable Diffusion model. For more information,
please have a look at the [Stable Diffusion](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion).

You can also export the model to [ONNX](https://huggingface.co/docs/diffusers/optimization/onnx), [MPS](https://huggingface.co/docs/diffusers/optimization/mps) and/or [FLAX/JAX]().

```python
#!pip install diffusers transformers scipy torch
from diffusers import StableDiffusionPipeline
import torch

model_id = "nitrosocke/Arcane-Diffusion"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

prompt = "arcane style, a magical princess with golden hair"
image = pipe(prompt).images[0]

image.save("./magical_princess.png")
```

# Gradio & Colab

We also support a [Gradio](https://github.com/gradio-app/gradio) Web UI and Colab with Diffusers to run fine-tuned Stable Diffusion models:
[![Open In Spaces](https://camo.githubusercontent.com/00380c35e60d6b04be65d3d94a58332be5cc93779f630bcdfc18ab9a3a7d3388/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f25463025394625413425393725323048756767696e67253230466163652d5370616365732d626c7565)](https://huggingface.co/spaces/anzorq/finetuned_diffusion)
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1j5YvfMZoGdDGdj3O3xRU1m4ujKYsElZO?usp=sharing)

![img](https://huggingface.co/nitrosocke/Arcane-Diffusion/resolve/main/magical_princess.png)

### Sample images from v3:
![output Samples v3](https://huggingface.co/nitrosocke/Arcane-Diffusion/resolve/main/arcane-v3-samples-01.jpg)
![output Samples v3](https://huggingface.co/nitrosocke/Arcane-Diffusion/resolve/main/arcane-v3-samples-02.jpg)
### Sample images from the model:
![output Samples](https://huggingface.co/nitrosocke/Arcane-Diffusion/resolve/main/arcane-diffusion-output-images.jpg)
### Sample images used for training:
![Training Samples](https://huggingface.co/nitrosocke/Arcane-Diffusion/resolve/main/arcane-diffusion-training-images.jpg)

**Version 3** (arcane-diffusion-v3): This version uses the new _train-text-encoder_ setting and improves the quality and edibility of the model immensely. Trained on 95 images from the show in 8000 steps.

**Version 2** (arcane-diffusion-v2): This uses the diffusers based dreambooth training and prior-preservation loss is way more effective. The diffusers where then converted with a script to a ckpt file in order to work with automatics repo.
Training was done with 5k steps for a direct comparison to v1 and results show that it needs more steps for a more prominent result. Version 3 will be tested with 11k steps.

**Version 1** (arcane-diffusion-5k): This model was trained using _Unfrozen Model Textual Inversion_ utilizing the _Training with prior-preservation loss_ methods. There is still a slight shift towards the style, while not using the arcane token.


# Improved Autoencoders

## Utilizing
These weights are intended to be used with the original [CompVis Stable Diffusion codebase](https://github.com/CompVis/stable-diffusion). If you are looking for the model to use with the 🧨 diffusers library, [come here](https://huggingface.co/CompVis/stabilityai/sd-vae-ft-ema).

## Decoder Finetuning
We publish two kl-f8 autoencoder versions, finetuned from the original [kl-f8 autoencoder](https://github.com/CompVis/latent-diffusion#pretrained-autoencoding-models) on a 1:1 ratio of [LAION-Aesthetics](https://laion.ai/blog/laion-aesthetics/) and LAION-Humans, an unreleased subset containing only SFW images of humans. The intent was to fine-tune on the Stable Diffusion training set (the autoencoder was originally trained on OpenImages) but also enrich the dataset with images of humans to improve the reconstruction of faces.
The first, _ft-EMA_, was resumed from the original checkpoint, trained for 313198 steps and uses EMA weights. It uses the same loss configuration as the original checkpoint (L1 + LPIPS).
The second, _ft-MSE_, was resumed from _ft-EMA_ and uses EMA weights and was trained for another 280k steps using a different loss, with more emphasis
on MSE reconstruction (MSE + 0.1 * LPIPS). It produces somewhat ``smoother'' outputs. The batch size for both versions was 192 (16 A100s, batch size 12 per GPU).
To keep compatibility with existing models, only the decoder part was finetuned; the checkpoints can be used as a drop-in replacement for the existing autoencoder..

_Original kl-f8 VAE vs f8-ft-EMA vs f8-ft-MSE_

## Evaluation
### COCO 2017 (256x256, val, 5000 images)
| Model    | train steps | rFID | PSNR         | SSIM          | PSIM          | Link                                                                              | Comments
|


# Waifu Diffusion v1.3

Waifu Diffusion is a latent text-to-image diffusion model that has been conditioned on high-quality anime images through fine-tuning.

- [Float 16 EMA Pruned](https://huggingface.co/hakurei/waifu-diffusion-v1-3/blob/main/wd-v1-3-float16.ckpt)
- [Float 32 EMA Pruned](https://huggingface.co/hakurei/waifu-diffusion-v1-3/blob/main/wd-v1-3-float32.ckpt)
- [Float 32 Full Weights](https://huggingface.co/hakurei/waifu-diffusion-v1-3/blob/main/wd-v1-3-full.ckpt)
- [Float 32 Full Weights + Optimizer Weights (For Training)](https://huggingface.co/hakurei/waifu-diffusion-v1-3/blob/main/wd-v1-3-full-opt.ckpt)

## Model Description

The model originally used for fine-tuning is [Stable Diffusion 1.4](https://huggingface.co/CompVis/stable-diffusion-v1-4), which is a latent image diffusion model trained on [LAION2B-en](https://huggingface.co/datasets/laion/laion2B-en). The current model has been fine-tuned with a learning rate of 5.0e-6 for 10 epochs on 680k anime-styled images.

[See here for an in-depth overview of Waifu Diffusion 1.3.](https://gist.github.com/harubaru/f727cedacae336d1f7877c4bbe2196e1)

## License

This model is open access and available to all, with a CreativeML OpenRAIL-M license further specifying rights and usage.
The CreativeML OpenRAIL License specifies:

1. You can't use the model to deliberately produce nor share illegal or harmful outputs or content
2. The authors claims no rights on the outputs you generate, you are free to use them and are accountable for their use which must not go against the provisions set in the license
3. You may re-distribute the weights and use the model commercially and/or as a service. If you do, please be aware you have to include the same use restrictions as the ones in the license and share a copy of the CreativeML OpenRAIL-M to all your users (please read the license entirely and carefully)
[Please read the full license here](https://huggingface.co/spaces/CompVis/stable-diffusion-license)

## Downstream Uses

This model can be used for entertainment purposes and as a generative art assistant.

## Team Members and Acknowledgements

This project would not have been possible without the incredible work by the [CompVis Researchers](https://ommer-lab.com/).

- [Anthony Mercurio](https://github.com/harubaru)
- [Salt](https://github.com/sALTaccount/)
- [Cafe](https://twitter.com/cafeai_labs)

In order to reach us, you can join our [Discord server](https://discord.gg/touhouai).

[![Discord Server](https://discordapp.com/api/guilds/930499730843250783/widget.png?style=banner2)](https://discord.gg/touhouai)


### Redshift Diffusion

This is the fine-tuned Stable Diffusion model trained on high resolution 3D artworks.
Use the tokens **_redshift style_** in your prompts for the effect.

**The name:** I used Cinema4D for a very long time as my go-to modeling software and always liked the redshift render it came with. That is why I was very sad to see the bad results base SD has connected with its token. This is my attempt at fixing that and showing my passion for this render engine.

**If you enjoy my work and want to test new models before release, please consider supporting me**
[![Become A Patreon](https://badgen.net/badge/become/a%20patron/F96854)](https://patreon.com/user?u=79196446)

**Characters rendered with the model:**
![Videogame Samples](https://huggingface.co/nitrosocke/redshift-diffusion/resolve/main/images/redshift-diffusion-samples-01s.jpg)
**Cars and Landscapes rendered with the model:**
![Misc. Samples](https://huggingface.co/nitrosocke/redshift-diffusion/resolve/main/images/redshift-diffusion-samples-02s.jpg)

#### Prompt and settings for Tony Stark:
**(redshift style) robert downey jr as ironman Negative prompt: glasses helmet**
_Steps: 40, Sampler: DPM2 Karras, CFG scale: 7, Seed: 908018284, Size: 512x704_

#### Prompt and settings for the Ford Mustang:
**redshift style Ford Mustang**
_Steps: 20, Sampler: DPM2 Karras, CFG scale: 7, Seed: 579593863, Size: 704x512_

This model was trained using the diffusers based dreambooth training by ShivamShrirao using prior-preservation loss and the _train-text-encoder_ flag in 11.000 steps.

### Gradio

We support a [Gradio](https://github.com/gradio-app/gradio) Web UI run redshift-diffusion:
[![Open In Spaces](https://camo.githubusercontent.com/00380c35e60d6b04be65d3d94a58332be5cc93779f630bcdfc18ab9a3a7d3388/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f25463025394625413425393725323048756767696e67253230466163652d5370616365732d626c7565)](https://huggingface.co/spaces/nitrosocke/Redshift-Diffusion-Demo)

### 🧨 Diffusers

This model can be used just like any other Stable Diffusion model. For more information,
please have a look at the [Stable Diffusion](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion).

You can also export the model to [ONNX](https://huggingface.co/docs/diffusers/optimization/onnx), [MPS](https://huggingface.co/docs/diffusers/optimization/mps) and/or [FLAX/JAX]().

```python
from diffusers import StableDiffusionPipeline
import torch

model_id = "nitrosocke/redshift-diffusion"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

prompt = "redshift style magical princess with golden hair"
image = pipe(prompt).images[0]

image.save("./magical_princess.png")
```

## License

This model is open access and available to all, with a CreativeML OpenRAIL-M license further specifying rights and usage.
The CreativeML OpenRAIL License specifies:

1. You can't use the model to deliberately produce nor share illegal or harmful outputs or content
2. The authors claims no rights on the outputs you generate, you are free to use them and are accountable for their use which must not go against the provisions set in the license
3. You may re-distribute the weights and use the model commercially and/or as a service. If you do, please be aware you have to include the same use restrictions as the ones in the license and share a copy of the CreativeML OpenRAIL-M to all your users (please read the license entirely and carefully)
[Please read the full license here](https://huggingface.co/spaces/CompVis/stable-diffusion-license)


![visitors](https://visitor-badge.glitch.me/badge?page_id=Cyberpunk_Anime_Diffusion)

# Cyberpunk Anime Diffusion

An AI model that generates cyberpunk anime characters!~

Based of a finetuned Waifu Diffusion V1.3 Model with Stable Diffusion V1.5 New Vae, training in Dreambooth

by [DGSpitzer](https://www.youtube.com/channel/UCzzsYBF4qwtMwJaPJZ5SuPg)

### 🧨 Diffusers

This repo contains both .ckpt and Diffuser model files. It's compatible to be used as any Stable Diffusion model, using standard [Stable Diffusion Pipelines](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion).

You can convert this model to [ONNX](https://huggingface.co/docs/diffusers/optimization/onnx), [MPS](https://huggingface.co/docs/diffusers/optimization/mps) and/or [FLAX/JAX](https://huggingface.co/blog/stable_diffusion_jax).

```python example for loading the Diffuser
#!pip install diffusers transformers scipy torch
from diffusers import StableDiffusionPipeline
import torch

model_id = "DGSpitzer/Cyberpunk-Anime-Diffusion"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

prompt = "a beautiful perfect face girl in dgs illustration style, Anime fine details portrait of school girl in front of modern tokyo city landscape on the background deep bokeh, anime masterpiece, 8k, sharp high quality anime"
image = pipe(prompt).images[0]

image.save("./cyberpunk_girl.png")
```

# Online Demo

You can try the Online Web UI demo build with [Gradio](https://github.com/gradio-app/gradio), or use Colab Notebook at here:

*My Online Space Demo*
[![Open In Spaces](https://camo.githubusercontent.com/00380c35e60d6b04be65d3d94a58332be5cc93779f630bcdfc18ab9a3a7d3388/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f25463025394625413425393725323048756767696e67253230466163652d5370616365732d626c7565)](https://huggingface.co/spaces/DGSpitzer/DGS-Diffusion-Space)

*Finetuned Diffusion WebUI Demo by anzorq*
[![Use Finetuned_Diffusion WebUI](https://camo.githubusercontent.com/00380c35e60d6b04be65d3d94a58332be5cc93779f630bcdfc18ab9a3a7d3388/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f25463025394625413425393725323048756767696e67253230466163652d5370616365732d626c7565)](https://huggingface.co/spaces/anzorq/finetuned_diffusion)

*Colab Notebook*
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/HelixNGC7293/cyberpunk-anime-diffusion/blob/main/cyberpunk_anime_diffusion.ipynb)[![GitHub](https://badgen.net/badge/icon/Github?icon=github&label)](https://github.com/HelixNGC7293/cyberpunk-anime-diffusion)

*Buy me a coffee if you like this project ;P ♥*
[![Buy me a coffee](https://badgen.net/badge/icon/Buy%20Me%20A%20Coffee?icon=buymeacoffee&label)](https://www.buymeacoffee.com/dgspitzer)

# **👇Model👇**

AI Model Weights available at huggingface: https://huggingface.co/DGSpitzer/Cyberpunk-Anime-Diffusion

# Usage

After model loaded, use keyword **dgs** in your prompt, with **illustration style** to get even better results.

For sampler, use **Euler A** for the best result (**DDIM** kinda works too), CFG Scale 7, steps 20 should be fine

**Example 1:**

```
portrait of a girl in dgs illustration style, Anime girl, female soldier working in a cyberpunk city, cleavage, ((perfect femine face)), intricate, 8k, highly detailed, shy, digital painting, intense, sharp focus
```

For cyber robot male character, you can add **muscular male** to improve the output.

**Example 2:**

```
a photo of muscular beard soldier male in dgs illustration style, half-body, holding robot arms, strong chest
```

**Example 3 (with Stable Diffusion WebUI):**

If using [AUTOMATIC1111's Stable Diffusion WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui)

You can simply use this as **prompt** with **Euler A** Sampler,  CFG Scale 7, steps 20, 704 x 704px output res:

```
an anime girl in dgs illustration style
```

And set the **negative prompt** as this to get cleaner face:

```
out of focus, scary, creepy, evil, disfigured, missing limbs, ugly, gross, missing fingers
```

This will give you the exactly same style as the sample images above.


# Anything V5 (https://civitai.com/models/9409)
# Uploaded by the Real Anything V3 Author
# Please try it


### Ghibli Diffusion

This is the fine-tuned Stable Diffusion model trained on images from modern anime feature films from Studio Ghibli.
Use the tokens **_ghibli style_** in your prompts for the effect.

**If you enjoy my work and want to test new models before release, please consider supporting me**
[![Become A Patreon](https://badgen.net/badge/become/a%20patron/F96854)](https://patreon.com/user?u=79196446)

**Characters rendered with the model:**
![Characters Samples](https://huggingface.co/nitrosocke/Ghibli-Diffusion/resolve/main/images/ghibli-diffusion-samples-01s.jpg)
**Cars and Animals rendered with the model:**
![Misc. Samples](https://huggingface.co/nitrosocke/Ghibli-Diffusion/resolve/main/images/ghibli-diffusion-samples-02s.jpg)
**Landscapes rendered with the model:**
![Landscape 1](https://huggingface.co/nitrosocke/Ghibli-Diffusion/resolve/main/images/ghibli-diffusion-samples-03s.jpg)
_ghibli style beautiful Caribbean beach tropical (sunset) - Negative prompt: soft blurry_
![Landscape 2](https://huggingface.co/nitrosocke/Ghibli-Diffusion/resolve/main/images/ghibli-diffusion-samples-04s.jpg)
_ghibli style ice field white mountains ((northern lights)) starry sky low horizon - Negative prompt: soft blurry_

#### Prompt and settings for the Strom Trooper:
**ghibli style (storm trooper) Negative prompt: (bad anatomy)**
_Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 3450349066, Size: 512x704_

#### Prompt and settings for the VW Beetle:
**ghibli style VW beetle Negative prompt: soft blurry**
_Steps: 30, Sampler: Euler a, CFG scale: 7, Seed: 1529856912, Size: 704x512_

This model was trained using the diffusers based dreambooth training by ShivamShrirao using prior-preservation loss and the _train-text-encoder_ flag in 15.000 steps.

<!-- ### Gradio

We support a [Gradio](https://github.com/gradio-app/gradio) Web UI run redshift-diffusion:
[![Open In Spaces](https://camo.githubusercontent.com/00380c35e60d6b04be65d3d94a58332be5cc93779f630bcdfc18ab9a3a7d3388/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f25463025394625413425393725323048756767696e67253230466163652d5370616365732d626c7565)](https://huggingface.co/spaces/nitrosocke/Ghibli-Diffusion-Demo)-->

### 🧨 Diffusers

This model can be used just like any other Stable Diffusion model. For more information,
please have a look at the [Stable Diffusion](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion).

You can also export the model to [ONNX](https://huggingface.co/docs/diffusers/optimization/onnx), [MPS](https://huggingface.co/docs/diffusers/optimization/mps) and/or [FLAX/JAX]().

```python
from diffusers import StableDiffusionPipeline
import torch

model_id = "nitrosocke/Ghibli-Diffusion"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

prompt = "ghibli style magical princess with golden hair"
image = pipe(prompt).images[0]

image.save("./magical_princess.png")
```

## License

This model is open access and available to all, with a CreativeML OpenRAIL-M license further specifying rights and usage.
The CreativeML OpenRAIL License specifies:

1. You can't use the model to deliberately produce nor share illegal or harmful outputs or content
2. The authors claims no rights on the outputs you generate, you are free to use them and are accountable for their use which must not go against the provisions set in the license
3. You may re-distribute the weights and use the model commercially and/or as a service. If you do, please be aware you have to include the same use restrictions as the ones in the license and share a copy of the CreativeML OpenRAIL-M to all your users (please read the license entirely and carefully)
[Please read the full license here](https://huggingface.co/spaces/CompVis/stable-diffusion-license)


Hassanblend1.4

# HassanBlend1.4 - fantasy.ai

Fantasy.ai is the official and exclusive hosted AI generation platform that holds a commercial use license for HassanBlend, you can use their service at https://Fantasy.ai/
I am hassan, I created HassansBlend, the latest version currently is 1.4. I continue to iterate and improve on this model over time. Feel free to check out our discord or rentry page for more examples with prompts and outputs generated.

I have also some custom created content such as enhancement hypernetworks/embeddings etc for patreons or KoFi subscribers only on my pages below
 Links

Patreon

KoFi

Discord

### Quicklinks:

* [Latest Setup](https://rentry.org/sdhassan#current-setup)
* [HassanBlend Model Finetune Updates](https://rentry.org/sdhassan#hassanblend-finetuning-updates)
* [Latest Patreon Posts](https://rentry.org/sdhassan#patreon-posts)
* [Models](https://rentry.org/sdhassan#merged-models)
	* [HassanBlend1.4](https://rentry.org/sdhassan#hassanblend14-downloads)
* [Prompts](https://rentry.org/sdhassan#prompts)
* [Photorealistic Tips](https://rentry.org/sdhassan#tips-for-photorealistic-images)
* [Embeddings](https://rentry.org/sdhassan#embeddings)
* [Hypernetworks](https://rentry.org/sdhassan#hypernetworks)
* [Wildcards](https://rentry.org/sdhassan#wildcards-i-made)
* [MyTools](https://rentry.org/sdhassan#my-tools)
* [Settings I use](https://rentry.org/sdhassan#settings)

Model details and examples with sample prompts: https://rentry.org/sdhassan

# Gradio Demo

We support a [Gradio](https://github.com/gradio-app/gradio) Web UI to run hassanblend1.4:
[![Open In Spaces](https://camo.githubusercontent.com/00380c35e60d6b04be65d3d94a58332be5cc93779f630bcdfc18ab9a3a7d3388/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f25463025394625413425393725323048756767696e67253230466163652d5370616365732d626c7565)](https://huggingface.co/spaces/akhaliq/hassanblend1.4)


Counterfeit is anime style Stable Diffusion model.
DreamBooth + Merge Block Weights + Merge LoRA
Please refer to the example below for your prompt.

# Counterfeit-V2.0 e.g.
 ((masterpiece, best quality)),a girl, solo, hat, blush,long hair, skirt, beret, sitting, bangs, socks, wariza, pink hair, light blue eyes, black headwear,holding,rifle,weapon, looking at viewer, white sailor collar, school uniform, closed mouth, black hat, sailor collar, holding weapon, long sleeves, pleated skirt, white socks,indoors,industrial
Negative prompt: (low quality, worst quality:1.4), (bad anatomy), (inaccurate limb:1.2),bad composition, inaccurate eyes, extra digit,fewer digits,(extra arms:1.2),
Steps: 20, Sampler: DPM++ SDE Karras, CFG scale: 8, Size: 576x384 or 576x448, Denoising strength: 0.6, Clip skip: 2, Hires upscale: 2, Hires upscaler: Latent
![sample1](https://huggingface.co/gsdf/Counterfeit-V2.0/resolve/main/sample_001.jpg)

((masterpiece, best quality)),a girl, solo, skirt, sky, sitting, pantyhose, serafuku, cloud,black gloves, outdoors, neckerchief ,day, bangs, fence, shirt, ahoge, rooftop, long hair, white pantyhose, black hair, school uniform, white sailor collar, red eyes, sailor collar, blue skirt, red neckerchief, blue serafuku, animal ears, blue sky, long sleeves, blue shirt, looking at viewer, closed mouth,cat ears, chain-link fence, pleated skirt, cloudy sky, trash can
Negative prompt: (low quality, worst quality:1.4), (bad anatomy), (inaccurate limb:1.2),bad composition, inaccurate eyes, extra digit,fewer digits,(extra arms:1.2),
Steps: 20, Sampler: DPM++ SDE Karras, CFG scale: 8, Size: 384x640, Denoising strength: 0.6, Clip skip: 2, Hires upscale: 2, Hires upscaler: Latent
![sample2](https://huggingface.co/gsdf/Counterfeit-V2.0/resolve/main/sample_002.jpg)

((masterpiece, best quality)), a girl, flower, dress, solo, lying, rain, butterfly, bug, water, bangs, frills, breasts, long hair, white dress, short sleeves, hair ornament, on back, outstretched arm, frilled dress, arm up, white flower, hair flower, grey eyes, white hair,looking away
Negative prompt: (low quality, worst quality:1.4), (bad anatomy), (inaccurate limb:1.2),bad composition, inaccurate eyes, extra digit,fewer digits,(extra arms:1.2),
Steps: 20, Sampler: DPM++ SDE Karras, CFG scale: 8, Size: 640x384, Denoising strength: 0.6, Clip skip: 2, Hires upscale: 2, Hires upscaler: Latent
![sample3](https://huggingface.co/gsdf/Counterfeit-V2.0/resolve/main/sample_003.jpg)

((masterpiece, best quality)), 2girls, barefoot, shorts, sitting, shirt, couch, indoors, messy room, t-shirt, holding, feet, pillow, controller, toes, gun, cup, bangs, soles, rifle, denim, table, camera, multiple girls, black hair, red hair, short hair, long hair, crossed legs, red eyes, short shorts, white shirt, black shorts, game controller, monitor, warm lighting
Negative prompt: (low quality, worst quality:1.4), (bad anatomy), (inaccurate limb:1.2),bad composition, inaccurate eyes, extra digit,fewer digits,(extra arms:1.2),
Steps: 20, Sampler: DPM++ SDE Karras, CFG scale: 8, Size: 640x384, Denoising strength: 0.6, Clip skip: 2, Hires upscale: 2, Hires upscaler: Latent
![sample4](https://huggingface.co/gsdf/Counterfeit-V2.0/resolve/main/sample_004.jpg)

((masterpiece, best quality)),a girl, solo, dress, standing, halo, alley, outdoors, bangs, white dress, white hair, long hair, black footwear, industrial pipe, looking at viewer, air conditioner,dark lighting, garbage, garbage bin
Negative prompt: (low quality, worst quality:1.4), (bad anatomy), (inaccurate limb:1.2),bad composition, inaccurate eyes, extra digit,fewer digits,(extra arms:1.2),
Steps: 20, Sampler: DPM++ SDE Karras, CFG scale: 8, Size: 640x384, Denoising strength: 0.6, Clip skip: 2, Hires upscale: 2, Hires upscaler: Latent
![sample5](https://huggingface.co/gsdf/Counterfeit-V2.0/resolve/main/sample_005.jpg)

((masterpiece, best quality)),a girl, solo, serafuku, thighhighs, skirt, lying, ribbon, upperbody, class room, indoors, shirt, neckerchief, school uniform, long hair, black thighhighs, looking at viewer, blue eyes, black serafuku, black skirt, red ribbon, long sleeves, pleated skirt, blonde hair, wood floor
Negative prompt: (low quality, worst quality:1.4), (bad anatomy), (inaccurate limb:1.2),bad composition, inaccurate eyes, extra digit,fewer digits,(extra arms:1.2),
Steps: 20, Sampler: DPM++ SDE Karras, CFG scale: 8, Size: 640x384, Denoising strength: 0.6, Clip skip: 2, Hires upscale: 2, Hires upscaler: Latent
![sample6](https://huggingface.co/gsdf/Counterfeit-V2.0/resolve/main/sample_006.jpg)

(masterpiece, best quality)),a girl, solo, twintails, shirt, skirt, petals, bowtie, earrings, jewelry, bangs, black hair, hair ornament, hair ribbon, red ribbon, red eyes, long hair, open mouth, white shirt, multicolored hair, black skirt, red hair, long sleeves, pink bowtie, hair between eyes, looking at viewer, collared shirt, upper body, hand up, falling petals, depth of field, strong bloom, red background
Negative prompt: (low quality, worst quality:1.4), (bad anatomy), (inaccurate limb:1.2),bad composition, inaccurate eyes, extra digit,fewer digits,(extra arms:1.2),
Steps: 20, Sampler: DPM++ SDE Karras, CFG scale: 8, Size: 640x384, Denoising strength: 0.6, Clip skip: 2, Hires upscale: 2, Hires upscaler: Latent
![sample7](https://huggingface.co/gsdf/Counterfeit-V2.0/resolve/main/sample_007.jpg)


This model is open access and available to all, with a CreativeML OpenRAIL-M license further specifying rights and usage.
The CreativeML OpenRAIL License specifies:

1. You can't use the model to deliberately produce nor share illegal or harmful outputs or content
2. Riffusion claims no rights on the outputs you generate, you are free to use them and are accountable for their use which must not go against the provisions set in the license
3. You may re-distribute the weights and use the model commercially and/or as a service. If you do, please be aware you have to include the same use restrictions as the ones in the license and share a copy of the CreativeML OpenRAIL-M to all your users (please read the license entirely and carefully)
Please read the full license carefully here: https://huggingface.co/spaces/CompVis/stable-diffusion-license

Please read the LICENSE to access this model

# Riffusion

Riffusion is an app for real-time music generation with stable diffusion.

Read about it at https://www.riffusion.com/about and try it at https://www.riffusion.com/.

* Code: https://github.com/riffusion/riffusion
* Web app: https://github.com/hmartiro/riffusion-app
* Model checkpoint: https://huggingface.co/riffusion/riffusion-model-v1
* Discord: https://discord.gg/yu6SRwvX4v

This repository contains the model files, including:

 * a diffusers formated library
 * a compiled checkpoint file
 * a traced unet for improved inference speed
 * a seed image library for use with riffusion-app

## Riffusion v1 Model

Riffusion is a latent text-to-image diffusion model capable of generating spectrogram images given any text input. These spectrograms can be converted into audio clips.

The model was created by [Seth Forsgren](https://sethforsgren.com/) and [Hayk Martiros](https://haykmartiros.com/) as a hobby project.

You can use the Riffusion model directly, or try the [Riffusion web app](https://www.riffusion.com/).

The Riffusion model was created by fine-tuning the **Stable-Diffusion-v1-5** checkpoint. Read about Stable Diffusion here [🤗's Stable Diffusion blog](https://huggingface.co/blog/stable_diffusion).

### Model Details
- **Developed by:** Seth Forsgren, Hayk Martiros
- **Model type:** Diffusion-based text-to-image generation model
- **Language(s):** English
- **License:** [The CreativeML OpenRAIL M license](https://huggingface.co/spaces/CompVis/stable-diffusion-license) is an [Open RAIL M license](https://www.licenses.ai/blog/2022/8/18/naming-convention-of-responsible-ai-licenses), adapted from the work that [BigScience](https://bigscience.huggingface.co/) and [the RAIL Initiative](https://www.licenses.ai/) are jointly carrying in the area of responsible AI licensing. See also [the article about the BLOOM Open RAIL license](https://bigscience.huggingface.co/blog/the-bigscience-rail-license) on which our license is based.
- **Model Description:** This is a model that can be used to generate and modify images based on text prompts. It is a [Latent Diffusion Model](https://arxiv.org/abs/2112.10752) that uses a fixed, pretrained text encoder ([CLIP ViT-L/14](https://arxiv.org/abs/2103.00020)) as suggested in the [Imagen paper](https://arxiv.org/abs/2205.11487).

### Direct Use
The model is intended for research purposes only. Possible research areas and
tasks include

- Generation of artworks, audio, and use in creative processes.
- Applications in educational or creative tools.
- Research on generative models.

### Datasets
The original Stable Diffusion v1.5 was trained on the [LAION-5B](https://arxiv.org/abs/2210.08402) dataset using the [CLIP text encoder](https://openai.com/blog/clip/), which provided an amazing starting point with an in-depth understanding of language, including musical concepts. The team at LAION also compiled a fantastic audio dataset from many general, speech, and music sources that we recommend at [LAION-AI/audio-dataset](https://github.com/LAION-AI/audio-dataset/blob/main/data_collection/README.md).

### Fine Tuning

Check out the [diffusers training examples](https://huggingface.co/docs/diffusers/training/overview) from Hugging Face. Fine tuning requires a dataset of spectrogram images of short audio clips, with associated text describing them. Note that the CLIP encoder is able to understand and connect many words even if they never appear in the dataset. It is also possible to use a [dreambooth](https://huggingface.co/blog/dreambooth) method to get custom styles.

## Citation

If you build on this work, please cite it as follows:

```
@article{Forsgren_Martiros_2022,
  author = {Forsgren, Seth* and Martiros, Hayk*},
  title = {{Riffusion - Stable diffusion for real-time music generation}},
  url = {https://riffusion.com/about},
  year = {2022}
}
```


# Stable Diffusion Image Variations Model Card

📣 V2 model released, and blurriness issues fixed! 📣

🧨🎉 Image Variations is now natively supported in 🤗 Diffusers! 🎉🧨

![](https://raw.githubusercontent.com/justinpinkney/stable-diffusion/main/assets/im-vars-thin.jpg)

## Version 2

This version of Stable Diffusion has been fine tuned from [CompVis/stable-diffusion-v1-4-original](https://huggingface.co/CompVis/stable-diffusion-v-1-4-original) to accept CLIP image embedding rather than text embeddings. This allows the creation of "image variations" similar to DALLE-2 using Stable Diffusion. This version of the weights has been ported to huggingface Diffusers, to use this with the Diffusers library requires the [Lambda Diffusers repo](https://github.com/LambdaLabsML/lambda-diffusers).

This model was trained in two stages and longer than the original variations model and gives better image quality and better CLIP rated similarity compared to the original version

See training details and v1 vs v2 comparison below.

## Example

Make sure you are using a version of Diffusers >=0.8.0 (for older version see the old instructions at the bottom of this model card)

```python
from diffusers import StableDiffusionImageVariationPipeline
from PIL import Image

device = "cuda:0"
sd_pipe = StableDiffusionImageVariationPipeline.from_pretrained(
  "lambdalabs/sd-image-variations-diffusers",
  revision="v2.0",
  )
sd_pipe = sd_pipe.to(device)

im = Image.open("path/to/image.jpg")
tform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Resize(
        (224, 224),
        interpolation=transforms.InterpolationMode.BICUBIC,
        antialias=False,
        ),
    transforms.Normalize(
      [0.48145466, 0.4578275, 0.40821073],
      [0.26862954, 0.26130258, 0.27577711]),
])
inp = tform(im).to(device).unsqueeze(0)

out = sd_pipe(inp, guidance_scale=3)
out["images"][0].save("result.jpg")
```

### The importance of resizing correctly... (or not)

Note that due a bit of an oversight during training, the model expects resized images without anti-aliasing. This turns out to make a big difference and is important to do the resizing the same way during inference. When passing a PIL image to the Diffusers pipeline antialiasing will be applied during resize, so it's better to input a tensor which you have prepared manually according to the transfrom in the example above!

Here are examples of images generated without (top) and with (bottom) anti-aliasing during resize. (Input is [this image](https://github.com/SHI-Labs/Versatile-Diffusion/blob/master/assets/ghibli.jpg))

![](alias-montage.jpg)

![](default-montage.jpg)

### V1 vs V2

Here's an example of V1 vs V2, version two was trained more carefully and for longer, see the details below. V2-top vs V1-bottom

![](v2-montage.jpg)

![](v1-montage.jpg)

Input images:

![](inputs.jpg)

One important thing to note is that due to the longer training V2 appears to have memorised some common images from the training data, e.g. now the previous example of the Girl with a Pearl Earring almosts perfectly reproduce the original rather than creating variations. You can always use v1 by specifiying `revision="v1.0"`.

v2 output for girl with a pearl earing as input (guidance scale=3)

![](earring.jpg)

# Training

**Training Procedure**
This model is fine tuned from Stable Diffusion v1-3 where the text encoder has been replaced with an image encoder. The training procedure is the same as for Stable Diffusion except for the fact that images are encoded through a ViT-L/14 image-encoder including the final projection layer to the CLIP shared embedding space. The model was trained on LAION improved aesthetics 6plus.

- **Hardware:** 8 x A100-40GB GPUs (provided by [Lambda GPU Cloud](https://lambdalabs.com/service/gpu-cloud))
- **Optimizer:** AdamW

- **Stage 1** - Fine tune only CrossAttention layer weights from Stable Diffusion v1.4 model
  - **Steps**: 46,000
  - **Batch:** batch size=4, GPUs=8, Gradient Accumulations=4. Total batch size=128
  - **Learning rate:** warmup to 1e-5 for 10,000 steps and then kept constant

- **Stage 2** - Resume from Stage 1 training the whole unet
  - **Steps**: 50,000
  - **Batch:** batch size=4, GPUs=8, Gradient Accumulations=5. Total batch size=160
  - **Learning rate:** warmup to 1e-5 for 5,000 steps and then kept constant

Training was done using a [modified version of the original Stable Diffusion training code](https://github.com/justinpinkney/stable-diffusion).

# Uses
_The following section is adapted from the [Stable Diffusion model card](https://huggingface.co/CompVis/stable-diffusion-v1-4)_

## Direct Use
The model is intended for research purposes only. Possible research areas and
tasks include

- Safe deployment of models which have the potential to generate harmful content.
- Probing and understanding the limitations and biases of generative models.
- Generation of artworks and use in design and other artistic processes.
- Applications in educational or creative tools.
- Research on generative models.

Excluded uses are described below.

 ### Misuse, Malicious Use, and Out-of-Scope Use

The model should not be used to intentionally create or disseminate images that create hostile or alienating environments for people. This includes generating images that people would foreseeably find disturbing, distressing, or offensive; or content that propagates historical or current stereotypes.

#### Out-of-Scope Use
The model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.

#### Misuse and Malicious Use
Using the model to generate content that is cruel to individuals is a misuse of this model. This includes, but is not limited to:

- Generating demeaning, dehumanizing, or otherwise harmful representations of people or their environments, cultures, religions, etc.
- Intentionally promoting or propagating discriminatory content or harmful stereotypes.
- Impersonating individuals without their consent.
- Sexual content without consent of the people who might see it.
- Mis- and disinformation
- Representations of egregious violence and gore
- Sharing of copyrighted or licensed material in violation of its terms of use.
- Sharing content that is an alteration of copyrighted or licensed material in violation of its terms of use.

## Limitations and Bias

### Limitations

- The model does not achieve perfect photorealism
- The model cannot render legible text
- The model does not perform well on more difficult tasks which involve compositionality, such as rendering an image corresponding to “A red cube on top of a blue sphere”
- Faces and people in general may not be generated properly.
- The model was trained mainly with English captions and will not work as well in other languages.
- The autoencoding part of the model is lossy
- The model was trained on a large-scale dataset
  [LAION-5B](https://laion.ai/blog/laion-5b/) which contains adult material
  and is not fit for product use without additional safety mechanisms and
  considerations.
- No additional measures were used to deduplicate the dataset. As a result, we observe some degree of memorization for images that are duplicated in the training data.
  The training data can be searched at [https://rom1504.github.io/clip-retrieval/](https://rom1504.github.io/clip-retrieval/) to possibly assist in the detection of memorized images.

### Bias

While the capabilities of image generation models are impressive, they can also reinforce or exacerbate social biases.
Stable Diffusion v1 was trained on subsets of [LAION-2B(en)](https://laion.ai/blog/laion-5b/),
which consists of images that are primarily limited to English descriptions.
Texts and images from communities and cultures that use other languages are likely to be insufficiently accounted for.
This affects the overall output of the model, as white and western cultures are often set as the default. Further, the
ability of the model to generate content with non-English prompts is significantly worse than with English-language prompts.

### Safety Module

The intended use of this model is with the [Safety Checker](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/safety_checker.py) in Diffusers.
This checker works by checking model outputs against known hard-coded NSFW concepts.
The concepts are intentionally hidden to reduce the likelihood of reverse-engineering this filter.
Specifically, the checker compares the class probability of harmful concepts in the embedding space of the `CLIPModel` *after generation* of the images.
The concepts are passed into the model with the generated image and compared to a hand-engineered weight for each NSFW concept.

## Old instructions

If you are using a diffusers version <0.8.0 there is no `StableDiffusionImageVariationPipeline`,
in this case you need to use an older revision (`2ddbd90b14bc5892c19925b15185e561bc8e5d0a`) in conjunction with the lambda-diffusers repo:

First clone [Lambda Diffusers](https://github.com/LambdaLabsML/lambda-diffusers) and install any requirements (in a virtual environment in the example below):

```bash
git clone https://github.com/LambdaLabsML/lambda-diffusers.git
cd lambda-diffusers
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

Then run the following python code:

```python
from pathlib import Path
from lambda_diffusers import StableDiffusionImageEmbedPipeline
from PIL import Image
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
pipe = StableDiffusionImageEmbedPipeline.from_pretrained(
"lambdalabs/sd-image-variations-diffusers",
revision="2ddbd90b14bc5892c19925b15185e561bc8e5d0a",
)
pipe = pipe.to(device)

im = Image.open("your/input/image/here.jpg")
num_samples = 4
image = pipe(num_samples*[im], guidance_scale=3.0)
image = image["sample"]

base_path = Path("outputs/im2im")
base_path.mkdir(exist_ok=True, parents=True)
for idx, im in enumerate(image):
    im.save(base_path/f"{idx:06}.jpg")
```

*This model card was written by: Justin Pinkney and is based on the [Stable Diffusion model card](https://huggingface.co/CompVis/stable-diffusion-v1-4).*


# Stable Diffusion Image Variations Model Card

This version of Stable Diffusion has been fine tuned from [CompVis/stable-diffusion-v1-3-original](https://huggingface.co/CompVis/stable-diffusion-v-1-3-original) to accept CLIP image embedding rather than text embeddings. This allows the creation of "image variations" similar to DALLE-2 using Stable Diffusion.

![](https://raw.githubusercontent.com/justinpinkney/stable-diffusion/main/assets/im-vars-thin.jpg)

## Example

To use this model requires a fork of the Stable Diffusion repo: [justinpinkney/stable-diffusion](https://github.com/justinpinkney/stable-diffusion)

```bash
git clone https://github.com/justinpinkney/stable-diffusion.git
cd stable-diffusion
mkdir -p models/ldm/stable-diffusion-v1
wget https://huggingface.co/lambdalabs/stable-diffusion-image-conditioned/resolve/main/sd-clip-vit-l14-img-embed_ema_only.ckpt -O models/ldm/stable-diffusion-v1/sd-clip-vit-l14-img-embed_ema_only.ckpt
pip install -r requirements.txt
python scripts/gradio_variations.py
```

For the version ported to huggingface Diffusers, see [this model](https://huggingface.co/lambdalabs/sd-image-variations-diffusers).

# Training

**Training Data**
The model developers used the following dataset for training the model:

- LAION-2B (en) and subsets thereof (see next section)

**Training Procedure**
This model is fine tuned from Stable Diffusion v1-3 where the text encoder has been replaced with an image encoder. The training procedure is the same as for Stable Diffusion except for the fact that images are encoded through a ViT-L/14 image-encoder including the final projection layer to the CLIP shared embedding space.

- **Hardware:** 4 x A6000 GPUs (provided by [Lambda GPU Cloud](https://lambdalabs.com/service/gpu-cloud))
- **Optimizer:** AdamW
- **Gradient Accumulations**: 1
- **Steps**: 87,000
- **Batch:** 6 x 4 = 24
- **Learning rate:** warmup to 0.0001 for 1,000 steps and then kept constant

Training was done using a [modified version of the original Stable Diffusion training code]((https://github.com/justinpinkney/stable-diffusion)

# Uses
_The following section is adapted from the [Stable Diffusion model card](https://huggingface.co/CompVis/stable-diffusion-v1-4)_

## Direct Use
The model is intended for research purposes only. Possible research areas and
tasks include

- Safe deployment of models which have the potential to generate harmful content.
- Probing and understanding the limitations and biases of generative models.
- Generation of artworks and use in design and other artistic processes.
- Applications in educational or creative tools.
- Research on generative models.

Excluded uses are described below.

 ### Misuse, Malicious Use, and Out-of-Scope Use

The model should not be used to intentionally create or disseminate images that create hostile or alienating environments for people. This includes generating images that people would foreseeably find disturbing, distressing, or offensive; or content that propagates historical or current stereotypes.

#### Out-of-Scope Use
The model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.

#### Misuse and Malicious Use
Using the model to generate content that is cruel to individuals is a misuse of this model. This includes, but is not limited to:

- Generating demeaning, dehumanizing, or otherwise harmful representations of people or their environments, cultures, religions, etc.
- Intentionally promoting or propagating discriminatory content or harmful stereotypes.
- Impersonating individuals without their consent.
- Sexual content without consent of the people who might see it.
- Mis- and disinformation
- Representations of egregious violence and gore
- Sharing of copyrighted or licensed material in violation of its terms of use.
- Sharing content that is an alteration of copyrighted or licensed material in violation of its terms of use.

## Limitations and Bias

### Limitations

- The model does not achieve perfect photorealism
- The model cannot render legible text
- The model does not perform well on more difficult tasks which involve compositionality, such as rendering an image corresponding to “A red cube on top of a blue sphere”
- Faces and people in general may not be generated properly.
- The model was trained mainly with English captions and will not work as well in other languages.
- The autoencoding part of the model is lossy
- The model was trained on a large-scale dataset
  [LAION-5B](https://laion.ai/blog/laion-5b/) which contains adult material
  and is not fit for product use without additional safety mechanisms and
  considerations.
- No additional measures were used to deduplicate the dataset. As a result, we observe some degree of memorization for images that are duplicated in the training data.
  The training data can be searched at [https://rom1504.github.io/clip-retrieval/](https://rom1504.github.io/clip-retrieval/) to possibly assist in the detection of memorized images.

### Bias

While the capabilities of image generation models are impressive, they can also reinforce or exacerbate social biases.
Stable Diffusion v1 was trained on subsets of [LAION-2B(en)](https://laion.ai/blog/laion-5b/),
which consists of images that are primarily limited to English descriptions.
Texts and images from communities and cultures that use other languages are likely to be insufficiently accounted for.
This affects the overall output of the model, as white and western cultures are often set as the default. Further, the
ability of the model to generate content with non-English prompts is significantly worse than with English-language prompts.

*This model card was written by: Justin Pinkney and is based on the [Stable Diffusion model card](https://huggingface.co/CompVis/stable-diffusion-v1-4).*


# AnimeBackgroundGAN (CartoonGAN by Chen et. al.)

<img src="https://m.media-amazon.com/images/M/MV5BZTExN2EwMmYtNDcwZS00ZmI1LTk1NGQtNTQ3YWFjMmY3YjQwXkEyXkFqcGdeQXVyNTU1OTUzNDg@._V1_.jpg" alt="5 Centimetres per Second directed by Makoto Shinkai" style="height: 300px;"/>

- [Makoto Shinkai （新海誠）](https://en.wikipedia.org/wiki/Makoto_Shinkai) pre-trained model from [CartoonGAN](http://openaccess.thecvf.com/content_cvpr_2018/CameraReady/2205.pdf) `[Chen et al., CVPR18]`.
- This model can transform real-life photos into Japanese-animation-like backgrounds, following the style of movies such as [Kimi no Na wa](https://en.wikipedia.org/wiki/Kimi_no_Na_wa) with a photorealistic painting style.
- The implementation is in PyTorch (see [source code here](https://huggingface.co/spaces/akiyamasho/AnimeBackgroundGAN/blob/main/network/Transformer.py)).
- Check out the demo here:

[![Demo in Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/akiyamasho/AnimeBackgroundGAN)

# Other pre-trained model versions

The other versions were also trained from movies of the different Japanese animation directors.

##### Mamoru Hosoda（細田守）
- director of  [Wolf Children](https://en.wikipedia.org/wiki/Wolf_Children), with a distinct mild and cool background style
- [Director Profile](https://en.wikipedia.org/wiki/Mamoru_Hosoda)
- **Model Repository**: https://huggingface.co/akiyamasho/AnimeBackgroundGAN-Hosoda

##### Satoshi Kon（今敏）
- director of [Paprika](https://en.wikipedia.org/wiki/Paprika_(2006_film)) with a distinct high contrast, reddish hue style
- [Director Profile](https://en.wikipedia.org/wiki/Satoshi_Kon)
- **Model Repository**: https://huggingface.co/akiyamasho/AnimeBackgroundGAN-Kon

##### Hayao Miyazaki（宮崎駿）
- director of [Howl's Moving Castle](https://en.wikipedia.org/wiki/Howl%27s_Moving_Castle_(film)) with a relatively soft and painterly style
- [Director Profile](https://en.wikipedia.org/wiki/Hayao_Miyazaki)
- **Model Repository**: https://huggingface.co/akiyamasho/AnimeBackgroundGAN-Miyazaki

### Credits

- Paper at [CartoonGAN: Generative Adversarial Networks for Photo Cartoonization](http://openaccess.thecvf.com/content_cvpr_2018/CameraReady/2205.pdf) `[Chen et al., CVPR18]`
- Original PyTorch implementation was created by [Yijun Li](https://github.com/Yijunmaverick/)
- Spaces/Models re-packaging and implementation by [Shō Akiyama](https://github.com/Yijunmaverick/).

##### Special Thanks
- [Nima Boscarino](https://github.com/NimaBoscarino)
- [Omar Sanseviero](https://github.com/osanseviero)


# MAXIM pre-trained on GoPro for image deblurring

MAXIM model pre-trained for image deblurring. It was introduced in the paper [MAXIM: Multi-Axis MLP for Image Processing](https://arxiv.org/abs/2201.02973) by Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, Yinxiao Li and first released in [this repository](https://github.com/google-research/maxim).

Disclaimer: The team releasing MAXIM did not write a model card for this model so this model card has been written by the Hugging Face team.

## Model description

MAXIM introduces a shared MLP-based backbone for different image processing tasks such as image deblurring, deraining, denoising, dehazing, low-light image enhancement, and retouching. The following figure depicts the main components of MAXIM:

![](https://github.com/google-research/maxim/raw/main/maxim/images/overview.png)

## Training procedure and results

The authors didn't release the training code. For more details on how the model was trained, refer to the [original paper](https://arxiv.org/abs/2201.02973).

As per the [table](https://github.com/google-research/maxim#results-and-pre-trained-models), the model achieves a PSNR of 32.86 and an SSIM of 0.961.

## Intended uses & limitations

You can use the raw model for image deblurring tasks.

The model is [officially released in JAX](https://github.com/google-research/maxim). It was ported to TensorFlow in [this repository](https://github.com/sayakpaul/maxim-tf).

### How to use

Here is how to use this model:

```python
from huggingface_hub import from_pretrained_keras
from PIL import Image

import tensorflow as tf
import numpy as np
import requests

url = "https://github.com/sayakpaul/maxim-tf/raw/main/images/Deblurring/input/1fromGOPR0950.png"
image = Image.open(requests.get(url, stream=True).raw)
image = np.array(image)
image = tf.convert_to_tensor(image)
image = tf.image.resize(image, (256, 256))

model = from_pretrained_keras("google/maxim-s3-deblurring-gopro")
predictions = model.predict(tf.expand_dims(image, 0))
```

For a more elaborate prediction pipeline, refer to [this Colab Notebook](https://colab.research.google.com/github/sayakpaul/maxim-tf/blob/main/notebooks/inference-dynamic-resize.ipynb).

### Citation

```bibtex
@article{tu2022maxim,
  title={MAXIM: Multi-Axis MLP for Image Processing},
  author={Tu, Zhengzhong and Talebi, Hossein and Zhang, Han and Yang, Feng and Milanfar, Peyman and Bovik, Alan and Li, Yinxiao},
  journal={CVPR},
  year={2022},
}
```


## Model description
This repo contains the model and the notebook [Low-light image enhancement using MIRNet](https://keras.io/examples/vision/mirnet/).

Full credits go to [Soumik Rakshit](https://github.com/soumik12345)

Reproduced by [Vu Minh Chien](https://www.linkedin.com/in/vumichien/) with a slight change on hyperparameters.

With the goal of recovering high-quality image content from its degraded version, image restoration enjoys numerous applications, such as photography, security, medical imaging, and remote sensing. The MIRNet model for low-light image enhancement is a fully-convolutional architecture that learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details
## Dataset
The [LoL Dataset](https://drive.google.com/uc?id=1DdGIJ4PZPlF2ikl8mNM9V-PdVxVLbQi6) has been created for low-light image enhancement. It provides 485 images for training and 15 for testing. Each image pair in the dataset consists of a low-light input image and its corresponding well-exposed reference image.
## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-04
- train_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: ReduceLROnPlateau
- num_epochs: 50

### Training results

- The results are shown in TensorBoard (Training metrics).

### View Model Demo

![Model Demo](./demo.png)

<details>

<summary> View Model Plot </summary>

  ![Model Image](./model.png)

</details>


## Notes
* This model is a trained version of the Keras Tutorial [Image Super Resolution](https://keras.io/examples/vision/super_resolution_sub_pixel/)
* The model has been trained on inputs of dimension 100x100 and outputs images of 300x300.

[Link to a pyimagesearch](https://www.pyimagesearch.com/2021/09/27/pixel-shuffle-super-resolution-with-tensorflow-keras-and-deep-learning/) tutorial I worked on, where we have used Residual blocks along with the Efficient sub pixel net.


## Zero-DCE for low-light image enhancement

**Original Author**: [Soumik Rakshit](https://github.com/soumik12345)

**Date created**: 2021/09/18

**HF Contribution**: [Harveen Singh Chadha](https://github.com/harveenchadha)

**Dataset**: [LOL Dataset](https://huggingface.co/Harveenchadha/low-light-image-enhancement/blob/main/lol_dataset.zip)

## [Spaces Demo](https://huggingface.co/spaces/Harveenchadha/low-light-image-enhancement)

## Description: Implementing Zero-Reference Deep Curve Estimation for low-light image enhancement.

Zero-Reference Deep Curve Estimation or Zero-DCE formulates low-light image enhancement as the task of estimating an image-specific tonal curve with a deep neural network. In this example, we train a lightweight deep network, DCE-Net, to estimate pixel-wise and high-order tonal curves for dynamic range adjustment of a given image.

Zero-DCE takes a low-light image as input and produces high-order tonal curves as its output. These curves are then used for pixel-wise adjustment on the dynamic range of the input to obtain an enhanced image. The curve estimation process is done in such a way that it maintains the range of the enhanced image and preserves the contrast of neighboring pixels. This curve estimation is inspired by curves adjustment used in photo editing software such as Adobe Photoshop where users can adjust points throughout an image’s tonal range.

Zero-DCE is appealing because of its relaxed assumptions with regard to reference images: it does not require any input/output image pairs during training. This is achieved through a set of carefully formulated non-reference loss functions, which implicitly measure the enhancement quality and guide the training of the network.

Sample Images:


# AnimeBackgroundGAN-Hosoda (CartoonGAN by Chen et. al.)

<img src="https://m.media-amazon.com/images/M/MV5BYjgxYjk4OTktZjU3Ni00YzE5LTkyMmItMzI4YzY1YTlhNDg2XkEyXkFqcGdeQXVyNzEyMDQ1MDA@._V1_.jpg" alt="Mirai directed by Mamoru Hosoda" style="height: 300px;"/>

- [Mamoru Hosoda（細田守）](https://en.wikipedia.org/wiki/Mamoru_Hosoda) pre-trained model from [CartoonGAN](http://openaccess.thecvf.com/content_cvpr_2018/CameraReady/2205.pdf) `[Chen et al., CVPR18]`.
- This model can transform real-life photos into Japanese-animation-like backgrounds, following the style of movies such as [Wolf Children](https://en.wikipedia.org/wiki/Wolf_Children), with a distinct mild and cool background style.
- The implementation is in PyTorch (see [source code here](https://huggingface.co/spaces/akiyamasho/AnimeBackgroundGAN/blob/main/network/Transformer.py)).
- Check out the demo here:

[![Demo in Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/akiyamasho/AnimeBackgroundGAN)

# Other pre-trained model versions

The other versions were also trained from movies of the different Japanese animation directors.

##### Makoto Shinkai （新海誠）
- director of [Kimi no Na wa](https://en.wikipedia.org/wiki/Kimi_no_Na_wa) with a photorealistic painting style
- [Director Profile](https://en.wikipedia.org/wiki/Makoto_Shinkai)
- **Model Repository**: https://huggingface.co/akiyamasho/AnimeBackgroundGAN-Shinkai

##### Satoshi Kon（今敏）
- director of [Paprika](https://en.wikipedia.org/wiki/Paprika_(2006_film)) with a distinct high contrast, reddish hue style
- [Director Profile](https://en.wikipedia.org/wiki/Satoshi_Kon)
- **Model Repository**: https://huggingface.co/akiyamasho/AnimeBackgroundGAN-Kon

##### Hayao Miyazaki（宮崎駿）
- director of [Howl's Moving Castle](https://en.wikipedia.org/wiki/Howl%27s_Moving_Castle_(film)) with a relatively soft and painterly style
- [Director Profile](https://en.wikipedia.org/wiki/Hayao_Miyazaki)
- **Model Repository**: https://huggingface.co/akiyamasho/AnimeBackgroundGAN-Miyazaki

### Credits

- Paper at [CartoonGAN: Generative Adversarial Networks for Photo Cartoonization](http://openaccess.thecvf.com/content_cvpr_2018/CameraReady/2205.pdf) `[Chen et al., CVPR18]`
- Original PyTorch implementation was created by [Yijun Li](https://github.com/Yijunmaverick/)
- Spaces/Models re-packaging and implementation by [Shō Akiyama](https://github.com/Yijunmaverick/).

##### Special Thanks
- [Nima Boscarino](https://github.com/NimaBoscarino)
- [Omar Sanseviero](https://github.com/osanseviero)


# AnimeBackgroundGAN-Miyazaki (CartoonGAN by Chen et. al.)

<img src="https://m.media-amazon.com/images/M/MV5BMTM4MTg2MjAzN15BMl5BanBnXkFtZTcwMTk1NzEyNw@@._V1_.jpg" alt="Howl's Moving Castle directed by Hayao Miyazaki" style="height: 300px;"/>

- [Hayao Miyazaki（宮崎駿）](https://en.wikipedia.org/wiki/Hayao_Miyazaki) pre-trained model from [CartoonGAN](http://openaccess.thecvf.com/content_cvpr_2018/CameraReady/2205.pdf) `[Chen et al., CVPR18]`.
- This model can transform real-life photos into Japanese-animation-like backgrounds, following the style of movies such as [Howl's Moving Castle](https://en.wikipedia.org/wiki/Howl%27s_Moving_Castle_(film)) with a relatively soft and painterly style.
- The implementation is in PyTorch (see [source code here](https://huggingface.co/spaces/akiyamasho/AnimeBackgroundGAN/blob/main/network/Transformer.py)).
- Check out the demo here:

[![Demo in Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/akiyamasho/AnimeBackgroundGAN)

# Other pre-trained model versions

The other versions were also trained from movies of the different Japanese animation directors.

##### Mamoru Hosoda（細田守）
- director of  [Wolf Children](https://en.wikipedia.org/wiki/Wolf_Children), with a distinct mild and cool background style
- [Director Profile](https://en.wikipedia.org/wiki/Mamoru_Hosoda)
- **Model Repository**: https://huggingface.co/akiyamasho/AnimeBackgroundGAN-Hosoda

##### Satoshi Kon（今敏）
- director of [Paprika](https://en.wikipedia.org/wiki/Paprika_(2006_film)) with a distinct high contrast, reddish hue style
- [Director Profile](https://en.wikipedia.org/wiki/Satoshi_Kon)
- **Model Repository**: https://huggingface.co/akiyamasho/AnimeBackgroundGAN-Kon

##### Makoto Shinkai （新海誠）
- director of [Kimi no Na wa](https://en.wikipedia.org/wiki/Kimi_no_Na_wa) with a photorealistic painting style
- [Director Profile](https://en.wikipedia.org/wiki/Makoto_Shinkai)
- **Model Repository**: https://huggingface.co/akiyamasho/AnimeBackgroundGAN-Shinkai

### Credits

- Paper at [CartoonGAN: Generative Adversarial Networks for Photo Cartoonization](http://openaccess.thecvf.com/content_cvpr_2018/CameraReady/2205.pdf) `[Chen et al., CVPR18]`
- Original PyTorch implementation was created by [Yijun Li](https://github.com/Yijunmaverick/)
- Spaces/Models re-packaging and implementation by [Shō Akiyama](https://github.com/Yijunmaverick/).

##### Special Thanks
- [Nima Boscarino](https://github.com/NimaBoscarino)
- [Omar Sanseviero](https://github.com/osanseviero)


# ROME: Realistic one-shot mesh-based head avatars
[Paper](https://arxiv.org/abs/2206.08343) | [Project Page](https://samsunglabs.github.io/rome) | [Github](https://github.com/SamsungLabs/rome)

## Model Description

The ROME models can be used to create a personal avatar from a single image. The resulted meshes can be animated and rendered with photorealistic quality.
To render a ROME avatar with pretrained weights the FLAME Model and DECA weights are required.

FLAME Project: https://flame.is.tue.mpg.de/modellicense.html

## Citations
```
@inproceedings{Khakhulin2022ROME,
  author    = {Khakhulin, Taras and Sklyarova,  Vanessa and Lempitsky, Victor and Zakharov, Egor},
  title     = {Realistic One-shot Mesh-based Head Avatars},
  booktitle = {European Conference of Computer vision (ECCV)},
  year      = {2022}
}
```


# DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation - Faces

Creators: Gwanghyun Kim, Taesung Kwon, Jong Chul Ye
Paper: https://arxiv.org/abs/2110.02711

<img src="https://github.com/submission10095/DiffusionCLIP_temp/raw/master/imgs/main1.png" alt="Excerpt from DiffusionCLIP paper showcasing comparison of DiffusionCLIP versus other methods for image reconstruction, manipulation, and style transfer." style="height: 300px;"/>

DiffusionCLIP is a diffusion model which is well suited for image manipulation thanks to its nearly perfect inversion capability, which is an important advantage over GAN-based models. This checkpoint was trained on the [CelebA-HQ Dataset](https://arxiv.org/abs/1710.10196), available on the Hugging Face Hub: https://huggingface.co/datasets/huggan/CelebA-HQ.

This checkpoint is most appropriate for manipulation, reconstruction, and style transfer on images of human faces using the DiffusionCLIP model. To use ID loss for preserving Human face identity, you are required to download the [pretrained IR-SE50 model](https://drive.google.com/file/u/1/d/1KW7bjndL3QG3sxBbZxreGHigcCCpsDgn/view) from [TreB1eN](https://github.com/TreB1eN/InsightFace_Pytorch). Additional information is available on [the GitHub repository](https://github.com/gwang-kim/DiffusionCLIP).

### Credits

- Code repository available at: https://github.com/gwang-kim/DiffusionCLIP

### Citation

```
@article{kim2021diffusionclip,
  title={Diffusionclip: Text-guided image manipulation using diffusion models},
  author={Kim, Gwanghyun and Ye, Jong Chul},
  journal={arXiv preprint arXiv:2110.02711},
  year={2021}
}
```


# Stable Diffusion Image Variations Model Card

This version of Stable Diffusion has been fine tuned from [CompVis/stable-diffusion-v1-3-original](https://huggingface.co/CompVis/stable-diffusion-v-1-3-original) to accept CLIP image embedding rather than text embeddings. This allows the creation of "image variations" similar to DALLE-2 using Stable Diffusion. This version of the weights has been ported to huggingface Diffusers, to use this with the Diffusers library requires the [Lambda Diffusers repo](https://github.com/LambdaLabsML/lambda-diffusers).

![](https://raw.githubusercontent.com/justinpinkney/stable-diffusion/main/assets/im-vars-thin.jpg)

## Example

First clone [Lambda Diffusers](https://github.com/LambdaLabsML/lambda-diffusers) and install any requirements (in a virtual environment in the example below):

```bash
git clone https://github.com/LambdaLabsML/lambda-diffusers.git
cd lambda-diffusers
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

Then run the following python code:

```python
from pathlib import Path
from lambda_diffusers import StableDiffusionImageEmbedPipeline
from PIL import Image
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
pipe = StableDiffusionImageEmbedPipeline.from_pretrained("lambdalabs/sd-image-variations-diffusers")
pipe = pipe.to(device)

im = Image.open("your/input/image/here.jpg")
num_samples = 4
image = pipe(num_samples*[im], guidance_scale=3.0)
image = image["sample"]

base_path = Path("outputs/im2im")
base_path.mkdir(exist_ok=True, parents=True)
for idx, im in enumerate(image):
    im.save(base_path/f"{idx:06}.jpg")
```

# Training

**Training Data**
The model developers used the following dataset for training the model:

- LAION-2B (en) and subsets thereof (see next section)

**Training Procedure**
This model is fine tuned from Stable Diffusion v1-3 where the text encoder has been replaced with an image encoder. The training procedure is the same as for Stable Diffusion except for the fact that images are encoded through a ViT-L/14 image-encoder including the final projection layer to the CLIP shared embedding space.

- **Hardware:** 4 x A6000 GPUs (provided by [Lambda GPU Cloud](https://lambdalabs.com/service/gpu-cloud))
- **Optimizer:** AdamW
- **Gradient Accumulations**: 1
- **Steps**: 87,000
- **Batch:** 6 x 4 = 24
- **Learning rate:** warmup to 0.0001 for 1,000 steps and then kept constant

Training was done using a [modified version of the original Stable Diffusion training code]((https://github.com/justinpinkney/stable-diffusion), the original version of the weights is [here](https://huggingface.co/lambdalabs/stable-diffusion-image-conditioned).

# Uses
_The following section is adapted from the [Stable Diffusion model card](https://huggingface.co/CompVis/stable-diffusion-v1-4)_

## Direct Use
The model is intended for research purposes only. Possible research areas and
tasks include

- Safe deployment of models which have the potential to generate harmful content.
- Probing and understanding the limitations and biases of generative models.
- Generation of artworks and use in design and other artistic processes.
- Applications in educational or creative tools.
- Research on generative models.

Excluded uses are described below.

 ### Misuse, Malicious Use, and Out-of-Scope Use

The model should not be used to intentionally create or disseminate images that create hostile or alienating environments for people. This includes generating images that people would foreseeably find disturbing, distressing, or offensive; or content that propagates historical or current stereotypes.

#### Out-of-Scope Use
The model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.

#### Misuse and Malicious Use
Using the model to generate content that is cruel to individuals is a misuse of this model. This includes, but is not limited to:

- Generating demeaning, dehumanizing, or otherwise harmful representations of people or their environments, cultures, religions, etc.
- Intentionally promoting or propagating discriminatory content or harmful stereotypes.
- Impersonating individuals without their consent.
- Sexual content without consent of the people who might see it.
- Mis- and disinformation
- Representations of egregious violence and gore
- Sharing of copyrighted or licensed material in violation of its terms of use.
- Sharing content that is an alteration of copyrighted or licensed material in violation of its terms of use.

## Limitations and Bias

### Limitations

- The model does not achieve perfect photorealism
- The model cannot render legible text
- The model does not perform well on more difficult tasks which involve compositionality, such as rendering an image corresponding to “A red cube on top of a blue sphere”
- Faces and people in general may not be generated properly.
- The model was trained mainly with English captions and will not work as well in other languages.
- The autoencoding part of the model is lossy
- The model was trained on a large-scale dataset
  [LAION-5B](https://laion.ai/blog/laion-5b/) which contains adult material
  and is not fit for product use without additional safety mechanisms and
  considerations.
- No additional measures were used to deduplicate the dataset. As a result, we observe some degree of memorization for images that are duplicated in the training data.
  The training data can be searched at [https://rom1504.github.io/clip-retrieval/](https://rom1504.github.io/clip-retrieval/) to possibly assist in the detection of memorized images.

### Bias

While the capabilities of image generation models are impressive, they can also reinforce or exacerbate social biases.
Stable Diffusion v1 was trained on subsets of [LAION-2B(en)](https://laion.ai/blog/laion-5b/),
which consists of images that are primarily limited to English descriptions.
Texts and images from communities and cultures that use other languages are likely to be insufficiently accounted for.
This affects the overall output of the model, as white and western cultures are often set as the default. Further, the
ability of the model to generate content with non-English prompts is significantly worse than with English-language prompts.

### Safety Module

The intended use of this model is with the [Safety Checker](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/safety_checker.py) in Diffusers.
This checker works by checking model outputs against known hard-coded NSFW concepts.
The concepts are intentionally hidden to reduce the likelihood of reverse-engineering this filter.
Specifically, the checker compares the class probability of harmful concepts in the embedding space of the `CLIPModel` *after generation* of the images.
The concepts are passed into the model with the generated image and compared to a hand-engineered weight for each NSFW concept.

*This model card was written by: Justin Pinkney and is based on the [Stable Diffusion model card](https://huggingface.co/CompVis/stable-diffusion-v1-4).*


### LegoStyle1.5 Dreambooth model trained with [TheLastBen's fast-DreamBooth](https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb) notebook

This is the fine-tuned Stable Diffusion model trained on lego set images.
Use the tokens **_LegoStyle style_** in your prompts for the effect.

Test the concept via A1111 Colab [fast-Colab-A1111](https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast_stable_diffusion_AUTOMATIC1111.ipynb)

Sample pictures of this concept img-to-img:

![joind3.jpg](https://s3.amazonaws.com/moonup/production/uploads/1673630119515-6387323e5c68cf2713b75239.jpeg)

```
Positive: LegoStyle style, smooth objects, high resolution
Negative: curve, circle, blurry, drawing, cartoon illustration
```

### 🧨 Diffusers

This model can be used just like any other Stable Diffusion model. For more information,
please have a look at the [Stable Diffusion](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion).

You can also export the model to [ONNX](https://huggingface.co/docs/diffusers/optimization/onnx), [MPS](https://huggingface.co/docs/diffusers/optimization/mps) and/or [FLAX/JAX]().

```python
from diffusers import StableDiffusionPipeline
import torch
model_id = "sivar/legostyle1-5"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")
prompt = "LegoStyle style, smooth objects, high resolution"
image = pipe(prompt).images[0]
image.save("./lego.png")
```

## License

This model is open access and available to all, with a CreativeML OpenRAIL-M license further specifying rights and usage.
The CreativeML OpenRAIL License specifies:

1. You can't use the model to deliberately produce nor share illegal or harmful outputs or content
2. The authors claims no rights on the outputs you generate, you are free to use them and are accountable for their use which must not go against the provisions set in the license
3. You may re-distribute the weights and use the model commercially and/or as a service. If you do, please be aware you have to include the same use restrictions as the ones in the license and share a copy of the CreativeML OpenRAIL-M to all your users (please read the license entirely and carefully)
[Please read the full license here](https://huggingface.co/spaces/CompVis/stable-diffusion-license)


# AnimeBackgroundGAN (CartoonGAN by Chen et. al.)

<img src="https://m.media-amazon.com/images/M/MV5BNjNjYTRkNGUtMGQ2MS00MTFiLTg0OTEtYTM3MmM1YTY1OTM1XkEyXkFqcGdeQXVyNjc3OTE4Nzk@._V1_.jpg" alt="Paprika directed by Satoshi Kon" style="height: 300px;"/>

- [Satoshi Kon（今敏）](https://en.wikipedia.org/wiki/Satoshi_Kon) pre-trained model from [CartoonGAN](http://openaccess.thecvf.com/content_cvpr_2018/CameraReady/2205.pdf) `[Chen et al., CVPR18]`.
- This model can transform real-life photos into Japanese-animation-like backgrounds, following the style of movies such as [Paprika](https://en.wikipedia.org/wiki/Paprika_(2006_film)) with a distinct high contrast, reddish hue style.
- The implementation is in PyTorch (see [source code here](https://huggingface.co/spaces/akiyamasho/AnimeBackgroundGAN/blob/main/network/Transformer.py)).
- Check out the demo here:

[![Demo in Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/akiyamasho/AnimeBackgroundGAN)

# Other pre-trained model versions

The other versions were also trained from movies of the different Japanese animation directors.

##### Mamoru Hosoda（細田守）
- director of  [Wolf Children](https://en.wikipedia.org/wiki/Wolf_Children), with a distinct mild and cool background style
- [Director Profile](https://en.wikipedia.org/wiki/Mamoru_Hosoda)
- **Model Repository**: https://huggingface.co/akiyamasho/AnimeBackgroundGAN-Hosoda

##### Makoto Shinkai （新海誠）
- director of [Kimi no Na wa](https://en.wikipedia.org/wiki/Kimi_no_Na_wa) with a photorealistic painting style
- [Director Profile](https://en.wikipedia.org/wiki/Makoto_Shinkai)
- **Model Repository**: https://huggingface.co/akiyamasho/AnimeBackgroundGAN-Shinkai

##### Hayao Miyazaki（宮崎駿）
- director of [Howl's Moving Castle](https://en.wikipedia.org/wiki/Howl%27s_Moving_Castle_(film)) with a relatively soft and painterly style
- [Director Profile](https://en.wikipedia.org/wiki/Hayao_Miyazaki)
- **Model Repository**: https://huggingface.co/akiyamasho/AnimeBackgroundGAN-Miyazaki

### Credits

- Paper at [CartoonGAN: Generative Adversarial Networks for Photo Cartoonization](http://openaccess.thecvf.com/content_cvpr_2018/CameraReady/2205.pdf) `[Chen et al., CVPR18]`
- Original PyTorch implementation was created by [Yijun Li](https://github.com/Yijunmaverick/)
- Spaces/Models re-packaging and implementation by [Shō Akiyama](https://github.com/Yijunmaverick/).

##### Special Thanks
- [Nima Boscarino](https://github.com/NimaBoscarino)
- [Omar Sanseviero](https://github.com/osanseviero)


## Model description
This repo contains the trained model for Style transfer using vgg16 as the backbone.

Full credits go to [Nhu Hoang](https://www.linkedin.com/in/nhu-hoang/)

Motivation: Style transfer is an interesting task with an amazing outcome.

## Training and evaluation data
### Training hyperparameters

The following hyperparameters were used during training:

| Hyperparameters | Value |
| :-- | :-- |
| name | Adam |
| learning_rate | 3e-5 |
| training_precision | float16 |


# MAXIM pre-trained on SIDD for image denoising

MAXIM model pre-trained for image denoising. It was introduced in the paper [MAXIM: Multi-Axis MLP for Image Processing](https://arxiv.org/abs/2201.02973) by Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, Yinxiao Li and first released in [this repository](https://github.com/google-research/maxim).

Disclaimer: The team releasing MAXIM did not write a model card for this model so this model card has been written by the Hugging Face team.

## Model description

MAXIM introduces a shared MLP-based backbone for different image processing tasks such as image deblurring, deraining, denoising, dehazing, low-light image enhancement, and retouching. The following figure depicts the main components of MAXIM:

![](https://github.com/google-research/maxim/raw/main/maxim/images/overview.png)

## Training procedure and results

The authors didn't release the training code. For more details on how the model was trained, refer to the [original paper](https://arxiv.org/abs/2201.02973).

As per the [table](https://github.com/google-research/maxim#results-and-pre-trained-models), the model achieves a PSNR of 39.96 and an SSIM of 0.96.

## Intended uses & limitations

You can use the raw model for image denoising tasks.

The model is [officially released in JAX](https://github.com/google-research/maxim). It was ported to TensorFlow in [this repository](https://github.com/sayakpaul/maxim-tf).

### How to use

Here is how to use this model:

```python
from huggingface_hub import from_pretrained_keras
from PIL import Image

import tensorflow as tf
import numpy as np
import requests

url = "https://github.com/sayakpaul/maxim-tf/raw/main/images/Denoising/input/0011_23.png"
image = Image.open(requests.get(url, stream=True).raw)
image = np.array(image)
image = tf.convert_to_tensor(image)
image = tf.image.resize(image, (256, 256))

model = from_pretrained_keras("google/maxim-s3-denoising-sidd")
predictions = model.predict(tf.expand_dims(image, 0))
```

For a more elaborate prediction pipeline, refer to [this Colab Notebook](https://colab.research.google.com/github/sayakpaul/maxim-tf/blob/main/notebooks/inference-dynamic-resize.ipynb).

### Citation

```bibtex
@article{tu2022maxim,
  title={MAXIM: Multi-Axis MLP for Image Processing},
  author={Tu, Zhengzhong and Talebi, Hossein and Zhang, Han and Yang, Feng and Milanfar, Peyman and Bovik, Alan and Li, Yinxiao},
  journal={CVPR},
  year={2022},
}
```


# Swin2SR model (image super-resolution)

Swin2SR model that upscales images x2. It was introduced in the paper [Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration](https://arxiv.org/abs/2209.11345)
by Conde et al. and first released in [this repository](https://github.com/mv-lab/swin2sr).

# Intended use cases

This model is intended for image super resolution.

# Usage

Refer to the [documentation](https://huggingface.co/docs/transformers/main/en/model_doc/swin2sr#transformers.Swin2SRForImageSuperResolution.forward.example).