initial commit

.gitattributes

@@ -25,7 +25,6 @@
 *.safetensors filter=lfs diff=lfs merge=lfs -text
 saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
 *.tflite filter=lfs diff=lfs merge=lfs -text
 *.tgz filter=lfs diff=lfs merge=lfs -text
 *.wasm filter=lfs diff=lfs merge=lfs -text

.vscode/settings.json

@@ -0,0 +1,6 @@
+{
+	"[python]": {
+		"editor.defaultFormatter": "ms-python.black-formatter"
+	},
+	"python.formatting.provider": "none"
+}

README.md

@@ -0,0 +1,94 @@
+---
+tags:
+- stable-diffusion
+- controlnet
+- image-to-image
+license: openrail++
+language:
+- en
+pipeline_tag: image-to-image
+---
+# QR Code Conditioned ControlNet Models for Stable Diffusion 2.1
+
+![1](https://www.dropbox.com/s/c1kx64v1cpsh2mp/1.png?raw=1)
+
+## Model Description
+
+This repo holds the safetensors & diffusers versions of the QR code conditioned ControlNet for Stable Diffusion v2.1.
+The Stable Diffusion 2.1 version is marginally more effective, as it was developed to address my specific needs. However, a 1.5 version model was also trained on the same dataset for those who are using the older version.
+
+## How to use with diffusers
+
+```bash
+pip -q install diffusers transformers accelerate torch xformers
+```
+
+```python
+import torch
+from PIL import Image
+from diffusers import StableDiffusionControlNetImg2ImgPipeline, ControlNetModel, DDIMScheduler
+from diffusers.utils import load_image
+
+controlnet = ControlNetModel.from_pretrained("DionTimmer/controlnet_qrcode-control_v11p_sd21",
+                                             torch_dtype=torch.float16)
+
+pipe = StableDiffusionControlNetImg2ImgPipeline.from_pretrained(
+    "stabilityai/stable-diffusion-2-1",
+    controlnet=controlnet,
+    safety_checker=None,
+    torch_dtype=torch.float16
+)
+
+pipe.enable_xformers_memory_efficient_attention()
+pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
+pipe.enable_model_cpu_offload()
+
+def resize_for_condition_image(input_image: Image, resolution: int):
+    input_image = input_image.convert("RGB")
+    W, H = input_image.size
+    k = float(resolution) / min(H, W)
+    H *= k
+    W *= k
+    H = int(round(H / 64.0)) * 64
+    W = int(round(W / 64.0)) * 64
+    img = input_image.resize((W, H), resample=Image.LANCZOS)
+    return img
+
+
+# play with guidance_scale, controlnet_conditioning_scale and strength to make a valid QR Code Image
+
+# qr code image
+source_image = load_image("https://s3.amazonaws.com/moonup/production/uploads/6064e095abd8d3692e3e2ed6/A_RqHaAM6YHBodPLwqtjn.png")
+# initial image, anything
+init_image = load_image("https://s3.amazonaws.com/moonup/production/uploads/noauth/KfMBABpOwIuNolv1pe3qX.jpeg")
+condition_image = resize_for_condition_image(source_image, 768)
+init_image = resize_for_condition_image(init_image, 768)
+generator = torch.manual_seed(123121231)
+image = pipe(prompt="a bilboard in NYC with a qrcode",
+             negative_prompt="ugly, disfigured, low quality, blurry, nsfw", 
+             image=init_image,
+             control_image=condition_image,
+             width=768,
+             height=768,
+             guidance_scale=20,
+             controlnet_conditioning_scale=1.5,
+             generator=generator,
+             strength=0.9, 
+             num_inference_steps=150,
+            )
+
+image.images[0]
+
+```
+
+## Performance and Limitations
+
+These models perform quite well in most cases, but please note that they are not 100% accurate. In some instances, the QR code shape might not come through as expected. You can increase the ControlNet weight to emphasize the QR code shape. However, be cautious as this might negatively impact the style of your output.**To optimize for scanning, please generate your QR codes with correction mode 'H' (30%).**
+
+To balance between style and shape, a gentle fine-tuning of the control weight might be required based on the individual input and the desired output, aswell as the correct prompt. Some prompts do not work until you increase the weight by a lot. The process of finding the right balance between these factors is part art and part science. For the best results, it is recommended to generate your artwork at a resolution of 768. This allows for a higher level of detail in the final product, enhancing the quality and effectiveness of the QR code-based artwork.
+
+## Installation
+
+The simplest way to use this is to place the .safetensors model and its .yaml config file in the folder where your other controlnet models are installed, which varies per application. 
+For usage in auto1111 they can be placed in the webui/models/ControlNet folder. They can be loaded using the controlnet webui extension which you can install through the extensions tab in the webui (https://github.com/Mikubill/sd-webui-controlnet). Make sure to enable your controlnet unit and set your input image as the QR code. Set the model to either the SD2.1 or 1.5 version depending on your base stable diffusion model, or it will error. No pre-processor is needed, though you can use the invert pre-processor for a different variation of results. 768 is the preferred resolution for generation since it allows for more detail.
+Make sure to look up additional info on how to use controlnet if you get stuck, once you have the webui up and running its really easy to install the controlnet extension aswell.

config.json

@@ -0,0 +1,47 @@
+{
+  "_class_name": "ControlNetModel",
+  "_diffusers_version": "0.18.0.dev0",
+  "act_fn": "silu",
+  "attention_head_dim": [
+    5,
+    10,
+    20,
+    20
+  ],
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "class_embed_type": null,
+  "conditioning_embedding_out_channels": [
+    16,
+    32,
+    96,
+    256
+  ],
+  "controlnet_conditioning_channel_order": "rgb",
+  "cross_attention_dim": 1024,
+  "down_block_types": [
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "global_pool_conditions": false,
+  "in_channels": 4,
+  "layers_per_block": 2,
+  "mid_block_scale_factor": 1,
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "num_class_embeds": null,
+  "only_cross_attention": false,
+  "projection_class_embeddings_input_dim": null,
+  "resnet_time_scale_shift": "default",
+  "upcast_attention": null,
+  "use_linear_projection": true
+}

control_v11p_sd21_qrcode.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:340ff6922e3f96d6311eb1e467dd63886463068cafedffc56a2d8bc5ff9f5563
+size 1456951266

control_v11p_sd21_qrcode.yaml

@@ -0,0 +1,85 @@
+model:
+  target: cldm.cldm.ControlLDM
+  params:
+    linear_start: 0.00085
+    linear_end: 0.0120
+    num_timesteps_cond: 1
+    log_every_t: 200
+    timesteps: 1000
+    first_stage_key: "jpg"
+    cond_stage_key: "txt"
+    control_key: "hint"
+    image_size: 64
+    channels: 4
+    cond_stage_trainable: false
+    conditioning_key: crossattn
+    monitor: val/loss_simple_ema
+    scale_factor: 0.18215
+    use_ema: False
+    only_mid_control: False
+
+    control_stage_config:
+      target: cldm.cldm.ControlNet
+      params:
+        use_checkpoint: True
+        image_size: 32 # unused
+        in_channels: 4
+        hint_channels: 3
+        model_channels: 320
+        attention_resolutions: [ 4, 2, 1 ]
+        num_res_blocks: 2
+        channel_mult: [ 1, 2, 4, 4 ]
+        num_head_channels: 64 # need to fix for flash-attn
+        use_spatial_transformer: True
+        use_linear_in_transformer: True
+        transformer_depth: 1
+        context_dim: 1024
+        legacy: False
+
+    unet_config:
+      target: cldm.cldm.ControlledUnetModel
+      params:
+        use_checkpoint: True
+        image_size: 32 # unused
+        in_channels: 4
+        out_channels: 4
+        model_channels: 320
+        attention_resolutions: [ 4, 2, 1 ]
+        num_res_blocks: 2
+        channel_mult: [ 1, 2, 4, 4 ]
+        num_head_channels: 64 # need to fix for flash-attn
+        use_spatial_transformer: True
+        use_linear_in_transformer: True
+        transformer_depth: 1
+        context_dim: 1024
+        legacy: False
+
+    first_stage_config:
+      target: ldm.models.autoencoder.AutoencoderKL
+      params:
+        embed_dim: 4
+        monitor: val/rec_loss
+        ddconfig:
+          #attn_type: "vanilla-xformers"
+          double_z: true
+          z_channels: 4
+          resolution: 256
+          in_channels: 3
+          out_ch: 3
+          ch: 128
+          ch_mult:
+          - 1
+          - 2
+          - 4
+          - 4
+          num_res_blocks: 2
+          attn_resolutions: []
+          dropout: 0.0
+        lossconfig:
+          target: torch.nn.Identity
+
+    cond_stage_config:
+      target: ldm.modules.encoders.modules.FrozenOpenCLIPEmbedder
+      params:
+        freeze: True
+        layer: "penultimate"

diffusion_pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0e671b5b6c993b99afa265ea88794b5ee40f56ee3b26fc06f75d21b4d8cdfcb
+size 1457051321

diffusion_pytorch_model.fp16.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b3daef6325b0c8e453a30b98cebc146bece31658ac15c178ce122071f7301c3
+size 728596455

diffusion_pytorch_model.fp16.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a68a2ee6172524be3f6d6758cf83e5e6c8b17af31ab617fdca933139fe0f6ca
+size 728496840

diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0bf31f4c634ae1118b517a4adfe0fd67ae00eb13e1bb97f77e7192f3a047a82b
+size 1456953560

handler.py

@@ -0,0 +1,128 @@
+import torch
+from PIL import Image
+from diffusers import (
+    StableDiffusionControlNetImg2ImgPipeline,
+    ControlNetModel,
+    DDIMScheduler,
+)
+from diffusers.utils import load_image
+import openai
+from io import BytesIO
+import base64
+import qrcode
+import random
+
+qrcode_data = "https://www.vertxdesigns.com/"
+prompt = "masterpiece, best quality, mecha, no humans, black armor, blue eyes, science fiction, fire, laser canon beam, war, conflict, destroyed city background"
+negative_prompt = "UnrealisticDream, FastNegativeEmbedding"
+
+
+qr = qrcode.QRCode(
+    version=1,
+    error_correction=qrcode.constants.ERROR_CORRECT_H,
+    box_size=10,
+    border=4,
+)
+qr.add_data(qrcode_data)
+qr.make(fit=True)
+img = qr.make_image(fill_color="black", back_color="white")
+
+# Resize image
+basewidth = 768
+wpercent = basewidth / float(img.size[0])
+hsize = int((float(img.size[1]) * float(wpercent)))
+qrcode_image = img.resize((basewidth, hsize), Image.LANCZOS)
+
+# Display the image
+qrcode_image
+# img.save('qrcode.png')
+
+
+# Initialize the control net model and pipeline.
+controlnet = ControlNetModel.from_pretrained(
+    "DionTimmer/controlnet_qrcode-control_v11p_sd21", torch_dtype=torch.float16
+)
+
+pipe = StableDiffusionControlNetImg2ImgPipeline.from_pretrained(
+    "stabilityai/stable-diffusion-2-1",
+    controlnet=controlnet,
+    safety_checker=None,
+    torch_dtype=torch.float16,
+)
+
+# Enable memory efficient attention.
+pipe.enable_xformers_memory_efficient_attention()
+
+# Set the scheduler for the pipeline.
+pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
+
+# Enable CPU offload for the model.
+pipe.enable_model_cpu_offload()
+
+
+# Resizes input_image to a specified resolution while maintaining the aspect ratio.
+def resize_for_condition_image(input_image: Image, resolution: int):
+    input_image = input_image.convert("RGB")
+    W, H = input_image.size
+    k = float(resolution) / min(H, W)
+    H *= k
+    W *= k
+    H = int(round(H / 64.0)) * 64
+    W = int(round(W / 64.0)) * 64
+    img = input_image.resize((W, H), resample=Image.LANCZOS)
+    return img
+
+
+def get_random_seed():
+    return random.randint(1, 1e8)  # random integer between 1 and 1,000,000.
+
+
+# Generate and store your seed.
+seed = get_random_seed()
+
+# Set the seed for the random number generator.
+generator = torch.manual_seed(seed)
+
+# Print the seed.
+print(seed)
+
+
+openai.api_key = "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"
+response = openai.Image.create(prompt=prompt, n=1, size="1024x1024")
+image_url = response.data[0].url
+print(image_url)
+
+
+init_image = load_image(image_url)
+
+# Set the control image to the qrcode image.
+control_image = qrcode_image
+
+# Resize the initial image
+init_image = resize_for_condition_image(init_image, 768)
+
+# Run the image generation process using the pipeline.
+image = pipe(
+    prompt=prompt,
+    negative_prompt=negative_prompt,
+    image=init_image,  # The initial image, set as a QR code image
+    control_image=control_image,  # QR code image
+    width=768,
+    height=768,
+    guidance_scale=7.5,  # The influence of the 'prompt' 0-50
+    controlnet_conditioning_scale=1.6,  # The influence of the qr code 1-5
+    generator=generator,  # Random seed for the generation process
+    strength=0.99,  # Noise added to the QR code 0-1
+    num_inference_steps=150,  # The number of steps in the image generation process
+)
+
+
+image.images[0]
+
+
+pil_image = image.images[0]
+buffered = BytesIO()
+pil_image.save(buffered, format="PNG")
+image_base64 = base64.b64encode(buffered.getvalue()).decode()
+print(f"First 10 characters: {image_base64[:10]}")
+print(f"Length of string: {len(image_base64):,}")