Spaces:

r3gm

/

DiffuseCraft

like 58

Running on Zero

Thanks for the heads-up @John6666
I’ve made some changes and will keep your tips in mind

Thank you for always maintaining the library.😸
I will close the Discussion.

BTW, I wonder if streamtest is going away and gone without being officially merged...
I've moved to dev2, but I've seen a few people using DiffuseCraft, which is a streamtest-only implementation version (yield images, seed).
Well, I'm sure it will be fixed in sync if they haven't modified it too much... just wondering.

Hello, I have added the function to the main branch

Hi, Thanks for the contact!
I'll try to transplant it.

Successfully confirmed that True/False in image_previews changes the number of return values (3 if on). However, since the Zero GPU space has recently been sped up, the step is often completed before the previews appear. (Like noise -> complete). It would be helpful to have Seed returned.
This would not be a problem for anyone using either the old or new implementation. Thanks.

https://huggingface.co/posts/cbensimon/747180194960645
The HF bug is about 70% fixed, but it's still there, and it's easy to have trouble with CUDA onloading and offloading, especially after starting the demo. stablepy is probably able to work around it thanks to frequent offloading, but libraries that assume only a CUDA environment are in trouble. If the tensor straddles the CPU and GPU, the program is sure to die.

By the way, this is totally unrelated.
It seems to be customary for Civitai and others to write metadata into image files.
I've added my own code for an image generation space that uses Serverless Inference, but as far as DiffuseCraft is concerned, I'd like to keep in line with the original as much as possible.

Although it would be so much easier if stablepy returned it, since stablepy returns images, not image files (And it should be.), I'm thinking it would be just before or after the yield in DiffuseCraft's app.py, but the timing is more difficult than I thought, especially considering the stream of the yield, even if it ends with a return...
Any ideas?

The metadata looks like this, which is probably originally WebUI's own format, so there should be no strict definition.
https://huggingface.co/spaces/cagliostrolab/animagine-xl-3.1/blob/main/app.py
https://huggingface.co/spaces/cagliostrolab/animagine-xl-3.1/blob/main/utils.py

    metadata = {
        "prompt": prompt,
        "negative_prompt": negative_prompt,
        "resolution": f"{width} x {height}",
        "guidance_scale": guidance_scale,
        "num_inference_steps": num_inference_steps,
        "seed": seed,
        "sampler": sampler,
        "sdxl_style": style_selector,
        "add_quality_tags": add_quality_tags,
        "quality_tags": quality_selector,
    }

        metadata["use_upscaler"] = None
        metadata["Model"] = {
            "Model": DESCRIPTION,
            "Model hash": "e3c47aedb0",
        }

def save_image(image, metadata, output_dir, is_colab):
    if is_colab:
        current_time = datetime.now().strftime("%Y%m%d_%H%M%S")
        filename = f"image_{current_time}.png"   
    else:
        filename = str(uuid.uuid4()) + ".png"
    os.makedirs(output_dir, exist_ok=True)
    filepath = os.path.join(output_dir, filename)
    metadata_str = json.dumps(metadata)
    info = PngImagePlugin.PngInfo()
    info.add_text("metadata", metadata_str)
    image.save(filepath, "PNG", pnginfo=info)
    return filepath

My version

https://huggingface.co/spaces/John6666/flux-lora-the-explorer/blob/main/mod.py

def save_image(image, savefile, modelname, prompt, height, width, steps, cfg, seed):
    import uuid
    from PIL import Image, PngImagePlugin
    import json
    try:
        if savefile is None: savefile = f"{modelname.split('/')[-1]}_{str(uuid.uuid4())}.png"
        metadata = {"prompt": prompt, "Model": {"Model": modelname.split("/")[-1]}}
        metadata["num_inference_steps"] = steps
        metadata["guidance_scale"] = cfg
        metadata["seed"] = seed
        metadata["resolution"] = f"{width} x {height}"
        metadata_str = json.dumps(metadata)
        info = PngImagePlugin.PngInfo()
        info.add_text("metadata", metadata_str)
        image.save(savefile, "PNG", pnginfo=info)
        return str(Path(savefile).resolve())
    except Exception as e:
        print(f"Failed to save image file: {e}")
        raise Exception(f"Failed to save image file:") from e

I'm setting up a lighter stream to improve performance. For the metadata, I'm currently using a plain text format like in automatic1111, but I might switch to a dictionary format like you suggested.
In the yield, I use if image_path: to identify the last loop for returning the metadata. I'm not sure if this will be helpful to you, though

I'm setting up a lighter stream to improve performance.

Preview is now perfect.

For the metadata, I'm currently using a plain text format like in automatic1111,

I think A1111 is more standard. You have already written...

I use if image_path: to identify the last loop for returning the metadata.

It can be a great help!

P.S.
The metadata is then successfully embedded.
Since Gradio's Image and Gallery components do not destroy the image metadata, I decided to insert the processing in this line, a little wildly, so that the GUI side does not need to be modified.
This is easy enough to rewrite in the event of a DiffuseCraft update.
Thank you very much!🤗
https://huggingface.co/spaces/John6666/DiffuseCraftMod

from PIL import Image
def save_images(images: list[Image.Image], metadatas: list[str]):
    from PIL import PngImagePlugin
    import uuid
    try:
        output_images = []
        for image, metadata in zip(images, metadatas):
            info = PngImagePlugin.PngInfo()
            info.add_text("metadata", metadata)
            savefile = f"{str(uuid.uuid4())}.png"
            image.save(savefile, "PNG", pnginfo=info)
            output_images.append(str(Path(savefile).resolve()))
        return output_images
    except Exception as e:
        print(f"Failed to save image file: {e}")
        raise Exception(f"Failed to save image file:") from e

                info_state = info_state + "<br>" + "GENERATION DATA:<br>" + "<br>-------<br>".join(metadata).replace("\n", "<br>")

                img = save_images(img, metadata)
                
            yield img, info_state

Works well
I found a way to do this with HTML, which could make things easier.

The HTML download link worked perfectly, but with the clientele in my space, sharing the images folder would have been kind of disastrous, so I had no choice but to omit it.😅

I'm reporting a future bug.

It's a long explanation because I know you don't understand it. A few weeks ago, I was forced to shorten Examples in my space because I was experiencing crashes on start-up if I included None in the Examples field.
The root cause is still unknown, although it was discussed on the HF forum.
I assumed that since it wasn't happening in your space, it must be my code, but I was wrong.

It can be reproduced with just the following changes. When I look at the behaviour in the logs, it's absurd, like trying to put numbers for different components in the dropdown...
However, this is not just a common Gradio bug, as it happens even when it is not Gradio.
Note that no workaround other than reducing the contents of Examples is currently known.

README.md

sdk_version: 4.44.0

Thanks so much for the info

Sorry. The few workarounds appear to have failed.

As a rule of thumb, this is more likely to happen with drop-downs than with text boxes or sliders.
Although it is somewhat obvious that dynamically generated elements are likely to be involved, it is hard to guess from the logic, as model selection and the like are fine.
When I tried to avoid this, I used the Russian roulette? method and removed them from one side to the other.

It's just a guess, but I suspect that one or more of the components is not well visible to the mysterious process on the VM management function (the bug culprit), and that after passing through the components, they are shifted one by one?

===== Application Startup at 2024-09-22 05:38:12 =====

The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.


0it [00:00, ?it/s]
0it [00:00, ?it/s]
You need an API key to download Civitai models.
[#dfebe7 200MiB/1.9GiB(9%) CN:16 DL:268MiB ETA:6s]
[#dfebe7 550MiB/1.9GiB(27%) CN:16 DL:316MiB ETA:4s]
[#dfebe7 910MiB/1.9GiB(44%) CN:16 DL:333MiB ETA:3s]
[#dfebe7 1.2GiB/1.9GiB(62%) CN:16 DL:340MiB ETA:2s]
[#dfebe7 1.5GiB/1.9GiB(79%) CN:16 DL:342MiB ETA:1s]
[#dfebe7 1.9GiB/1.9GiB(96%) CN:16 DL:343MiB]

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
dfebe7|OK  |   340MiB/s|models/milkyWonderland_v40.safetensors

Status Legend:
(OK):download completed.
[#5dd1ca 0B/0B CN:1 DL:0B]
[#5dd1ca 0B/0B CN:1 DL:0B]
[#5dd1ca 291MiB/319MiB(91%) CN:16 DL:295MiB]

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
5dd1ca|OK  |   289MiB/s|vaes/sdxl_vae-fp16fix-c-1.1-b-0.5.safetensors

Status Legend:
(OK):download completed.
[#d39e0b 232MiB/319MiB(72%) CN:16 DL:312MiB]

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
d39e0b|OK  |   305MiB/s|vaes/sdxl_vae-fp16fix-blessed.safetensors

Status Legend:
(OK):download completed.
[#f9221f 159MiB/159MiB(99%) CN:1 DL:211MiB]

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
f9221f|OK  |   210MiB/s|vaes/vividReal_v20.safetensors

Status Legend:
(OK):download completed.
[#11bcc6 0B/0B CN:1 DL:0B]
[#11bcc6 87MiB/159MiB(54%) CN:16 DL:275MiB]

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
11bcc6|OK  |   278MiB/s|vaes/vae-ft-mse-840000-ema-pruned_fp16.safetensors

Status Legend:
(OK):download completed.
You need an API key to download Civitai models.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
fb2215|OK  |   303MiB/s|loras/Coloring_book_-_LineArt.safetensors

Status Legend:
(OK):download completed.
You need an API key to download Civitai models.
You need an API key to download Civitai models.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
58f467|OK  |    74MiB/s|loras/anime-detailer-xl.safetensors

Status Legend:
(OK):download completed.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
09a4c5|OK  |   103MiB/s|loras/style-enhancer-xl.safetensors

Status Legend:
(OK):download completed.
You need an API key to download Civitai models.
[#e06807 240MiB/256MiB(93%) CN:16 DL:300MiB]

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
e06807|OK  |   293MiB/s|loras/Hyper-SD15-8steps-CFG-lora.safetensors

Status Legend:
(OK):download completed.
[#980881 220MiB/750MiB(29%) CN:16 DL:282MiB ETA:1s]
[#980881 548MiB/750MiB(73%) CN:16 DL:309MiB]

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
980881|OK  |   305MiB/s|loras/Hyper-SDXL-8steps-CFG-lora.safetensors

Status Legend:
(OK):download completed.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
7af41f|OK  |   8.1MiB/s|embedings/bad_prompt_version2.pt

Status Legend:
(OK):download completed.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
2a51bd|OK  |    11MiB/s|embedings/EasyNegativeV2.safetensors

Status Legend:
(OK):download completed.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
30a175|OK  |   1.3MiB/s|embedings/bad-hands-5.pt

Status Legend:
(OK):download completed.
FILE: embedings/bad_prompt_version2.pt
FILE: embedings/EasyNegativeV2.safetensors
FILE: embedings/bad-hands-5.pt
FILE: models/milkyWonderland_v40.safetensors
FILE: loras/Coloring_book_-_LineArt.safetensors
FILE: loras/anime-detailer-xl.safetensors
FILE: loras/style-enhancer-xl.safetensors
FILE: loras/Hyper-SD15-8steps-CFG-lora.safetensors
FILE: loras/Hyper-SDXL-8steps-CFG-lora.safetensors
FILE: vaes/sdxl_vae-fp16fix-c-1.1-b-0.5.safetensors
FILE: vaes/sdxl_vae-fp16fix-blessed.safetensors
FILE: vaes/vividReal_v20.safetensors
FILE: vaes/vae-ft-mse-840000-ema-pruned_fp16.safetensors
🏁 Download and listing of valid models completed.
Loading model...
[INFO] >> Default VAE: madebyollin/sdxl-vae-fp16-fix
[DEBUG] >> The deprecation tuple ('no variant default', '0.24.0', "You are trying to load the model files of the `variant=fp16`, but no such modeling files are available.The default model files: {'unet/diffusion_pytorch_model.safetensors', 'text_encoder/model.safetensors', 'text_encoder_2/model.safetensors', 'vae/diffusion_pytorch_model.safetensors'} will be loaded instead. Make sure to not load from `variant=fp16`if such variant modeling files are not available. Doing so will lead to an error in v0.24.0 as defaulting to non-variantmodeling files is deprecated.") should be removed since diffusers' version 0.30.2 is >= 0.24.0
[DEBUG] >> Loading model without parameter variant=fp16


Loading pipeline components...:   0%|          | 0/7 [00:00<?, ?steps/s]

Loading pipeline components...:  29%|██▊       | 2/7 [00:01<00:03,  1.40steps/s]

Loading pipeline components...:  71%|███████▏  | 5/7 [00:04<00:01,  1.22steps/s]
Loading pipeline components...: 100%|██████████| 7/7 [00:04<00:00,  1.49steps/s]
[DEBUG] >> Default VAE
[DEBUG] >> Base sampler: EulerAncestralDiscreteScheduler {
  "_class_name": "EulerAncestralDiscreteScheduler",
  "_diffusers_version": "0.30.2",
  "beta_end": 0.012,
  "beta_schedule": "scaled_linear",
  "beta_start": 0.00085,
  "interpolation_type": "linear",
  "num_train_timesteps": 1000,
  "prediction_type": "epsilon",
  "rescale_betas_zero_snr": false,
  "sample_max_value": 1.0,
  "set_alpha_to_one": false,
  "skip_prk_steps": true,
  "steps_offset": 1,
  "timestep_spacing": "leading",
  "trained_betas": null
}

/usr/local/lib/python3.10/site-packages/gradio/components/dropdown.py:188: UserWarning: The value passed into gr.Dropdown() is not in the list of choices. Please update the list of choices to include: Lineart or set allow_custom_value=True.
  warnings.warn(
/usr/local/lib/python3.10/site-packages/gradio/components/dropdown.py:188: UserWarning: The value passed into gr.Dropdown() is not in the list of choices. Please update the list of choices to include: txt2img or set allow_custom_value=True.
  warnings.warn(
/usr/local/lib/python3.10/site-packages/gradio/components/dropdown.py:188: UserWarning: The value passed into gr.Dropdown() is not in the list of choices. Please update the list of choices to include: 512 or set allow_custom_value=True.
  warnings.warn(
/usr/local/lib/python3.10/site-packages/gradio/components/dropdown.py:188: UserWarning: The value passed into gr.Dropdown() is not in the list of choices. Please update the list of choices to include: 100 or set allow_custom_value=True.
  warnings.warn(
Traceback (most recent call last):
  File "/home/user/app/app.py", line 1142, in <module>
    gr.Examples(
  File "/usr/local/lib/python3.10/site-packages/gradio/helpers.py", line 61, in create_examples
    examples_obj = Examples(
  File "/usr/local/lib/python3.10/site-packages/gradio/helpers.py", line 281, in __init__
    self._get_processed_example(example)
  File "/usr/local/lib/python3.10/site-packages/gradio/helpers.py", line 290, in _get_processed_example
    prediction_value = component.postprocess(sample)
  File "/usr/local/lib/python3.10/site-packages/gradio/components/file.py", line 204, in postprocess
    orig_name=Path(value).name,
  File "/usr/local/lib/python3.10/pathlib.py", line 960, in __new__
    self = cls._from_parts(args)
  File "/usr/local/lib/python3.10/pathlib.py", line 594, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/usr/local/lib/python3.10/pathlib.py", line 578, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not int
The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.


0it [00:00, ?it/s]
0it [00:00, ?it/s]
You need an API key to download Civitai models.

Good evening?

I'm here to suggest that if Flux and Pony are going to enter the competition, it will soon be difficult to determine the model type by name.
And Pony7 was declared as a likely AuraFlow...
This is not usable in a local environment, but for a demo, it would be okay.

model_type_dict = {
    "diffusers:StableDiffusionPipeline": "SD 1.5",
    "diffusers:StableDiffusionXLPipeline": "SDXL",
    "diffusers:FluxPipeline": "FLUX",
}

def get_model_type(repo_id: str):
    from huggingface_hub import HfApi
    api = HfApi() # api = HfApi(token=HF_READ_TOKEN) # if use private or gated model
    default = "SD 1.5"
    try:
        model = api.model_info(repo_id=repo_id, timeout=5.0)
        tags = model.tags
        for tag in tags:
            if tag in model_type_dict.keys(): return model_type_dict.get(tag, default)
    except Exception:
        return default
    return default

model_type = get_model_type(model_name)

And while we're at it, we've got a semi-official model that doesn't require certification, which is a modified version of someone else's dev that multimodalart has made into Diffusers.
Maybe this one will be the base for training LoRA.
https://huggingface.co/multimodalart/FLUX.1-dev2pro-full

Thanks a lot, I’ve used this in the GUI