riffusion

/

riffusion-model-v1

like 550

how to use localy ?

I use it in voldys sd web-ui

I tried to fine-tune the model using the HF train_text_to_image.py script, but when I load it in Riffusion I'm hitting an error:

Traceback (most recent call last):
  File "/home/james/anaconda3/envs/riffusion/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script
    exec(code, module.__dict__)
  File "/home/james/src/somms/riffusion/riffusion/streamlit/pages/text_to_audio.py", line 102, in <module>
    render_text_to_audio()
  File "/home/james/src/somms/riffusion/riffusion/streamlit/pages/text_to_audio.py", line 78, in render_text_to_audio
    image = streamlit_util.run_txt2img(
  File "/home/james/anaconda3/envs/riffusion/lib/python3.9/site-packages/streamlit/runtime/caching/cache_utils.py", line 428, in wrapper
    return get_or_create_cached_value()
  File "/home/james/anaconda3/envs/riffusion/lib/python3.9/site-packages/streamlit/runtime/caching/cache_utils.py", line 401, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
  File "/home/james/src/somms/riffusion/riffusion/streamlit/util.py", line 102, in run_txt2img
    output = pipeline(
  File "/home/james/anaconda3/envs/riffusion/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/james/anaconda3/envs/riffusion/lib/python3.9/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 531, in __call__
    noise_pred = self.unet(latent_model_input, t, encoder_hidden_states=text_embeddings).sample
  File "/home/james/anaconda3/envs/riffusion/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/james/anaconda3/envs/riffusion/lib/python3.9/site-packages/diffusers/models/unet_2d_condition.py", line 421, in forward
    sample = self.conv_in(sample)
  File "/home/james/anaconda3/envs/riffusion/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/james/anaconda3/envs/riffusion/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/james/anaconda3/envs/riffusion/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (c10::Half) and bias type (float) should be the same

I also tried training with "--mixed_precision=fp16", in case that was an issue, but same error.
Any ideas how to fix this?

(I can push the error down the line by calling sample = self.conv_in(sample) with autocast, but it's not really a solution... I just get a different type error later.)

Are there special settings for the HF script, or for saving/loading the model in Riffusion that I should know about?

Ugh... I just noticed that you get a special prize if you let the script complete it's full run of num_train_epochs... heh...
I'm so used to early stopping and grabbing the checkpoint...
I'm guessing the output of pipeline.save_pretrained(args.output_dir) is going to make a lot more sense when loading in Riffusion... (gulp)

@jbmaxwell did you fix this issue ?

@ogkalu , Sorry, I was being a bit obscure. Just let the training run to completion and it will save the full checkpoint, with everything you need to load in Riffusion.