How to combine with HED?

#3
by redromnon - opened

I'm new to this and aware that TemporalNet works best with other controlnet conditionings like HED. What tensor/image I should be using for the HED edge detection?

Here's how I intended to apply HED in the code:

  1. Add multiple conditioning controlnet in pipe:
    controlnet=[ ControlNetModel.from_pretrained("wav/TemporalNet2", torch_dtype=torch.float16), ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-hed", torch_dtype=torch.float16) ]

  2. Initiliase HED detector

  3. In stylize_video(), convert flow_img tensor to PIL.Image using torchvision.transforms.ToPILImage

  4. Get HED image from above PIL.Image

  5. In pipe, use arguments:
    control_image=[control_img, hed_image],
    controlnet_conditioning_scale=[1.0, 1.0]

Sign up or log in to comment