Spaces:
Running
on
Zero
prompt to trigger the lora?
I'm not understanding the prompt we should use to trigger the trained lora. by defaut it is tok, but in the model description it says we should use this:
to trigger concept TOK → use
so if we want to use multiple loras in the same prompt, do we use the same trigger words twice? is this just for hugging face or a1111?
thx, great tool btw
The LoRAs trained in this Space follow a special regime called Pivotal Tuning. This means special tokens are trained alongside the model, so instead of using a trigger word like sks or [V] with existing tokens, a new textual embedding is trained to be used with the model.
If you are using AUTOMATIC1111 or ComfyUI, besides your LoRA, you gotta download the embeddings file (saved with a _emb) in your models, put it into the embeddings folder and use it with the same name of the file.
Now, if you are using diffusers, you gotta load this new embeddings into the model and use it into the prompt. The standard loading is <s0><s1> but that is something you can customize:
from diffusers import AutoPipelineForText2Image
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
pipeline = AutoPipelineForText2Image.from_pretrained('stabilityai/stable-diffusion-xl-base-1.0', torch_dtype=torch.float16).to('cuda')
pipeline.load_lora_weights('multimodalart/medieval-animals-lora', weight_name='pytorch_lora_weights.safetensors')
embedding_path = hf_hub_download(repo_id='multimodalart/medieval-animals-lora', filename='medieval-animals-lora_emb.safetensors' repo_type="model")
state_dict = load_file(embedding_path)
pipeline.load_textual_inversion(state_dict["clip_l"], token=["<s0>", "<s1>"], text_encoder=pipeline.text_encoder, tokenizer=pipeline.tokenizer)
pipeline.load_textual_inversion(state_dict["clip_g"], token=["<s0>", "<s1>"], text_encoder=pipeline.text_encoder_2, tokenizer=pipeline.tokenizer_2)
image = pipeline('in the style of <s0><s1>').images[0]
You can read more about how this works on training and inference in our blog: https://huggingface.co/blog/sdxl_lora_advanced_script