multimodalart HF staff commited on
Commit
ab75230
1 Parent(s): be866c4

Upload folder using huggingface_hub

Browse files
Files changed (4) hide show
  1. README.md +75 -0
  2. embeddings.pti +0 -0
  3. lora.safetensors +3 -0
  4. special_params.json +1 -0
README.md ADDED
@@ -0,0 +1,75 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: creativeml-openrail-m
3
+ tags:
4
+ - text-to-image
5
+ - stable-diffusion
6
+ - lora
7
+ - diffusers
8
+ base_model: stabilityai/stable-diffusion-xl-base-1.0
9
+ instance_prompt: <s0><s1>
10
+ inference: false
11
+ ---
12
+ # sdxl-zelda64 LoRA by [jbilcke](https://replicate.com/jbilcke)
13
+ ### A SDXL LoRA inspired by Zelda games on Nintendo 64
14
+
15
+ ![lora_image](https://tjzk.replicate.delivery/models_models_cover_image/c8b21524-342a-4dd2-bb01-3e65349ed982/image_12.jpeg)
16
+ >
17
+
18
+ ## Inference with Replicate API
19
+ Grab your replicate token [here](https://replicate.com/account)
20
+ ```bash
21
+ pip install replicate
22
+ export REPLICATE_API_TOKEN=r8_*************************************
23
+ ```
24
+
25
+ ```py
26
+ import replicate
27
+
28
+ output = replicate.run(
29
+ "sdxl-zelda64@sha256:435913219645a80ee6743ca500940ab8708889172ca5c4c71bbb701309bb4a60",
30
+ input={"prompt": "Link working as a pizza delivery driver, on a scooter, in new york, in the style of TOK"}
31
+ )
32
+ print(output)
33
+ ```
34
+ You may also do inference via the API with Node.js or curl, and locally with COG and Docker, [check out the Replicate API page for this model](https://replicate.com/jbilcke/sdxl-zelda64/api)
35
+
36
+ ## Inference with 🧨 diffusers
37
+ Replicate SDXL LoRAs are trained with Pivotal Tuning, which combines training a concept via Dreambooth LoRA with training a new token with Textual Inversion.
38
+ As `diffusers` doesn't yet support textual inversion for SDXL, we will use cog-sdxl `TokenEmbeddingsHandler` class.
39
+
40
+ The trigger tokens for your prompt will be `<s0><s1>`
41
+
42
+ ```shell
43
+ pip install diffusers transformers accelerate safetensors huggingface_hub
44
+ git clone https://github.com/replicate/cog-sdxl cog_sdxl
45
+ ```
46
+
47
+ ```py
48
+ import torch
49
+ from huggingface_hub import hf_hub_download
50
+ from diffusers import DiffusionPipeline
51
+ from cog_sdxl.dataset_and_utils import TokenEmbeddingsHandler
52
+ from diffusers.models import AutoencoderKL
53
+
54
+ pipe = DiffusionPipeline.from_pretrained(
55
+ "stabilityai/stable-diffusion-xl-base-1.0",
56
+ torch_dtype=torch.float16,
57
+ variant="fp16",
58
+ ).to("cuda")
59
+
60
+ load_lora_weights("multimodalart/sdxl-zelda64", weight_name="lora.safetensors")
61
+
62
+ text_encoders = [pipe.text_encoder, pipe.text_encoder_2]
63
+ tokenizers = [pipe.tokenizer, pipe.tokenizer_2]
64
+
65
+ embedding_path = hf_hub_download(repo_id="multimodalart/sdxl-zelda64", filename="embeddings.pti", repo_type="model")
66
+ embhandler = TokenEmbeddingsHandler(text_encoders, tokenizers)
67
+ embhandler.load_embeddings(embedding_path)
68
+ prompt="Link working as a pizza delivery driver, on a scooter, in new york, in the style of <s0><s1>"
69
+ images = pipe(
70
+ prompt,
71
+ cross_attention_kwargs={"scale": 0.8},
72
+ ).images
73
+ #your output image
74
+ images[0]
75
+ ```
embeddings.pti ADDED
lora.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f35a67040dff8262b605a79cebe105a83797da93e47e70d262869327a7e3d8a5
3
+ size 185968776
special_params.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"TOK": "<s0><s1>"}