twodgirl/flux-text-encoder-neutered

Flux TE Neutered

A text encoder designed to serve as a lightweight alternative to the T5 model. For use with the Flux Dev model. Trained in the shadow of a giant.

Prompt

fantasy art depicts a redhead woman, red scales, evil face, black dragon horns, black lipstick

Prompt

drunk man hugging a barmaid next to him at the table, man wearing a hat, next to him, a woman in maid dress, cheerful scene, medieval, daylight, indoors, pub, man holding a beer

Prompt

short redhead hair, pixie cut, freckles, beautiful symmetric eyes, filigree details, sharp focus, highly detailed skin texture, subsurface scattering, reflections, empty background, lipgloss, mascara, eyeshadow, round face, pale skin, full shot, from below, looking at viewer, blue eyes, gown, bracelet, earrings

Prompt

realistic, cyberpunk woman in the shadows, photo of a Korean woman, cinematic, colored inner hair, green black streaked hair, black leather stylish jacket, stars and sparkles, black background, collar on her neck, stylish punk leather jacket, futuristic scene

Prompt

the devil girl within the chamber in the shadows, eyeshadow, sitting, Korean woman, colored inner hair, her reflections shown on the surface of the deep blue water, looks up to the sky, peaking at viewer, from above, black leather stylish jacket, stars and sparkles, black background, collar on her neck, stylish punk leather jacket, futuristic scene

Prompt

a gothpunk witch with a gothpunk hairstyle with pink highlights, coherently detailed perfectly manicured hands, wearing a big Halloween themed witch hat, a long goth dress with lace and black and pink checkered pattern, a broom and bats. The background is a gradient

Prompt

He is dressed in dark, distressed clothing such as a black leather jacket over a dark, graphic tee and ripped jeans. His hair is styled with an emo fringe and dyed in darker tones. The makeup includes dark eyeliner and a pale foundation, reflecting a brooding, melancholic mood. His expression is one of deep sadness and introspection, contrasting with the usual confident demeanor. The background is a moody, dimly lit

Prompt

In the middles of this fantastical landscape, a vintage silver structure is nestled in deep purple. With intricate floral patterns that sparkle against its surface and the glowing orb of fire swirl from branch to branch, beckoning all who gaze upon it with unforgiving expression before continuing to grow into something else.

While the model can still follow prompts to some extent, its performance in this area is noticeably worse than the original text encoder.

Download

Download the T5 variant, the optional preview decoder and the FP8 quant.

Lightweight requirements:

text_encoder.py
text_encoder_2.safetensors

Setup

pip install accelerate diffusers einops optimum-quanto protobuf sentencepiece transformers

Inference

from diffusers import FluxPipeline, FluxTransformer2DModel
from optimum.quanto.models import QuantizedDiffusersModel, QuantizedTransformersModel
import torch
from text_encoder import t5_config, T5EncoderModel, PretrainedTextEncoder

class Flux2DModel(QuantizedDiffusersModel):
    base_class = FluxTransformer2DModel

if __name__ == '__main__':
    t5 = PretrainedTextEncoder(t5_config, T5EncoderModel(t5_config)).to(dtype=torch.float16)
    t5.load_model('text_encoder_2.safetensors')
    # transformer = Flux2DModel.from_pretrained('./flux-fp8')._wrapped
    pipe = FluxPipeline.from_pretrained('black-forest-labs/FLUX.1-dev',
                                        text_encoder_2=t5,
                                        # transformer=transformer
                                       )
    pipe.enable_model_cpu_offload()
    image = pipe('a black cat wearing a Pikachu cosplay', num_inference_steps=10, output_type='pil').images[0]
    image.save('cat.png')

The example saves a preview image.

from diffusers import FluxPipeline, FluxTransformer2DModel
from optimum.quanto.models import QuantizedDiffusersModel, QuantizedTransformersModel
import torch
from tea_model import TeaDecoder
from text_encoder import t5_config, T5EncoderModel, PretrainedTextEncoder

class Flux2DModel(QuantizedDiffusersModel):
    base_class = FluxTransformer2DModel

def preview_image(pipe, latents):
    latents = FluxPipeline._unpack_latents(latents,
                                           pipe.default_sample_size * pipe.vae_scale_factor,
                                           pipe.default_sample_size * pipe.vae_scale_factor,
                                           pipe.vae_scale_factor)
    tea = TeaDecoder(ch_in=16)
    load_model(tea, './vae_decoder.safetensors')
    tea = tea.to(device='cuda')
    output = tea(latents.to(torch.float32)) / 2.0 + 0.5
    preview = transforms.ToPILImage()(output[0].clamp(0, 1))

    return preview

if __name__ == '__main__':
    t5 = PretrainedTextEncoder(t5_config, T5EncoderModel(t5_config)).to(dtype=torch.float16)
    t5.load_model('text_encoder_2.safetensors')
    # transformer = Flux2DModel.from_pretrained('./flux-fp8')._wrapped
    pipe = FluxPipeline.from_pretrained('black-forest-labs/FLUX.1-dev',
                                        text_encoder_2=t5,
                                        # transformer=transformer
                                       )
    pipe.enable_model_cpu_offload()
    latents = pipe('cat playing piano', num_inference_steps=10, output_type='latent').images
    preview = preview_image(pipe, latents)
    preview.save('cat.png')

Disclaimer

Use of this code and the model requires citation and attribution to the author via a link to their Hugging Face profile in all resulting work.