zhoudoe23/MCDiffusion-35M

Minecraft Skin Diffusion (64x64 Pixel-Space Text-to-Skin)

This model is a lightweight, pixel-space diffusion model trained to generate 64x64 RGBA Minecraft skin textures directly from natural language prompts or structured tag streams.

Unlike conventional latent diffusion pipelines (e.g., Stable Diffusion), this model does not use a VAE. It operates directly in the discrete 64x64 RGBA pixel coordinate space, preserving hard-edged pixel art boundaries and the multi-layer skin format (Base layer + Outer overlay) ready for in-game use.

---

Model Architecture & Training

The model was developed via a two-stage pretraining and fine-tuning strategy:

1. Stage 1 (Unconditional / Class-Conditional Pretraining):

   • Backbone: UNet2DModel (~35M parameters, $64 \times 64 \times 4$ RGBA input/output).
   • Data: Pretrained on ~1.1 million uncaptioned Minecraft skins (MihaiPopa-1/minecraft-skins-1.1m-deduped-64x64-1.5) to thoroughly learn Minecraft UV geometry, limb symmetries, and pixel art shading techniques.
   • Conditioning: Learned class embeddings for classic (4px Steve arms) and slim (3px Alex arms).

2. Stage 2 (Text-Conditioned Full Fine-Tuning):

   • Text Encoder: zhoudoe23/siglip-so400m-mcskin-fused (SigLIP-So400M fine-tuned on Minecraft UV maps, 1152-dim text embeddings).
   • Data: ~106,000 skins labeled with structured annotations (captions, short summaries, visual tags, keywords).
   • Training: Inherited Stage 1 convolutional prior; all convolutional layers and Cross-Attention layers (attn2) were fully unfrozen and trained end-to-end with dynamic weighted prompt sampling and Classifier-Free Guidance (CFG).

---

Known Limitations & Quirks

This model is an experimental community project and has clear limitations inherent to continuous diffusion operating on discrete low-resolution pixel grids:

1. Sub-pixel Detail Loss (Eyes & Tiny Accessories)

• Eye Color Issue: Minecraft eye pixels are tiny (1x1 or 2x2 pixels on an 8x8 face). In the U-Net's feature space (16x16 and 8x8), the entire face occupies only 2x2 or 1x1 tokens. As a result, cross-attention often fails to isolate eye pixels from surrounding hair and skin, leading to blurred eyes, missing irises, or eyes adopting surrounding hair tones (e.g., green eyes turning hazel/brown).
• Single-pixel Accessories: Requests like "a green leaf clipped by one ear" or tiny pins are below the receptive field threshold and are usually omitted.

2. "Harmonic Palette" Bias (Color Bleeding / Homogenization)

• Human-made Minecraft skins strongly prioritize global color harmony (e.g., soft pastel palettes). The Stage 1 prior deeply learned this aesthetic consistency.
• If a prompt specifies a dominant color (e.g., a light mint dress) alongside a starkly contrasting local element (e.g., "tall dark brown boots"), the model tends to harmonize the boots into a matching pale cream/green tone rather than stark dark leather.

3. Contrastive Text Encoder "Bag-of-Words" Effect

• Like all models conditioned on CLIP/SigLIP, complex syntactic modifiers can bleed. Prompts like "a man in a purple suit with a pale face" can occasionally cause purple color to bleed into skin tones unless explicitly separated by tags or negative prompts.

4. Representation Bias

• The underlying 1.1M dataset is heavily dominated by aesthetic anime/human female skins. The model excels at aesthetic hair gradients, hoodies, and dresses, but is weaker at non-human mobs (e.g., Creepers, robots, armored knights).

5. Random Seed Limitation

• Lower random seeds (e.g. 21, 42, 35) often produce much better results than higher random seeds (e.g. 12345678987, 1794051925).

Quickstart (Inference Code)

import torch
import numpy as np
from PIL import Image
from diffusers import UNet2DConditionModel, DPMSolverSinglestepScheduler
from transformers import AutoTokenizer, SiglipTextModel

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
MODEL_ID = "zhoudoe23/MCDiffusion-35M"
SIGLIP_ID = "zhoudoe23/siglip-so400m-mcskin-fused"

# 1. Load Models
unet = UNet2DConditionModel.from_pretrained(MODEL_ID, torch_dtype=torch.float16).to(DEVICE)
unet.eval()

tokenizer = AutoTokenizer.from_pretrained(SIGLIP_ID)
text_model = SiglipTextModel.from_pretrained(SIGLIP_ID, torch_dtype=torch.float16).to(DEVICE)
text_model.eval()

# 2. Setup DPM-Solver++ 2S Scheduler
scheduler = DPMSolverSinglestepScheduler(
    num_train_timesteps=1000,
    beta_start=0.0001,
    beta_end=0.02,
    beta_schedule="linear",
    algorithm_type="dpmsolver++",
    solver_order=2,
)
scheduler.set_timesteps(25, device=DEVICE)

# 3. Prompt Setup
prompt = "bright green eyes, pale face, long pink hair, cat ears, black cat hoodie, white skirt"
negative_prompt = "faceless, no eyes, blind, blurry face, flat skin"
model_type = "slim" # "classic" (4px arms) or "slim" (3px arms)

with torch.no_grad():
    cond_inputs = tokenizer([prompt], padding="max_length", max_length=64, truncation=True, return_tensors="pt").to(DEVICE)
    cond_emb = text_model(**cond_inputs).last_hidden_state
    
    neg_inputs = tokenizer([negative_prompt], padding="max_length", max_length=64, return_tensors="pt").to(DEVICE)
    uncond_emb = text_model(**neg_inputs).last_hidden_state

text_embs = torch.cat([uncond_emb, cond_emb], dim=0)
class_id = 1 if model_type == "slim" else 0
class_labels = torch.tensor([class_id, class_id], device=DEVICE)

# 4. Latent Sampling (with Clamping for stability)
g = torch.Generator(device=DEVICE).manual_seed(42)
raw_latents = torch.randn((1, 4, 64, 64), generator=g, device=DEVICE, dtype=torch.float16)
latents = raw_latents.clamp(-2.5, 2.5) # Clean outlier noise

guidance_scale = 3.0

with torch.no_grad():
    for t in scheduler.timesteps:
        latent_input = torch.cat([latents] * 2)
        noise_pred = unet(latent_input, t, encoder_hidden_states=text_embs, class_labels=class_labels).sample
        uncond, text = noise_pred.chunk(2)
        noise_pred = uncond + guidance_scale * (text - uncond)
        latents = scheduler.step(noise_pred, t, latents).prev_sample

# 5. Export 64x64 RGBA Skin Texture
samples = ((latents.float() + 1.0) / 2.0).clamp(0, 1) * 255.0
arr = samples[0].permute(1, 2, 0).cpu().numpy().astype("uint8")

# Binarize outer layer Alpha channel
arr[:, :, 3] = np.where(arr[:, :, 3] > 128, 255, 0)

skin_image = Image.fromarray(arr, mode="RGBA")
skin_image.save("generated_skin.png")

---

Disclaimer & Acknowledgements

• Not an Official Minecraft Product: This project is not approved by or associated with Mojang or Microsoft.
• Built using Diffusers and Transformers.
• Pretrained on datasets sourced from public community skin archives.