Llama-3.2-1B-Chiptune-MIDI

A LoRA adapter that fine-tunes slseanwu/MIDI-LLM_Llama-3.2-1B to generate 8-bit/chiptune-style video game music from text prompts.

Built with Llama.

Hear the Results

Baseline means the music which generated using MIDI-LLM models (before finetuned). Finetuned wav extension means the music generated by Llama-3.2-1B-Chiptune-MIDI (after finetuned) The baseline files contain music generated using the original MIDI-LLM model (before fine0tuning). The fine-tuned files feature music generated by the Llama-3.2-1B-Chiptune-MIDI adapter.

Prompt : "A cheerful 8-bit video game chiptune with fast arpeggios"

Prompt : "An 8-bit platformer game theme, upbeat and adventurous"

Model Details

Model Description

MIDI-LLM extends Llama 3.2 1B's vocabulary with 55,030 tokens representing MIDI events (using the Anticipatory Music Transformer's arrival-time tokenization), turning it into a text-to-MIDI generator. The base checkpoint was trained on general-purpose MIDI (Lakh MIDI + MidiCaps captions), so its zero-shot output leans toward generic pop/classical/rock textures even when asked for "chiptune."

This adapter fine-tunes that base model on a collection of chiptune MIDI transcriptions from retro video game soundtracks, shifting generation toward chiptune-characteristic pitch ranges, note density, and instrumentation (square/pulse-wave leads, synth bass, drums) — see Results below for a quantitative before/after comparison.

• Developed by: omayib
• Funded by: N/A (personal/independent project)
• Shared by: omayib
• Model type: LoRA adapter for LlamaForCausalLM (extended-vocabulary text-to-MIDI model)
• Language(s): English (text prompts/captions)
• License: Llama 3.2 Community License (inherited from the base model — see Meta's license terms for redistribution/use requirements)
• Finetuned from model: slseanwu/MIDI-LLM_Llama-3.2-1B

Model Sources

• Base model repository: https://github.com/slSeanWU/MIDI-LLM
• Base model paper: Wu, S.-L., Carlton, D., Mikayawa, R., Kim, Y., Donahue, C., Huang, C.-Z. A. "MIDI-LLM: Improving text-to-MIDI music generation via adapting large language models." ISMIR 2026.

Uses

Direct Use

Generating short (~10-30 second) chiptune/8-bit-style MIDI clips from a text description — e.g. "a cheerful 8-bit platformer theme," "a tense chiptune boss battle track with fast pulse-wave arpeggios." Intended for prototyping, hobbyist game-jam music, and as a fine-tuning case study.

Out-of-Scope Use

• Not a general-purpose music generator. Fine-tuning on chiptune narrows the model's range; expect degraded quality on prompts asking for other genres compared to the base MIDI-LLM checkpoint.
• Not suitable for commercial release without review. The training data (see below) is derived from fan-made transcriptions of copyrighted video game soundtracks; the copyright status of model outputs trained on such material is legally unsettled. Users taking this beyond a hobby/research context should seek their own legal guidance.
• Not a transcription or arrangement tool. It generates new sequences stylistically influenced by the training data; it does not reproduce specific existing game tracks on request, though outputs may occasionally resemble source material given the training data's nature.
• Not tuned for bass-heavy or drum-heavy compositions. See Limitations below — bass instrumentation did not transfer as strongly as lead melody during fine-tuning.

Bias, Risks, and Limitations

• Training data provenance: the fine-tuning data consists of MIDI transcriptions of copyrighted video game soundtracks (fan-made rips), not licensed source material. This model card discloses that fact for transparency; it does not constitute a claim of rights over the underlying compositions.
• Instrumentation gaps: comparing baseline vs. fine-tuned generations against a reference set of real chiptune MIDI showed the model learned to favor square-wave lead instruments (matching the reference distribution well), but did not reliably pick up characteristic synth-bass usage — bass instrumentation in outputs is closer to the vanilla base model's habits than to real chiptune's.
• Note density undershoot: fine-tuned outputs average ~17.4 notes/sec vs. ~21.2 notes/sec in the real chiptune reference set (vanilla baseline overshot at ~29.3 notes/sec) — outputs may feel slightly sparser than typical chiptune.
• Occasional off-genre leakage: a minority of generations still use non-chiptune instruments (e.g. acoustic piano) inherited from the base model's pretraining.
• Small-scale fine-tune: trained with a LoRA adapter (rank 16, ~0.2% of parameters) on a POC-scale dataset — not a large-scale or exhaustively validated training run.

Recommendations

Listen to and review generations before use; do not assume outputs are free of copyright encumbrance given the training data's nature; expect occasional off-genre or under-characteristic (e.g. bass-light) outputs.

How to Get Started with the Model

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, LogitsProcessor, LogitsProcessorList
from anticipation.convert import events_to_midi
import torch

base_model = AutoModelForCausalLM.from_pretrained(
    "slseanwu/MIDI-LLM_Llama-3.2-1B", dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base_model, "omayib/Llama-3.2-1B-Chiptune-MIDI")
tokenizer = AutoTokenizer.from_pretrained("slseanwu/MIDI-LLM_Llama-3.2-1B", pad_token="<|eot_id|>")
model.eval()

LLAMA_VOCAB_SIZE = 128256
AMT_GPT2_BOS_ID = 55026
MIDI_BOS_ID = AMT_GPT2_BOS_ID + LLAMA_VOCAB_SIZE  # 183282
ALLOWED_TOKEN_IDS = range(LLAMA_VOCAB_SIZE, LLAMA_VOCAB_SIZE + AMT_GPT2_BOS_ID)
SYSTEM_PROMPT = (
    "You are a world-class composer. Please compose some music according "
    "to the following description: "
)

class RestrictToMidiVocab(LogitsProcessor):
    def __init__(self, allowed_ids, eos_token_id):
        self.mask = None
        self.allowed_ids = allowed_ids
        self.eos_token_id = eos_token_id

    def __call__(self, input_ids, scores):
        if self.mask is None:
            mask = torch.full((scores.shape[-1],), float("-inf"), device=scores.device)
            mask[self.allowed_ids.start:self.allowed_ids.stop] = 0.0
            mask[self.eos_token_id] = 0.0
            self.mask = mask
        return scores + self.mask

@torch.inference_mode()
def generate_chiptune(prompt, max_new_tokens=1024, top_p=0.9, temperature=1.0, output_path="generated.mid"):
    full_prompt = SYSTEM_PROMPT + prompt + " "
    input_ids = tokenizer(full_prompt, return_tensors="pt", add_special_tokens=True)["input_ids"].to(model.device)
    input_ids = torch.cat([input_ids, torch.tensor([[MIDI_BOS_ID]], device=model.device)], dim=1)
    logits_processor = LogitsProcessorList([RestrictToMidiVocab(ALLOWED_TOKEN_IDS, tokenizer.eos_token_id)])
    output = model.generate(
        input_ids, max_new_tokens=max_new_tokens, do_sample=True, top_p=top_p, temperature=temperature,
        pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id, logits_processor=logits_processor,
    )
    generated = output[0][input_ids.shape[1]:].tolist()
    midi_tokens = [t - LLAMA_VOCAB_SIZE for t in generated if t != tokenizer.eos_token_id]
    events_to_midi(midi_tokens).save(output_path)
    return output_path

generate_chiptune("A cheerful 8-bit video game chiptune with fast arpeggios", output_path="my_chiptune.mid")

Requires pip install peft 'anticipation @ git+https://github.com/jthickstun/anticipation.git@af37397922665a0fb8d474d7988b0f3755a38d45'.

Training Details

Training Data

A private collection of chiptune MIDI transcriptions from retro video game soundtracks, chunked into ~20-second segments and captioned automatically:

• Objective musical features (tempo, key, instrument roles, note density, register) were extracted programmatically from each chunk.
• Captions were generated from those features using an LLM constrained to only describe extracted facts (not to invent genre/mood beyond what the data supports), in the style of the MidiCaps caption format the base model was trained on.
• Split into train/val/test by source song (not by chunk) to prevent near-duplicate leakage across splits.

The dataset is not published alongside this model, since the underlying MIDI transcriptions are derived from copyrighted commercial video game soundtracks (see Limitations).

Training Procedure

Preprocessing

• MIDI cleaned: tracks with fewer than 10 notes dropped (typically leftover automation/SFX channels); instrument program numbers normalized to a small canonical chiptune-relevant set.
• Chunked into ~20-30 second windows.
• Tokenized with anticipation.convert.midi_to_events (Anticipatory Music Transformer's arrival-time tokenization), then shifted by +128256 into the base model's extended vocabulary space.
• Text captions wrapped in the same system prompt format used by the base model at inference: "You are a world-class composer. Please compose some music according to the following description: " + caption + " ".
• A MIDI-specific BOS token (id 183282) inserted between the text prompt and MIDI tokens, matching the base model's inference-time format.

Training Hyperparameters

• LoRA config: rank 16, alpha 32, targeting q_proj, k_proj, v_proj, o_proj; dropout 0.05 (3,407,872 trainable params, ~0.2% of the 1.7B total)
• Effective batch size: 8 (per-device batch size 2, gradient accumulation 4)
• Learning rate: 1e-4, cosine schedule, 3% warmup
• Weight decay: 0.01
• Epochs: ceiling of 15, with early stopping (patience 3 evaluations on eval loss) — training stopped at epoch 8.56, best checkpoint at epoch 7.75
• Training regime: bf16 mixed precision

Speeds, Sizes, Times

• Hardware: single NVIDIA A100 GPU (Google Colab)
• Adapter size: ~13MB (LoRA weights only, base model not included)

Evaluation

Testing Data, Factors & Metrics

Evaluated on a held-out validation split (chunks from source songs not seen during training), using the same text-caption-to-MIDI-token format as training.

Metrics

• Eval loss (cross-entropy on MIDI token predictions, text portion masked out)
• Perplexity (exp(eval_loss))
• Token-level top-1 accuracy on the MIDI token portion

Results

Best checkpoint (epoch 7.75): eval loss 0.8835, perplexity 2.42, token accuracy 79.77%.

A separate qualitative comparison generated matched prompts from the vanilla base model and this fine-tuned adapter, then compared both against a reference set of real chiptune MIDI on objective musical features:

Metric	Reference (real chiptune)	Baseline (vanilla)	Fine-tuned (this model)
Pitch mean	47.9	52.5	43.4
Pitch std dev	25.8	15.7	25.2
Pitch min	1	30	1
Pitch max	108	100	103
Notes/sec	21.2	29.3	17.4
Top instrument	Lead 1 (square)	Drums / Piano	Drums / Lead 1 (square)

Summary

Fine-tuning measurably shifted pitch range/spread and instrumentation toward the real chiptune reference distribution — notably, square-wave lead usage went from near-absent in the baseline to the model's #2 most-used instrument. Note density and bass-instrument usage did not fully converge to the reference (see Limitations).

Technical Specifications

Model Architecture and Objective

LlamaForCausalLM (1.4B base params) with a vocabulary extended to 183,286 tokens (128,256 text + 55,030 MIDI event tokens from the Anticipatory Music Transformer vocabulary). This adapter adds a rank-16 LoRA to the attention projection layers; the causal language modeling objective is unchanged, trained on next-MIDI-token prediction conditioned on a text caption.

Compute Infrastructure

Hardware

Single NVIDIA A100 GPU, Google Colab.

Software

• PyTorch (CUDA 12.8 build)
• Transformers (note: TrainingArguments.warmup_ratio was removed in Transformers v5 in favor of a unified warmup_steps argument that accepts a float ratio)
• PEFT 0.20.0
• Anticipation (pinned to commit af37397) for AMT MIDI tokenization/detokenization

Citation

If you use this adapter, please also cite the base MIDI-LLM model this work builds on:

BibTeX:

@inproceedings{wu2026midillm,
  title={{MIDI-LLM}: Improving text-to-{MIDI} music generation via adapting large language models},
  author={Wu, Shih-Lun and Carlton, Dave and Mikayawa, Ryan and Kim, Yoon and Donahue, Chris and Huang, Cheng-Zhi Anna},
  booktitle={International Society for Music Information Retrieval Conference (ISMIR)},
  year={2026}
}

Model Card Contact

See the model repository for contact information.

Framework versions

• PEFT 0.20.0