echoforest/yt-chat-gemma4-12b-screen-heads-base

YouTube live-chat market classifier: Gemma 4 12B, screening arm heads-base

Screening checkpoint, not a production model. This is one of four arms trained on the same 15,000-turn sample for one epoch to choose the configuration for the full training run. Scores are on a fixed 4,000-turn validation sample, before calibration. The model classifies one author's turn in Korean bitcoin-futures YouTube live chat into eight labels.

• Base model: google/gemma-4-12B (base checkpoint), text decoder only.
• Arm: LoRA adapter on the text decoder plus eight linear heads (heads.safetensors) on the final-norm hidden state of the last token. One forward pass per turn.
• Labels: stored reference labels from an LLM labeller. Every score measures agreement with those labels, not ground truth.

Screening results

Correct direction views at the lowest gate τ with ≥80% val precision, and mean field agreement, on the same 4,000 val turns. Both backbones were screened with the same recipe and data; the production model is the Qwen3.5-9B heads-instruct arm retrained in full (echoforest/yt-chat-classifier-qwen3.5-9b).

Arm	Gemma 4 12B (this screen): correct views / agreement	Qwen3.5-9B: correct views / agreement
heads-base (this model)	936 / 88.3%	707 / 86.4%
heads-instruct	0 / 68.9%	717 / 86.2%
gen-base	903 / 88.6%	705 / 86.6%
gen-instruct	876 / 88.5%	624 / 86.1%

This arm: mean field agreement 88.3%; all eight fields identical on 46.0% of turns; direction views: 936 correct of 1170 accepted (precision 80.0%, recall 68.3%) at τ 0.48.

Per-field val agreement: relevance 95.3 · direction 81.3 · direction_basis 84.1 · target 90.7 · own_position_or_trade 94.2 · pnl_state 97.4 · expressed_emotion 76.0 · sarcasm 87.7.

Fields

Field	Options (in order)
relevance	market, off_topic, unclear
direction	bullish, bearish, rangebound, mixed, none, unclear
direction_basis	explicit_forecast, own_position, recommendation_to_others, wish_or_hope, reaction_to_move, not_applicable
target	bitcoin, ethereum, named_altcoin, altcoins_general, crypto_market, traditional, multiple, none
own_position_or_trade	long, short, took_profit, stopped_or_liquidated, hedged, flat_waiting, none
pnl_state	profit, loss, liquidated_or_large_loss, none
expressed_emotion	fear, fomo, greed_euphoria, enthusiasm, frustration, regret, mockery, uncertainty, calm, mixed, none
sarcasm	likely, not_evident, unclear

Input format

Transcripts must be rendered exactly as in training (the project's N1 template): a stream preamble with a slang glossary, recent earlier chat, and the target author's lines fenced as <<<...>>>. Gemma needs exactly one <bos> at the start.

Usage

import json, torch
from huggingface_hub import snapshot_download
from peft import PeftModel
from safetensors.torch import load_file
from transformers import AutoConfig, AutoTokenizer, Gemma4UnifiedForCausalLM

repo = snapshot_download("echoforest/yt-chat-gemma4-12b-screen-heads-base")
base = "google/gemma-4-12B"
lm = Gemma4UnifiedForCausalLM.from_pretrained(
    base, config=AutoConfig.from_pretrained(base).text_config, dtype=torch.bfloat16, device_map="cuda",
    # Text decoder only. The checkpoint stores it as model.language_model.*, so rename the keys,
    # otherwise every weight is silently re-initialised.
    key_mapping={r"^model\.language_model\.": "model."})
text = PeftModel.from_pretrained(lm.model, repo).eval()
labels = json.load(open(f"{repo}/labels.json"))
heads = {k: v.cuda().float() for k, v in load_file(f"{repo}/heads.safetensors").items()}
tok = AutoTokenizer.from_pretrained(base)
piece = lambda s: tok(s, add_special_tokens=False)["input_ids"]

def classify(transcript):
    ids = torch.tensor([[tok.bos_token_id] + piece(transcript + "\n### LABELS")], device="cuda")  # exactly one <bos>
    with torch.no_grad():
        h = text(input_ids=ids).last_hidden_state[0, -1].float()
    return {f: dict(zip(labels["options"][f],
                     torch.softmax(h @ heads[f + ".weight"].T + heads[f + ".bias"], -1).tolist()))
            for f in labels["fields"]}

Needs transformers ≥ 5.20 (gemma4_unified) and peft ≥ 0.21. No custom kernels are needed. Gemma caps its output logits at 30·tanh(x/30); the generative snippet applies the same cap. Probabilities are uncalibrated (screening runs fit no temperatures).

Training

• Data: export v20261016-1835 (splits by whole stream). A stream-proportional 15,000-turn train sample (seed 13), 1 epoch, max length 1024 tokens, nothing truncated. Turns that needed stream audio trained at half weight.
• Method: LoRA r 32 / α 32 / dropout 0.05 on attention q/k/v/o and MLP gate/up/down; heads trained in full with dropout 0.1, cross-entropy summed over fields, direction ×2. LoRA LR 0.0002, heads LR 0.001, cosine schedule, 3% warmup, AdamW, weight decay 0.01, 32 sequences per step, bf16, gradient checkpointing.
• Compute: 1× H100 80GB, about 3,100 tokens/s.
• Versions: torch 2.8.0+cu128, transformers 5.20.1, peft 0.21.2.

Limitations

• Screening checkpoint: trained on about a fifth of the data for one epoch, uncalibrated, and evaluated only on a val sample from 2 streams. The test set is reserved for the final run.
• Narrow data: 15 train streams over 6 days (2026-10-10 to 10-15), all Korean crypto-futures chat.
• Chat text that imitates the transcript fences or gives instructions can move labels; the robustness suite is run on the final model only.