scrapegoat/butterfly-tipping-point-50B

Butterfly Tipping Point 50B

A dual-track parallel transformer with learned inter-track gating

---

Overview

Butterfly Tipping Point (BTP) is a dual-track parallel transformer that runs two complete attention/MLP streams simultaneously at every layer and blends their outputs with a learned per-token gate — inspired by the corpus callosum connecting the brain's hemispheres.

Unlike traditional transformers where every layer applies the same attention mechanism, BTP maintains two parallel computation tracks:

Track	Name	Mechanism	Role
A	Butterfly Effect	Gated Delta Net (linear attention)	Fast, O(1) per-token state — captures local patterns and streaming context
B	Tipping Point	Full GQA self-attention	Precise, O(n²) — captures long-range dependencies and global reasoning

A learned corpus callosum gate (attn_track_gate) produces a 2-logit sigmoid that determines how much each track contributes at every position, allowing the model to dynamically allocate computation.

Architecture

                              |
                              v
              +-------------------------------+
              |           EMBEDDING           |
              |   Vocab: 248,320              |
              |   Dimension: 5,120            |
              +---------------+---------------+
                              |
                              v
        +---------------------------------------------+
        |             DUALTRACK LAYER x64             |
        |                                             |
        |   +----------------+    +----------------+  |
        |   |    TRACK A     |    |    TRACK B     |  |
        |   |                |    |                |  |
        |   |      GDN       |    |      GQA       |  |
        |   |    (linear)    |    |     (full)     |  |
        |   +-------+--------+    +--------+-------+  |
        |           |                      |          |
        |           +----------+-----------+          |
        |                      |                      |
        |                 GATE [2]                    |
        |                      |                      |
        |                      v                      |
        |          +-------------------------+        |
        |          |     DUAL MLP BLEND      |        |
        |          |                         |        |
        |          | gA * MLP_A + gB * MLP_B |        |
        |          +------------+------------+        |
        |                       |                     |
        +-----------------------+---------------------+
                                |
                                v
                  +---------------------------+
                  |   RMSNorm -> LM Head      |
                  +-------------+-------------+
                                |
                                v
                            OUTPUT

Layer Layout

The 64 layers use a 3:1 alternation pattern (matching Qwen3.5's native design):

Layer Type	Count	Layers
Linear Attention (Track A)	48	0,1,2, 4,5,6, 8,9,10, ...
Full Attention (Track B)	16	3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47, 51, 55, 59, 63

Every layer runs both tracks in parallel and blends via the learned gate. Full-attention layers handle the heavy lifting of global reasoning; linear-attention layers provide efficient streaming and local context.

Key Specifications

Parameter	Value
Total Parameters	~50B
Hidden Size	5,120
Num Layers	64
Attention Heads	24
KV Heads (GQA)	4
Head Dim	256
Intermediate Size	17,408
Max Context	262,144 tokens
Vocab Size	248,320
Normalization	GemmaRMSNorm (ε=1e-6)
Activation	SiLU
RoPE	Default θ=10M, partial rotary (25%)
Precision	bfloat16 (FP8 quantized serving)
Output Gate	Swish-gated LM head

Track A: Gated Delta Net (Linear Attention)

Based on Qwen3.5's GatedDeltaNet, Track A implements a linear-complexity recurrent attention mechanism:

• 16 key heads × 128-dim (vs. Track B's 4 KV heads × 256-dim)
• 48 value heads × 128-dim (richer value representation)
• Conv1d kernel (dim=4) for local context mixing
• 1D state-space recurrence — maintains a fixed-size hidden state per head, enabling O(1) per-token inference
• Processes the full sequence in a single pass (no KV cache needed)

Track B: Full GQA Self-Attention

Standard Grouped Query Attention with:

• 24 query heads × 256-dim
• 4 KV heads × 256-dim (6:1 query/KV ratio)
• QK-Norm via GemmaRMSNorm for training stability
• Sigmoid output gate (attn_output_gate=true)
• Full causal masking over the context window
• KV cache for autoregressive generation

Corpus Callosum Gate

The inter-track gate is a learned linear layer producing 2 logits per token:

gate = sigmoid(hidden_states @ attn_track_gate.weight.T + attn_track_gate.bias)
# gate ∈ [0, 1]² per token — per-track blending weights

Initialization: Quantile-Balanced — re-centers gate weights to zero mean and L2-normalizes them so logits are content-driven O(1) rather than saturated. This prevents the gate from collapsing to a hard 0/1 routing (all-A or all-B) early in training.

MLP blending: Both tracks have their own MLP (Qwen2MoEMLP with gated SiLU). The final MLP output is:

mlp_output = gate[:, 0] * MLP_A(hidden) + gate[:, 1] * MLP_B(hidden)

Training Checkpoint

The gate checkpoint (s2_000100.pt) is trained via a secondary S2 phase after the base model is frozen. The BTP_GATE_ALPHA and BTP_GATE_BIAS_B environment variables allow runtime adjustment of gate bias for ablation studies.

Serving

With SGLang

python3 -m sglang.launch_server \
  --model-path scrapegoat/butterfly-tipping-point-50B \
  --served-model-name butterfly_tipping_point \
  --host 0.0.0.0 --port 30000 \
  --attention-backend triton \
  --quantization fp8 \
  --mem-fraction-static 0.95 \
  --trust-remote-code

Environment Variables

Variable	Default	Description
BTP_ORACLE	none	oracle = alternation mode (B at layers %4==3, A elsewhere)
BTP_FORCE_TRACK	both	a = Track A only, b = Track B only
BTP_GATE_SCALE	1.0	Scaling factor for gate logits
BTP_GATE_ALPHA	0	Linear bias for Track B gate weight
BTP_GATE_BIAS_B	0	Constant bias added to Track B gate
BTP_DEQUANT_LOAD	0	1 = load FP8 weights as bf16 (debugging)
BTP_DEBUG_STATS	0	1 = log per-layer activation statistics

API Usage

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="dummy")

response = client.chat.completions.create(
    model="butterfly_tipping_point",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain the butterfly effect in chaos theory."}
    ],
    max_tokens=512,
    temperature=0.7,
)

print(response.choices[0].message.content)

Benchmark Results

How It Works

The Dual-Track Hypothesis

Traditional transformers process every token through the same mechanism at every layer. This is computationally expensive (quadratic attention) and doesn't allow the model to specialize different layers for different roles.

BTP addresses this by maintaining two parallel computation streams at every layer:

1. Track A (Butterfly Effect) — Fast, linear-complexity attention that maintains a fixed-size state. Like small perturbations in a chaotic system, it captures the "butterfly effect" of local token interactions.

2. Track B (Tipping Point) — Full quadratic attention that provides precise global reasoning. Like a phase transition at the "tipping point," it enables the model to synthesize information across the entire context.

The learned gate dynamically routes each token to the appropriate track based on content — simple tokens might get 90% Track A (fast), while complex reasoning tokens might get 50/50 or even 70% Track B.

Quantile-Balanced Gate Initialization

Without careful initialization, the gate quickly collapses to hard routing (always A or always B), defeating the purpose of dual tracks. The Quantile-Balanced initialization:

1. Re-centers each gate weight row to zero mean
2. L2-normalizes to unit norm
3. Scales by BTP_GATE_SCALE

This keeps both tracks active (~35-65%) and ensures the gate is driven by input content rather than weight magnitude.

Citation

@article{butterfly-tipping-point,
  title={Butterfly Tipping Point: Dual-Track Parallel Transformers with Learned Inter-Track Gating},
  author={ScrapeGoat Research},
  year={2026}
}

License

Apache 2.0

Acknowledgments

Built on top of SGLang serving framework and Qwen3.5 architecture. The GatedDeltaNet component is based on Yang et al., 2025.