pragmaticcs/SignOfFour

A four-way MoE merge of the Qwen 35B-A3B architecture, fusing task vectors from three specialized fine-tunes into a base anchor via DARE-TIES with sinusoidal depth modulation.

Designed specifically to consolidate software engineering, code synthesis, and agentic tool execution capabilities. Multimodal vision weights and Multi-Token Prediction (MTP) heads were stripped to reduce VRAM footprint and maximize throughput during coding tasks.

---

Contents

• Architectural Specifications
• Composition
• Merge Methodology
• Layer-Stratified Policies
• Chat Template
• Generation Parameters
• How to Use
• Lineage
• References

---

Architectural Specifications

Spec	Value
Total parameters	35B
Active parameters / token	3B
Decoder layers	40
Routed experts	256
Shared experts	1
Attention	Gated DeltaNet hybrid linear attention
Merge algorithm	DARE-TIES + sine depth scaling

Composition

Jackrong/Qwopus3.6-35B-A3B-Coder serves as the base anchor (W₀); the remaining three models contribute task vectors at the listed weights.

Model	Role	Task Weight (α)
Jackrong/Qwopus3.6-35B-A3B-Coder	Base anchor (W₀)	1.00
ornith-ai/Ornith-1.5-35B-A3B	Donor (D₁)	0.30
Kwaipilot/KAT-Coder-V2.5-Dev	Donor (D₂)	0.25
Qwen/Qwen-AgentWorld-35B-A3B	Donor (D₃)	0.20

---

Merge Methodology

For each floating-point parameter, a task delta is computed per donor $kk$:

$${\mathrm{\Delta }}_k=D_k-W_0\backslash Delta\_k\; =\; D\_k\; -\; W\_0$$

DARE pruning. A Bernoulli mask at retention density $pp$ zeroes out low-magnitude updates; surviving values are rescaled by $p^{-1}p^\{-1\}$:

$${\tilde{\mathrm{\Delta }}}_k=\frac{1}{p}({\mathrm{\Delta }}_k\odot M_k),M_k\sim \text{Bernoulli}(p)\backslash tilde\{\backslash Delta\}\_k\; =\; \backslash frac\{1\}\{p\}\; \backslash left(\backslash Delta\_k\; \backslash odot\; M\_k\backslash right),\; \backslash quad\; M\_k\; \backslash sim\; \backslash text\{Bernoulli\}(p)$$

TIES sign election. A consensus sign $\mathrm{\Gamma }\backslash Gamma$ is computed via weighted vote across donors, and any donor update conflicting with it is dropped before averaging:

$$\mathrm{\Gamma }=\mathrm{sgn}\left(\sum _{k=1}^K{\alpha }_k{\tilde{\mathrm{\Delta }}}_k\right)\backslash Gamma\; =\; \backslash operatorname\{sgn\}\backslash left(\backslash sum\_\{k=1\}^K\; \backslash alpha\_k\; \backslash tilde\{\backslash Delta\}\_k\backslash right)$$

$${\mathrm{\Delta }}_{\text{TIES}}=\frac{\sum _{k=1}^K{\alpha }_k{\tilde{\mathrm{\Delta }}}_k\odot \mathbb{I}(\mathrm{sgn}({\tilde{\mathrm{\Delta }}}_k)=\mathrm{\Gamma })}{\sum _{k=1}^K{\alpha }_k\cdot \mathbb{I}(\mathrm{sgn}({\tilde{\mathrm{\Delta }}}_k)=\mathrm{\Gamma })+ϵ}\backslash Delta\_\{\backslash text\{TIES\}\}\; =\; \backslash frac\{\backslash sum\_\{k=1\}^K\; \backslash alpha\_k\; \backslash tilde\{\backslash Delta\}\_k\; \backslash odot\; \backslash mathbb\{I\}\backslash left(\backslash operatorname\{sgn\}(\backslash tilde\{\backslash Delta\}\_k)\; =\; \backslash Gamma\backslash right)\}\{\backslash sum\_\{k=1\}^K\; \backslash alpha\_k\; \backslash cdot\; \backslash mathbb\{I\}\backslash left(\backslash operatorname\{sgn\}(\backslash tilde\{\backslash Delta\}\_k)\; =\; \backslash Gamma\backslash right)\; +\; \backslash epsilon\}$$

Depth-scaled reconstruction. The merged weight is reconstructed as:

$$W_{\text{final}}=W_0+\lambda (l)\cdot {\mathrm{\Delta }}_{\text{TIES}}W\_\{\backslash text\{final\}\}\; =\; W\_0\; +\; \backslash lambda(l)\; \backslash cdot\; \backslash Delta\_\{\backslash text\{TIES\}\}$$

where the layer scaling factor $\lambda (l)\backslash lambda(l)$ across decoder layer index $l\in [0,39]l\; \backslash in\; [0,\; 39]$ is defined as:

$$\lambda (l)=\beta \cdot (0.5+0.5\sin \left(\pi \frac{l}{39}\right))\backslash lambda(l)\; =\; \backslash beta\; \backslash cdot\; \backslash left(0.5\; +\; 0.5\; \backslash sin\backslash left(\backslash pi\; \backslash frac\{l\}\{39\}\backslash right)\backslash right)$$

This keeps input/output projections closer to the base and applies the strongest task transfer to middle layers $(l\in [12,28])(l\; \backslash in\; [12,\; 28])$.

---

Layer-Stratified Policies

Parameter Group	Match Substring	Policy	Density (p)	Base Scale (β)
Embeddings / LM head	embed_tokens, lm_head	Linear	—	1.00
Norms / biases	norm, bias, 1D tensors	Linear	—	1.00
DeltaNet recurrent state	a_log, dt_bias, conv1d	Linear	—	1.00
MoE router gate	mlp.gate.weight, block_sparse_moe.gate	Linear	—	1.00
MoE shared expert	shared_expert	DARE‑TIES	0.70	0.60
Attention projections	attn, rotary, in_proj, out_proj, x_proj	DARE‑TIES	0.75	0.60
Routed experts (×256)	experts, mlp	DARE‑TIES	0.65	0.55

• Router protection: Gate weights use linear interpolation (~57% base, ~43% donors) rather than DARE to avoid destabilizing expert routing.
• DeltaNet stability: Recurrent state kernels are excluded from DARE to prevent divergence in the linear-attention state space.
• MTP removed: Multi-token-prediction heads beyond the 40 primary decoder blocks were stripped for standard CausalLM inference.

---

Chat Template

This model uses the Improved Chat Template for Qwen 3.x by Olivia Rossi to support multi-tier Chain-of-Thought (CoT) reasoning, dual-format agentic tool execution, automatic error-recovery heuristics, and strict token-waste elimination.

---

Recommended Generation Parameters

For code generation and agentic task trajectories, avoid high temperatures to maintain routing stability and syntax validity.

Parameter	Coding / Terminal Agent	Creative Reasoning
Temperature	0.6	1.0
Top-P	0.95	0.95
Min-P	0.0	0.01
Repetition Penalty	off	1.05

---

How to Use

Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pragmaticcs/SignOfFour"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a precise agentic software engineer. Solve problems concisely."},
    {"role": "user", "content": "Write an asynchronous Python queue consumer with retry backoff."}
]

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

output = model.generate(
    inputs,
    max_new_tokens=1024,
    temperature=0.6,
    top_p=0.95,
    min_p=0.01,
    do_sample=True,
)

print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))

---

Lineage

Qwen/Qwen3.6-35B-A3B
└── pragmaticcs/SignOfFour
    ├── base:  Jackrong/Qwopus3.6-35B-A3B-Coder
    ├── donor: ornith-ai/Ornith-1.5-35B-A3B
    ├── donor: Kwaipilot/KAT-Coder-V2.5-Dev
    └── donor: Qwen/Qwen-AgentWorld-35B-A3B

---

Citation & References

• Jackrong/Qwopus3.6-35B-A3B-Coder
• ornith-ai/Ornith-1.5-35B-A3B
• Kwaipilot/KAT-Coder-V2.5-Dev
• Qwen/Qwen-AgentWorld-35B-A3B
• Improved Chat Template for Qwen 3.x

@inproceedings{yu2024dare,
  title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch},
  author={Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2024}
}

@inproceedings{yadav2023ties,
  title={Resolving Interference When Merging Models},
  author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2023}
}