mp-juuuns/qwen35-standalone4l-classification-base

Qwen3.5 Standalone 4L Classification Base

This repository publishes a headless, multilingual-input, classification-ready four-layer Qwen3.5 text backbone. The root model has no task labels and no classification head. It is a compact starting point for user-trained single-label or multilabel classifiers, not a chat model or a ready-made universal classifier.

The repository follows one integrated 1-1-1 layout:

1. Repository root: task-agnostic four-layer base trained on unlabeled general text.
2. models/semeval-propaganda/: ready-to-use SemEval-derived propaganda classifier preserved as a separate task-specific lineage.
3. distillation/: reusable 24L→8L→6L→4L platform for user-owned data.

Model at a glance

Item	Root model
Upstream	Qwen/Qwen3.5-0.8B
Runtime class	Qwen3_5TextModel in Transformers 5.13.0
Text layers	4: linear_attention → full_attention → linear_attention → full_attention
Hidden / FFN	1,024 / 3,584
Vocabulary	full upstream Qwen3.5 tokenizer, 248,320 entries
Root parameters	334,096,704
Root BF16 weights	668,198,976 bytes (637.24 MiB)
Output	hidden states; no LM head, task head, labels, or thresholds
Task-free KD data	4,096 unlabeled WikiText-103 raw rows
Downstream evidence	one English 14-label SemEval-derived transfer probe

Load the headless base

from transformers import AutoModel, AutoTokenizer

model_id = "mp-juuuns/qwen35-standalone4l-classification-base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
backbone = AutoModel.from_pretrained(model_id, trust_remote_code=False)

The published root weight file contains no score.weight. A fresh task head and labeled fine-tuning are required before outputs have class meaning.

Make your own classifier

git clone https://huggingface.co/mp-juuuns/qwen35-standalone4l-classification-base
cd qwen35-standalone4l-classification-base/distillation
python -m pip install -e .

qwen35-distill finetune \
  --checkpoint .. \
  --train examples/multilabel_train.jsonl \
  --labels examples/labels.json \
  --mode multilabel \
  --output my-classifier-4l

For a full task-specific compression run, fine-tune a 24-layer teacher, then repeat materialize-classifier and distill-classifier for 24to8, 8to6, and 6to4. Exact JSONL formats, split rules, manifests, and commands are in distillation/README.md.

How the root weights were made

The text backbone and full tokenizer were extracted from the upstream Qwen3.5-0.8B model. Students were trained sequentially through 24→8→6→4 using hidden-boundary and final-representation matching on unlabeled text.

The task-free training set was exactly 4,096 deterministically shuffled, non-empty rows from the first locally cached WikiText-103 raw training shard. No SemEval examples, labels, logits, thresholds, classification heads, or evidence/span objectives were read while training the root weights.

KD stage	Selected source layers	Mean total loss	Mean interface loss	Mean final loss
24→8	0,4,6,11,13,16,20,23	0.176227	0.215628	0.166264
8→6	0,1,3,4,6,7	0.033821	0.044861	0.031147
6→4	0,2,3,5	0.047020	0.070484	0.043120

These are training-objective values, not downstream quality scores. Exact parameter counts, BF16 file sizes, hashes, and stage values are in benchmark/compression_ladder.csv and provenance/task_agnostic_stages/.

The chart reports the actual unsharded safetensors parameter counts and BF16 file sizes. It does not imply that downstream quality scales monotonically with layer count.

Downstream benchmark contract

Because the root is headless, it cannot be scored as a classifier without adaptation. Its task-appropriate benchmark is therefore a controlled transfer probe:

• attach a fresh randomly initialized 14-label classification head;
• train all parameters for five epochs on the same SemEval-derived train split;
• select one global threshold on the epoch-5 calibration split;
• evaluate once per seed on the already-opened 55-article test split;
• compare against the same four-layer structural copy without task-free KD.

This derived task measures article-level presence of 14 propaganda techniques. It is not the official SemEval span- or fragment-level task. Results are exploratory because the public test split had already been opened.

Frozen protocol

Setting	Value
Seeds	41, 42, 43
Train / calibration / test articles	260 / 56 / 55
Test windows	434
Epochs	5, no early stopping
Input windows	max length 256, overlap stride 128
Article aggregation	label-wise maximum probability
Batch	micro 1, gradient accumulation 32, effective 32
Optimizer	AdamW, LR 2e-5, weight decay 0.01, gradient clip 1.0
Precision / device	BF16 / NVIDIA GeForce RTX 5070 Ti
Threshold selection	epoch-5 calibration Macro-F1, then Micro-F1

Overall results by seed

Initialization	Seed	Threshold	Macro-F1	Micro-F1	Exact match	Predicted positive rate
Task-agnostic KD base	41	0.15	0.58831	0.64681	0.05455	0.57143
Task-agnostic KD base	42	0.30	0.60284	0.66552	0.03636	0.40909
Task-agnostic KD base	43	0.10	0.57128	0.64653	0.00000	0.51558
Structural copy, no task-free KD	41	0.05	0.51694	0.57740	0.00000	0.71299
Structural copy, no task-free KD	42	0.20	0.53494	0.60834	0.00000	0.62078
Structural copy, no task-free KD	43	0.20	0.50238	0.58564	0.00000	0.59610

Three-seed summary

Values are mean ± sample SD across the three training seeds. The SD is not a confidence interval.

Initialization	Macro-F1	Micro-F1	Exact match	Predicted positive rate
Task-agnostic KD base	0.58748 ± 0.01579	0.65295 ± 0.01088	0.03030 ± 0.02777	0.49870 ± 0.08248
Structural copy, no task-free KD	0.51809 ± 0.01631	0.59046 ± 0.01603	0.00000 ± 0.00000	0.64329 ± 0.06161
Existing specialized 4L, separate lineage	0.58757 ± 0.00717	see nested reports	see nested reports	see nested reports

The task-agnostic KD initialization improved same-seed mean Macro-F1 by +0.06939 over the structural-copy control. This is an observed comparison under this protocol, not a significance or universal-superiority claim.

Bars are three-seed means and whiskers are sample SD. The specialized 4L point is a separate historical lineage rather than a fine-tuned child of the root base.

Task-agnostic base: per-label test results

Precision, recall, and F1 are three-seed means. Support is the fixed number of positive test articles, not a three-seed sum.

Label	Support	Precision	Recall	F1
Appeal_to_Authority	11	0.3959	0.5758	0.4657
Appeal_to_fear-prejudice	24	0.5540	0.7361	0.6278
Bandwagon,Reductio_ad_hitlerum	7	0.4852	0.8095	0.6007
Black-and-White_Fallacy	12	0.3668	0.4722	0.4054
Causal_Oversimplification	18	0.5429	0.7963	0.6447
Doubt	22	0.5224	0.9848	0.6802
Exaggeration,Minimisation	22	0.4621	0.7727	0.5727
Flag-Waving	17	0.5643	0.8235	0.6465
Loaded_Language	45	0.8649	0.9852	0.9209
Name_Calling,Labeling	33	0.7381	0.9091	0.8125
Repetition	23	0.5197	0.7536	0.6148
Slogans	14	0.5161	0.7381	0.5835
Thought-terminating_Cliches	7	0.3492	0.3333	0.3190
Whataboutism,Straw_Men,Red_Herring	10	0.2762	0.5000	0.3304

Per-seed values and sample SDs for all labels and all arms are available in benchmark/per_label_metrics.csv and benchmark/per_label_summary.csv.

Labels are ordered by their three-seed mean F1. Support remains visible because several classes have only 7–14 positive test articles.

Efficiency and resource measurements

These numbers belong to the fresh-head transfer checkpoints, not a standalone headless-root latency benchmark.

Measure	Task-agnostic transfer model
Parameters with 14-label head	334,111,040
Test inference	0.08131 ± 0.00243 s/article
Test batch	55 articles / 434 windows
Peak CUDA allocated	3,380,975,616 bytes (3.15 GiB)
Total train-and-evaluate wall time	1,591.69 s mean (26.53 min)

The timing is specific to the recorded RTX 5070 Ti, BF16, batch-1, sliding-window protocol and includes article aggregation. It is not a claim about CPU, mobile, UNO Q, or other deployment performance. Full seed-level resource fields are in benchmark/resource_metrics.csv.

These plots compare the recorded fresh-head transfer checkpoints only. Apparent differences must not be generalized to the headless root on untested hardware.

Machine-readable benchmark bundle

• benchmark/BENCHMARK_CARD.md: generated benchmark report and interpretation
• benchmark/summary.json: complete aggregate
• benchmark/seed_metrics.csv: overall metrics per arm and seed
• benchmark/arm_summary.csv: three-seed means and sample SDs
• benchmark/per_label_metrics.csv: per-label values per arm and seed
• benchmark/per_label_summary.csv: per-label means and sample SDs
• benchmark/resource_metrics.csv: parameters, CUDA memory, wall time, and test timing
• benchmark/reports/: normalized full frozen reports, including article-level probability vectors
• benchmark/figures/: Matplotlib PNG/PDF figures, per-figure source CSVs, and a hash manifest

Multilingual scope

The root retains the upstream Qwen3.5 tokenizer and multilingual architecture. The official Qwen3.5-0.8B model card states expanded support for 201 languages and dialects and reports multilingual upstream benchmarks such as MMMLU, MMLU-ProX, NOVA-63, INCLUDE, Global PIQA, PolyMATH, WMT24++, and MAXIFE. This is why this repository carries the Hugging Face multilingual language and model tag.

However, this four-layer root was distilled on English WikiText and its only downstream transfer probe is English SemEval-derived data. Tokenizer coverage and inherited architecture support do not establish retained classification quality in all 201 languages. No multilingual downstream score is reported for this root release. Users should fine-tune and evaluate on each intended language and domain before making performance claims.

Separate SemEval model

The ready-to-use propaganda classifier is under models/semeval-propaganda/. It preserves the published seed-41/42/43 checkpoints and links to the original repository at immutable revision e7e0ff1.

That specialized model is not claimed to descend from this task-free base. It has its own SemEval task-specific shrink/distillation and fine-tuning history and a reduced 128k vocabulary, while the root retains the full upstream tokenizer. Its complete reports remain under models/semeval-propaganda/benchmark/.

Limitations

• A classification head and labeled training are required before root-model predictions have task meaning.
• Task-free KD used only 4,096 rows from one English WikiText shard.
• Only one downstream task and one hardware/software setting were measured.
• The 55-article test split was previously opened; results are exploratory.
• Per-label support ranges from 7 to 45 articles and rare-label estimates are unstable.
• Three seeds describe run-to-run variation; they are not a confidence interval.
• Multilingual input support has not been validated as multilingual downstream classification quality for this four-layer root.
• Default layer maps are documented structural choices, not universal optima.
• Long-context, calibration, robustness, fairness, and production safety have not been established.
• Do not use either model as a fact checker, safety oracle, or autonomous decision maker.

License and attribution

The released code and model artifacts use Apache-2.0, subject to upstream model and data terms. Qwen3.5 is attributed to Qwen. Transformers, PyTorch, Hugging Face Hub, WikiText, and SemEval remain the work of their respective authors. No WikiText or SemEval source records are redistributed.

The WikiText page currently has a license wording discrepancy: metadata lists CC BY-SA 3.0 and GFDL, while prose says CC BY-SA 4.0. Users should inspect Salesforce/wikitext directly.