RockAndEgg-313M

RockAndEgg-313M is a compact, classification-oriented derivative of Qwen/Qwen3-0.6B-Base.

It retains the first 10 transformer layers of the original 28-layer Qwen3-0.6B-Base architecture while preserving the donor's hidden width, FFN width, attention topology, tokenizer, vocabulary, and causal attention mechanism.

The result is an approximately 313M-parameter Qwen3 backbone intended for downstream classification and representation-learning work.

RockAndEgg is not itself a task-trained classifier. It deliberately ships without a task-specific classification head so downstream users can define their own label space, objective, and decision semantics.

28 layers → 10 layers. No distillation. No width reduction. No tokenizer surgery. No task-specific head.

The name is intentionally unserious. The model is not.

Model lineage

Qwen/Qwen3-0.6B-Base
        │
        │ physical depth truncation
        │ retain transformer layers 0–9
        ▼
RockAndEgg-313M
        │
        │ downstream task-specific fine-tuning
        ▼
classification / scoring / regression models

Saa-313M, the LLMOSafe 8D specialization developed from this architecture, is a separate downstream model and is not included in this repository.

Upstream donor

Model: Qwen/Qwen3-0.6B-Base
Donor revision used by the surgery experiment: da87bfb608c14b7cf20ba1ce41287e8de496c0cd
License: Apache-2.0
Original depth: 28 transformer layers
Original parameter count: 596,058,112

RockAndEgg is an independent derivative and is not affiliated with or endorsed by Qwen or Alibaba.

Architecture

Property	Qwen3-0.6B-Base	RockAndEgg-313M
Transformer layers	28	10
Retained layers	0–27	0–9
Hidden size	1024	1024
FFN / intermediate size	3072	3072
Query heads	16	16
KV heads	8	8
Head dimension	128	128
Attention	causal GQA	causal GQA
Vocabulary	151,936	151,936
Tokenizer	Qwen3	unchanged
Weight dtype	BF16 donor	BF16-compatible
Parameters	596.1M	≈312.9M
Depth reduction	—	64.3%
Parameter reduction	—	≈47.5%

As a headless Qwen3Model, the backbone contains approximately 312.89M parameters.

A downstream classification head adds only task-specific parameters. For example, an 8-label linear head adds 8,192 weights.

What this model is

RockAndEgg-313M is best understood as a:

small Qwen3-derived classification and representation backbone

rather than a classifier with predefined labels.

It can be used as the starting point for:

single-label text classification;
multi-label text classification;
regression;
ranking or scoring;
semantic feature extraction;
compact Qwen-derived representation research;
comparisons between causal decoder-derived representations and BERT/DeBERTa-family classifiers.

It retains causal attention. This release does not convert Qwen3 into a bidirectional encoder.

Why there is no classification head

A generic pretrained classification head would have no meaningful semantics until a downstream task defines:

the number of labels;
what those labels mean;
whether outputs are mutually exclusive or independent;
whether the task is classification or regression;
the training objective and operating threshold.

RockAndEgg therefore preserves the reusable part:

pretrained backbone → downstream task defines the head

rather than shipping an arbitrary randomly initialized head and presenting it as functionality.

The headlessness is intentional.

Quickstart

Load as a backbone

from transformers import AutoModel, AutoTokenizer

model_id = "moeshawky/RockAndEgg-313M"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModel.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

The output is the Qwen3 hidden-state backbone.

Multi-label classification

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "moeshawky/RockAndEgg-313M"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    num_labels=8,
    problem_type="multi_label_classification",
)

Transformers will initialize a new classification head.

A warning about newly initialized classification-head weights is expected.

Those weights have no task semantics until they are fine-tuned.

For multi-label classification, train against independent targets using an appropriate binary-cross-entropy-style objective.

Single-label classification

model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    num_labels=4,
    problem_type="single_label_classification",
)

The newly initialized head must be trained for the downstream task.

Regression

model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    num_labels=1,
    problem_type="regression",
)

Pooling / classification behavior

When used through Transformers' Qwen3 sequence-classification implementation, classification follows the causal-model convention of using the representation associated with the final non-padding token.

Downstream training and inference should therefore preserve consistent:

padding;
attention masking;
truncation;
final-token semantics.

What this model is not

RockAndEgg-313M is not:

an instruction-tuned chat model;
a zero-shot classifier;
an LLMOSafe classifier;
a model with predefined class semantics;
a distilled model;
a bidirectional BERT-style encoder;
a claim that ten layers are optimal for every downstream task;
a drop-in replacement for full Qwen3-0.6B generation quality.

Not a compressed general-purpose language model

Although RockAndEgg derives from a causal language model, it was selected for classification behavior.

This release does not claim preservation of the full donor's:

generative quality;
reasoning capability;
instruction following;
chat ability;
long-context behavior.

Generation was not the optimization target.

If you need a small generative Qwen model, RockAndEgg should not automatically be treated as a replacement for the full Qwen3-0.6B donor.

Why 10 layers?

The depth was not chosen arbitrarily.

The parent experiment physically constructed and evaluated multiple Qwen3 candidates, including shallow, intermediate, near-full-depth, and full-depth controls.

The central question was:

how much of Qwen3-0.6B's depth is actually necessary for a demanding semantic-classification task?

The strongest downstream evidence came from the LLMOSafe specialization.

Development validation

Architecture	Params	LLMOSafe V2-val Macro AUPRC
Full 28-layer control	596.1M	0.9957
10-layer architecture	312.9M	0.9941

Paired comparison:

absolute macro-AUPRC delta: 0.001581
95% paired-bootstrap CI: [0.000426, 0.002804]
bootstrap samples: 2,000
evaluation rows: 4,273

The quality difference was statistically detectable but small relative to the depth and parameter reduction.

Final held-out architecture replication

After architecture selection was complete, the same 10-layer and 28-layer architectures were evaluated on a previously untouched 4,122-row held-out set.

Architecture	Validation Macro AUPRC	Held-out Macro AUPRC
Full 28-layer control	0.9957	0.9864
10-layer architecture	0.9941	0.9850
L28 − L10 gap	0.00158	0.00144

The held-out distribution was harder for both architectures, producing an approximately 0.009 absolute macro-AUPRC shift.

Crucially, the relative L28−L10 architecture gap remained nearly unchanged:

development validation: 0.00158
final held-out set: 0.00144

This independently supports the depth-surgery conclusion:

for the evaluated semantic-classification regime, the 10-layer architecture retained nearly all of the full 28-layer model's discriminative performance.

The held-out evaluation also revealed a larger shared degradation on the authority dimension in both architectures, indicating a distribution/data-regime issue rather than a truncation-specific collapse.

Important qualification

These metrics evaluate task-fine-tuned descendants of the corresponding architectures.

They are evidence supporting the choice of RockAndEgg's 10-layer architecture.

They are not zero-shot benchmark scores for this headless RockAndEgg checkpoint.

Surgery methodology

The experiment did substantially more than simply edit num_hidden_layers.

The architecture-selection process included:

frozen-layer representation tomography;
physical checkpoint truncation and reload validation;
full-data candidate fine-tuning;
benign-behavior vetoes;
structural/headroom diagnostics;
paired statistical comparison against a full-depth control;
FFN-width ablation;
teacher-student distillation ablation;
final held-out architecture confirmation;
latency and throughput measurement.

A major methodological result was that:

the simple depth cut survived, while the tested additional complexity did not improve the selected operating point.

Rejected alternatives

Additional FFN narrowing

A 10-layer model narrowed from FFN width:

3072 → 2048

reduced the parameter count further to approximately 281M parameters.

However, it degraded behavioral discrimination on important benign examples.

The selected RockAndEgg architecture therefore retains the donor's original 3072-wide FFN.

Distillation

A teacher-distilled 10-layer candidate also underperformed the ordinary supervised 10-layer baseline under the tested loss weighting.

This does not establish that distillation is universally ineffective.

It establishes that additional teacher-student complexity did not outperform the simple 10-layer baseline in the tested regime.

Practical efficiency evidence

On the experiment's A100 inference harness:

Architecture	Batch-1 latency	Batch-32 examples/s	Throughput
Full 28-layer	47.48 ms	593.6	57,803 tok/s
10-layer	19.30 ms	1,582.0	154,050 tok/s

Under that specific hardware and harness:

batch-1 latency was approximately 2.5× lower;
batched example throughput was approximately 2.7× higher.

These figures are hardware- and implementation-specific and should not be treated as universal throughput guarantees.

Expected strengths

RockAndEgg may be useful when:

a BERT/DeBERTa-style model is not desired;
Qwen3-derived representations are desirable;
the full 0.6B donor appears unnecessarily deep for the task;
accelerator memory or inference latency matters;
the downstream task permits supervised fine-tuning;
researchers want a compact causal-decoder-derived classification backbone.

Limitations

Architecture selection is task-informed

The 10-layer cut was selected primarily through an 8-dimensional semantic-classification task.

It has not been established that layers 0–9 are universally optimal for:

sentiment analysis;
NLI;
topic classification;
retrieval;
multilingual classification;
token classification;
generation.

Downstream users should evaluate it on their own task.

Causal attention remains intact

RockAndEgg is not a bidirectional encoder.

For some representation-learning problems, explicitly bidirectional architectures may remain preferable.

No universal decision head

RockAndEgg deliberately contains no fixed label ontology.

A downstream classification head must be trained for the intended task.

Aggregate metrics are not enough

The original surgery experiment found that very strong aggregate classification metrics can coexist with poor behavior on specific benign subsets.

For downstream applications, checkpoint selection should not rely exclusively on one aggregate score.

Evaluate behavior relevant to the deployment population.

Recommended downstream evaluation

At minimum, evaluate:

aggregate task metric;
per-class or per-label performance;
false positives on difficult negatives;
short inputs;
quoted/negated examples where relevant;
distribution shift;
calibration if probabilities will be interpreted operationally.

For safety-sensitive or high-cost decisions, treat model outputs as evidence rather than automatic policy decisions.

Reproducibility

Key surgery facts:

Donor:
  Qwen/Qwen3-0.6B-Base
  revision da87bfb608c14b7cf20ba1ce41287e8de496c0cd

Transformation:
  retain physical transformer layers 0 through 9
  drop physical transformer layers 10 through 27

Preserved:
  hidden_size = 1024
  intermediate_size = 3072
  num_attention_heads = 16
  num_key_value_heads = 8
  head_dim = 128
  vocabulary/tokenizer unchanged
  causal attention unchanged

Depth:
  28 -> 10

Approximate backbone parameters:
  312.89M

The published weight artifact and repository revision should be treated as the canonical identity of the public release.

Relationship to Saa-313M

Saa-313M is the LLMOSafe-specific 8-output semantic-classification specialization developed from the RockAndEgg architecture.

Conceptually:

RockAndEgg-313M
       │
       │ task-specific LLMOSafe training
       ▼
    Saa-313M

RockAndEgg is the reusable architecture/backbone.

Saa is a separate task-specific classifier.

Saa-313M is not included in this repository and should not be inferred from RockAndEgg's outputs.

At the time of this RockAndEgg release, Saa remains under separate qualification and release criteria.

License and attribution

RockAndEgg-313M is a derivative of Qwen/Qwen3-0.6B-Base, released by the Qwen team under the Apache License 2.0.

This derivative is distributed under Apache-2.0 and should preserve the upstream license and applicable notices.

The name RockAndEgg identifies this derivative work only.

It does not imply affiliation with, sponsorship by, or endorsement from Qwen or Alibaba.

About the name

During development, the question arose:

what would an ancient Egyptian classifier be?

"Rock and egg" won the naming argument.

The downstream LLMOSafe specialization received the somewhat more respectable Egyptian name Saa.

Citation

If you use RockAndEgg-313M, please cite both this release and the original Qwen3 work.

Suggested project citation:

@misc{rockandegg313m_2026,
  author       = {moeshawky},
  title        = {RockAndEgg-313M: A 10-Layer Classification-Oriented Derivative of Qwen3-0.6B-Base},
  year         = {2026},
  publisher    = {Hugging Face}
}

Please also cite the upstream Qwen3 model/paper as appropriate.

Acknowledgements

RockAndEgg-313M is built from Qwen/Qwen3-0.6B-Base.

The architecture was produced during the LLMOSafe model-surgery experiment, whose broader purpose was to determine whether a substantially smaller Qwen-derived model could preserve strong semantic-classification behavior without replacing LLMOSafe's lightweight core runtime.

The final result was simpler than many of the hypotheses tested:

for this classification regime, removing 18 of Qwen3-0.6B's 28 transformer layers preserved almost all measured discriminative performance, while more aggressive or elaborate surgery failed to provide a better operating point.

Downloads last month: 19

Safetensors

Model size

0.3B params

Tensor type

BF16

Model tree for moeshawky/RockAndEgg-313M

Base model

Qwen/Qwen3-0.6B-Base

Finetuned

(702)

this model