StupidQwen3: Extreme Pruning and 1-Bit Quantization Experiment

This repository contains an experimental GGUF model derived from Qwen3-1.7B. It is the result of a controlled structural degradation test, combining aggressive layer pruning with extreme low-bit quantization to observe the mechanics of large language model collapse.

⚠️ WARNING: This model is not intended for practical use, inference, or deployment. It has been intentionally crippled and will produce high-entropy, non-sensical output. It serves strictly as a technical demonstration of model degradation.

🔬 Experimental Methodology

The base model (Qwen3-1.7B) was subjected to two sequential destructive operations:

1. Structural Pruning:
   • Layer 0 (initial token embedding projection) was removed.
   • Layer 27 (final pre-logit projection) was removed.
   • Rationale: Removing the input layer disrupts the initial mapping of tokens to the hidden state space, while removing the final layer prevents the model from coherently projecting its internal states back into the vocabulary distribution.
2. Extreme Quantization:
   • The remaining weights were quantized to Q1_0 (~1 bit per parameter) using llama.cpp.
   • Rationale: Forcing continuous float16 weights into a ternary/binary state without calibration or fine-tuning destroys the fine-grained magnitude relationships required for meaningful attention mechanisms.

Final Artifact Specifications:

• Original Size: ~3.4 GB (FP16)
• Final Size: ~466 MB (GGUF Q1_0 + tokenizer overhead)
• Effective Parameters: ~1.27B (due to layer removal)

📉 Observed Behavior (Model Collapse)

When prompted with a standard instruction (e.g., "Write a simple clicker game in Python"), the model does not fail silently. Instead, it exhibits a fascinating cascade of failure modes before settling into a degenerate attractor state:

1. Semantic Hallucination: The model outputs fragmented, out-of-context technical vocabulary (e.g., inert gas, polynomial bound, difference, ToObject, Cl()).
2. Vocabulary Bleeding: Due to corrupted logits, rare tokens from the extended vocabulary are randomly activated, including CJK characters (e.g., 钋 [Polonium], 砰 [bang], 巅 [peak]).
3. Degenerate Attractor Collapse: Within ~50-100 tokens, the softmax distribution collapses entirely. The hidden states become trapped in a local minimum, resulting in an infinite, repetitive loop of a single subword token (e.g., imbimbimbimb...).

Example Output

Prompt: Write a simple clicker game in Python Generation:

A gas gas gasA gasimb signs initial initial inert gas gas difference yesimb initial differenceampployment polynomial/__ inert imbimbimbimbinge bound bound inert sign initially difference polynomial same bound bound/__ bound difference bound bound/__ difference polynomial __ initiallyployment relevance imbimbimbimbimbimbimbimbinge/__ /__/__ initialA bound bound bound bound bound bound bound bound bound bound bound bound bound __ inert sign bound sign heatimbimbimbimbimbimb,varimbimbimbimbimbimbourcesimbimbimbimbimbimbimbimb 砰 bound/__ bound bound bound inert sign/__ bound/__ bound/__ bound惯imbagrid sign initial initially difference/__ bound ges flexible页 sign inert/__ bounduras bound __ initial free initial inert sign sign difference initial sign/__ /__ initially/__ bound sign inertA same/__ bound/__ bound/__ __ sign majority์ imbimbimbimbimbimbimb巅 imbimbagrid __ __ sign,$_imbimbimbagrid bound bound sign matter initial imbimbimbimb巅 imbimbimbimb,varimbimbimbimb巅 imb钋 imbimbimbimb砰 pageimb,var imbimbourcesimb钋 imbimbimbimbimb,varimbimbimbimb巅 imbimbagrid bound __ bound bound /_/_ _ bound/_ signs inert bound/__ bound/_/_ ever sign sign page imbimbimbimbimbimbimbToObjecturas sign flexible/__ signuras/_ /__With mon页 sign sign _) mon bound bound bound book/__ bound/__ bound bound ever sign flexible bound,$_页 /__ signhipsimbimbimbimbimbimb砰 imbimbimbimbourcesimbimbimb巅 imbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimbimb......

🛠️ Usage Instructions

This model is provided in GGUF format and requires llama.cpp (or a compatible GGUF inference engine) to run.

Prerequisites:

• A compiled version of llama.cpp
• The stupidqwen3.gguf file downloaded from this repository

Inference Command:

./llama-cli -m stupidqwen3.gguf -p "Write a simple clicker game in Python" -n 256 --temp 0.8

Note: Adjust -n (number of tokens) as desired. The model will inevitably enter the imb repetition loop regardless of temperature settings.

📚 Technical Context

This experiment demonstrates that LLMs do not degrade gracefully under extreme compression combined with structural damage. Unlike uniform noise, the output reveals the underlying "ghosts" of the training data (e.g., programming syntax, scientific terms) fighting against the mathematical collapse of the residual stream.

For comparison, a standard, healthy Gemma 3 1B at Q4_K_M quantization occupies ~810 MB and maintains coherent functionality. This artifact achieves a 43% smaller footprint at the total expense of utility, serving as a boundary test for quantization limits.

📜 License

This derivative work is released under the Apache 2.0 License, inheriting the licensing terms of the original Qwen3-1.7B model.

🙏 Acknowledgments

• Base Model: Qwen Team (Alibaba Cloud)
• Tooling: Hugging Face transformers, llama.cpp
• Methodology Assistance: Claude (for pruning script generation)