smoll-swarm/SmolCompressor-360M

🐝 SMOL COMPRESSOR 360M

The ultra-dense context compression and RAG-pruning engine for SMOL SWARM.

360M Parameters · 70–80% Context Reduction · Zero Metric Loss · Runs Locally on CPU

---

📌 Overview

Passing large 2,000-token document chunks, meeting transcripts, and noisy search results straight into primary reasoning models causes slow responses, burns token budgets, and leads to context dilution.

SmolCompressor-360M is a specialized causal language model fine-tuned to act as a lightweight RAG middleware and document pruner.

It takes messy, verbose source text and shrinks it by 70–80% while preserving 100% of the metrics, dates, names, error codes, and key decisions in a structured JSON payload.

🎯 How it Differs from SmolExtractor

Feature	SmolCompressor-360M (This Model)	SmolExtractor-1.7B
Core Job	Prunes context & summarizes long docs	Maps entities into rigid database tables
Output Type	Dense natural language paragraph + entities	Typed database fields (int, bool, float)
Needs a Schema?	No (Auto-compresses any document)	Yes (Requires target schema keys)
Target Consumer	Downstream LLMs, RAG pipelines, human readers	SQL databases, webhooks, backend APIs

---

⚡ Quick Start

import json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "smoll-swarm/SmolCompressor-360M"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
    device_map="auto"
)

SYSTEM_PROMPT = """You are SmolCompressor, an ultra-dense context compression engine.
Your job is to compress long, noisy documents, transcripts, and RAG context chunks by 70-80% while preserving 100% of facts, names, dates, metrics, and key decisions.
You must respond ONLY with a valid JSON object matching this schema:
{"compressed_context": "dense factual summary", "key_entities": ["list", "of", "entities"], "reduction": "percentage"}"""

raw_text = """Retrieved Document Chunk #102: In Q3 2024, CloudCorp reported total revenue of $84.2M, 
representing a 28% year-over-year increase. Operating margins expanded by 240 basis points to 22%. 
The cloud infrastructure division led by VP Sarah Connor accounted for $34M of total bookings. 
The company reaffirmed FY2025 guidance of $250M in annual recurring revenue."""

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": f"Compress and prune this context:\n\"\"\"{raw_text}\"\"\""}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=150,
        do_sample=False,  # Deterministic factual compression
        pad_token_id=tokenizer.eos_token_id
    )

output_ids = outputs[0][inputs.input_ids.shape[1]:]
result_json = json.loads(tokenizer.decode(output_ids, skip_special_tokens=True).strip())

print(json.dumps(result_json, indent=2))

---

📊 Output Schema

{
  "compressed_context": "CloudCorp Q3 2024 revenue: $84.2M (+28% YoY). Operating margin: 22%. Cloud division led by Sarah Connor generated $34M. FY2025 ARR guidance reaffirmed at $250M.",
  "key_entities": [
    "CloudCorp",
    "Q3 2024",
    "Sarah Connor",
    "FY2025 ARR"
  ],
  "reduction": "78%"
}

---

🏗️ Swarm Pipeline Architecture

In the Smol Swarm local multi-agent setup, SmolRouter sends summarization requests straight to SmolCompressor:

                       Incoming Document / RAG Chunk
                                     │
                                     ▼
                          ┌─────────────────────┐
                          │   SmolRouter-135M   │  ~ Intent Classifier
                          └──────────┬──────────┘
                                     │
                 ┌───────────────────┴───────────────────┐
                 ▼ route == 'compressor'                 ▼ route == 'extractor'
      ┌───────────────────────┐               ┌───────────────────────┐
      │  SmolCompressor-360M  │               │  SmolExtractor-1.7B   │
      │  (Context Pruner)     │               │  (JSON Specialist)    │
      └──────────┬────────────┘               └───────────────────────┘
                 │
                 ▼
      70–80% Smaller Context
      (Passed to Main LLM / RAG)

---

🧪 Verified Compression Examples

• Financial Earnings Call:

  • Raw Input (53 words): `"In Q3 2024, CloudCorp reported total revenue of $84.2M, representing a 28% year-over-year increase. Operating margins expanded by 240 basis points to 22%. The cloud infrastructure division led by VP Sarah Connor accounted for $34M of total bookings..."*
  • Compressed Output: {"compressed_context": "CloudCorp Q3 2024 revenue: $84.2M (+28% YoY). Operating margin: 22%. Cloud division led by Sarah Connor generated $34M. FY2025 ARR reaffirmed at $250M.", "key_entities": ["CloudCorp", "Q3 2024", "Sarah Connor", "FY2025 ARR"], "reduction": "78%"}

• Engineering Meeting Notes:

  • Raw Input (56 words): `"Alex discussed the ongoing database migration from Postgres 14 to Postgres 16. Timeline was delayed by 2 weeks due to index lock contention on table 'audit_logs'. Maria agreed to allocate 3 senior backend engineers to assist with schema partitioning..."*
  • Compressed Output: {"compressed_context": "Postgres 14->16 migration delayed 2 weeks due to index locking on audit_logs. Maria allocating 3 engineers for schema partitioning.", "key_entities": ["Postgres 16", "Maria", "Alex", "audit_logs"], "reduction": "73%"}

---

🛠️ Training Specifications

• Dataset: 2,500 domain-balanced context compression pairs (RAG retrieval chunks, incident post-mortems, meeting notes, customer support logs).
• Loss Masking: Prompt tokens masked using -100; gradient loss calculated exclusively on the target compression JSON payload.
• Hardware: NVIDIA RTX PRO 6000 Blackwell Server Edition (95 GB VRAM).
• Precision: Full Bfloat16 with TF32 Tensor Core optimizations.
• License: Apache 2.0

---

📦 Model Footprint

Format	File Size	Memory (RAM / VRAM)	Target Environment
Bfloat16 / FP16	~720 MB	~800 MB	PyTorch, Transformers, Python backends
Q8_0 GGUF	~380 MB	~450 MB	High-precision local CPU inference
Q4_K_M GGUF	~280 MB	~320 MB	Edge devices, laptops, mobile, Ollama

---

🐝 The Smol Swarm Colony

• SmolRouter-135M — Ultra-fast intent routing switchboard
• SmolExtractor-1.7B — Deterministic typed JSON extraction engine
• SmolCompressor-360M — Lossless context pruning for RAG pipelines (You are here)
• SmolPlanner-1.7B — Multi-step task decomposition and roadmapping
• SmolCoder-1.7B — Utility script and regex generation

---

Built with 🐝 by the SMOL SWARM community

Small models. Big swarm. Zero cloud bills.

```