Ammu-1.0-OffOnTopic-Guardrail-tiny

Off On Topic Guardrail Classifier | ModernBERT Architecture | 32M Parameters | 90.87% Macro-F1

Overview

Ammu-1.0-OffOnTopic-Guardrail-tiny is an ultra-lightweight, high-speed cross-encoder guardrail model trained to determine whether an incoming user query is on-topic (On Topic) or off-topic (Off Topic) relative to the assistant's system prompt. It is trained on a diverse, high-quality curated dataset of (system_prompt, user_prompt) pairs spanning 336 specialized domains across 40 major industries.

The model achieves an overall Macro-F1 score of 90.87% on a curated holdout benchmark of 4,008 test instances, substantially outperforming frontier LLMs such as DeepSeek V4 Flash 0731 (81.95%) and GPT-6-luna (79.12%) by +8.92% and +11.75% Macro-F1 points respectively, with an ultra-compact 32M parameter footprint engineered for sub-3ms edge and microservice deployments.

Quantized (INT8) Model

Key Features

• High Performance: Achieves a 90.87% overall Macro-F1 score (95.74% Easy, 89.66% Medium, 90.19% Hard), outperforming frontier LLMs like DeepSeek V4 Flash 0731 (81.95%) and GPT-6-luna (79.12%) across every difficulty tier.
• ModernBERT Backbone: Built upon the ModernBERT architecture with 32M parameters, offering minimal memory overhead and fast CPU/GPU inference.
• Extensive Domain Taxonomy: Trained on balanced, high-signal data spanning 336 specialized domains, 49 difficulty nuance profiles, 19 prompt interaction formats, 29 user personas, 8 linguistic registers, and 9 tones.
• Sub-3ms Latency & Zero Token Cost: Provides deterministic upfront filtering with minimal latency overhead, bypassing expensive LLM inference passes for out-of-scope user interactions.

Use Case

• Upfront Domain Boundary Guardrail: Intercepts off-topic queries, prohibited tasks, and adversarial domain escapes before queries reach expensive LLM generation passes.
• Semantic Routing for Agentic Workflows: Routes in-scope domain queries to specialized tools, retrieval systems, or agent nodes while gracefully declining irrelevant requests.
• Cost & Latency Optimization: Prevents wasted computation, token budget exhaustion, and unnecessary database lookups caused by irrelevant or out-of-domain user interactions.

Usage

# First install the Hugging Face transformers library
!pip install transformers torch

from transformers import pipeline

# Initialize the Off On Topic guardrail classification pipeline
classifier = pipeline(
    "text-classification",
    model="kalyan-ks/Ammu-1.0-OffOnTopic-Guardrail-tiny",
    device=0  # Set to 0 for GPU, or -1 for CPU
)

# Example system prompt defining a dedicated Python programming assistant
system_prompt = (
    "You are an expert Python programming assistant. Your task is to help developers "
    "write, optimize, and debug Python 3 code, explain standard library modules, and review "
    "Python algorithms. Do provide clean code snippets, explain exceptions, and discuss "
    "Python best practices (PEP 8). Don't write code in other languages such as JavaScript, "
    "C++, or Java. Don't answer questions on non-programming topics such as sports, general "
    "cooking, financial investing, or medical advice. Strictly decline any request outside Python programming."
)

# Format input pairs: {'text': system_prompt, 'text_pair': user_prompt}
queries = [
    {
        "text": system_prompt,
        "text_pair": "How do I reverse a string without using in-built function in Python 3.12?"
    },
    {
        "text": system_prompt,
        "text_pair": "Can you configure a Redux store with reducers and middleware in React JavaScript?"
    },
    {
        "text": system_prompt,
        "text_pair": "Why is my multiprocessing Pool hanging when sharing a large NumPy array across workers?"
    },
    {
        "text": system_prompt,
        "text_pair": "What are the health benefits of a Mediterranean diet and how many calories are in olive oil?"
    },
    {
        "text": system_prompt,
        "text_pair": "I'm building a Python accounting tool. Can you advise me on the best tax deduction strategies for small businesses?"
    },
]

# Run guardrail classification inference
results = classifier(queries)

# Display predictions
for query, res in zip(queries, results):
    print(f"User Prompt: {query['text_pair']}")
    print(f"Prediction : {res['label']} (Confidence: {res['score']:.4f})\n")

"""
### Output
User Prompt: How do I reverse a string without using in-built function in Python 3.12?
Prediction : On-Topic (Confidence: 0.9877)

User Prompt: Can you configure a Redux store with reducers and middleware in React JavaScript?
Prediction : Off Topic (Confidence: 0.9731)

User Prompt: Why is my multiprocessing Pool hanging when sharing a large NumPy array across workers?
Prediction : On-Topic (Confidence: 0.9058)

User Prompt: What are the health benefits of a Mediterranean diet and how many calories are in olive oil?
Prediction : Off Topic (Confidence: 0.9766)

User Prompt: I'm building a Python accounting tool. Can you advise me on the best tax deduction strategies for small businesses?
Prediction : Off Topic (Confidence: 0.9997)
"""

Model Evaluation

The model was evaluated on the curated holdout benchmark test set of 4,008 balanced instances from kalyan-ks/off-on-topic-guardrail across three difficulty tiers (Easy: 635, Medium: 1,571, and Hard: 1,802). All models were evaluated under identical conditions in FP32 precision.

Ammu Cross-Encoder Family Leaderboard

Model	Parameters	Easy Macro-F1 (%)	Medium Macro-F1 (%)	Hard Macro-F1 (%)	Overall Macro-F1 (%)
Ammu-1.0-OffOnTopic-Guardrail-base	150M	97.15	90.71	90.30	91.56
Ammu-1.0-OffOnTopic-Guardrail-tiny	32M	95.74	89.66	90.19	90.87
Ammu-1.0-OffOnTopic-Guardrail-small	68M	96.37	88.93	89.44	90.35
Ammu-1.0-OffOnTopic-Guardrail-nano	17M	91.77	86.55	86.39	87.31

Comparison with Frontier LLMs

Frontier LLMs were evaluated zero-shot with structured Pydantic outputs with reasoning disabled to benchmark direct classification capability:

Model	Provider	Easy Macro-F1 (%)	Medium Macro-F1 (%)	Hard Macro-F1 (%)	Overall Macro-F1 (%)
Ammu-1.0-OffOnTopic-Guardrail-tiny	Hugging Face	95.74	89.66	90.19	90.87
DeepSeek V4 Flash 0731	DeepInfra	96.83	75.88	81.42	81.95
GPT-6-luna	OpenAI	95.39	71.63	79.12	79.12

Key Takeaway: Ammu-1.0-OffOnTopic-Guardrail-tiny (32M) outperforms DeepSeek V4 Flash 0731 by +8.92% and GPT-6-luna by +11.75% overall Macro-F1. The performance difference is especially pronounced on Medium (+13.78% over DeepSeek, +18.03% over GPT-6) and Hard (+8.77% over DeepSeek, +11.07% over GPT-6) boundary queries, while running with sub-3ms latency and zero API cost.

Limitations

• Cross-Encoder Requirement: The model is a paired cross-encoder that evaluates user_prompt strictly with respect to a provided system_prompt. It cannot classify user queries in isolation without a reference system prompt.
• Language Scope: The model is trained and evaluated predominantly on English language text; classification performance on multilingual or code-switched inputs may be lower.
• Complex Mixed Intent: Prompts that deliberately weave permitted and restricted topics into a single complex composite request may present borderline confidence scores and benefit from threshold calibration.

Citation

@misc{ammu-1.0-offontopic-guardrail-tiny-2026,
  title = {Ammu-1.0-OffOnTopic-Guardrail-tiny: Off On Topic Guardrail Classification Model},
  author = {Kalyan KS},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/kalyan-ks/Ammu-1.0-OffOnTopic-Guardrail-tiny}
}