OpenPangram-2B: Token-Level AI-Text Detection & Provenance

Replication of Pangram 4's Multi-Stage AI Detection Architecture on Qwen3.5-2B

1. Overview & Key Capabilities

OpenPangram-2B is an open reproduction of the Pangram 4 AI-text detection architecture, trained on top of the Qwen3.5-2B hybrid linear-attention transformer backbone.

While conventional AI detectors produce only a single, blunt document-level binary score (often prone to false positives on human non-native writers), OpenPangram provides fine-grained, interpretable provenance:

Continuous Degree Calibration (0–100%): Predicts calibrated continuous AI involvement using an ordinal 15-bin regression head ($DOC\ \rho = 0.948$, Window $MAE = 6.7%$).
Token-Level Provenance Highlighting: Classifies every token into {Human, AI-Assisted, AI-Generated} using Repeat2 bidirectional causal context (83.23% token accuracy, 95.2% human recall, 90.6% AI recall).
Mixed-Authorship Window Detection: Identifies collaborative, human-AI co-authored passages (89.71% accuracy, 0.7097 binary F1).
Heuristic Span Smoothing: Employs contiguous span filtering to eliminate single-token argmax flicker without full CRF inference latency.

2. Training Methodology (Based on Pangram Technical Reports)

OpenPangram-2B was trained following the multi-stage training methodology formulated in the Pangram technical reports (Pangram 4 and Pangram 3 series), which replaces brittle perplexity heuristics with multi-task supervision:

[ Human & AI Reference Corpus (~60k Documents) ]
                    │
      Clause Alignment & Similarity
   (Lexical N-Gram Overlap + Embedding Cosine)
                    │
      Multi-Stage Curriculum Training
                    │
 ┌──────────────────┴──────────────────┐
 │                                     │
 ▼                                     ▼
Stage 1: Degree Calibration      Stage 2: Repeat2 Tokenwise
- Sequence-level supervision     - Duplicated sequence [x, x]
- 15-bin ordinal degree head     - Bidirectional causal context
- 4-bucket score head            - Multi-task token + mixed losses
- DOC rho = 0.948 achieved       - 972K validation tokens

Stage 1: Continuous Degree Calibration

Supervision: Documents were supervised with continuous teacher degree targets derived from clause-level lexical and semantic edit distances against reference texts.
Objective: Multi-task cross-entropy across a 15-bin ordinal degree head (segment_head) and a 4-ordered-bucket head (score_head).
Results: Achieved a document-level Spearman correlation of DOC $\rho = 0.948$ against ground-truth AI degree on held-out validation sets.

Stage 2: Repeat2 Token-Level Provenance & Mixed Authorship

Backbone Fusion: Stage 1 degree weights were permanently merged into the base backbone.
Repeat2 Context Duplication: Each input window $x$ (up to 384 tokens) was duplicated into $[x, x]$ (length 768). Causal attention runs across the concatenated sequence, with supervision applied exclusively to the second copy. This grants every supervised token full bidirectional visibility over the entire passage within a causal model.
Multi-Task Loss Balancing: $$\mathcal{L}{total} = 1.0 \times \mathcal{L}{token} + 0.5 \times \mathcal{L}{mixed} + 0.25 \times \mathcal{L}{segment} + 0.25 \times \mathcal{L}_{score}$$

Critical Engineering Fixes

Head Persistence: Ensured all custom heads (score_head, segment_head, token_head, mixed_head) were registered under PEFT's modules_to_save, preventing head freezing during adapter training.
Attention Mask Sanitization: Fixed unmasked mean pooling to prevent pad tokens from corrupting pooled representations.
Native bfloat16 Dynamics: Bypassed PyTorch GradScaler to leverage bfloat16's native 8-bit dynamic exponent range.

3. Official Validation Benchmarks

Evaluated on the full held-out validation suite across 2,400 documents (3,711 windows, 972,065 tokens):

Benchmark Progression

Evaluation Metric	Baseline / Initial	Stage 1 (Degree)	Stage 2 (Tokenwise + Degree)	Significance
DOC Gate $\rho$ (vs Teacher Degree)	0.315	0.948	0.948	Zero decay from Stage 1
Window Degree $\rho$ (15 Bins)	0.250	0.915	0.914	Continuous rank correlation
Window Degree $\rho$ (4 Buckets)	-0.443	0.907	0.910	Continuous rank correlation
Window Degree MAE	39.3%	6.8%	6.7%	Average prediction error
Tokenwise Accuracy	N/A	N/A	83.23%	Token-level accuracy
Tokenwise Macro-F1	N/A	N/A	0.6554	Balanced across 3 classes
Mixed-Authorship Window F1	N/A	N/A	0.7097	Binary detection (89.71% Acc)

Tokenwise Classification Breakdown (972,065 Tokens)

              precision    recall  f1-score   support

       Human      0.862     0.952     0.905    467,769
 AI-Assisted      0.470     0.126     0.198    119,710
AI-Generated      0.823     0.906     0.863    384,586

    accuracy                          0.832    972,065
   macro avg      0.718     0.661     0.655    972,065
weighted avg      0.798     0.832     0.801    972,065

4. Interactive Evaluation Dashboards

Interactive HTML evaluation reports and architectural walkthroughs are included directly within this repository under the evals/ directory:

Stage 2 Full Benchmark Report (evals/stage2_evaluation_report.html): Complete interactive dashboard featuring benchmark progression, class confusion matrix, and a live toggle comparing raw argmax tokens with heuristic span smoothing.
Training Approach Guide (evals/approach.html): Visual explanation of the multi-stage training pipeline, teacher signals, and validation gates.
Label Distribution Audit (evals/stage1_full_label_audit.html): Audit of 60k training records, degree distributions, and clause-level alignment samples.
Architectural Deep-Dive (evals/modelcard_with_diagrams.html): Visual diagrams illustrating Repeat2 sequence duplication, multi-head output routing, and decoding strategies.

5. How to Use

Installation

pip install torch transformers accelerate safetensors

Python Quickstart

You can load and query the model directly via transformers with trust_remote_code=True:

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "sandeshrajx/openpangram-2b"
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, trust_remote_code=True, dtype=torch.bfloat16).to(device).eval()

text = "While artificial intelligence models can synthesize text rapidly, human prose carries distinct structural variance."

# Tokenize
inputs = tokenizer(text, return_tensors="pt").to(device)

with torch.no_grad():
    # Forward pass with Repeat2 context duplication
    out = model(inputs["input_ids"], inputs["attention_mask"], repeat2=True)
    
    # 1. Continuous Degree (0 to 100%)
    probs_seg = torch.softmax(out["logits_segment"][0], dim=-1)
    degree = float((probs_seg * torch.arange(15, device=device)).sum() / 14.0)
    
    # 2. Mixed Authorship
    probs_mixed = torch.softmax(out["logits_mixed"][0], dim=-1)
    is_mixed = bool(probs_mixed.argmax().item() == 1)
    
    # 3. Tokenwise Labels (0: Human, 1: AI-Assisted, 2: AI-Generated)
    token_classes = ["Human", "AI-Assisted", "AI-Generated"]
    token_preds = [token_classes[idx] for idx in out["logits_token"][0].argmax(dim=-1).tolist()]

print(f"Continuous AI Involvement: {degree * 100:.1f}%")
print(f"Mixed-Authorship Detected: {is_mixed} (p={probs_mixed[1].item():.3f})")
print(f"Analyzed {len(token_preds)} tokens.")

Command-Line Interface (CLI)

The included detect.py provides formatted terminal output and optional JSON export:

# Analyze text directly
python detect.py "The quick brown fox jumps over the lazy dog."

# Analyze a document with span smoothing (min length 3)
python detect.py --file paper.txt --smooth 3

# Output structured JSON
python detect.py --file essay.txt --json

Running the Interactive WebUI

An interactive Gradio WebUI with color-coded provenance heatmaps is provided in the repository:

python openpangram/app.py

Then navigate to http://127.0.0.1:7860 in your browser.

6. Model Architecture Specification

Component	Specification
Backbone	Qwen3.5-2B Text (24 layers, 2048 hidden dim, hybrid linear-attention)
Weights	Fused bfloat16 `model.safetensors` (~3.76 GB)
Vocabulary Size	248,320 (BPE)
Context Length	512 tokens ($2 \times 384$ in Repeat2 mode)
Head A (`score_head`)	`Linear(2048, 4)` • 4-bucket involvement score
Head B (`segment_head`)	`Linear(2048, 15)` • 15-bin ordinal continuous regression
Head C (`token_head`)	`Linear(2048, 3)` • Per-token {Human, Assisted, AI} provenance
Head D (`mixed_head`)	`Linear(2048, 2)` • Window-level binary mixed-authorship
Head E (`humanizer_head`)	`Linear(2048, 4)` • Stop-gradient adversarial probe (initialized)

7. Limitations & Responsible Use

Advisory Beta for High-Stakes Settings: While Document Degree correlation is high ($\rho = 0.948$), token-level AI-Assisted recall is currently 12.6%. The model is intended for triage, editorial assistance, and provenance exploration, not automated disciplinary sanctions (e.g. academic integrity violations) without human adjudication.
Domain Shift & Non-Native English: Non-native English (ESL) text can exhibit statistical regularities that naive detectors penalize. We recommend calibrating operating thresholds to maintain $<0.1%$ false-positive rates on target domain baselines.
Short Texts: Passages under 50 words contain limited statistical signal; scores on very short text should be interpreted cautiously.

Citation & Acknowledgments

Trained following the multi-stage detection formulation from the Pangram technical reports:

Pangram 4 Technical Report (Pangram Labs, 2026).
Repeat2: Efficient Bidirectional Attention via Prefix Repetition (Leviathan et al., 2025).

Downloads last month: -

Safetensors

Model size

2B params

Tensor type

BF16