brief-risk-classifier

A distilbert-base-uncased model fine-tuned to predict, from a campaign brief alone, whether OmniBrandStudio's own judge pipeline is likely to mark generated content as auto_approve, flag, or auto_reject. Intended as a pre-flight risk check — surfacing likely-risky briefs to a creator before the full generation + LLM-judge pipeline runs, so obviously non-compliant campaigns can be caught before paying for generation and judging.

Model Details

• Base model: distilbert-base-uncased (66M parameters)
• Task: 3-class text classification
• Labels: auto_approve, flag, auto_reject
• Input: campaign brief text only (objective, key messages, raw brief text, channels, audience segments) — never generated content or the brand guide directly, matching what a real pre-flight check would have available before generation happens
• Fine-tuned by: OmniBrandStudio project team, trained locally on Apple Silicon (MPS backend)
• License: Apache 2.0 (inherited from base model)

Intended Use

Intended: a lightweight, fast, pre-flight signal to warn a campaign creator that a brief is likely to be flagged or rejected by the real judge panel, before spending time/cost on full content generation. Not a replacement for the judge panel — a triage step in front of it.

Not intended: as a standalone compliance/moderation system, as a substitute for human or LLM-judge review, or for any content type outside marketing campaign briefs in the domain this brand guide covers. It has not been validated on out-of-domain text.

Training Data

Labels come from OmniBrandStudio's own real judge pipeline (Claude, GPT-4o, and Llama judges scoring against a specific brand guide), not from human annotation or an external model's guess — the classifier is learning to predict this specific system's behavior.

287 labeled examples, three sources:

Source	Description
synthetic	Deliberately varied synthetic briefs run through the real (judge-only) pipeline
redteam	Real judge verdicts from a 22-case adversarial red-team suite (health claims, financial guarantees, competitor comparisons, etc.)
live	Real campaigns run through the production app (organic, passively logged)

Label distribution (excluded=false rows only):

Label	Count	%
auto_reject	162	56.4%
flag	93	32.4%
auto_approve	32	11.2%

Excluded data: 15 examples are retained in the source database but excluded from training/eval (flagged, never deleted) — an early batch of auto_approve-targeted briefs used a rigid long-format template that created a real length confound with the (much shorter) auto_reject examples, risking the model learning "long brief = safe" instead of the actual compliance signal. Flagged with a documented exclusion_reason rather than deleted, in case they're useful later (e.g. a length-ablation study).

Training Procedure

• Loss: class-weighted cross-entropy (inverse-frequency weighting via sklearn.utils.class_weight.compute_class_weight("balanced", ...)) — chosen over undersampling after an earlier experiment showed undersampling to balance classes (32/32/32) improved the minority class but discarded 190 real, valid examples and hurt the majority class's score on an already-small validation set. Class weighting keeps every example.
• Epochs: 8
• Batch size: 8
• Learning rate: 2e-5
• Max sequence length: 256 tokens
• Hardware: Apple Silicon (MPS), local training, no cloud compute
• Evaluation: stratified 5-fold cross-validation (not a single train/test split) — a single 80/20 split on this dataset size produced accuracy estimates ranging from 0.60 to 0.78 depending purely on the random split, which is not a trustworthy number to report on its own.

Evaluation Results

5-fold stratified cross-validation, class-weighted loss, all 287 examples used (mean ± std across folds):

Model	Accuracy	Macro F1
Untrained base model (random classification head, zero training)	0.201 ± 0.184	0.101 ± 0.071
Majority-class baseline (always predict auto_reject)	0.564 ± 0.004	0.240 ± 0.001
Fine-tuned (this model)	0.707 ± 0.060	0.682 ± 0.037

Macro F1 lift over untrained base model: +0.581 Macro F1 lift over majority-class baseline: +0.441

Per-class performance (fine-tuned model, mean ± std across folds):

Class	Precision	Recall	F1
auto_approve	0.652 ± 0.068	0.714 ± 0.129	0.678 ± 0.088
flag	0.634 ± 0.101	0.581 ± 0.110	0.601 ± 0.095
auto_reject	0.759 ± 0.054	0.778 ± 0.105	0.766 ± 0.072

Limitations

• Small dataset. 287 examples across 3 classes is thin for a from-scratch fine-tune — expect meaningful variance (the ±0.037 to ±0.184 std figures above are real uncertainty, not rounding). More organic (live) data will narrow this over time.
• Noisy labels. Ground truth comes from an LLM judge panel, which itself shows real disagreement between judges on some content (observed directly in production logs) — not a human-curated gold standard. The classifier cannot be more consistent than the label source it was trained to predict.
• Class imbalance. auto_approve is the rarest class (11% of data) and has the widest per-fold variance in this model's performance.
• Domain-specific. Trained on one brand guide's rules (CTA requirements, tone-per-persona, channel constraints, absolute prohibitions). Will not generalize to a different brand's compliance rules without retraining on that brand's judge verdicts.
• Brief-only signal. By design the model never sees the brand guide or generated content — it has to infer brand-specific constraints (exact required phrasing, tone rules) purely from patterns in brief text, which is inherently harder than what the real judge panel does (which sees the full brand guide via RAG retrieval).

How to Use

from transformers import pipeline

classifier = pipeline("text-classification", model="<repo-id>")
classifier("Create an email campaign for our new shoe. Premium tone. CTA: 'Explore the Collection'. No invented claims.")
# [{'label': 'auto_approve', 'score': 0.87}]

Training Code

Full training pipeline, data generation scripts, and methodology writeup (including the length-confound bug, the k-fold vs. single-split comparison, and the class-weighting experiment) are documented in backend/scripts/training/HANDOFF.md in the OmniBrandStudio repository.