lowdown-labs/fela-moderator

N.B. To use this model with a convenient interface check us out on Github!

FELA-Moderation: CPU native content moderation and PII detection

FELA-Moderation reads a piece of text and does two things with it. It scores how toxic the text is across a set of moderation categories, and it points out the spans that look like personal information: names, emails, phone numbers, credit cards, and the like. It is small enough to run on a plain CPU, so that whole moderation and PII pass can live inside your own service or app instead of going out to a moderation API and billing you per call.

It reads text as raw bytes rather than words. That has a nice side effect: there is no vocabulary file to ship, and the model is not locked to any one language.

The model covers a 19 category moderation taxonomy, the 6 native Jigsaw labels, byte level PII over 56 entity types, and four extra classifier heads for spam, jailbreak, NSFW severity, and targeted identity. It is about 18.9M parameters and ships as model.safetensors (fp32) plus an int8 export of roughly 19 MB for on device and WASM serving. We ship weights only and never redistribute the source text.

What goes in, what comes out

• Input: any UTF-8 text string. It is byte tokenized to [CLS] b0 b1 ... (each byte is its own token, ids 0 to 255, plus 256=PAD, 257=CLS, 258=SEP) and truncated to 512 tokens. The encode_text and pad_batch helpers in modeling.py do this for you.
• Output 1 (taxonomy): 19 independent probabilities over the moderation taxonomy (the 11 OpenAI aligned categories plus identity_attack, insult, profanity_obscene, threat, sexual_explicit, severe_toxicity, dehumanize, incitement_violence). Full list in config.json. Sigmoid, multi label.
• Output 2 (jigsaw): 6 independent probabilities over the native Jigsaw labels (toxic, severe_toxic, obscene, threat, insult, identity_hate). Sigmoid, multi label.
• Output 3 (pii): per token BIO tags over 56 entity types (113 tags: O plus B- and I- per entity).
• Output 4 (safety heads): spam / scam / phishing, jailbreak / prompt injection, NSFW severity, and targeted identity.

The head is chosen with task= in the forward call (taxonomy, jigsaw, pii, spam, jailbreak, nsfw, identity, or both).

Why we built it this way

The model is built on the same Fourier Neural Operator the rest of the FELA family uses: a filter that mixes the whole sequence at once in the frequency domain, with no attention matrix that balloons as the text gets longer.

Alongside it rides a gated linear attention layer that picks up the content dependent details, and its working memory stays a fixed size. Because none of this leans on the usual all pairs attention, a moderation plus PII pass stays fast on an ordinary CPU. That speed is the whole cost argument against paying a per call API.

It is deliberately small, about 18.9M parameters. One shared backbone feeds several light heads (taxonomy, Jigsaw, PII, spam, jailbreak, NSFW, identity), and the byte vocabulary is only 259 symbols, which quantizes to int8 well.

Performance

Format and footprint. The fp32 weights ship as model.safetensors (about 75 MB). INT8 dynamic quantization of the Linear layers shrinks the model to roughly 19 MB while the FFT and GLA einsum paths stay in float; the int8 export holds accuracy on the held out slices (see below). Because the encoder carries no KV cache and a fixed size state, working memory does not grow with input length.

Accuracy

Held out numbers are on permissively licensed splits: Jigsaw on a fixed leak free slice of the Jigsaw training data, Civil Comments validation for taxonomy, and nemotron plus synthetic for PII. Headline held out:

head	AUROC / acc	note
PII (byte token acc)	0.96
Jigsaw (mean AUROC)	0.94
taxonomy: hate / harassment	0.84 / 0.77
taxonomy: sexual	0.67	data limited, explicit data not in the training set
spam / jailbreak / nsfw	0.94 / 0.88 / 0.82
identity	0.64	weaker, use as a soft signal only

The int8 export holds these numbers. Final training loss 0.69. The model's advantage is privacy on local hardware, speed, and clean provenance at roughly comparable quality, not topping a specialist leaderboard.

The one honest gap is the sexual category (0.67). It does not close with more training: it is a data limitation. Explicit sexual data was deliberately kept out of the training set, so that category is supplied only by synthetic examples. We report it rather than hide it.

How to run it

import torch
from modeling import load_model, encode_text, pad_batch

m = load_model("model.safetensors")
ids, _ = encode_text("You are awful. Email me at jane.doe@example.com", m.cfg.max_len)
input_ids, mask = pad_batch([ids], m.cfg.max_len)
out = m(input_ids, mask, task="both")

tax = torch.sigmoid(out["taxonomy"][0])   # 19 taxonomy probabilities
jig = torch.sigmoid(out["jigsaw"][0])     # 6 Jigsaw probabilities
pii = out["pii"][0].argmax(-1)            # per token BIO tag id

Loading with standard tooling

The repo ships config.json (architecture hyperparameters plus the full label and tag maps and the per category toxicity thresholds) and a self contained modeling.py with a load_model / from_pretrained entry point:

from modeling import load_model
m = load_model("/path/to/weights_dir")  # OR
m = load_model("lowdown-labs/fela-moderator")

The fp32 weights are shipped as model.safetensors.

Serving artifacts

• model.safetensors plus config.json for the safetensors load path (fp32).
• tier_full_int8.safetensors plus tier_full_scales.json for the roughly 19 MB int8 export.
• manifest.json and NOTICE.txt describe the tiers and the training source attributions.
• verify.py runs a fixed sample input and checks the output shapes and a verification value.

For serving at scale, use the separate CPU native FELA server (https://github.com/Lowdown-Labs/fela_server). It runs this model on CPU with no GPU required.

Training data

We train on the sources below and distribute model weights only. No source text or derived dataset is redistributed, so the CC-BY-SA-3.0 status of Wikipedia derived comment text (under Jigsaw and Civil Comments) never triggers share alike. data/licenses.py carries the verified license of every source, and NOTICE.txt records the attributions.

• PII: nvidia/nemotron-pii (CC-BY-4.0, 55+ types, character offset spans) plus gretelai/synthetic_pii_finance_multilingual (Apache-2.0) plus synthetic generation via Faker (MIT) and the Presidio generator (MIT, driven from Faker).
• Toxicity and taxonomy: google/civil_comments (CC0-1.0), google/jigsaw_toxicity_pred (CC0-1.0 annotations), ucberkeley-dlab/measuring-hate-speech (CC-BY-4.0), allenai/real-toxicity-prompts (Apache-2.0).
• Spam: redasers/difraud (MIT) and ucirvine/sms_spam (CC-BY-4.0).
• Jailbreak and prompt injection: cyberec/llm-prompt-injection-attacks (Apache-2.0), deepset/prompt-injections (Apache-2.0), jackhhao/jailbreak-classification (Apache-2.0), Lakera/gandalf_ignore_instructions (MIT).
• NSFW: google/civil_comments sexual_explicit signal, with mmathys/openai-moderation-api-evaluation (MIT) used for evaluation only.
• Targeted identity: ucberkeley-dlab/measuring-hate-speech (CC-BY-4.0).

Full split definitions, size assertions, and per source license verdicts are reproduced in train.py and data/licenses.py.

Expanded taxonomy (19). Beyond the 11 OpenAI aligned categories the model adds identity_attack, insult, profanity_obscene, threat, sexual_explicit, severe_toxicity, dehumanize, and incitement_violence, each sourced from a permissive dataset. Growing the head does not change the byte level backbone, so the int8 size is unchanged. Five categories (self_harm, self_harm_intent, self_harm_instructions, sexual_minors, violence_graphic) have no permissive source: they are masked from real data and supplied by synthetic and reviewed examples, and reported separately so coverage is never overclaimed.

Byte level PII BIO with exact spans. Character offsets from the real sources and byte offsets from the synthetic generator are aligned to byte tokens in data/bio.py (verified 100% correct on 1,251 spans). The head is trained across context lengths (128 to 8192 bytes) with PII needles buried in benign carrier text, so it finds a single email or one toxic span inside a long document.

How it was trained

The model was trained CPU only via DiLoCo (data parallel with infrequent parameter averaging coordinated through S3), on commodity CPU instances with no GPU. Weights checkpoint to S3 periodically, so a spot interruption resumes rather than restarts. The taxonomy and Jigsaw heads train every step; the spam, jailbreak, NSFW, and identity heads rotate. Run python train.py --smoke for an offline end to end check that trains a few steps on synthetic data and writes the full artifact set.

The model took about two hours on a single CPU instance, on the order of 0.05 kg CO2e - a few phone charges of energy, and millions of times less than training a frontier language model.

Intended use, limitations, and safety

What it is for: the moderation and PII inference core inside an on premises or on device text pipeline, where sending user text to a third party API is a cost or privacy problem. It flags toxic content and personal information for a downstream policy layer.

What it is not for: this is a small model and a decision support signal, not a final arbiter. Do not use it as the sole gate on user safety without human review and per category threshold tuning (sensible starting thresholds ship in config.json).

Known limitations:

• The taxonomy head is scored on validation, not a held out competition test; treat those numbers as in distribution. The sexual category is data limited (0.67), and the targeted identity head is weaker (0.64) and should be used as a soft signal only.
• English centric supervision: the byte tokenizer is multilingual by construction, but the training labels are predominantly English, so non English accuracy is not established.
• The PII head reflects the nemotron and Presidio aligned entity catalog and its distribution; entity types and formats outside that corpus may be missed.

How to cite

@misc{lowdownlabs_felamoderation,
  title  = {FELA-Moderation: CPU native byte level content moderation and PII detection},
  author = {Lowdown Labs},
  year   = {2026},
  note   = {Model card}
}

You should also cite the datasets and baselines used (Jigsaw / Conversation AI, nemotron, civil_comments, measuring-hate-speech, real-toxicity-prompts, and Detoxify / Presidio); full references and licenses are in data/licenses.py and the references below.

Acknowledgements and references

• Fourier Neural Operator: Li, Z., et al. (2021). Fourier Neural Operator for Parametric Partial Differential Equations. ICLR. https://arxiv.org/abs/2010.08895
• Gated Linear Attention: Yang, S., et al. (2024). Gated Linear Attention Transformers with Hardware-Efficient Training. https://arxiv.org/abs/2312.06635
• Byte level modeling: Xue, L., et al. (2022). ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. TACL. https://arxiv.org/abs/2105.13626
• PyTorch: Paszke, A., et al. (2019). NeurIPS. https://arxiv.org/abs/1912.01703

Model family

This is part of the FELA family from Lowdown Labs: one Fourier Neural Operator architecture across many modalities, all CPU native and subquadratic. This repo is pushed as lowdown-labs/fela-moderator. Sibling repos share no weights, so none carries a base_model link.

License

Apache-2.0. The model weights and the code are both under Apache-2.0 (see LICENSE and NOTICE.txt): no separate model license and no non commercial restriction. Every training source is permissively licensed and commercial safe. We distribute weights only, never the source text.