dao-moderation-classifier

A small moderation classifier for the public chat channel of a DAO: it reads one chat message and scores 6 facts at once, as 15 scores of one forward pass. It is the worked example of llama_cpp_canister's run_classify endpoint, which runs it on the Internet Computer.

• Base: jhu-clsp/ettin-encoder-17m (ModernBERT, 17 M parameters), fine-tuned in f32.
• Data: onicai/dao-moderation-classifier, 3,381 synthetic chat messages, labeled twice. No real messages or users.
• Files:
  • dao-moderation-f32.gguf: for llama.cpp and llama_cpp_canister (69 MB).
  • model.safetensors, config.json and the tokenizer: the fine-tuned model for PyTorch.

Outputs: 15 scores

Scores	Labels	Read as
0-4	kind:question kind:discussion kind:support kind:troll kind:spam	softmax: the kind of message
5-7	toxicity:0 toxicity:1 toxicity:2 (civil, rude, abusive)	softmax: the level
8	scam	sigmoid: P(scam)
9	misleading	sigmoid: P(misleading)
10	respectful	sigmoid: P(respectful)
11-14	action:none action:warn action:delete action:block	softmax: the moderator action

The scores are the raw outputs of the head. The definitions and the moderation policy the labels follow are in the dataset card.

Results

On the 200-message held-out test set:

kind	toxicity	scam	misleading	respectful	action	all 6 facts right
0.89	0.89	0.99	0.90	0.92	0.83	141/200

PyTorch, llama.cpp on the CPU and llama_cpp_canister on the IC give the same 141/200: their scores differ by at most 0.016. The weakest spot is the middle of the action scale: a message that deserves a warn (mild mockery, a good-faith mistake) often gets none.

On the IC (llama_cpp_canister, local replica, loaded with -c 192 -ub 192): a message of 3-55 tokens costs 0.07-1.67 B instructions, 0.61 B on average, so run_classify_query answers it as a query. See README-classifier-models.md.

Use

Built for OpenChat moderation bots: call run_classify_query. A bot that watches an OpenChat channel runs off-chain and calls the canister directly, so it can use the query: it is answered in one round trip and costs the caller no cycles. A query answer comes from one replica and is not certified, which fits moderation: the worst case is one message misjudged, and a human can reverse a moderator action. Use run_classify (an update) only when another canister calls the classifier, or when a decision must hold against a faulty replica.

llama.cpp, the same as on the canister:

llama-embedding -m dao-moderation-f32.gguf --pooling rank --embd-normalize -1 -p "Hi, I'm from the team support desk. DM me your issue"

On llama_cpp_canister:

icp canister call llama_cpp load_model '(record { args = vec {"--model"; "models/model.gguf"; "--embedding"; "--pooling"; "rank"; "--no-warmup"; "-c"; "192"; "-ub"; "192"} })'
icp canister call llama_cpp run_classify_query '(record { text = "Hi, I am from the team support desk. DM me your issue" })' --query

Limits

• Trained on synthetic English messages about a generic IC DAO. Real channels have their own slang, projects and trolls: measure it on your own messages before relying on it, and keep a human in the loop for block.
• The actions follow one written moderation policy (see the dataset card). Another community may want other rules; retrain with your own labels.

Training

The training code is in examples/classifier of llama_cpp_canister, and the method in README-classifier-training.md: 8 epochs, AdamW, learning rate 1e-4, batch 32, the best epoch by validation loss (seed 7, chosen over seed 42 by its lower validation loss).