dharshannn/CommentGuard-Qwen3-4B-MLX-LoRA

CommentGuard — Qwen3-4B MLX LoRA

Native MLX LoRA adapter for binary toxicity classification, fine-tuned with Unsloth Studio on an Apple M5 Pro with 24GB unified memory. Adapter only: requires the pinned base model and Apple Silicon MLX runtime. This is not a merged model or a Transformers/PEFT adapter, and uploading it does not provision a hosted inference endpoint.

Results

3,000 held-out Civil Comments examples, including239 toxic comments. Each model used its own validation-selected threshold; no threshold was fitted on test.

Model	Precision	Recall	F1	Average precision
Base Qwen under the same label-scoring protocol	.1739	.6402	.2735	.1892
TF-IDF + logistic regression	.2491	.5858	.3496	.3663
CommentGuard adapter	.6416	.6067	.6237	.6808

Selected model confusion counts: TP145, FP81, FN94, TN2680. The model misses39.3% of toxic test comments. Baselines were evaluated after the adapter test result was known, with their thresholds fitted on validation before baseline test scoring. Baselines were not exhaustively tuned. Base results describe this fixed scoring protocol, not optimal base prompting. See evaluation_results.json and baseline_comparison.json for exact metrics and timing.

Usage on Apple Silicon

Tested inference versions are in requirements.txt (Python3.13). Download the adapter repository, install dependencies, then run its scoring example:

python -m pip install -r requirements.txt
python classify.py "Thanks for explaining that clearly."

To download the repository first:

from huggingface_hub import snapshot_download
adapter_path = snapshot_download("dharshannn/CommentGuard-Qwen3-4B-MLX-LoRA")
print(adapter_path)  # run the commands above from this directory

The example fetches the exact base revision and computes the same conditional label score used in evaluation. It returns label0(non-toxic) or1(toxic), score, threshold and truncation status. The score is not a calibrated probability. Ordinary chat generation uses a different protocol and should not be assumed to reproduce these results.

Training and selection

• Base revision: 50d427756c6b1b2fe0c0a10f67fbda1fc8e82c1b.
• Civil Comments revision: f2970eb3a55777454c94069077cc8d9b5866312d.
• 5,000-example training pool, 1,250 toxic /3,750 non-toxic;1,000 validation examples.
• Binary target: annotation toxicity >=0.5. Normalized exact-text deduplication with held-out split precedence; near duplicates are not resolved.
• 4-bit base; rank8, alpha16 (MLX scale2), dropout0; context1024; LR1e-4, linear schedule, warmup50; batch/accumulation1/1; seed42.
• Selected checkpoint4,000 from the one-epoch run by minimum validation token loss. Matched two-epoch/dropout experiments did not replace it.
• Adapter SHA256: 43a29b0b731b257f4c728eb1888f4200fe6fa8b4db79f00883208cdd223f06f6.

The saved classification_config.json is the historical pre-test freeze snapshot; its “test not yet evaluated” status describes that point in time. The completed results above supersede that status, while retaining the original snapshot for provenance.

Scoring contract

Use the pinned chat template and system instruction from classification_config.json. User input is Comment:\n plus comment text. Append <think>\n\n</think>\n\n, compute next-token logits for label IDs15 and16, and apply float32 softmax over only those two candidates. Toxic when score >=0.4073334336280823. Trim only the comment suffix to fit1,023 prompt tokens. No EOS scoring, search or tools.

Limitations and intended use

Research and educational use. Not validated for automated enforcement, other languages, dialect robustness, subgroup fairness or out-of-domain comments. Annotations are subjective; pretraining overlap with this public dataset is unknown. Validation was reused for development and only one seed was used. Test performance is a held-out sample result, not a production guarantee. The larger model is substantially slower than the logistic baseline.

License and attribution

Adapter release: Apache-2.0. The upstream Qwen3 model and MLX quantization are Apache-2.0; their terms remain applicable. LICENSE is the upstream Qwen license text. Base weights and Civil Comments data are not bundled. Attribution: Qwen team, MLX Community, Unsloth, and Civil Comments contributors. The included inference example is released under Apache-2.0 as part of this adapter package.