mehedi4/SafeAge-XLM-R-Base

SafeAge XLM-R Base

This is the original fully fine-tuned XLM-R base comparison model evaluated in the SafeAge study. The SafeAge authors fine-tuned FacebookAI/xlm-roberta-base and its binary classification head on the SafeAge training partition. The pretrained architecture and representations are credited to Conneau et al. and the XLM-R team. This is a trained baseline, distinct from SafeAge Guard, the study's Qwen3-1.7B LoRA reference adaptation.

The repository contains the complete original classifier weights, tokenizer, validation-selected decision threshold, loading example, evaluation metadata, and provenance. No base-model download or LoRA merge is required. The original tensors and tokenizer/configuration files are unchanged.

Decisions and intended use

The model classifies a complete request and its supplied visible context under SafeAge's contextual allow/block policy. It was trained on Bangla, Bangla-English mixed text, romanized Bangla (Banglish), and an aligned English reference form.

The original saved labels are 0 = benign (allow) and 1 = attack (block). The block class includes harmful facilitation and unauthorized actions as well as prompt injection. Compute the two-class softmax and block when the class-1 probability is greater than or equal to 0.99. This threshold was selected using validation only and frozen before test evaluation. The generic classification pipeline's argmax or default 0.5 rule does not reproduce this operating point.

Provide the raw request text and the context needed to decide whether the requested action is permitted. Do not apply a chat template. Labels do not measure age suitability, generated-response safety, or complete agent security. Untrusted text asserting permission does not itself establish authorization.

Loading and inference

Install requirements.txt and download inference.py from this repository:

pip install -r requirements.txt
python inference.py --text "Explain how to protect my own account from phishing."
python inference.py --text-file request.txt

Programmatic usage, with inference.py in your working directory:

from inference import load_classifier, classify_texts

classifier = load_classifier("mehedi4/SafeAge-XLM-R-Base")
print(classify_texts(["Explain how to protect my own account from phishing."], classifier))

An exact release commit can be passed as revision or --revision. The complete model can also be loaded with AutoModelForSequenceClassification.from_pretrained; apply the saved 0.99 threshold rather than taking the maximum class.

The saved tensors are FP32. The original run used Transformers 5.5.4, PyTorch 2.11.0, and FP16 mixed precision on CUDA. The example keeps FP32 weights and uses FP16 autocast on CUDA by default, matching that mixed-precision setting; CPU inference uses FP32. --precision fp32 explicitly selects full precision. Precision, hardware, library versions, batching, and attention kernels may cause small numerical differences, so alternative execution settings do not promise bitwise equality with the archived probabilities.

Inputs use a 512-token maximum, right padding, and padding to a multiple of 8. The loader explicitly uses right truncation, the library default used by the original worker, which did not set a truncation-side override. The classifier architecture uses the first-token XLM-R classification head. The loader preserves the saved architecture and the library's attention selection used by the original loading code. Long inputs are truncated; important context beyond the limit may be lost.

Training and data

SafeAge contains 24,508 labeled requests from 6,127 aligned four-form cases. The fixed partitions contain 22,096 training, 1,208 validation, and 1,204 test requests, with related cases grouped within partitions. Validation and test contain author-reviewed constructed cases from synthetic scenario expansions. They are not independently collected user traffic.

The two authors annotated disjoint assigned subsets and repeatedly reviewed their own examples. The same procedure applied to validation and test. No independent double-annotation set was used, so inter-annotator agreement was not estimated.

Setting	Value
Pretrained base	FacebookAI/xlm-roberta-base
Exact pretrained revision	e73636d4f797dec63c3081bb6ed5c7b0bb3f2089
Training method	Full fine-tuning, including a binary classification head
Parameters	278,045,186
Saved tensor dtype / training precision	FP32 / FP16 mixed
Optimizer / learning rate	AdamW / 0.00002
Weight decay / warmup	0.01 / 10%
Effective batch size	32
Completed epochs / selected epoch	5 / 5
Training seed	42
Maximum input length	512 tokens
Checkpoint selection	Highest validation macro F1 at threshold 0.5
Decision threshold selection	Validation macro F1 on a 0.01-spaced grid; ties closest to 0.5, then lower threshold

Held-out evaluation

At the frozen threshold 0.99, the original checkpoint achieved the following on 1,204 SafeAge test requests (301 cases; 199 connected groups):

Metric	Result
Accuracy	91.03%
Macro F1	90.14%
Block precision	93.15%
Block recall	81.92%
Benign false positive rate	3.57%
Brier score	0.08267

Confusion counts: 729 true negatives, 27 false positives, 81 false negatives, and 367 true positives. evaluation/test_metrics.json contains the original run's results at thresholds 0.5 and 0.99; evaluation/test_by_language_variant.csv records language slices. These are archived measurements, not a new inference run during release. The paper's confidence intervals use connected-group resampling and do not estimate training-seed variation.

Verification and limitations

All seven original checkpoint files were checked against the archived run's SHA256 manifest. Five inference-required original files are released byte-for-byte; trainer_state.json and training_args.bin are training-only artifacts and are not required for inference.

The complete model.safetensors SHA256 is:

050b292486894af1e841a0d4a9f6a3992f5bb3413a4ddeb59806d6a458dce442

The file contains 201 FP32 tensors, including the trained classifier's dense and output projection weights and biases. The archived eight-validation-row reload check passed with a maximum positive-probability difference of 4.27e-08, as recorded in provenance/model_reload_check.json. Release preparation checks artifact identity and structure; it does not add a fresh model evaluation.

Results use a single seed on an author-curated benchmark with synthetic held-out scenarios and no measured independent annotation agreement. Template regularities, translation style, and truncation may affect performance. The model has not established universal safety performance or deployment reliability. Its validation-selected threshold does not guarantee the same error rates on other data.

License and attribution

The pinned upstream XLM-R model card declares MIT. The original fairseq MIT copyright notice is included as LICENSE.base-model.mit; the source URL and revision are recorded in provenance/upstream_license_source.json. The SafeAge fine-tuned release and accompanying original materials use MIT, consistent with the authors' SafeAge Guard release, while preserving upstream attribution. The SafeAge dataset has separate release terms; this model license does not relicense the dataset.

Dataset and adaptation authors: Md. Rehanul Bary and Mehedi Hasan Emon. Associated manuscript: SafeAge: Context-Sensitive Safety Classification for Bangla LLMs and Agentic Systems across Four Language Forms. No conference acceptance or publication is claimed here.

Historical experiment identifiers in provenance retain their original version strings for traceability. The dataset's display name is SafeAge.