nikolina-p/intention-request-to-talk-to

Prompt Authenticator (Intent to Talk to Named Person)

Overview

This model is a fine-tuned version of distilbert-base-uncased on the nikolina-p/intention-request-to-talk-to dataset.

It is a binary classifier that detects whether a user prompt expresses intent to engage in a live conversation with a specific named person.

It is designed as a guardrail component in a multi-stage authentication pipeline.

---

Task

• 1 (Positive): User intends to talk/speak/connect or get live help from a named individual
• 0 (Negative): No such intent (e.g., message relay, references to teams, offices, or unnamed individuals, indirect or future/async communicationinfo requests, non-person targets)

---

Intended Use

This model is not a final decision-maker, but a first-stage filter in an authentication pipeline. 👉 The classifier is intentionally permissive to avoid false negatives
👉 Final authorization is handled downstream

---

Limitations

• Synthetic data may not reflect real-world distribution
• Struggles with implicit intent and long prompts
• Mild overconfidence on some edge cases
• English only

---

Training

• Base: distilbert-base-uncased
• Task: Binary classification
• Training: Hugging Face Trainer
• Early stopping enabled (best model ~epoch 4–5)

---

Training and evaluation data

The dataset is synthetically generated and iteratively refined via error analysis nikolina-p/intention-request-to-talk-to

Covers:

• direct and indirect requests
• imperative phrasing
• assist/help vs message/relay
• multi-sentence and adversarial cases

---

Evaluation results

The best model corresponds to the checkpoint from epoch 4 and achieves the following results on the evaluation set:

Loss	Accuracy	Precision (0)	Precision (1)	Recall (0)	Recall (1)	F1 (0)	F1 (1)	Model Select Score
0.1105	0.9826	1.0000	0.9623	0.9688	1.0000	0.9841	0.9808	0.9797

---

Training procedure

The classifier is intentionally somewhat permissive to reduce false negatives on genuine request-to-speak prompts. This was achieved mainly through labeling policy and iterative dataset refinement: borderline valid requests were labeled as positive, and common false-negative patterns were added back into training data. Early stopping used F1 on the positive class to keep a reasonable balance between recall and precision.

The best model is selected from checkpoints using:

• highest F1_1 (primary criterion)
• lowest evaluation loss (secondary criterion in case of ties or plateau)

---

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 8

---

Training results

Epoch	Step	Training Loss	Validation Loss	Accuracy	Precision 0	Precision 1	Recall 0	Recall 1	F1 0	F1 1	Model Select Score
1.0	65	0.5199	0.3770	0.8522	0.9608	0.7656	0.7656	0.9608	0.8522	0.8522	0.8484
2.0	130	0.1652	0.1187	0.9652	0.9688	0.9608	0.9688	0.9608	0.9688	0.9608	0.9596
3.0	195	0.0692	0.1149	0.9739	0.9841	0.9615	0.9688	0.9804	0.9764	0.9709	0.9697
4.0	260	0.0248	0.1105	0.9826	1.0	0.9623	0.9688	1.0	0.9841	0.9808	0.9797
5.0	325	0.0159	0.1007	0.9739	0.9841	0.9615	0.9688	0.9804	0.9764	0.9709	0.9699

---

Framework versions

• Transformers 5.2.0
• Pytorch 2.10.0+cu128
• Datasets 4.5.0
• Tokenizers 0.22.2