nedhassan/foursight-intent-classifier

FourSight Intent Classifier (XLM-R base, 7 intents)

Deterministic sequence classification model fine-tuned for first-hop intent routing in logistics and supply chain assistant systems (FourSight).

Routes a single user question into one of 7 mutually exclusive intents:

1. shipment_information — Shipment visibility and location checks ("where is my shipment?").
2. realtime_query — Shipment questions concerning live ETAs, current statuses, and reference numbers.
3. analytics — Aggregated reasoning, reporting, and queries over historical supply chain data.
4. disruptions — Weather, port, border, carrier capacity, or supply chain disruption queries.
5. chitchat — Greeting, capabilities, and conversational exchanges with the assistant.
6. knowledge_base — Product documentation, how-to guides, and configuration help.
7. other — Out-of-scope queries or questions that fit none of the categories above.

---

Quickstart & Usage

from transformers import pipeline

# Load pipeline directly from Hugging Face Hub
classifier = pipeline(
    "text-classification",
    model="nedhassan/foursight-intent-classifier"
)

# Test query
query = "where is my shipment 8006907462?"
prediction = classifier(query)
print(prediction)
# Output: [{'label': 'shipment_information', 'score': 0.9644}]

---

Training Data

• 1,904 labelled, anonymised questions from the official training corpus (train.csv, single-turn, no conversation history).
• 55 hand-crafted synthetic examples (data/augment.csv: 35 shipment_information, 20 other) written from patterns observed in the ground-truth distribution to improve rare and minority class representation.
• The final model is trained on all available verified data; model selection was conducted using a stratified 80/20 train/validation split.

---

Training Procedure & Hyperparameters

• Base Architecture: xlm-roberta-base (multilingual RoBERTa)
• Epochs: 10
• Learning Rate: 3e-5 (AdamW)
• Batch Size: 16
• Max Sequence Length: 128
• Loss Function: Square-root inverse class-weighted Cross-Entropy loss
• Random Seed: 42

---

Validation Results (Stratified 20% Split, 381 queries, 3 seeds)

• OVERALL ($0.6 \times \text{Macro-F1} + 0.4 \times \text{Accuracy}$): 0.674 ± 0.019
• Macro-F1: ~0.64
• Overall Accuracy: 0.72 (72%)

Benchmark Comparison

Model / Pipeline	Validation OVERALL
TF-IDF + Logistic Regression	0.605
XLM-RoBERTa (without augmentations)	0.647 ± 0.023
XLM-RoBERTa (with synthetic augmentations)	0.674 ± 0.019

Per-Class F1 Performance (Mean across runs)

Intent Class	Mean F1-Score
knowledge_base	0.88
chitchat	0.87
realtime_query	0.77
disruptions	0.73
analytics	0.65
shipment_information	0.41
other	0.18

(Note: disruptions, chitchat, and other contain 6, 3, and 11 validation instances respectively, resulting in higher variance for these classes).

---

Intended Use

First-hop routing of user questions in a multi-backend assistant architecture to ensure queries reach the appropriate backend store (operational store, analytical warehouse, or knowledge base) deterministically, coupled with a confidence threshold for human-in-the-loop review.

---

Limitations

• Boundary Ambiguities: The boundary between shipment_information, realtime_query, and analytics exhibits natural label overlap in real-world log data; near-identical queries can carry different operational labels depending on caller intent.
• Single-Turn Context: Model processes single-turn inputs without conversation memory. In production, multi-turn dialogue state tracking is recommended for queries dependent on prior turns (e.g., "what about last week?").
• Rare Classes: The other intent is minimally represented and shares linguistic features with knowledge_base.