Intent Palette TF-IDF
A small, CPU-only English intent classifier for timers, alarms, reminders, task lists and shopping lists. It maps a request to one of eight categories, or returns abstain.
The model combines word and character TF-IDF features with logistic regression. The complete fitted pipeline is 1.68 MB, runs offline after installation, and needs no large language model or GPU.
This release includes the model, a Python package and CLI, buildable source, and results from experiments on training-data coverage and rejection thresholds.
Quick start
Download the wheel, then install it with Python 3.12:
python3.12 -m venv ./tfidf-env
./tfidf-env/bin/python -m pip install ./intent_palette_tfidf-0.1.1-py3-none-any.whl
./tfidf-env/bin/intent-palette-tfidf "show my shopping list"
from intent_palette_tfidf import load_predictor
model = load_predictor()
model.predict("show my shopping list")
model.predict(["set a timer", "remind me to call Alex"])
Each prediction contains a label and a score. The CLI also accepts JSON strings, one per line:
printf '"show my shopping list"\n"set a timer"\n' | ./tfidf-env/bin/intent-palette-tfidf --jsonl
See examples.py and requirements.txt. Inference uses scikit-learn, NumPy, SciPy, joblib and threadpoolctl. Dependencies are downloaded during installation; inference itself is offline.
Categories
| Label | Request |
|---|---|
timer |
Start a duration timer |
alarm |
Set or enable a clock-time alarm; wake-up request |
reminder |
View or query existing reminders |
reminder_update |
Create or update a reminder |
todo_list |
View or query tasks and errands |
todo_list_update |
Add tasks or clear a task list |
shopping_list |
View or query a shopping list |
shopping_list_update |
Add or remove shopping items |
The output is a category, not an executed action or extracted time/item. The classifier has an additional internal unsupported class. It returns abstain when that class wins or when the winning supported score is below 0.6691898252521505. Scores are logistic-regression outputs, not calibrated probabilities of correctness.
How it works
- Word TF-IDF features: unigrams and bigrams.
- Character TF-IDF features: 3–5 character sequences.
- Balanced logistic regression with C=1.
- Training: 1,600 CLINC OOS examples—800 supported, 250 out-of-scope and 550 from other intents.
- Fitted vocabulary, IDF values, preprocessing and acceptance policy are included in the package.
The original pipeline measured approximately 0.52 ms warm p95 on an Apple M4 Pro CPU with four compute threads. This measures the loaded pipeline rather than CLI startup.
Evaluation
The released model is candidate A. We also tested fuller-data and operation-focused variants to explore the tradeoff between recognizing supported requests and rejecting unsupported ones.
| Evaluation of A | Supported requests correct | Precision among accepted requests | Unsupported requests accepted |
|---|---|---|---|
| Original CLINC benchmark | 158/240 (65.8%) | 158/164 (96.3%) | 2/5,260 |
| Separately sourced MASSIVE subset | 30/64 (46.9%) | 30/34 (88.2%) | 4/176 |
Rejections count as incorrect in the supported-request column. The MASSIVE row uses 240 cases with agreement between two AI annotators; another 130 uncertain or conflicting cases are reported separately. The text comes from human-elicited commands, with labels mapped to this project's categories.
The experiments showed a recurring difficulty: distinguishing an operation from its topic—for example, setting an alarm versus querying or deleting one. Larger training coverage and stricter rejection policies changed that tradeoff, but the follow-up candidates did not meet the combined targets. A remains the released reference.
EVALUATION.md explains the experiments, datasets and results. evaluation-summary.json contains the aggregate counts. These are corpus evaluations, not measurements from deployed user traffic.
Files and license
- Python wheel: ready-to-install API and CLI.
- Source archive: inference code, tests, dependency lock and build metadata.
- A.joblib, config.json, policy.json: fitted pipeline and settings.
- SHA256SUMS: file-integrity checksums.
The model uses native sklearn/joblib format, not Transformers. Load joblib files only from trusted sources: pickle-based formats can execute code.
MIT for this project's code and model weights. Dataset attribution and dependency licenses are listed in NOTICES.md.
- Downloads last month
- -