vivekkopthsd/vajra-nano

TxCat-Nano

A 15,434-parameter transformer for transaction categorization, trained from scratch in raw PyTorch. No HF Trainer, no pretrained weights, no transfer learning.

Built as the smallest honest transformer experiment: how far can a stripped-down transformer go on a small transaction dataset, and what does the accuracy-versus-memory frontier actually look like?

Trained on a Kaggle P100 (torch 2.4.0+cu121), 25 epochs, about 8 seconds.

Model

Param	Value
Parameters	15,434
Layers / heads / d_model	1 / 2 / 32
FF dim	64
Context length	32 tokens
Vocabulary	45 (char-level, built from train split only)
Amount encoding	log-scaled into 64 buckets

Inputs: merchant/description text (char-level) plus transaction amount. Token and amount embeddings are injected into a CLS token, fed through one nn.TransformerEncoder layer, then a linear head over 10 classes.

Results (1,000 held-out rows, stratified 80/20, seed 42)

Metric	Value
Accuracy	0.9610
Macro-F1	0.9609
Weighted-F1	0.9609
ECE (calibration)	0.0276
Log-loss	0.1487
Brier	0.0065
Top-2 accuracy	0.9880
Inference	1.06 ms/sample, single CPU core

The model is slightly under-confident (mean confidence 0.934 vs accuracy 0.961), which is the opposite of the usual small-model failure mode.

Baselines and the frontier

Model	Params	Size	Held-out acc
TxCat-Nano (this)	15,434	60 KB	0.9610
TxCat-Micro	80,714	315 KB	0.9960
TxCat-Mini	292,490	1.1 MB	0.9970
TF-IDF + Logistic Regression	—	15,939 features	1.0000
Majority baseline	—	—	0.1000

On this dataset a linear model on TF-IDF features beats every transformer. The transformers' result is the size frontier: Nano is 28,185x smaller than DeBERTa-large (1,659 MB) at 96% of its accuracy, with better calibration. The comparison is reported because it is the honest one.

Usage

import torch, json
from huggingface_hub import hf_hub_download

bin_path = hf_hub_download("vivekkopthsd/txcat-tiny-transformer-nano", "pytorch_model.bin")
cfg = json.load(open(hf_hub_download("vivekkopthsd/txcat-tiny-transformer-nano", "config.json")))
sd = torch.load(bin_path, map_location="cpu")

# Build the TinyTransformer (class definition in the companion Kaggle kernel),
# load_state_dict(sd), tokenize merchant text + amount, forward, argmax.

Files

File	Contents
pytorch_model.bin	State dict (68 KB)
config.json	Architecture, tokenizer, training config, all metrics above
vocab.json	45-char vocab

Reproducibility

• Data: bipin2/transaction-categorization on Kaggle (10 classes, 5,000 samples; 4,000 train / 1,000 val).
• Training kernel: vivekkopthsd/txcat-tiny-transformer on Kaggle. Includes micro/mini variants, TF-IDF baseline, majority baseline, and the frontier plot.
• Weight persistence kernel: vivekkopthsd/txcat-nano-persist. The RNG stream was replicated from the original kernel so the persisted weights reproduce the exact 0.9610 run (a first attempt with a different init stream gave 0.9490).
• Hardware: P100, torch 2.4.0+cu121. The Kaggle default image ships cu128 builds with no Pascal kernels; the pin is required.

History

Three failures were debugged during development, each documented in the training kernel log:

1. cu128 torch build on P100: CUDA error: no kernel image is available for execution on the device. cuda.is_available() returned True; the error surfaced at the first operation. Fixed by pinning torch 2.4.0+cu121 and adding a CUDA smoke test.
2. Initial pip install without internet access failed at DNS.
3. A cosmetic cell divided by the TF-IDF baseline's size (0 KB) and crashed the report step; baselines were excluded from the "smallest trained transformer" logic.

Known limitation

The dataset reuses merchant names across rows, so every validation merchant also appears in training. Part of the 0.961 accuracy reflects merchant-name memorization rather than pure text understanding. A merchant-disjoint split would report a lower, more honest generalization number; it is the planned next step.

License

Weights trained from scratch on the public Kaggle dataset bipin2/transaction-categorization (see its dataset page for terms). Educational use.