muril-hinglish-cache-verifier

MuRIL fine-tuned as a cross-encoder verifier: given (new question, cached question) it outputs the probability that the cached answer can be reused. Used to block wrong cache hits in a verified semantic cache for Hinglish.

Part of the BCA (Honours, AI/ML) dissertation Verified Semantic Caching for Hinglish and Code-Mixed LLM Queries, Symbiosis Institute of Computer Studies and Research (SICSR), Pune. Team: Mohammad Jamal Yusuf, Myshaa Khan, Dhiren (SICSR, Pune); mentor: Prof. Arun Kumar.

Results

Test ROC-AUC (correct vs wrong best match)	English	Hinglish	Hindi
this model	0.81	0.81	0.72
bge-reranker-v2-m3 (not fine-tuned)	0.82	0.75	0.72

About 3 ms per check on an Apple M2 CPU.

Splits are by intent: no test intent was seen in training. Full pipeline, ablations and negative results are in the project repository and the W&B project hinglish-semcache-dissertation/hinglish-semcache.

Use

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained('jamal10110/muril-hinglish-cache-verifier')
m = AutoModelForSequenceClassification.from_pretrained('jamal10110/muril-hinglish-cache-verifier')
x = tok(['wallet ka PIN kaise reset karu?'], ['How do I reset my account password?'], return_tensors='pt')
print(torch.sigmoid(m(**x).logits[0, -1]).item())   # low = do not reuse

Limitations

The customer-support benchmark is synthetic and small (about 265 test questions per language); the near-miss probes were written by the authors; Hindi (Devanagari) retrieval is slightly lower than with the original model.