Lilavati: Pocket Mathematician

A 95M-parameter English model, trained from scratch by Udgam Labs, that solves maths word problems by writing a short Python program and running it. It is small enough to run on a laptop CPU.

The name comes from Līlāvatī (लीलावती), the book of arithmetic word problems that the mathematician Bhāskara II wrote in 1150 CE.

More: try it in your browser · code and evaluation on GitHub · the checker model

Question: A chai stall sells 45 cups of tea in the morning and twice as many in the evening.
          Each cup costs 12 rupees. How many rupees does the stall earn in a day?

Lilavati writes:                                         Running it gives:
    morning_sales = 45                                   1620
    evening_sales = morning_sales * 2
    cup_price = 12
    total_sales = (morning_sales + evening_sales) * cup_price
    result = total_sales
    return result

Developer	Udgam Labs
Parameters	94.7M (12 layers, width 640, tied embeddings)
Context	4,096 tokens
Input / output	an English word problem → a Python function whose return value is the answer
Training	from scratch: public synthetic maths datasets with Python solutions, plus general English text; then further training on word problems with execution-verified Python solutions
Companion	Lilavati-Checker (95M) rates programs so that many tries can be combined
Licence	Apache-2.0

Quick start

The lilavati package writes the programs, runs them in a restricted sandbox and combines the answers:

pip install torch transformers
git clone https://github.com/UdgamLabs/lilavati && cd lilavati && pip install -e .

from lilavati import Lilavati
lv = Lilavati()      # downloads UdgamLabs/Lilavati and UdgamLabs/Lilavati-Checker

r = lv.solve("A shop sells pencils at 8 rupees each. Riya buys 6 pencils and 2 erasers at 5 rupees each. "
             "She pays with a 100 rupee note. How much change does she get?")
print(r["answer"])        # 42
print(r["agreement"])     # share of tries that gave this answer
print(r["program"])       # the program behind the answer

• lv.solve(q, tries=1) is one greedy try: fastest, no checker.
• lv.solve(q, tries=8) (default) writes 8 programs at temperature 0.7, runs them all, and picks the answer with the most checker-weighted support. More tries are slower but more accurate (see the evaluation).

Using the model directly. The prompt format is:

<bos>Question: {question}
Answer (Python):
```python

The model continues with a function and closes the code block. For several tries, sample at temperature 0.7 with top-k and top-p turned off, as in the evaluation. Requires transformers 5.18 or newer. Never run generated code without a sandbox. The lilavati package checks each program's structure (maths-only imports, no file or network access, no dunder access), then runs it in a separate process with time and memory limits, and under the macOS sandbox where available.

What it is good at, and what it is not

It is good at grade-school word problems: shopping and change, rates and speeds, percentages, discounts and tax, averages, simple interest, areas and perimeters, and multi-step arithmetic.

It is weak at:

• irrelevant information: a number that should be ignored often gets used anyway;
• working backwards and problems that add an extra step to a familiar pattern;
• unit conversions and some ratio questions;
• unanswerable questions: it always returns a number, even when the question has no answer;
• anything that is not a maths word problem. It is not a chatbot and has little general knowledge.

solve() returns agreement and confidence. When they are low (below 0.5), treat the answer as a guess.

Evaluation

All numbers were measured by us. "Code executed" means the model's program is run and its return value is compared with the reference answer.

One try (greedy), code executed:

Test set	Lilavati
GSM8K test (1,319)	54.7
GSM-Plus testmini, the 7 answerable perturbation types (2,100)	39.2
SVAMP (1,000)	69.9
ASDiv, numeric answers (2,159)	74.0
GSM-Hard (1,319)	43.7

Many tries with the checker (GSM8K test, 48 programs per problem at temperature 0.7):

How the answer is chosen	Score
Checker-weighted vote (the default in solve())	72.3
The checker's single top-rated program	66.7
Plain majority vote	65.7
At least one of the 48 programs is right (an upper bound)	89.5

With 16 tries, the checker-weighted vote scores 68.8.

Comparison with other open models up to 150M parameters

Measured by us on 15 October 2026, greedy, with one harness. Other models answer in text, their native format, and are scored on their final number.

Model	Params	GSM8K test	Notes
Lilavati	95M	54.7 (57.3 on the first 300)	code executed
cagliostro-v3.5-chat	146M	21.3	first 300 items
Falcon-H1-Tiny-R-90M	90M	18.7	all items, 8K-token cap. SVAMP: 40.1 (Lilavati 69.9)
MobileLLM-R1.5-140M	140M	6.7	first 300 items; 99% of its answers ran past our 8K-token cap. Its developers report 8.3
GPT-X3-150M	150M	2.3
GPT-X2.5-135M	135M	1.7
Supra2-100M-Instruct	100M	1.6
JugnuLM-110M-R2plus	110M	1.3

Published results for models whose weights are not released: TinyGSM (Liu et al., 2023, arXiv:2312.09241) reports a 125M model, first pretrained on Phi-1.5 data, at 63.1% with one try and 68.9% when a 125M verifier picks one of 48 programs. Under that exact protocol (the checker's single top pick), Lilavati scores 66.7%. TinyGSM's 125M model trained from random initialisation reports 53.1%.

Contamination audit

We checked every training document against each test set for shared 13-word sequences:

Test set	Items sharing a 13-gram with training data
GSM8K test	0
GSM-Hard	0
GSM-Plus testmini	0.8%
SVAMP	2.2%
ASDiv	7.1%

Scores on the items with no overlap are the same as or higher than on the overlapping items, so overlap does not explain the results. Problems added in later training were filtered against all these test sets (8-word sequences).

How it works

1. Lilavati (this model) reads the question and writes a short Python function.
2. The program passes a structure check and runs in a restricted sandbox. Its return value is a candidate answer.
3. For more accuracy, Lilavati writes several programs. Lilavati-Checker, a second 95M model, reads the question and each program with its result and rates how likely it is to be right. The answer with the most checker-weighted support wins.

Training data:

• TinyGSM (MIT): about 12M grade-school problems with Python solutions;
• tinygsm_mixtral_12M: TinyGSM-style problems with solutions written by Mixtral;
• general English web and educational text;
• the GSM8K training split (MIT), and problems generated with large language models and verified by executing their solutions.

No test-set problems were used for training.

Intended use and limitations

Intended use: research and education on small reasoning models, program-aided maths, offline maths help on small devices, and as a starting point for fine-tuning.

Not suitable for: grading, financial or medical calculations, or anything where a wrong number could cause harm, without a person checking the program. A program that runs is not proof that it is correct.

The training problems are mostly in the style of English grade-school maths. Expect lower accuracy on other styles, longer problems, and other languages.

Licence

Weights and code: Apache-2.0.

Citation

@misc{lilavati2026,
  title  = {Lilavati: a 95M pocket mathematician that solves word problems by writing Python},
  author = {Udgam Labs},
  year   = {2026},
  url    = {https://huggingface.co/UdgamLabs/Lilavati}
}