Llama-3.1-8B-Aleph

Built with Llama. A reasoning model made from Meta's Llama 3.1 8B base by one person on a few dollars of compute. It thinks inside a marked block, then answers. Every number on this page was measured on problems the model never trained on, with the same prompts' rules and the same grader for every model in the table.

Version 2 is out: Llama-3.1-8B-Aleph-2, from the same trainer in one longer run, better on every test on this page. This page stays as the record of version 1.

What it is

• The start: Meta's Llama 3.1 8B base model (not the Instruct).
• A warmup: supervised fine-tuning (SFT), then a few short rounds of ORPO. It teaches the model to think inside a marked block and, above all, to stop when it is done. The warmup model is published too: Llama-3.1-8B-Aleph-warmup.
• The training: the Aleph trainer, which is my own. It is reinforcement learning from verifiable rewards with one idea at its center: for each problem it does not collect separate answers, it grows a tree of them, the Aleph tree. Answers share their beginnings and split along the way, so the trainer can see where in an answer the outcome was decided, and it learns from the difference between branches that ended right and branches that ended wrong. The only reward is a program's verdict on the final answer: no judge model, no human labels, nothing that scores the reasoning itself. An answer that goes in circles is ended and counted as wrong.

The numbers

Solved, percent. One greedy answer a problem (temperature 0), no examples in the prompt, the same grader for every model. Meta's Instruct is read in its own chat format and asked to box its final answer.

test	problems	the warmup model	this model	Meta's Llama 3.1 8B Instruct
GSM8K, grade-school math word problems	1,319	52.3	72.3	85.8
a second math set, harder and mixed (from Big-Math)	1,009	30.6	36.2	63.2
MBPP, Python tasks with hidden tests	500	27.4	30.2	54.0
HumanEval, Python functions with hidden tests	164	35.4	38.4	70.1
IFEval, following written instructions	541	33.3	40.5	74.7
tool calls: choosing a function and its arguments, as JSON	199	69.8	84.4	82.4
extraction: filling a JSON record from a text	200	61.0	85.5	90.0
following several written constraints at once (own test)	200	15.0	14.5	47.5
text puzzles (own test)	200	10.5	12.0	31.0

GSM8K read the way Meta's card reads it (one greedy answer after 8 solved examples): this model 69.4, the Instruct 83.9. Meta's card says 84.5 for the Instruct, so this page's way of reading reproduces their number.

How to read the table:

• Against Meta's Instruct: level on tool calls and on extraction (the differences are inside the tests' noise), 13.5 points behind on GSM8K, and far behind on harder math, code, IFEval and the two last tests. Meta's model had a full instruction tuning; this one had a warmup and about fifty rounds of training on one GPU.
• Against its own start: 20 points gained on GSM8K, 15 and 25 on tool calls and extraction, 7 on IFEval, 6 on the harder math set. On code and on the two last tests the gain is inside the noise.
• A difference under about 3 points between two models on one test is inside the noise of one pass.
• The last four tests were made for this project (generated tasks, a program checks each answer). The others are public tests with their own checkers.
• The 8 examples do not help this model (69.4 against 72.3 without them): it answers in the form it was trained in.
• On the harder math set this model's answers stop by themselves 97 percent of the time; the Instruct's 88 percent.

How to use it

The chat template writes the system line the model was trained with when you give none, and opens the think block. The answer comes after </think>.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "ombd/Llama-3.1-8B-Aleph"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "A shop sells pens at 3 for $2. How much do 12 pens cost?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt",
                                 return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=1536, do_sample=False)
print("<think>" + tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

With vLLM:

vllm serve ombd/Llama-3.1-8B-Aleph --dtype bfloat16 --max-model-len 4096

Greedy decoding (temperature 0) is the reading of the table above. The model was trained with answers of up to 1,536 tokens.

If you fine-tune it with an adapter and then merge: merge in 32-bit floats, not in 16-bit. Small changes to a weight are rounded away when they are added to 16-bit weights directly.

What it was trained on

• Math: GSM8K's training split and problems from Big-Math (checked one by one first: about a tenth of that set is flawed, and those were dropped).
• Code: the easy level of APPS, each answer run against hidden tests.
• Tool calls, extraction, instruction following and text puzzles: tasks generated by programs written for this project, each answer checked by a program.

No test problem on this page was in training: every training row of every stage was compared with every test problem, and the few that matched are left out of the numbers (15 of the second math set's 1,024 problems, 1 of the 200 tool-call problems). MBPP, HumanEval and IFEval were never trained on in any form.

Limits

• Code is behind. This is not a code model.
• Hard math is still hard. On problems well above grade-school level it is right about a third of the time.
• Long answers can go in circles. An answer that runs to the length limit is almost never a deep one.
• English only in training and in every test here.
• It can be confidently wrong, like any model of its size. Check what matters.

Acknowledgments

The trainer's idea and design, the experiments and every decision are the author's. Claude (Anthropic) was the engineering partner for the code, the tests and the records.

Licence

Llama 3.1 Community License. The licence and Meta's acceptable use policy are in this repo.

Author: Omar Badran.