TinyBrainBot-360M-A75M-v5.1-Python

A tiny Python coding assistant. The v5.1 360M-A75M mixture-of-experts model (383M parameters, 24 experts per layer, 4 active, so 75M parameters work per token), trained further on Python: a 2B-token code anneal, a Python instruction SFT, and one round of rejection-sampling fine-tuning where it learned from its own answers that passed unit tests. Ask it for a function or paste broken code; it answers with a ```python block. It loads as a standard MixtralForCausalLM.

You:

This function should return the sum of the even numbers, but it's wrong. Fix it:

def sum_even(nums):
    total = 0
    for n in nums:
        if n % 2 == 1:
            total += n
    return total

TinyBrainBot:

The issue with the previous function is that it's not checking if the number is even or not. It's checking if the number is divisible by 2 without a remainder.

Here's the corrected function:

def sum_even(nums):
    total = 0
    for n in nums:
        if n % 2 == 0:
            total += n
    return total

This function works by iterating through each number in the input list and checking if it's even. If it is, it adds it to the total. Finally, it returns the total sum.

---

You:

Write a function that returns the factorial of n.

TinyBrainBot:

Here is the code to solve this problem:

def factorial(n):
    if n == 0:
        return 1
    else:
        return n * factorial(n-1)

You:

Now add type hints and a docstring.

TinyBrainBot:

Here's a revised version of the function:

def factorial(n: int) -> int:
    """
    Returns the factorial of n.

    Args:
        n (int): The input integer.

    Returns:
        int: The factorial of n.

    Raises:
        TypeError: If n is not an integer.
    """
    if not isinstance(n, int):
        raise TypeError("Input must be an integer.")
    if n < 0:
        raise ValueError("Input must be a non-negative integer.")
    if n == 0:
        return 1
    return n * factorial(n-1)

Code benchmarks

All four models were run through the same harness: chat mode (the task goes in the chat template, the answer's

| Model | Params | Training tokens | HumanEval | MBPP |
|---|---|---|--:|--:|
| **TinyBrainBot-360M-A75M-v5.1-Python (this)** | 383M (75M active) | 20B | 14.6 | 32.7 |
| [SmolLM2-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct) | 362M | 4T | 16.5 | 33.5 |
| [SmolLM2-135M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct) | 135M | 2T | 7.9 | 10.9 |
| [Qwen2.5-Coder-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct) | 494M | Qwen2.5 + 5.5T code | 52.4 | 46.3 |

It lands next to SmolLM2-360M-Instruct after about 1/200th of the training data. Qwen2.5-Coder-0.5B is a bigger,
code-specialized model trained on trillions of tokens; it's here as a reference point, not a peer.

| General benchmarks (0-shot, full set) | acc_norm | *(raw acc)* |
|---|--:|--:|
| HellaSwag | 37.49 | *32.59* |
| ARC-Easy | 57.74 | *61.45* |
| ARC-Challenge | 34.64 | *33.28* |
| PIQA | 65.67 | *64.58* |
| ArithMark-3 | 46.60 | *46.60* |
| Open SLM Leaderboard Intelligence Index | 26.02 | *24.06* |

30 unedited answers (15 requests, greedy and sampled, with pass/fail) are in [`code_samples.md`](code_samples.md).

## Quick start

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "nkthebass/tinybrainbot-360m-a75m-v5.1-python"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
msgs = [{"role": "user", "content": "Write a function is_prime(n) that returns True if n is prime."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids, max_new_tokens=400)        # defaults from generation_config.json
text = tok.decode(out[0], skip_special_tokens=True)  # decode WITH the prompt, then cut it off (see below)
print(text[len(tok.decode(ids[0], skip_special_tokens=True)):].strip())

Decode the answer together with the prompt and cut the prompt off, like above. The tokenizer's decoder drops one leading space from whatever it decodes, so decoding only the new tokens can turn the first 4-space indent into 3.

GGUF (llama.cpp, LM Studio, Jan, Ollama): Q8_0 (409 MB), Q4_K_M (239 MB) or F16 (768 MB). The chat template is embedded, prompts with indented code tokenize exactly like transformers, and greedy F16 output matches transformers.

llama-cli -m tinybrainbot-360m-a75m-v5.1-python-Q8_0.gguf -cnv --temp 0.4 --top-p 0.9 --repeat-penalty 1.15 --logit-bias 241-inf --logit-bias 255-inf

Recommended sampling: temperature 0.4, top-p 0.9, repetition penalty 1.15 (the generation_config.json defaults). Plain greedy decoding without a repetition penalty can get stuck repeating itself on questions that aren't code.

How it was made

Stage	Tokens	Data
Code anneal (from v5.1-Base)	2.0B	45% codeparrot-clean, 25% OpenCoder annealing corpus (algorithmic Python + synthetic Q&A), 5% the-stack-smol Python, 25% replay of the v5.1 base's last training stage
Python SFT	262M	70% Python instructions (OpenCoder SFT stage 2, self-oss-instruct-sc2, Tested-143k-Python-Alpaca, Magicoder OSS-Instruct), 30% the v5.1-Instruct chat mix
Rejection-sampling fine-tune	35M	40% the model's own answers that passed unit tests, 30% Python SFT, 30% chat mix

Indentation. The TinyBrainBot tokenizer used to collapse all leading spaces, which makes Python impossible: the v5.1 base scores 0.0 on both benchmarks because every function body loses its indentation. For this model each run of 4 spaces is one token (" ", id 24) and tabs become 4 spaces, so indentation survives training and generation.

Rejection sampling. 26,480 Python problems from KodCode-V1 (easy and medium, one function each, unit tests checked against the reference solution), 8 attempts each. 21% of the attempts passed their tests; 12,419 problems got at least one pass. Up to two distinct passing answers per problem (17,080 total) went back into training.

Contamination. Every SFT and RFT example that shares a 13-word sequence with HumanEval (prompts, solutions, tests) or any MBPP split was dropped, as were KodCode problems that KodCode itself flags as close to a benchmark. No benchmark data was trained on.

Licenses: KodCode-V1 is CC BY-NC 4.0, and the code corpora carry their own source licenses; check those terms before any commercial use.

Limitations

• It's a 75M-active-parameter model. It handles short, single-function tasks (string and list processing, simple loops, recursion, binary search, fixing an obvious bug), but tasks with several steps often come out wrong: a class with error handling, reading a CSV, sorting a dict by value all failed in the samples.
• It explains code badly. Asked "what does this code do?", it tends to write different code instead.
• Asked for tests, it writes a few asserts, sometimes with wrong expected values.
• Greedy decoding can loop on non-code questions; use the recommended sampling.
• Python only. English only. 2,048-token context. Two or more spaces inside a line (not at the start) collapse into one.
• Like any model this size, it makes things up. Run the code before you trust it.