Machynlleth-3B-Instruct

A small instruction-following model for English and the languages of Britain and Ireland: Welsh, Irish, Scottish Gaelic and Scots. It is built on britllm/britllm-3b-v0.1, published as "Caernarfon 3B", which has 2.67B parameters and a Llama architecture.

It's named after Machynlleth, a small Welsh town where Owain Glyndŵr held a Welsh parliament in 1404. The base model is named after a larger town, Caernarfon, and this one is small too.

Licence: non-commercial (CC BY-NC 4.0). One instruction dataset, no_robots, is CC BY-NC 4.0; see Licence.

What changed from the base

Two stages, both LoRA, merged into the weights:

1. Continued pre-training on 22.4M tokens of real Celtic and Scots text, with English mixed back in so the model doesn't forget it. LoRA r=128, α=256, on all attention and MLP projections, for 342 steps of 2,048-token blocks.

   Text	Unique tokens	Share of training
   Books in Welsh, Irish and Scottish Gaelic, Burns, and Douglas's 1513 Eneados, from Project Gutenberg and British Library digitised books	2.6M	27%
   Wikipedia in Welsh, Irish, Gaelic and Scots	4.3M	19%
   Comparable Wikipedia article pairs, each language with English	4.2M	19%
   English from FineWeb-UK's FineWeb-Edu .uk selection	7.8M	35%

2. Instruction tuning on 41,025 conversations, in ChatML. LoRA r=32, α=64, for 2 epochs. It reuses BritLLM-3B-Instruct v0.1's English, Irish and identity data, with these changes:

   • Machine-translated Welsh and Gaelic is cut to 300 examples per language, with at most 3 per prompt opening, to stop the model copying stock phrases.
   • 2,572 cross-lingual examples ask the model to explain a topic in a given language, answered with human-written Wikipedia text.
   • Requests for poems are answered with real public-domain verse.

Results

BritEval

BritEval is a set of standard benchmarks machine-translated into the four languages. Every model below was scored zero-shot with the same script, using lm-evaluation-harness prompt formats and pinned dataset revisions.

The headline is the mean of ARC-Easy, PIQA and XNLI, the three tasks llm.org.uk's BritEval tables use. Higher is better.

Model	Welsh	Irish	Scots	Scottish Gaelic
BritLLM-3B v0.1 (base, "Caernarfon 3B")	44.5	41.2	54.3	41.0
+ continued pre-training (before instruction tuning)	48.7	45.7	54.8	43.1
BritLLM-3B-Instruct v0.1	48.4	45.2	56.1	42.5
Machynlleth-3B-Instruct	48.5	45.4	57.0	43.2

Compared with the base, that is +4.0 in Welsh, +4.2 in Irish, +2.7 in Scots and +2.2 in Scottish Gaelic. With 2,000–3,500 questions per test, differences under about 1 point may be noise.

Per task for this model (ARC and PIQA are length-normalised accuracy):

Task	Chance	Welsh	Irish	Scots	Scottish Gaelic
ARC-Easy	25	39.8	35.1	53.5	32.4
ARC-Challenge	25	25.5	27.1	34.6	25.7
PIQA	50	58.6	56.3	65.2	55.1
TruthfulQA mc1	~22	21.1	29.7	17.0	26.7
TruthfulQA mc2	–	40.6	42.7	31.0	43.9
XNLI	33	47.1	44.8	52.3	42.1
COPA-cy	50	62.8	–	–	–

These numbers aren't directly comparable with other published BritEval scores. Our TruthfulQA is zero-shot (the harness uses a 6-shot primer), and XNLI uses English connective words because the harness has none for these languages. Compare the models in this table with each other.

Continued pre-training

Measured on held-out text before instruction tuning, as a change in loss compared with the base model. Lower is better.

Held-out text	Change
Welsh book	−19.3%
Welsh Wikipedia	−19.0%
Scots Wikipedia	−17.9%
Scottish Gaelic book	−18.4%
Scottish Gaelic Wikipedia	−11.7%
Irish Wikipedia	−5.9%
Irish book	−4.9%
Burns	−13.5%
Douglas	−35.0%
English	−0.1%

Use

ChatML, with <|im_end|> as the stop token:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Beth yw prifddinas Cymru?<|im_end|>
<|im_start|>assistant
Prifddinas Cymru yw Caerdydd.<|im_end|>

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("jonpojonpo/Machynlleth-3B-Instruct")
model = AutoModelForCausalLM.from_pretrained("jonpojonpo/Machynlleth-3B-Instruct", dtype="bfloat16")
enc = tok.apply_chat_template([{"role": "user", "content": "Who are you?"}], add_generation_prompt=True, return_tensors="pt", return_dict=True)
print(tok.decode(model.generate(**enc, max_new_tokens=100)[0][enc["input_ids"].shape[1]:]))

GGUFs for llama.cpp and LM Studio are in gguf/ (Q4_K_M, Q6_K, Q8_0) and include the chat template. Their tokenisation was corrected to match training exactly, so if you convert the model yourself, check yours tokenises identically.

Limitations

• It calls itself BritLLM. The identity data from v0.1 wasn't changed.
• 2,048-token context, the base model's limit.
• Small: 2.67B parameters. Weak at arithmetic, reasoning and specific facts, and it states wrong things confidently. ARC-Challenge, a science test, is at chance in every language.
• Scots instruction data is minimal. Scots improved through continued pre-training, but the model may still answer Scots questions in English.
• Welsh and Gaelic instruction data is partly machine-translated, so fluency and facts in those languages vary.
• Known defects carried over from v0.1's English data: a US-to-UK spelling pass changed some proper nouns (for example "World Trade Centre", "Pearl Harbour"), and many American words are untouched ("cookie", "soccer").
• No preference tuning or safety training, beyond what the upstream datasets did.

Licence

CC BY-NC 4.0, research and non-commercial use. Sources and their licences:

Source	Licence
Base model, britllm/britllm-3b-v0.1	ODC-By 1.0
no_robots	CC BY-NC 4.0, which makes this model non-commercial
databricks-dolly-15k	CC BY-SA 3.0
ultrachat_200k	MIT (generated by GPT-3.5)
aya_dataset, aya_collection_language_split	Apache 2.0
Wikipedia	CC BY-SA 4.0
Books (Project Gutenberg, British Library digitised books)	Public domain or CC0 in their source collections. UK rights weren't reviewed work by work
English replay (FineWeb-UK, from FineWeb-Edu)	ODC-By 1.0, subject to Common Crawl's terms

Whether a model's weights count as a derivative of its training data is legally unsettled. This card follows the most restrictive licence in the mix. A commercially usable version would need retraining without no_robots, and ideally without dolly-15k too.