yaser-aj/bge-m3-qatar-1908

bge-m3 fine-tuned on Lorimer's Gazetteer (Qatar, 1908)

The search model behind Qafal, a map of Qatar in 1908 and today where you can ask questions in Arabic or English and get answers that cite the page of J. G. Lorimer's Gazetteer of the Persian Gulf they come from.

Qafal's chat model never answers from memory: it reads the 4 passages the search hands it and cites them. So the answer can only be as good as the search. This model is BAAI/bge-m3 fine-tuned to find the right Lorimer passage for a visitor's question, above all for Arabic questions about English 1908 text with spellings like Dohah, Wakrah and Bida'.

Results

172 test questions (88 English, 84 Arabic) about 83 passages, searched against all 567 Lorimer passages the app holds. Claude (Anthropic's assistant) wrote them one at a time, apart from the training questions, which a different model wrote. Recall@k is how often the right passage is in the top k.

Model	Recall@1	Recall@3	Recall@10	MRR@10
bge-m3 (base)	59.9%	78.5%	94.2%	0.713
Fine-tuned without ever seeing the 83 test passages (branch heldout)	78.5%	91.9%	97.7%	0.861
Fine-tuned on all passages (this model)	82.6%	94.8%	98.8%	0.892

By language, Recall@1:

Model	English	Arabic
bge-m3 (base)	80.7%	38.1%
Fine-tuned, test passages never seen	83.0%	73.8%
Fine-tuned on all passages	86.4%	78.6%

Present-day search was not trained on but improved too: on 400 questions about 200 Wikipedia, Wikidata and OpenStreetMap passages, Recall@3 went from 63.2% to 73.3%.

The heldout row is the honest generalisation number: that model never saw the test passages, not as answers and not as negatives. The main model trained on every passage (with other questions) because the app's archive is fixed: new questions about known passages are its real workload. Its test questions were written separately from its training questions.

Inside the app

Qafal does not use dense search alone. It mixes in keyword (BM25) scores and place-name rules, then hands the chat model 4 passages. With the int8 file inside that full search, on the same 172 test questions:

Search	Right passage among the 4	English	Arabic
bge-m3 (base, int8)	76.2%	84.1%	67.9%
Fine-tuned (this model, int8)	80.8%	86.4%	75.0%

No test question lost its right passage, and the app's earlier 84-question retrieval test stays at 100%.

Answers: on the 133 test questions where the two searches hand over different passages, Qwen3.6-35B-A3B wrote the app's answer from each set, and the same model, shown the reference passage but not which search was used, graded every answer. Correct answers rose from 91 to 98 (Arabic 37 to 44, English 54 both times) and replies saying "my sources don't say" fell from 27 to 20. When the right passage was among the 4, 90% of answers were correct; when it was missing, most answers said so rather than guess.

Use

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("yaser-aj/bge-m3-qatar-1908")
q = model.encode(["كم قارب غوص كان لأهل الخور؟"])
p = model.encode(["Lorimer, Gazetteer Vol. II (1908), p. 1754, 'Shaqiq (Khor)'\nThe village consists of about "
                  "400 mud and stone houses of Mahandah, half of the Al Hasan and half of the Misandah section, "
                  "to whom belong 80 pearl boats, 90 other sea-going vessels, and 30 fishing boats, besides 100 camels."])
print(model.similarity(q, p))

Dense vectors only (the [CLS] state, normalised), like bge-m3's dense mode. For CPU without PyTorch, onnx/model_int8.onnx is the same model quantised to int8 (571 MB), the file Qafal loads; take the [CLS] row of last_hidden_state and normalise it.

Training

• Data: yaser-aj/qatar-1908-search-questions. Qwen3.6-35B-A3B on HF Inference Providers wrote 4 English and 4 Arabic questions for each of the 567 passages (4,529 pairs after cleaning), given the Arabic and modern names of the places and people in the passage.
• Split: by original Gazetteer entry, seed 1908. 15% test, 10% dev (to choose the number of epochs), 75% train.
• Negatives: one hard negative per question, mined with the base model from the 10 best-scoring wrong passages, skipping parts of the same entry and anything within 0.02 of the right passage's score.
• Loss: CachedMultipleNegativesRankingLoss, scale 50 (temperature 0.02, as bge-m3 was trained), batch 64, no duplicate passages in a batch.
• Schedule: learning rate 1e-5, linear decay, 10% warm-up, bf16. Dev nDCG@10 peaked at 2.3 epochs (0.772 base, 0.907 best); the final model trained 2.25 epochs.
• Compute: one Hugging Face Job on an Nvidia A10G (a10g-large), 19 minutes for both trainings, the evaluations and the int8 export. About $0.50, plus about $0.25 of Inference Providers calls to write the questions and check answers.
• Code: scripts/search_finetune/ in github.com/DustyYak286/qafal.

Limits

• Tuned for one archive: Lorimer's 1908 entries on Qatar and the History of Qatar. It may help other Gulf historical text but was not tested there.
• The training questions are machine-written. The written test is small (172 questions), so differences of 2 to 3 points are within noise.
• int8 vectors agree with the full model at cosine 0.974 on average (the published Xenova/bge-m3 int8: 0.980). int8 Recall@1 on the written test: 62.8% base, 83.7% fine-tuned.
• The answer check is small and noisy: per question, 13 answers got better and 6 worse, mostly from the chat model's own variation. The judge is the same model that wrote the answers.

Licence and sources

MIT, like bge-m3. Lorimer's Gazetteer (1908-1915) is in the public domain; passages come from yaser-aj/qatar-1908 (CC BY-SA 4.0).