Crab-1 — a 1.7B OSINT agent you can actually test

Crab-1 is a Qwen3-1.7B fine-tuned (QLoRA SFT) to build a factual profile of a French company from its name alone — official website, administrative department, sector — using three tools: the official French company registry, web search, and page extraction. It runs locally on a consumer GPU, costs nothing per profile, and finishes in 4 tool-calling turns (9 s/profile on an RTX 3060 Ti).

Everything needed to verify the published claims is public: the evaluation harness, the 30-company ground-truth set, and the scoring function live in the GitHub repository. The ground-truth set is also published as a dataset: gaidar12345/crab-1-benchmark. The training-data factory (teacher generation, trajectory filtering, dataset build) is deliberately not published.

Files

File	Description
crab1-v7.f16.gguf	GGUF F16 weights (3.4 GB) — for Ollama/llama.cpp
lora-adapter/	PEFT LoRA adapter (67 MB) — for transformers
LICENSE	Apache-2.0 (derivative of Qwen3-1.7B)

Published numbers — with dates, because they matter

Model (same harness, n=30, 8-turn cap)	2026-07-22	2026-08-30
Crab-1 (this model)	76.7 %	66.7 %
Claude Haiku 4.5	—	66.7 %
Claude Opus 4.5	—	56.7 %
Claude Sonnet 4.5	—	53.3 %
Qwen3-1.7B untrained	56.7 %	70.0 %
Llama 3.1 8B, plain prompting	0 %	—

Pass rate = reward ≥ 0.7 (TLD-strict website match + department-level location, dynamic denominator — see harness/reward.py in the GitHub repo).

⚠️ This benchmark is perishable. The tools hit the live web. The same weights scored 76.7 % in July and 66.7 % in late August — nothing changed but the search results. The untrained baseline moved even more (56.7 % → 70.0 %). Two consequences:

1. Only compare models measured the same day.
2. Your numbers will differ from ours. Expect the ballpark, not the digit.

What stays stable across dates: Crab-1 always submits a profile within the turn budget (100 % submit rate vs 60–80 % for the frontier models, which tend to keep verifying past the 8-turn cap), and it's 2–4× faster end-to-end. Fine-tuning didn't buy intelligence — it bought protocol discipline: stop on time, fill the exact schema. For a fleet of cheap local crawlers, that's the property that matters.

Out-of-distribution check (2026-09-01, 20 known companies)

Because the 30-company benchmark is all startup-style companies, we also ran 20 well-known French companies the model never saw in any form — CAC40 groups, regional SMEs, tricky brand names (Michelin, Airbus, Back Market, Fermob, Saint James…). Set + raw results: data/ood_*.json in the GitHub repo; run your own list with --eval-file.

Metric (n=20, same-day, v7 vs untrained base)	Crab-1 v7	Qwen3-1.7B base
Website accuracy	0.78	0.75
Location accuracy	0.26	0.26
Pass rate	0.25	0.30

• Website finding generalizes: 0.78 accuracy, most partial misses are defensible TLD variants (michelin.fr vs .com).
• Location does not — for either model (0.26 for both, vs 0.87–0.97 on the startup benchmark). The cause is the tool, not the fine-tune: registry lookup by name returns the wrong entity for famous brands.
• Pass rate is a statistical tie — out of distribution the fine-tune behaves like the base with the same protocol discipline.
• Known single failure worth naming: "Armor Lux" → the model invented a hyphen (armor-lux.com does not exist). The base found the right domain. That's the class of error to expect from a 1.7B.

Usage

Ollama (recommended)

# 1. Get the weights, then from the GitHub repo:
ollama create crab1-v7 -f Modelfile

# 2. One profile, interactively:
python quickstart.py "Doctolib"

# 3. The full 30-company eval (~5 min, live web):
python eval/run_eval.py --model crab1-v7 --out results/crab1_v7.json

Transformers (LoRA adapter)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B")
model = PeftModel.from_pretrained(base, "gaidar12345/crab-1/lora-adapter")

Honest limitations

• Trained and evaluated on French companies only; the registry tool is France-specific.
• The eval set is 30 companies — big enough to rank models, too small for decimal-point bragging.
• Name collisions in the registry (brand name ≠ legal name) are the main remaining failure mode.
• The model is a specialist. Ask it anything outside "profile this French company with these tools" and the base 1.7B is what you get.

License

Weights: Apache-2.0 (derivative of Qwen3-1.7B). Code in the GitHub repo: MIT.

Write-up with the full story: https://yegorgaidar.org/blog/crab1-slm-osint/