minhhien0811/agentic-search

Agentic Search — retrieval-only benchmark

This project compares retrieval controllers over one shared, offline passage corpus. It never asks a model to generate a final answer: a future evaluation will only test whether the returned contexts contain a normalized reference answer (and, separately, gold-document recall/MRR).

The comparable systems are:

• fixed: fixed BM25 + BGE-M3 + RRF + shared reranker.
• adaptive_router: non-LLM sparse/dense routing heuristic.
• one_shot_planner: one LLM JSON retrieval plan, without an observation loop.
• agentic: a tool-observation controller; each step chooses a retrieval SDK primitive, observes contexts, then continues or finishes.
• retrieve_verify_refine: retrieve, have the LLM assess evidence sufficiency, then retrieve targeted follow-ups for at most three turns.
• parallel_fanout: plan subqueries once, run sparse and dense fan-out, merge, then rerank.
• sac: Search-as-Code; Qwen writes retrieval code, sees the contexts, and may refine it for at most three turns before emitting DONE.
• sac_latency: one compact SaC program with a 384-token code budget, one hybrid pass, smaller candidate sets, and no refine call.
• sac_accuracy: SaC initial program followed by a structured evidence-state verifier and one targeted retrieval-code patch when evidence is incomplete.

Every system uses the same bounded SDK primitives, trace schema, candidate budget, reranker, context count, latency accounting, and estimated-cost fields.

Data and global corpus

The corpus is the globally deduplicated union of all passages supplied with three multihop benchmarks:

• HotpotQA distractor validation;
• 2WikiMultiHopQA validation;
• MuSiQue development.

Passage identity is SHA-256(title, normalized passage text), deliberately independent of dataset and sample position. Thus exact duplicate passages are embedded once, and the same document ID can be used when each benchmark is evaluated against the common index. The combined-corpus manifest records raw and unique per-source counts plus checksums of every input.

Prepare all raw data and build the common corpus:

export PATH=/home/clouduser/miniconda3/envs/vllm-nightly/bin:$PATH
uv sync --extra dev --extra retrieval-models
uv run agentic-search data download-all
uv run agentic-search data build-combined-corpus

Learned retrieval index

The configured retrieval backend is BAAI/bge-m3, not a token hash. The index command encodes every passage with BGE-M3, L2-normalizes the vectors, and writes the vector matrix, ordered document IDs, metadata, and a FAISS IndexFlatIP index. At query time it embeds the query with the same model and searches that persisted FAISS index; reranking is configured for BAAI/bge-reranker-v2-m3.

Build the global index on an available GPU (GPU 4 is the default command below):

CUDA_VISIBLE_DEVICES=4 uv run agentic-search index dense \
  --corpus data/processed/multihop_context_union.jsonl \
  --index-dir data/indices/multihop_bge_m3

The legacy BLAKE2b hashing implementation exists only as a small, deterministic unit-test fixture. It is neither the default configuration nor a valid benchmark-retrieval result. Do not compare prior reports produced with it to learned BGE retrieval.

Model serving for controller baselines

The locally cloned Qwen model is hosted with four independent OpenAI-compatible vLLM replicas, one on each GPU 4–7. The launcher uses a 32k context window.

export MODEL_PATH=/mnt/data/hub/Qwen3.6-35B-A3B
./scripts/host_qwen36_replicas.sh
source artifacts/servers/<run-name>/endpoints.env
export MODEL_API_KEY=EMPTY

Only stop replicas created by this project:

SERVER_RUN_DIR=artifacts/servers/<run-name> ./scripts/stop_qwen36_replicas.sh

Fair Protocol 500 results

The published experiment fair_protocol500_20260711_114000 evaluates all nine systems on the first 500 examples from each benchmark with qwen3.6-35b-a3b as the controller backbone. It contains an accuracy pass, isolated latency runs at concurrency 1, and loaded latency runs at concurrency 16. The public artifact set deliberately includes only:

• aggregate summary.csv and summary.json reports;
• per-example results.jsonl records for all three passes;
• the detailed Vietnamese LaTeX report and its compiled PDF.

Large event traces, model-server logs, caches, learned indices, and raw or processed benchmark data are excluded from the public repository. The report is available at artifacts/reports/fair_protocol500_20260711_114000_report.pdf.

Headline macro results over the three datasets are 76.2% Answer Containment@10 for sac and 57.8% Joint Evidence@10 for sac_accuracy. See the report for per-dataset results, paired tests, resource usage, latency, failure accounting, and important limitations of the retrieval-only protocol.

Development checks

uv run ruff check src tests
uv run pytest
uv run agentic-search config show