muradil211/AetherSearch_DPO

🔭 AetherSearch DPO

A preference-aligned search agent for retrieval-augmented reasoning

Trained from AetherSearch SFT with direct preference optimization on all 2,126 preference pairs in the canonical AetherSearch DPO dataset.

🏠 Project · 🧪 Training code · 📚 Preference data · 🧠 SFT checkpoint

🔌 Bring your own retriever. AetherSearch DPO is a search-agent policy, not a self-contained question-answering service. The host runtime must execute each <search>...</search> request and return evidence inside <information>...</information>.

✨ Highlights

• 🎯 Preference aligned — trained to favor stronger search decisions, evidence use, and final answers over paired alternatives.
• 🔎 Search native — produces explicit reasoning, retrieval requests, and grounded answers using the AetherSearch XML protocol.
• 🔁 Multi-step retrieval — can request additional evidence when an existing trajectory is insufficient.
• 🧾 Environment-aware loss — retrieved information remains available as context while being excluded from preference log-probability sums.
• 🧪 Reproducible stage — canonical data identity, strict trainer, hardware-independent launcher, DeepSpeed configuration, and source checks are public.

🧠 How it works

Question
   │
   ▼
<think>reason about what is missing</think>
   │
   ▼
<search>focused retrieval query</search> ─────► Search / RAG backend
   ▲                                                  │
   └──── <information>retrieved evidence</information> ◄────┘
   │
   ├── repeat the search loop when more evidence is needed
   ▼
<answer>evidence-grounded final answer</answer>

The model produces reasoning, search, and answer spans. The surrounding runtime parses each completed <search> span, runs retrieval, appends the result as <information>, and resumes generation until the model emits an <answer> span.

📊 Model at a glance

Field	Value
🧬 Base checkpoint	muradil211/AetherSearch_SFT
🧬 Base revision	437aca474d3966e57e82af565db95d0ad64aa24d
📚 Training data	muradil211/AetherSearch_DPO
🔗 Training code	AetherSearch/dpo
🏗️ Architecture	Qwen2 causal language model
🔢 Parameters	3,085,938,688
🎛️ Weight dtype	BF16
📏 Context window	32,768 positions; training sequences capped at 4,096
🎯 Alignment method	Direct Preference Optimization

🧬 Checkpoint identity

This checkpoint was trained in one DPO stage from the pinned AetherSearch SFT checkpoint over all 2,126 records in the canonical train.jsonl, using the public code under the GitHub repository's dpo/ directory. Training was performed on a separate server; this repository contains the final model artifacts rather than server-local run logs or optimizer state.

Dataset SHA-256

c42adcb0f194cff3126134b37afd85e4b89aa9917e5c98dda4b09904509f61e9

🎯 Preference contract

Every training row supplies one shared prompt_text, one chosen continuation, and one rejected continuation.

• Shared prompt tokens are masked on both sides.
• Environment-provided <information>...</information> spans inside either continuation are masked while remaining visible as context.
• All remaining continuation tokens contribute to summed sequence log probabilities.
• Answer-terminal continuations supervise a final <|im_end|> token.
• Search-terminal continuations do not append that token because the runtime must provide retrieved information before generation resumes.
• The frozen reference and initial policy use the same pinned SFT checkpoint.

The strict public trainer rejects checksum drift, row-count drift, duplicate questions, malformed trajectories, identical pairs, all-masked continuations, and sequence truncation before allocating model weights.

🛠️ Training recipe

Setting	Value	Setting	Value
Epochs	1	Learning rate	5e-7
DPO beta	0.1	Scheduler	Cosine
Global batch size	12 pairs	Per-device batch	1 pair
Precision	BF16	Max sequence length	4,096
Warmup ratio	0.03	Weight decay	0.0
Distributed optimizer	DeepSpeed ZeRO-3	Seed	42

The launcher discovers devices already made visible by the surrounding runtime and derives gradient accumulation to preserve global batch 12. It does not embed physical GPU IDs, node addresses, communication-fabric settings, allocator tuning, or server-local paths. The policy and frozen reference are both sharded under ZeRO-3.

The complete implementation and reproduction commands are in the AetherSearch DPO directory.

🚀 Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "muradil211/AetherSearch_DPO"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.config.use_cache = True

For end-to-end use, wrap generation in the retrieval loop shown above and preserve the XML protocol exactly.

📦 Repository contents

The repository contains two BF16 SafeTensors shards, the shard index, model and generation configuration, tokenizer assets, this model card, and the AetherSearch logo. Intermediate checkpoints, optimizer state, report bundles, and server-local logs are not included.

⚠️ Limitations

Generated searches and answers can be incorrect, unsupported, or unsafe. Retrieval quality, evidence validation, answer verification, and deployment safeguards remain the caller's responsibility.

📜 Terms

No additional blanket license is asserted here. Review the Qwen2.5-3B-Instruct license, the AetherSearch SFT terms, and the AetherSearch DPO data attribution before redistribution or downstream use.

---

Built for agentic search and retrieval-augmented reasoning. 🔎✨