mhmsadegh/rectom-causal-reasoning-retriever-pilot

RecToM Causal-ToM Reasoning-Aware Retriever (Pilot)

Motivation

RecToM is a dialogue-based Theory-of-Mind movie-recommendation benchmark. A prior blind multi-judge audit (project docs127) found that off-the-shelf semantic retrieval (all-MiniLM-L6-v2) returns causally incompatible cases as its top-1 hit 74-82% of the time, despite high cosine similarity -- i.e. semantic similarity between dialogues is not the same as similarity in the causal Theory-of-Mind reasoning structure required to answer the question. This model is a pilot fine-tune testing whether structure-aware supervision closes that gap, following RAR-B (Xiao et al., 2024) and RaDeR (Das et al., 2025), adapted to RecToM's causal ToM setting.

Supervision signal

Training labels come from intervention-validated causal paths (project docs126): each causal edge was tested via a real counterfactual/control intervention on the answering model, not just claimed by an LLM. Edges whose intervention outcome was REJECTED or CONFOUNDED were dropped before deriving each item's ordered relation_type sequence.

Training-data construction

• Source pool: 80-item leakage-clean RecToM Case Bank, filtered to 56 items with a non-empty validated relation-type sequence.
• Same-task pairs only, labeled deterministically (no LLM call):
  • MATCH: identical relation-type sequence -- 49 pairs.
  • PARTIAL: one sequence is a prefix of the other -- 81 pairs, held out, NOT used in this training run.
  • MISMATCH: sequences diverge. Hard negatives = MISMATCH pairs with frozen-baseline cosine similarity >= 0.75 (61 pairs, all kept); remaining MISMATCH pairs subsampled to 127.
• Input text = Dialogue History + Question + Choices ONLY. Causal paths are supervision-only.

Leakage controls

• Training pool never includes any item from RecToM's 150-item evaluation set (docs124-127).
• Train/validation split is group-disjoint by dialogue_id, including 2 internal collision groups found and kept intact.
• 45 items in train, 11 in validation.

Backbone and training

• Backbone: BAAI/bge-small-en-v1.5
• Loss: ContrastiveLoss (binary MATCH/MISMATCH + mined hard negatives)
• Epochs: 10, batch size: 8, learning rate: 2e-05, best epoch: 1
• Seed: 42

Retrieval metrics (validation split, group-disjoint from training)

Model	MATCH@1	MATCH@3	Hit@1	Hit@3	Hit@5	MRR
Frozen all-MiniLM-L6-v2 (project baseline)	0.222	0.222	0.222	0.222	0.222	0.222
Frozen BAAI/bge-small-en-v1.5	0.222	0.222	0.222	0.222	0.222	0.222
This model (fine-tuned)	0.222	0.222	0.222	0.222	0.222	0.222

Important limitation

This is a pilot trained on a small (n=156 training pairs) intervention-validated Case Bank. It has NOT been evaluated on downstream RecToM answer-generation accuracy yet (Stage 2, pending). Metrics above measure representation/retrieval quality only, on a held-out validation split of the same small Case Bank -- not RecToM's official 150-item evaluation set, which this model has never seen in any form.