mhmsadegh/rectom-causal-reasoning-retriever-stage2

RecToM Causal-ToM Reasoning-Aware Retriever -- Stage 2 (Expanded Dataset)

Second fine-tuning run on a substantially larger, better-balanced, canonicalized Case Bank (docs126's 80 original items + docs128's 447 new intervention-validated items = 266 eligible items after filtering, vs. Stage 1's 56). Fixes two issues found in Stage 1: (1) the evaluation methodology (now group-disjoint 5-fold cross-validation instead of a degenerate 9-query split), and (2) relation_type wording inconsistencies across two discovery models are canonicalized before pair labeling.

Per-task item balance (this was the weakest point of Stage 1)

Task	Stage 1 eligible items	Stage 2 eligible items
belief	16	39
desire	11	59
intention	5	65
judgement	11	57
prediction	13	46

Retrieval metrics (group-disjoint held-out fold, n=53 queries)

Model	MATCH@1	Hit@3	MRR	pos-neg cosine gap
Frozen all-MiniLM-L6-v2	0.057	0.208	0.189	-0.007
Frozen BAAI/bge-small-en-v1.5	0.075	0.226	0.204	+0.001
This model (fine-tuned, epoch 3)	0.113	0.302	0.248	+0.050

Limitation

Causal paths came from deepseek-v4-flash (not gpt-5.4, which this project has banned for cost reasons) for both discovery and intervention validation. Quality was smoke-tested and manually spot-checked before the full run, but this is a different model than docs126's original discovery model, and only 1 intervention was typically designed per item (vs up to 2 in docs126), which lowered the item-level VALIDATED rate (vs. PARTIAL) somewhat mechanically. Still has NOT been evaluated on downstream RecToM answer-generation accuracy (Stage 3, pending).