LLMSR @ XLLM 25: Less is More: Enhancing Structured Multi-Agent Reasoning via Quality-Guided Distillation (2025)

Abstract: The LLMSR@XLLM25 formulates a low-resource structural reasoning task that challenges LLMs to generate interpretable, step-by-step rationales with minimal labeled data. We present Less is More , the third-place winning approach in the LLMSR@XLLM25, which focuses on structured reasoning from only 24 labeled examples. Our approach leverages a multi-agent framework with reverse-prompt induction, retrieval-augmented reasoning synthesis via GPT-4o, and dual-stage reward-guided filtering to distill high-quality supervision across three subtasks: question parsing, CoT parsing, and step-level verification. All modules are fine-tuned from Meta-Llama-3-8B-Instruct under a unified LoRA+ setup. By combining structure validation with reward filtering across few-shot and zero-shot prompts, our pipeline consistently improves structure reasoning quality. These results underscore the value of controllable data distillation in enhancing structured inference under low-resource constraints. Our code is available at https://github.com/JhCircle/Less-is-More .

## introduction structured reasoning tasks-such as decomposing a question into logical constraints or verifying a chain of deductions-pose unique challenges for large language models (llms) @xcite , especially under extreme low-resource conditions. the llmsr@xllm25 targets this very challenge, requiring participants to generate interpretable and verifiable reasoning processes from only 24 labeled examples. each instance involves four intertwined subtasks: extracting question conditions (question parsing), identifying reasoning steps and their justifications (cot parsing), and validating whether evidence supports each inferred statement @xcite . this setting presents two core challenges: (1) insufficient labeled data to fine-tune high-capacity models, and (2) the need to maintain step-level consistency and logical coherence across multiple reasoning modules. prior work on cot-style prompting typically relies on large-scale instruction tuning or heuristic prompting, which falters when supervision is scarce and structural granularity is essential. to tackle these challenges, we introduce less is more-a structured multi-agent framework that transforms minimal supervision into high-quality training signals through three key stages: (i) prompt induction via reverse thinking @xcite to derive task-specific instructions; (ii) retrieval-augmented reasoning synthesis with gpt-4o to generate contextually grounded annotations at scale @xcite ; and (iii) dual-stage filtering, which integrates lightweight structural pruning and reward-based selection to ensure semantic fidelity. each reasoning module is fine-tuned independently from meta-llama-3-8b-instruct on distilled cot data generated by gpt-4o @xcite @xcite , enabling modular, interpretable reasoning under lowresource settings. our approach ranked third in such shared task, outperforming several strong baselines. through detailed experiments across diffrent reward-filtering stratgies, we show that data quality-not quantity-is the key to enhancing structured reasoning. this highlights the promise of controllable, quality-centric distillation in advancing llm reasoning under real-world data scarcity. ## methodology we present less is more, a structured multiagent reasoning framework designed to address data scarcity through quality-guided distillation. given only 24 labeled examples in the llmsr@xllm25, we construct a scalable pipeline for data synthesis and filtering. our approach focuses on two reasoning subtasks, each treated as an instruction-following generation problem: demo pool logiqa prompt induction top-k ra-icl prompt induction p qp p ucot reasoning synthesis structure filtering reward r(x)>0 qp dataset cp dataset cv dataset sft meta-llama-3-8b-instruct qp agent cp agent cv agent cot paring: ["cp1","cp2",...] question paring: ["qp1","qp2",...] each instance is generated using only a small seed set and guided by reverse prompt induction @xcite . the full pipeline includes prompt design, reasoning synthesis via retrieval-augmented in-context learning, and reward-based filtering to ensure output quality. ## inference pipeline: multi-agent structured reasoning we deploy a structured inference pipeline that mirrors our modular training architecture, comprising three dedicated agents: parser, decomposer, and verifier, each instantiated as a finetuned llama3-8b-instruct model. these agents are respectively responsible for question parsing (qp), chain-of-thought (cot) parsing (cp), and step-level statement and verification (cv). at inference time, given a test instance x, we encode it into a dense embedding h x using a multilingual encoder foot_5 and retrieve semantically similar exemplars r(x) from the distillation pool. these demonstrations are reused across all agents via task-specific prompting templates, ensuring consistency and contextual alignment. the full reasoning pipeline unfolds as a cascade of agent interactions: ## conclusion we present less is more, a structured multi-agent framework for interpretable reasoning under lowresource supervision. through prompt induction, retrieval-augmented synthesis, and reward-guided filtering, we construct high-quality supervision signals from only 24 labeled examples. our system achieves third place in the llmsr@xllm25, demonstrating that data quality @xcite @xcite rather than quantity is the key driver of performance in structured reasoning tasks. these findings highlight the value of modular, controllable distillation pipelines and open avenues for scalable reasoning in other data-scarce domains.