LLMSR @ XLLM 25: A Language Model-Based Pipeline for Structured Reasoning Data Construction (2025)

Abstract: In this paper, we present a novel pipeline for the XLLM Shared Task-III: Large Language Model for Structural Reasoning (LLM-SR). Our pipeline addresses key challenges in automatic process-reward training data construction, such as high manual annotation costs, limited accuracy of large models in structured data processing, and dependency on auxiliary information for validation. To overcome these limitations, we first decompose the construction process into extraction and validation phases. Leveraging model-generated annotations, we produce pseudo-labeled data and iteratively refine model performance. Second, by analyzing structured data patterns, we encode structural constraints into a rule-based module and fine-tune the model with Gradient Reward Policy Optimization (GRPO), significantly improving structured data extraction success rates. Finally, we train the model to generate critical responses that assess evidence-conclusion relationships, thus enhancing validation reliability. Experimental results demonstrate that our pipeline outperforms models with an order of magnitude more parameters and achieves the first position on the task.

## introduction large language models (llms) have demonstrated remarkable capabilities in mathematical and logical reasoning tasks, particularly when employing chain-of-thought (cot) prompting to decompose problems into multi-step reasoning processes @xcite @xcite . however, even stateof-the-art models often produce unreliable intermediate reasoning steps, leading to cascading errors that compromise final outputs @xcite . to mitigate this issue, existing research has introduced step-wise verification methods @xcite . for instance, process reward models (prm) can evaluate reasoning paths during training, identify erroneous steps, and offer precise corrective feedback @xcite . alternatively, step-wise analysis of cot data from both correctness and redundancy perspectives can generate high-quality reasoning traces for training @xcite . these approaches not only enhance reasoning reliability but also improve overall data quality through generating constructive critiques of flawed reasoning steps, thereby providing valuable optimization signals for model refinement. however, developing such step-wise verification models faces three fundamental challenges: acquiring high-quality training data requires labor-intensive step-by-step annotation of reasoning processes with correctness feedback. for instance, the prm800k dataset @xcite utilizes expert annotators to provide process supervision annotations. this heavy dependence on skilled annotators significantly hinders both the development and practical application of step-wise verification models. moreover, existing verification approaches typically employ simplistic segmentation method (e.g. explicit "step:" markers, double line breaks, or periods) to parse reasoning steps @xcite . such rigid segmentation fails to capture the nuanced compositional structure inherent in natural cot reasoning. this limitation fundamentally constrains the verification fidelity in real-world applications, where reasoning complexity often exceeds template-based patterns. additionally, verification models struggle with complex problems, as even large-scale models frequently fail to accurately determine step correctness, revealing critical limitations in current verification paradigms. to generate high quality dataset, marh-shepherd @xcite uses rejection sampling to generate multiple reasoning paths starting from certain steps, approximating step correctness based on the accuracy of the final answer. rstar-math @xcite constructs positivenegative sample pairs to train ppm, guiding the model towards correct solutions. although these methods effectively create step-level supervision data and improve prm capabilities, their data primarily consist of synthetic samples and require substantial computational resources and additional messages for construction. more importantly, they cannot accurately parse the naturally occurring cot processes. to tackle these challenges, xllm shared task-iii: llm for structural reasoning requires participants to extract all conditions, statements, and their corresponding evidence from given problems and associated cot processes, then determine whether the evidence sufficiently supports each extracted statement-evidence pair. this approach achieves fine-grained cot analysis to enhance the generation of more coherent and accurate reasoning processes. in this work, we propose a fine-grained analysis pipeline for cot reasoning processes. the method decomposes the task into two components: extraction and verification. for the extraction task, following the construct of aiflow @xcite , we identify the problem conditions, statements, and supporting evidence from the question and cot process. to address data scarcity, we first employ prompt engineering and preliminary fine-tuning to generate high-quality pseudo-labels. these extraction patterns are then formalized as prior knowledge into rule-base reward, and the model is further trained using grpo @xcite to streamline the extraction process and improve extraction accuracy. for the verification task, inspired by positive-incentive noise @xcite , we reformulate it as generating concise yet effective critiques for statement-evidence pairs to determine whether the evidence supports the statement. our method achieved first place in this competition. while maintaining low resource consumption, our model improves extraction capability by 20% compared to baselines. our findings demonstrate the feasibility of using prior knowledge as rule reward to enhance model performance on specific nlu tasks like text extraction and recognition, as well as transforming verification tasks into critique generation tasks to improve verification capabilities of smaller models. ## related work llms for information extraction. the emergence of large language models has introduced new solutions and research directions for information extraction. @xcite proposed the multistage structured entity extraction method, which enhances effectiveness and efficiency by breaking down the task. pivoine @xcite focuses on the issue of open-world ie, improving the model's instruction-following ability by constructing the instructopenwiki dataset, and demonstrates excellent generalization to unseen instructions. llms for text2sql @xcite also provide a new avenue for exploring their role in information extraction. ## methodology in this section, we first introduce the detailed task description (section 3.1), followed by presenting the complete extraction pipeline architecture along with the specific extraction methods for each module and the approach for generating pseudo-labeled datasets (section 3.2). subsequently, we elaborated on the application of grpo for the extraction task (section 3.3) and the method of employing verification to validate the statement-evidence pairs (section 3.4). our final solution is an llm-powered pipeline for structured reasoning data construction, as shown in figure 1 . ## conclusion in this article, we propose an effective method for the xllm shared task-iii in llm for structural reasoning. we present a novel pipeline for finegrained analysis of cot processes that achieves extraction and verification performance comparable to state-of-the-art models while maintaining low resource requirements. our work demonstrates grpo's potential for enhancing llm performance on traditional nlu tasks and validates the feasibility of using critique to develop model verification capabilities. the proposed framework opens new possibilities for structured reasoning analysis in resource-constrained scenarios while maintaining competitive accuracy, with future work planned to explore applications to broader reasoning tasks and further optimization of the verification component.