LLMSR @ XLLM 25: Integrating Reasoning Prompt Strategies with Structural Prompt Formats for Enhanced Logical Inference (2025)

Abstract: This paper illustrates our NBTailee team sys- tem approach in XLLM-ACL 2025 Task-III: LLM for Structural Reasoning (LLM-SR), aim- ing to solve both Task: Question parsing and CoT parsing. The process of extracting state- ments and evidence is similar to Discourse Pars- ing. Correct extraction of statements or evi- dence from the COT is crucial at the outset. Next, the pairwise relationship between a spe- cific statement and its corresponding evidence is assessed (a statement should be followed by its related evidence from the CoT). Both seman- tic and lexical similarity are used to evaluate the accuracy of statements and evidence predic- tions. Finally, once a statement-evidence pair is correctly extracted, it is evaluated to deter- mine whether the evidence can logically deduce the statement. To tackle Question Parsing and CoT Parsing, we implement and investigate var- ious solutions, including (1) applying different structural prompt formats like JSON, Mark- down, or XML. (2) utilising various prompt techniques: Few-shot, Chain of thought, and Multi-hop prompting. (3) Taking advantage of Natural Language Inference (NLI) model for the Statement Verification step. Our best of- ficial result is a 243.047 mean score for test phases A and B, and finally, we rank 7th on the final leaderboard.

## introduction the advent of large language models (llms) has significantly advanced natural language processing, enabling sophisticated reasoning capabilities across diverse tasks. however, ensuring that these models produce structured, interpretable, and logically coherent reasoning remains a formidable challenge. addressing this, the xllm-acl 2025 task-iii: llm for structural reasoning (llm-sr) focuses on evaluating the abilities of llm to generate structured reasoning processes by parsing questions and corresponding chains of thought (cot) into distinct components: major premises, minor premises, and their interrelations. in this paper, we present the approach developed by our team, for the llm-sr task. our methodology targets two primary subtasks: question parsing and cot parsing. we conceptualise the extraction of statements and evidence as analogous to discourse parsing, emphasizing the accurate identification of these elements as a foundational step. subsequently, we assess the pairwise relationships between specific statements and their corresponding evidence, ensuring that each statement is logically supported by its related evidence within the cot. to enhance the structural reasoning capabilities of llms, we investigate several strategies: our system achieved a mean score of 243.047 across test phases a and b, securing the 7th position on the final leaderboard. these results underscore the effectiveness of combining structured prompt formats with advanced prompting techniques and verification models to enhance the structural reasoning abilities of llms. ## related work recent advancements in prompt engineering have demonstrated that providing large language models with a small number of in-context examples can yield strong zero and few-shot performance gains @xcite . explicitly eliciting intermediate reasoning steps via chain-of-thought prompting has been shown to further boost complex arithmetic and commonsense reasoning @xcite . techniques such as self-consistency decoding, which samples multiple reasoning paths and aggregates the most consistent answer, markedly improve chain of thought accuracy @xcite . decomposition and multi hop strategies like self ask break down complex queries into subquestions for greater interpretability @xcite . hybrid reasoning-acting prompts (react) interleave "thought" and "action" steps to ground llms in external environments or tools @xcite . pipeline optimisations such as lm-bff automate prompt template selection to refine fewshot tuning @xcite . constraining llm outputs to structured formats (json, xml) via schema-based or grammar-based decoding ensures machine-readable consistency for downstream extraction tasks @xcite . methods from discourse parsing-segmenting text into elementary discourse units and labelling their rhetorical relations-provide algorithms analogous to statement and evidence extraction @xcite . finally, natural language inference frameworks trained on large-scale corpora such as snli and multinli underpin the verification of logical entailment between extracted evidence and hypothesis statements @xcite . ## task description the xllm-acl 2025 shared task-iii: llm for structural reasoning (llm-sr) challenges participants to generate a controllable and interpretable reasoning process via step-by-step inference @xcite . it comprises two subtasks: question parsing, which extracts all conditions necessary for solving a given question, and cot parsing, which segments a provided chain-of-thought into distinct statements and their corresponding evidence @xcite . for each statement-evidence pair, systems must predict a binary verification label indicating whether the evidence logically entails the statement @xcite . the training set contains 24 annotated examples derived from the logiqa logical reasoning benchmark @xcite . participants are restricted to using the llama-3-8b-instruct backbone model for all subtasks (xll, 2025)(meta, 2024). evaluation is performed on two public test phases (a and b), with macro f1 computed over question parsing, statement parsing, statement-evidence pairing, and verification predictions @xcite . submissions are evaluated and scored through the codabench platform, ensuring reproducibility and standardized scoring @xcite . the xllm dataset (shuyi-zsy, n.d.) is conducted by the organizer. ## full pipeline this is our full best pipeline, which is shown in 1. the proposed pipeline constitutes a modular and interpretable architecture tailored for the llm-sr task, effectively addressing both question parsing and statement-evidence verification through a sequence of structured components. it begins with an input consisting of a question and its corresponding chain-of-thought (cot) rationale. a parsing prompt is applied to extract candidate statements and evidence segments from the cot, yielding a structured intermediate representation. this initial decomposition step is critical, as it transforms unstructured natural language into discrete units that downstream components can process more reliably. next, a similarity check module evaluates the lexical and semantic coherence between extracted statements and their corresponding evidence spans. this filtering mechanism ensures that only aligned pairs proceed to the next stage, thereby minimizing noise and reducing the likelihood of spurious relations. following this, the pipeline incorporates two reasoning pathways in parallel: chain-of-thought prompting and multi-hop prompting. chain-of-thought prompting improves interpretability and promotes stepwise deduction by explicitly modeling intermediate reasoning steps. in contrast, multi-hop prompting decomposes complex inference into smaller, interdependent sub-questions, enabling more accurate retrieval of distributed evidence and enhancing logical consistency. the outputs from these reasoning modules are routed into a natural language inference (nli) model, which performs the verification step by determining whether each evidence segment entails its associated statement. this dedicated verification layer isolates the decision-making process from generation, improving both reliability and transparency. by leveraging pre-trained nli models, the system achieves strong verification performance without additional supervision. overall, this pipeline exemplifies best practices in prompt-based llm system design. its modular structure allows for independent tuning and component replacement, fostering adaptability and ease of maintenance. the integration of structured prompting strategies, semantic similarity filtering, and nli-based verification results in a robust and scalable solution for structural reasoning tasks. moreover, the pipeline supports transparency and interpretability at each stage, making it suitable for highstakes domains where explanation and traceability are essential. ## main results first, the combination of few-shot, chainof-thought, multi-hop prompting, nli verification, and similarity checking, all delivered in a markdown format with four prompt calls, achieves the highest overall accuracy (question parsing: 67.96%, statement parsing: 39.21%, statement-evidence matching: 12.04%, reasoning: 3.83). this demonstrates that layering multiple complementary prompting techniques yields significant gains, particularly for the most challenging subtasks of identifying and aligning statements with their evidential support. second, reducing the prompt format to pure markdown without the similarity check module causes a modest drop in performance finally, the simplest few-shot-only configurations (two or four calls) produce the lowest scores across all subtasks (question parsing: 56.02-59.24%, statement parsing: 29.17-38.50%, statement-evidence matching: 2.07-2.29%, reasoning: 1.07-1.84), confirming that advanced prompting strategies are critical to unlock the full structural reasoning capabilities of llms. in summary, these results underscore the effectiveness of integrating multiple prompting techniques within a markdown format and highlight the trade-offs inherent in prompt format selection, prompt complexity, and verification strategy. the highest-performing configuration (four calls, markdown, full-technique suite) is the reference point for further improvements in structural reasoning pipelines. finally, our best score on test set b are 73.24, 47.07, 15.59, and 10.22. our team achieve rank 7th i xllm-acl 2025 task-iii: llm for structural reasoning (llm-sr). ## conclusion and future work in this study, we presented a comprehensive and modular pipeline for addressing the llm-sr task, targeting both question parsing and chainof-thought (cot) parsing subtasks. through a series of experiments, we demonstrated that the integration of structured prompt formats, advanced prompting strategies (few-shot, chain-of-thought, multi-hop), and a dedicated nli-based verification step significantly enhances the model's structural reasoning performance. our analysis highlights the effectiveness of markdown as a prompt format and the importance of leveraging multiple complementary reasoning techniques to ensure accurate statement-evidence alignment and logical verification. in future work, we plan to explore dynamic prompt selection mechanisms that adapt based on question complexity, as well as integrating retrievalaugmented generation (rag) components to support knowledge-intensive reasoning. we are also interested in fine-tuning or instruction-tuning smaller models to perform verification and parsing steps more efficiently.