LLMSR @ XLLM 25: An Empirical Study of LLM for Structural Reasoning (2025)

Abstract: We present Team asdfo123’s submission to the XLLM@ACL 2025–LLM-SR shared task, which evaluates large language models on producing fine-grained, controllable, and interpretable reasoning processes. Systems must extract all problem conditions, decompose a chain of thought into statement–evidence pairs, and verify the logical validity of each pair. Leveraging only the off-the-shelf Meta-Llama-3-8B-Instruct, we craft a concise few-shot, multi-turn prompt that first enumerates all conditions and then guides the model to label, cite, and adjudicate every reasoning step. A lightweight post-processor based on regular expressions normalises spans and enforces the official JSON schema. Without fine-tuning, external retrieval, or ensembling, our method ranks 5th overall, achieving macro- F_{1} scores on par with substantially more complex and resource-consuming pipelines. We conclude by analysing the strengths and limitations of our approach and outlining directions for future research in structural reasoning with LLMs. Our code is available at https://github.com/asdfo123/LLMSR-asdfo123

## introduction large language models (llms) have recently shown impressive performance on complex reasoning tasks, spurred in part by chain-of-thought (cot) prompting, which asks the model to externalise intermediate steps before giving an answer @xcite . subsequent variants-such as zero-shot cot @xcite , selfconsistency decoding @xcite , treeof-thought search @xcite , and automatically generated demonstrations @xcite -further boost accuracy, yet these free-form rationales remain difficult to evaluate and prone to hallucinations @xcite . the llmsr@xllm25 shared task tackles this limitation by framing reasoning as a constrained cot process: systems must (i) extract every explicit problem condition, (ii) segment a rationale into aligned statement-evidence pairs, and (iii) judge whether each evidence span logically entails its statement. such fine-grained structure "improves the transparency and reliability of the process" (task description) and enables detailed diagnosis of model behaviour. moreover, the step-level labels provide dense supervision for process reward modeling (prm), which optimises how a solution is reached rather than merely what answer is produced @xcite . structured parsing of reasoning brings three concrete benefits. first, it enhances debuggability: developers can pinpoint the exact step where a hallucination or logical slip occurs. second, it supplies explicit training signals for prm, shown to yield more coherent and truthful solutions on mathematical benchmarks @xcite . third, it promotes trustworthy ai: users can audit or amend individual steps, a requirement for safetycritical deployments and formal logic tasks such as entailmentbank proofs @xcite or logicbench diagnostics @xcite . in this report we present team asdfo123's lightweight submission, which relies solely on the untuned meta-llama-3-8b-instruct @xcite . a compact few-shot, multi-turn prompt guides the model through all three subtasks, while a minimal post-processor enforces the official json schema. despite its simplicity, our approach ranks 5th overall, demonstrating that careful prompt design and constrained reasoning can rival far more elaborate pipelines. ## experiments we conduct all experiments on the official llmsr@xllm25 test sets foot_0 . the shared task provides a fine-grained chain-of-thought (cot) analysis corpus derived from logiqa @xcite . it contains only 24 fully annotated training instances, each accompanied by both question-parsing and cot-parsing labels. from the 24 training instances, we heuristically select a small subset of demonstrations that spans the major logical patterns; these serve as the few-shot exemplars in our prompts. the evaluation follows a two-stage protocol. first, we perform a k-shot ablation for question parsing (qp), varying the number of in-context demonstrations. after selecting the best qp setting, we keep it fixed and sweep k again for cot parsing & verification (cp) to determine its optimal demonstration budget. ## conclusion we showed that a carefully crafted, few-shot prompting pipeline-backed by lightweight postprocessing-can tackle the llmsr@xllm25 shared task without fine-tuning or external tools, ranking 5 th overall. while meta-llama-3-8b handles condition extraction well, its verification accuracy remains limited, underscoring the need for stronger or specialised reasoners and process-level training signals. future work should pair stronger base models with dedicated entailment critics and reward models that explicitly value step-by-step correctness.