Self-Correction Makes LLM s Better Parsers (2025)

Abstract: Large language models (LLMs) have achieved remarkable success across various natural language processing (NLP) tasks. However, recent studies suggest that they still face challenges in performing fundamental NLP tasks essential for deep language understanding, particularly syntactic parsing. In this paper, we conduct an in-depth analysis of LLM parsing capabilities, delving into the underlying causes of why LLMs struggle with this task and the specific shortcomings they exhibit. We find that LLMs may be limited in their ability to fully leverage grammar rules from existing treebanks, restricting their capability to generate syntactic structures. To help LLMs acquire knowledge without additional training, we propose a self-correction method that leverages grammar rules from existing treebanks to guide LLMs in correcting previous errors. Specifically, we automatically detect potential errors and dynamically search for relevant rules, offering hints and examples to guide LLMs in making corrections themselves. Experimental results on three datasets using various LLMs demonstrate that our method significantly improves performance in both in-domain and cross-domain settings.

## introduction llms have exhibited significant success across a wide range of nlp tasks and applications @xcite @xcite . however, in the field of syntax parsing, enabling llms to achieve a deep structural understanding of sentences remains challenging @xcite @xcite . although non-llm parsers have achieved over 95% f-scores in the newswire domain, the performance of llms falls significantly short. this discrepancy inevitably raises two questions: where exactly does the parsing capability of llms fall short? what specific knowledge are llms lacking? * corresponding author. prior works have provided some findings about the parsing capabilities of llms. for instance, @xcite pointed out hallucinations in llm outputs, while @xcite highlighted that they tend to be much flatter than gold trees. however, existing works primarily analyze high-level performance metrics without delving into the detailed shortcomings or the underlying causes. to address these gaps, we conduct an in-depth analysis of llm parsing capabilities, examining both the structural characteristics of their outputs and the specific types of errors they exhibit. our findings suggest that the inability to fully utilize grammar rules in the training data is a potential reason why llms fail. unlike non-llm parsers, which can comprehensively learn rules from extensive training data, llms solely rely on limited contextual examples provided during few-shot learning. this fundamental difference distinguishes llm parsing performance from non-llm parsers. motivated by the above analysis, we are interested in methods that allow llms to learn structural knowledge without additional training. the method is not intended for making llm parsers practical (competitive to non-llm parsers), but for analyzing the performance of llms on the parsing task. therefore, in this work, we propose a selfcorrection method that first identifies errors in the initial answer, then searches the treebank for relevant examples based on the characteristics of different types of errors, and finally guides llms in correcting errors via hints and examples. this method effectively enables llms to reduce all types of errors in their parsing results, thereby improving the overall parsing performance. experimental results prove that our method achieves significant improvement in both in-domain and cross-domain settings. in conclusion, our contributions are three-fold: our code is available at https://github.com/ zzy0509/self-correction-parsing . ## in-depth analysis of llm parsing capability llms have demonstrated impressive performance in a wide range of nlp tasks @xcite . however, recent studies indicate that llms face significant challenges when performing syntactic parsing tasks, such as producing parse trees that are flatter than gold trees @xcite . although previous studies have explored the parsing capability of llms to some extent, they remain relatively limited in providing deep insights into the potential reason why llms fail. both studies lack a comprehensive analysis of the reasons behind the weak performance and the specific types of errors llms make. to gain a deeper understanding of the constituency parsing capacity of llms, we conduct an in-depth analysis from the perspectives of the overall parsing performance, the characteristics of their parsing results, and the types of errors they make. specifically, we first evaluate the parsing performance of different llms under the few-shot setting, comparing these results to those from one of the non-llm sota models, the berkeley parser @xcite . second, we compare the parsing results of llms with both gold trees and those produced by sota non-llm parsers, aiming to analyze the distinct characteristics of llms and non-llm parsers and uncover reasons for the significant performance gap between them. finally, we categorize the parsing errors made by llms into four types and analyze the error distribution, and conduct an in-depth examination to reveal which types of errors llms commonly make. ## our self-correction approach the above results and analysis help us better understand the shortcomings of llm parsing results and encourage us to propose targeted methods for enhancing llm parsing capabilities. since section 2.2 has proved that the low performance of llms may stem from their lack of known rules in the existing treebanks, thereby we are interested in exploring methodologies that enable llms to effectively assimilate knowledge from the existing treebanks without the need for additional training to improve their constituency parsing capabilities. furthermore, we aim at a method based on the four types of parsing errors mentioned in section 2.3 to effectively address all errors made by the llm. in this work, we propose a self-correction method based on the rules from the existing treebank, guiding llms to correct errors in the previous answers. consistent with the common process of self-correction in previous works @xcite , we first prompt llm to generate a base answer, identify potential errors in it, and then guide llm to correct these errors through specific hints and examples. specifically, based on the characteristics of parsing results, we categorize self-correction into two parts: sentence mismatch correction and tree structure correction. first, we perform mismatch correction to ensure the alignment of the leaf nodes between the predicted trees and gold trees. then we make structure correction, which addresses four types of structural errors introduced in 2.3. ## experiments we conduct experiments to verify the effectiveness of our self-correction method and perform a detailed analysis to gain insights into what llms have learned through our method. all experimental settings remain consistent with those in direct llm-based parsing, as described in section 2.1. ## conclusion in this paper, we conduct an in-depth analysis of llm parsing capability from the perspective of the overall parsing performance, the characteristics of their parsing results based on the rules, and the types of errors they made. based on the analysis that the low performance of llms may stem from their lack of known rules in existing treebanks, we propose a self-correction method that enables llms to efficiently acquire rules from treebanks without additional training, thereby correcting errors in the previous answer. experimental results demonstrate that our method consistently improves performance, effectively guiding llms to correct all types of errors made by themselves.