Large Language Models Are No Longer Shallow Parsers (2024)

Abstract: The development of large language models (LLMs) brings significant changes to the field of natural language processing (NLP), enabling remarkable performance in various high-level tasks, such as machine translation, question-answering, dialogue generation, etc., under end-to-end settings without requiring much training data. Meanwhile, fundamental NLP tasks, particularly syntactic parsing, are also essential for language study as well as evaluating the capability of LLMs for instruction understanding and usage. In this paper, we focus on analyzing and improving the capability of current state-of-the-art LLMs on a classic fundamental task, namely constituency parsing, which is the representative syntactic task in both linguistics and natural language processing. We observe that these LLMs are effective in shallow parsing but struggle with creating correct full parse trees. To improve the performance of LLMs on deep syntactic parsing, we propose a three-step approach that firstly prompts LLMs for chunking, then filters out low-quality chunks, and finally adds the remaining chunks to prompts to instruct LLMs for parsing, with later enhancement by chain-of-thought prompting. Experimental results on English and Chinese benchmark datasets demonstrate the effectiveness of our approach on improving LLMs’ performance on constituency parsing.

## introduction recently, the development of large language models (llms) @xcite @xcite significantly reshapes the field of natural language processing (nlp), where they achieve impressive performance on a variety of nlp applications such as question-answering and dialogue interaction. nevertheless, in the era of llm, fundamental nlp tasks that involve linguistic analysis still play essential roles in the field of nlp @xcite . for example, syntax parsing provides structural understanding of sentences and thus is valuable in evaluating the capability of llms to understand and use instructions @xcite @xcite . in general, syntactic parsing is an essential nlp task that has been studied for decades @xcite @xcite @xcite @xcite and has two widely used conventional variants with phrase-structure and dependency grammar, of which the former drives constituency parsing that provides deep structural analysis of sentences with informative phrase and function markups. recent studies @xcite reveal that llms are ineffective in such parsing task, and we also confirm the fact in figure 1 with parse trees generated by @xcite under various prompts, where they all look very different from human-crafted gold-standard trees, indicating syntactic parsing is challenging to llms. although some studies @xcite @xcite design prompts to instruct llms for better parsing, it is still limited in performing one-step (i.e., end-to-end) parsing, which does not fully benefit from the advantages brought by the strengths of llms. in this paper, we explore the effectiveness of directly using state-of-the-art llms for constituency parsing, and find that they generate parse trees that are shallow and with relatively low recalls on retrieving correct text spans, while in the meantime llms show relatively better performance in chunking, especially in identifying short chunks. therefore, based on our analyses and motivated by existing studies that decompose complex tasks into several steps to improve llm's performance ( @xcite @xcite @xcite , we propose a three-step approach to improve llms for parsing. the approach firstly prompts llms to extract chunks in the text, which is considered partial parsing and is demonstrated to be an essential step for full parsing @xcite . then, our approach filters out long chunks that are more likely to carry noise than the short ones. finally, it uses the remaining chunks as conditions in the prompts to instruct llm for parsing, allowing llms to directly use the chunks to generate the parse tree of a sentence. we validate our approach on several benchmark datasets, including english penn treebank 3 (ptb) @xcite , chinese penn treebank 5 (ctb5) @xcite , and genia @xcite , where the results prove the effectiveness of our approach over existing llms used directly for constituency parsing. the contributions of this paper are summarized as follows. first, we provide comprehensive analyses of existing state-of-the-art llms on constituency parsing. second, we propose a threestep approach that involves chunking, filtering, and parsing, to improve llms for constituency parsing. third, the experiment results and further analyses demonstrate the effectiveness of the proposed approach for constituency parsing. the performance of llms, which can be divided into two groups. the first group includes gpt-3.5 and gpt-4, which are accessible only through online api. the second group contains publicly available llms; namely, the 7b and 65b versions of llama-2 for english @xcite , and the 7b and 13b versions of alpaca for chinese @xcite (we denote these english and chinese llms as llama for the sake of convenience). 2 we use these llms to parse the test sets of ptb, ctb5, and genia, where ptb and ctb5 are general domain english and chinese datasets, respectively, and genia is an english medical dataset. the statistics of the datasets are reported in prompt template: the following is a sentence that has already been tokenized. words are separated by white spaces. please parse the sentence with given words. <few-shot @xmath0: <@xmath1: input: is this what the home builders want ? output: (sq (vbz is) (np (dt this)) (sbar (whnp (wp what)) (s (np (dt the) (nn home) (nns builders)) (vp (vbp want)))) (. ?)) generally, there are two approaches to linearizing a parse tree into a string to facilitate llm processing @xcite . the first is the bracket-based approach, which uses brackets to represent the structure of a parse tree, such as "(s (np he) (vp jumps))". the second is the transition-based approach that utilizes the sequence of transition actions to construct the parse tree, such as "shift he; np; shift jumps; vp; reduce; s". according to @xcite , when working with llms, the bracket-based approach outperforms the transition-based one; therefore, we adopt the bracket-based representation in our study. motivated by @xcite , we design prompts (see ## related work constituency parsing is a fundamental nlp task that is generally performed via chart-or transitionbased approaches @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite , where extra resources are leveraged to improve the parsing performance @xcite @xcite @xcite . recently, llms have achieved remarkable success in many nlp tasks, which motivates researchers to apply them to parsing. existing approaches that leverage llms for parsing generally follow the sequence-to-sequence settings @xcite , where the goal of llms is to produce the parse tree in the bracket format. the llm-based approaches fine-tune llms on the training data or design prompts to instruct figure 6 : two examples from the test set of ptb. the first three columns show the gold-standard parse tree; the parse tree generated by the gpt-4 baseline and our approach, respectively; the last column lists some chunks produced in step 1 of our approach. the red boxes mark the parts in the generated parse tree that do not match the gold standard. the parse trees by our approach for these sentences happen to be identical to the gold-standard trees. llms to generate valid parse trees @xcite . for example, @xcite directly train llama @xcite on the training set of ptb and show inferior performance than the conventional chart-and transitionbased approaches that leverage small pre-trained language models such as bert @xcite and xlnet @xcite . they also perform zero-shot and few-shot parsing but the results are unsatisfactory, which confirms the ineffectiveness of llms for parsing. there are studies that select appropriate training data @xcite to further improve constituency parsing with llm. in this study, we propose to break parsing into three steps: chunking, filtering chunks, and parsing with chunks, and demonstrate this 3-step approach improve system performance significantly. ## conclusion in this paper, we apply llms to parsing and observe that the current state-of-the-art llms are shallow parsers in that they are effective in perform-ing simple syntactic tasks such as chunking but are ineffective at conducting full constituency parsing due to their auto-regressive nature of text modeling approach. given this, we propose to break parsing into three steps: chunking, filtering, and parsing with a chunk list. experiments on english, chinese, and domain benchmark datasets show that the 3step approach enables llms to produce deeper and better parse trees and thus they are no longer shallow parsers. this study demonstrates the benefits of decomposing a complex task (e.g., parsing) into subtasks and the importance of including additional mechanisms (e.g., filtering chunks in step 2 and treating given chunks as soft constraints in step 3) to alleviate the risk of the error propagation. we plan to apply this idea to other nlp tasks in the future.