LLM -enhanced Self-training for Cross-domain Constituency Parsing (2023)

Abstract: Self-training has proven to be an effective approach for cross-domain tasks, and in this study, we explore its application to cross-domain constituency parsing. Traditional self-training methods rely on limited and potentially low-quality raw corpora. To overcome this limitation, we propose enhancing self-training with the large language model (LLM) to generate domain-specific raw corpora iteratively. For the constituency parsing, we introduce grammar rules that guide the LLM in generating raw corpora and establish criteria for selecting pseudo instances. Our experimental results demonstrate that self-training for constituency parsing, equipped with an LLM, outperforms traditional methods regardless of the LLM’s performance. Moreover, the combination of grammar rules and confidence criteria for pseudo-data selection yields the highest performance in the cross-domain constituency parsing.

## introduction constituency parsing, a fundamental task in natural language processing (nlp), has achieved remarkable progress on in-domain benchmarks @xcite @xcite @xcite , indicating the growing competence of parsers in capturing the underlying syntactic structures. however, opendomain constituency parsing is notably challenging @xcite . in diverse, open-domain scenarios, constituency parsing faces complexities beyond the well-defined task. addressing these challenges is crucial for its broader real-world nlp applications. to address the issue of domain shift, selftraining-based unsupervised domain adaptation has emerged as a promising approach @xcite @xcite @xcite @xcite . this method utilizes a source domain model to automatically label a large-scale raw corpus from the target domain during each iteration. high-confidence pseudo data is then selected as additional training data to improve target domain performance. however, the quality and quantity of raw corpus cannot always be guaranteed for low-resource domains @xcite @xcite , which limits the use of self-training approaches. traditional methods struggle to construct fine-grained sentences that facilitate knowledge transfer. the large language model (llm), with its powerful generative capabilities, can serve as a potential solution to the challenge of the raw corpus quantity and quality for the target domain (as shown in figure 1 ). it's important to note that our experiments revealed that llms exhibit limited performance for constituency parsing. to tackle the challenges of llms' flexibility and hallucination problems @xcite in generating sentences, we employ grammar rules as instructions for llms to generate target domain sentences. grammar rules have proven effective in cross-domain data generation @xcite and are closely related to constituency parsing @xcite . by incorporating grammar rules, we aim to generate a large volume of high-quality raw sentences highly relevant to the cross-domain constituency parsing task. furthermore, we dynamically embed llms into the iterative process of self-training, enhancing their adaptability and flexibility. in each iteration of self-training, we 1) use llm with grammar rules extracted from currently available training instances to generate target domain raw corpus; 2) train a constituency parser with this data; 3) parse the raw corpus using the trained parser; and 4) select high-quality pseudo data from the parsed trees as additional training instances for parse the constituency tree for the sentence: terms were n't desclosed . generate a sentence by the grammars: @xmath0, @xmath1... the next iteration. for data selection, we employ multiple criteria such as token-based, confidencebased, and grammar-rule-based selection. experimental results demonstrate the superior constituency parsing performance of our llmenhanced self-training approach. the raw corpus generated by llm proves quite effective for the cross-domain self-training tasks, and combining grammar-rule-based and confidence-based selection of pseudo data performs best. this criterion considers both structural information and ensures reliability. additionally, open-source llms can effectively replace closed-source llms, achieving comparable performance in our methods. throughout the self-training iterations, updated grammar rules and selected pseudo data progressively move closer to the target domain. to the best of our knowledge, we are the first to dynamically incorporate llms into the selftraining for constituency parsing. our model utilize llms to create a fine-grained raw corpus that gradually adapts from the source to the target domain. we also introduce an optimal pseudodata selection criterion, combining grammar-rulebased and confidence-based criteria. this selection effectively contributes to our llm-enhanced self-training method for cross-domain constituency parsing. furthermore, we observed that llms underperform in few-shot settings compared to both our baselines and our proposed method. self-training researchers @xcite have attempted to directly transfer source domain parsers to target domains. however, such direct transfer approaches are insufficient for target domains with large gaps compared to the source domain. self-training @xcite @xcite @xcite is a simple and early bootstrapping approach for domain adaptation, applied to tasks such as classification @xcite ; ye et al. (2020), sequence labeling wang et al. (2021, 2020), dependency parsing yu et al. (2015); rotman and reichart (2019); guo et al. (2022), sequence generation he et al. (2019), and qa sachan and xing ( 2018 ) and so on. however, self-training has not been applied to cross-domain constituency parsing. llms' parsing llms have been successfully applied to various nlp tasks @xcite @xcite @xcite . while llms exhibit limitations in structured extraction tasks @xcite , such as cross-domain constituency parsing, their generative capabilities can be leveraged to provide a raw corpus for self-training constituency parsing. to address the issue of hallucinations in llms, we propose employing grammar rules to regulate the structure of generated sentences. we also provide a small set of domain-specific sentences as style references and length limits to further guide the llm generation process. prior research @xcite has shown a close relationship between grammar rules and constituency parser performance, while @xcite successfully used grammar rules to generate cross-domain data. our work uniquely integrates llms into self-training iterations, generating fine-grained raw corpus based on the grammar rules, making the process more flexible and controlled. pseudo-data selection strategy when applying the self-training method to different tasks, researchers typically develop task-specific criteria for pseudo-data selection, taking into account the unique characteristics of each task. for example, @xcite use an ensemble learning model for reliable newly labeled data selection in classification. @xcite show that sampling strategies may vary across datasets, and @xcite propose using out-of-vocabulary numbers to measure source-target domain distance in chinese word segmentation and pos tagging. in this work, we propose adopting grammar rules as a selection criterion and combining it with confidence-based selection. this method ensures the pseudo-data contains rich structural information and exhibits high reliability, ultimately improving cross-domain constituency parsing performance. ## analysis to conduct a thorough analysis and gain deeper insights into our methods, we have chosen the review domain for the detailed exploration. due to space constraints, we placed the comparison between open-source and closed-source llms approaches in the appendix a.3 ## conlusion in this study, we introduced an innovative llmenhanced self-training method for cross-domain adaptation in constituency parsing. by harnessing the generation of llms and integrating them into the self-training process, we showed that our approach considerably enhances constituency parsing performance across various domains. our method effectively merges high-confidence selection cri