Challenges to Open-Domain Constituency Parsing (2022)

Abstract: Neural constituency parsers have reached practical performance on news-domain benchmarks. However, their generalization ability to other domains remains weak. Existing findings on cross-domain constituency parsing are only made on a limited number of domains. Tracking this, we manually annotate a high-quality constituency treebank containing five domains. We analyze challenges to open-domain constituency parsing using a set of linguistic features on various strong constituency parsers. Primarily, we find that 1) BERT significantly increases parsers’ cross-domain performance by reducing their sensitivity on the domain-variant features.2) Compared with single metrics such as unigram distribution and OOV rate, challenges to open-domain constituency parsing arise from complex features, including cross-domain lexical and constituent structure variations.

## introduction constituency parsing is a fundamental task in nlp that has received constant research attention @xcite @xcite . as shown in figure 1 , given a sentence, the task is to identify hierarchical phrase structures that reflect its syntax, such as prepositional phrases (pp; e.g., "in late 1991"), noun phrases (np; e.g., "late 1991") and verb phrases (vp; e.g., "scheduled for delivery in late 1991"). constituent structures have been shown useful for downstream tasks including machine translation @xcite , natural language inference @xcite ), text summarization @xcite . in addition, they can be transformed into dependency tree structures @xcite , which have been shown to be useful for a wide range of nlp tasks. the dominant approach to constituency parsing employs a neural model with pre-trained token representation @xcite , training the network parameters over manually labeled constituent structures from the penn treebank (ptb) @xcite . as labeled constituent trees can be costly to obtain, most work makes use of the ptb data for training, which is financial news. the current state-of-the-art f-scores reach over 95% on the training domain (i.e., newswire) and are around 88% for biomedical and web test data @xcite . compared with parser performance decades ago, accuracies around 90% nowadays is much more useful for downstream applications. @xcite showed that pretraining is a key factor that brings consistent crossdomain performance improvements by using bert @xcite . ideally, a constituency parser should give robust performance in the open domain, so that both domain-specific applications @xcite and open-domain nlp tasks @xcite can benefit from syntactic structures. the above observations, however, are made on a rather limited (i.e., 3) number of domains. in addition, there has been relatively little study on the key factors to the performance gap between financial news test and test on other domains, when the model is trained on financial news. it remains an interesting research question to understand the performance of constituency parsing with regard to a wider range of domains and text genres in order to understand the boundaries and existing techniques and identify the main challenges for robust open-domain constituent parsing. such knowledge can be informative for guiding the design of robust open-domain parsers. to this end, we evaluate three strong constituency parsers on these domains, as well as the existing news, biomedical and web domains. the parsers include the non-neural bllip parser @xcite , the in-order transitionbased parser @xcite and the berkeley neural chart-based parser @xcite . for the test domains, we include the vast majority of existing cross-domain test data in the literature, which cover the biomedical, web text, literature fiction and telephone conversations. in addition, given much research interest in nlp for dialogue @xcite , law @xcite and review @xcite ) domains, we manually label constituent structures for five typical domains (i.e., dialogue, forum, law, literature, review), resulting in a test set of 1,000 sentences for each domain. empirically, we aim to answer the following research questions. first, what are the parser performances in the open domain, and which domains are the most challenging for constituent parsing? we find that the parser performance varies from 83% to 93% under different domains, and the most challenging text genres are review, dialogue and literature. the low results on these domains mean that open-domain constituency parsing is still a challenge. second, what are the relative strengths of different parser models, and does bert give similar improvements for all domains? we find that the parsers that give stronger results on ptb do not necessarily give stronger results on various other domains, which reflects limitations of evaluating parser performances only on ptb data. besides, we show that bert benefits parsers on crossdomain performance by reducing their sensitivity on domain-variant features. third, what are the main challenges for crossdomain parsing? by analyzing a set of linguistic features, we find that compared with single metrics such as unigram distribution and oov rate, challenges to cross-domain constituency parsing arise from combinations of factors, including crossdomain lexical and constituent structure variations. to our knowledge, we are the first to construct constituency parsing test data for the forum and law domains and the first to analyze the factors that make open-domain parsing challenging by extensive empirical evaluation. we release our dataset and results at https://github.com/ringos/ multi-domain-parsing-analysis . ## conclusion we investigated the challenges of cross-domain constituent parsing by making use of a large number of test domains, which include newswire, biomedicine, prose, web-text, conversational speeches, as well as give new test domains including dialogue, forum, law, literature and review, for each of which we construct a test set of 1,000 sentences. results show that the dominant parsers can achieve 83% to 93% accuracies for different domains, and cross-domain parsing is still a challenge, where different domains exhibit varying types of difficulty. we further find that the difficulty for cross-domain parsing lies more in comprehensive distribution differences involving multiple factors such as grammar rules and patterns, as compared to single factors such as oov rate and token ngram distribution variations. in addition, bert helps neural parsers improve cross-domain performance by reducing their sensitivity to domain-variant features. our results show that toward robust opendomain constituent parsing, more work should be done on addressing out-of-distribution generalization in representation learning.