Genres, Parsers, and BERT : The Interaction Between Parsers and BERT Models in Cross-Genre Constituency Parsing in E nglish and S wedish (2021)

Abstract: Genre and domain are often used interchangeably, but are two different properties of a text. Successful parser adaptation requires both cross-domain and cross-genre sensitivity (Rehbein and Bildhauer, 2017). While the impact domain differences have on parser performance degradation is more easily measurable in respect to lexical differences, impact of genre differences can be more nuanced. With the predominance of pre-trained language models (LMs; e.g. BERT (Devlin et al., 2019)), there are now additional complexities in developing cross-genre sensitive models due to the infusion of linguistic characteristics derived from, usually, a third genre. We perform a systematic set of experiments using two neural constituency parsers to examine how different parsers behave in combination with different BERT models with varying source and target genres in English and Swedish. We find that there is extensive difficulty in predicting the best source due to the complex interactions between genres, parsers, and LMs. Additionally, the influence of the data used to derive the underlying BERT model heavily influences how best to create more robust and effective cross-genre parsing models.

## introduction the performance degradation of models trained on one data set when used on another has been well established @xcite . however, how we define the source of the problem (e.g. out-of-domain differences) is problematic. within domain adaption, even the term domain is incredibly loosely defined @xcite . this has allowed conflating several different properties of texts, as such properties can be difficult to distinguish given some of their inherent overlap. relevant for this work is how we define the distinction between genre and domain. we use @xcite as a template and define genre dealing with more abstract and linguistic characteristics used within a text @xcite . domain, however, is more about the topics and content words used. much parsing work has actively focused on handling domain differences, such as reducing lexical gap issues between target and source domains (e.g. @xcite ), while explicit handling of genre differences is not as heavily researched, nor as well understood. while much parsing literature uses the terms interchangeably @xcite , they are not, however, identical concepts. by doing so we are are not effectively identifying which out-ofdomain differences should be contributed more to out of genre or domain difference. for example, wikipedia articles are written in a style following more of an encyclopedia. pages on medicine and languages may contain very different vocabularies, but the linguistic characteristics are most likely similar given the encyclopedic style of writing. however, responses in a forum on medical advice may share a large amount of vocabulary overlap with wikipedia medical pages, but would share considerably less linguistic structure given the dialogue nature of a forum (e.g. more interrogative sentences). a treebank (in our case constituency treebanks) often contains many noticeable domains, but it is harder to gauge how many distinct genres are present. sometimes they are explicitly marked in the annotation @xcite , while others explicitly separate out the different genres out @xcite . yet a treebank is often times a concatenation of various texts that may ultimately represent slightly different linguistic characteristics (even if annotated strictly on newspaper). domain differences often result in a high lexi-cal divergence. the best performing chart-based grammar-based constituency parsers were predominantly unelexicalized @xcite , which helped reduce issues with lexical differences. however, current state-of-the-art neural span-based chart-based parsers have substantially changed this paradigm. with the use of lexical embeddings, there is now a great deal of lexicalization within the modeling architecture to an extent that was not seen before, with some state-of-the-art parsers not utilizing pos tags @xcite . the use of characters and subtoken information has been shown to be beneficial in reducing lexical sparsity issues @xcite , which ultimately reduces domain difference disparities. however, what impact this degree of lexical contextualization has on cross-genre parsing remains unclear. additionally, the use of language models (lms), such as bert @xcite , to derive contextualized embeddings presents yet another variable in selecting the best source. given that lms are derived on large, unannotated texts, they implicitly capture various linguistic properties of the these texts @xcite ) which they they infuse into the parser via contextualized embeddings. how embeddings derived from, often, a single genre behave as a bridge between two, often, other genres presents an interesting issue. we are interested in examining the following in cross-genre experiments: more recent work has been focused on creating domain specific embeddings. the use of domain embeddings in chinese dependency parsing by @xcite built on the previous research by @xcite . both showed domain and treebank specific embeddings respectively yielded better performance over direct treebank concatenation, as this allows for the capturing of domain specific and general features. results were further improved upon with adversarial methods and bert fine-tuning by @xcite . @xcite found that contextualized embeddings have substantially reduced the difficulty in handling lexical gap issues between domains when the target and source are syntactically similar, but then employ additional strategies to handle more syntactically dissimilar ones. additional work by @xcite showed that while pretrained lms improved parser performance over several english domains, the improvements for outof-domain results were not relatively larger. as @xcite note, parser adaptation requires both genre and domain adaption, but that content features, such as topics, do not generalize well for genre modeling, suggesting different techniques are needed for cross-genre modeling. while the incorporation of pre-trained lms has become standard in many nlp tasks, understanding how different models interact with different tasks is still an area of active research. work by @xcite on french shows that a smaller french specific lm model derived from more diversified source data can compete with models substantially larger across a variety of downstream nlp tasks. specifically, tasks which showed more divergence from wikipedia benefited the most from a mixed genre lm. the importance of source diversification is also seen in lms for finnish @xcite and chinese @xcite , each of which contain more text sources than simply wikipedia. the impact of source diversification can also be seen in domain specific lms, such as finbert @xcite , which was derived from several types of financial sources, as different financial texts are radically different stylistically. one additional benefit by explicitly looking at specific genres is it can help further our understanding of linguistic properties of different texts, forcing us to re-evaluate earlier annotation schemes predominantly designed for an original treebank genre @xcite . ## methodology we perform a systematic set of experiments for english and swedish, using different neural constituency parsing architectures in combination with various bert models to examine how this impacts cross-genre parsing. english is widely used in cross-domain and cross-genre research. swedish, however, is not as thoroughly examined, yet possess multiple genre treebanks as well as bert models, making it suitable for our research interests. ## delexicalized and subtoken divergence as noted in section 2, delexicalized comparisons of treebanks have been used to identify treebank similarity for source selection. an established delexicalized method is kl divergence @xcite of pos trigrams @xcite . in ## discussion the different parsing architectures interact differently with the underlying latent properties of the embeddings in their parsing decisions. supar, however, does seem to show the most consistent stable performance across all experiments, and in a majority of cases, is the best performing model. whether pos information is needed in neural constituency parsing is seemingly a complicated picture in terms of performance, though it has been shown to benefit certain neural dependency parsing architectures @xcite . however, we can see the impact the inclusion of the pos loss has in terms of parser source preferences, as seldom were the behaviors of bnpno and bnp similar. this is to be expected, as the source of the underlying lm may have implicitly different pos distributions than either the target or source genre, and a pos loss is most likely sensitive to these differences. bnp showed much more stable source preferences across genres and experiments, indicating how the pos task is seemingly is able to mitigate, to some degree, the influence of the lm, though whether this is positive or negative is unclear. this may indicate that embeddings derived from more mono-genre texts interact in a more consistent way when using pos information, stablizing source preferences. this is seen in both english and swedish experiments to a degree. however, once the lm has more genre representations, this stabilizing factor no longer holds as the inherent pos distributions are most likely far more varied, as seen with kbbert. we can also see how other architectural choices besides the inclusion of pos information are important, as otherwise we would expect bnpno and supar to behave similarly, which they do not. a clear distinction is how the two parsers incorporate bert embeddings. the choice of scalar mixing @xcite , embedding averages @xcite , and different subtoken selection @xcite have all shown to impact performance on nlp tasks. another factor may be the additional word boundaries mlp layers created in supar's architecture, providing more context for an individual parsing decision, making it more robust to slight variations in syntactic distributions. the influence of the lm's genre is perhaps most seen in the swedish wikipedia genre experiments with swbert and bnp seen in fig. 4 . all the sources produce in similar results when wikipedia is the target. it may simply be that when the target genre is too similar to the genre of the lm, the impact of similar sizes of different source genres is minimized, as there now exist too much latent and explicit wikipedia data. however, in the all setting, we see a substantial increase where now there is not only more data, but more diversity to counterbalance the wikipedia derived lm. additionally we see that wikipedia is the preferred source for all genres outside of blog for bnp, and results are substantially better than their bnpno counterparts. however, this does not hold across parsers, given that supar shows completely different behaviors for swbert than the berkeley experiments. this further emphasizes the difficulty of transferring knowledge of one parser's source preference behaviors to another. for both languages there can be substantial deviation of the best performing source genre from the closest source genre on both a delexicalized and subtoken level. overall gains specific sources show for an individual target source can also be incredibly inconsistent across experiments. why this is, is further complicated by the source genres interaction with the lm. an english only bert model yielded some improvements while a swedish only model showed varying results depending upon the lm. this can be due to several factors. the most obvious one is due to the size of the lm, as swbert is substantially smaller, yet it still yields results close to the much larger mbert for swedish. however, kbbert is approximately the same size as bertbc, and produces much larger absolute gains than bertbc did for english, providing counter evidence that size is not the only factor. the difficulty in identifying the reasons is due to many interacting aspects such as higher baselines for english, treebank sizes, and annotation scheme complexity. however, kbbert was derived from a far more diverse set of genres, many of which overlap with the set, compared to bertbc, which was derived from mostly two distinct genres, neither of which overlap with the ewt substantially. importantly, we also see the impact a lm model has on closing performance gaps between parsing architectures, as kbbert results for the berkeley parsers are overall more on par with supar. this demonstrates how interactions between three distinct genres makes optimal source selection far more difficult when using lms and different parsers than established delexicalized approaches. ## conclusion we have performed a set of detailed experiments that explored the interaction between genres, parsers, and bert models. we have shown that the lm plays the pivotal role in successful genre-sensitive parsing within our chosen parsing architectures. in addition, we have also shown that different architectures often behave dissimilarly, making determining best sources for a specific target reliant on better understanding the underlying architectures, and not transferring direct behavior of one parser to another. treebanks are rather static, particularly constituency treebanks. while we have often seen incremental performance gains with every new parser, how successful we are at cross-genre parsing will, for the time being, be more related to our exploitation of various other sources and methods. lms, for example, can be trained on vast amounts of unannotated data, allowing for the the lm to become far more sensitive to genre differences than any small treebank, especially as we have control over the creation of an lm, and less so than with a treebank. perhaps the most practical way to currently create genre-sensitive parsing models is to better mix distinct genres within the data used to derive the lm. the lm itself does not even have to be overtly large, rather even small mixtures of other genres and domains provides noticeable benefits @xcite . future research will look to create multilingual cross-genre models that work across treebanks and genres. a full english results for parsers and bert models