A Second Wave of UD H ebrew Treebanking and Cross-Domain Parsing (2022)

Abstract: Foundational Hebrew NLP tasks such as segmentation, tagging and parsing, have relied to date on various versions of the Hebrew Treebank (HTB, Sima’an et al. 2001). However, the data in HTB, a single-source newswire corpus, is now over 30 years old, and does not cover many aspects of contemporary Hebrew on the web. This paper presents a new, freely available UD treebank of Hebrew stratified from a range of topics selected from Hebrew Wikipedia. In addition to introducing the corpus and evaluating the quality of its annotations, we deploy automatic validation tools based on grew (Guillaume, 2021), and conduct the first cross domain parsing experiments in Hebrew. We obtain new state-of-the-art (SOTA) results on UD NLP tasks, using a combination of the latest language modelling and some incremental improvements to existing transformer based approaches. We also release a new version of the UD HTB matching annotation scheme updates from our new corpus.

## introduction treebanks (tbs) form a fundamental resource for nlp and computational linguistics research: they provide high quality annotated data for tokenization, sentence splitting, pos tagging and syntactic/semantic relation extraction. for morphologically rich languages (mrls, @xcite , high quality morphosyntactically annotated data is particularly crucial, since basic search on a wordmatching level cannot work before morphological segmentation has occurred. in languages such as hebrew, where vowels are not well represented in the script, the need for reliable morphosyntactic disambiguation is particularly strong, since stringlevel ambiguity is very frequent and high. this is demonstrated by the following frequently cited example from @xcite , which has a large number of possible analyses for just a four character sequence (note hebrew is right-to-left). when such sequences contain multiple subtokens @xcite the.image'), we follow universal dependencies foot_0 terminology in referring to the larger unit as a multi-word token (mwt), and the sub-parts, each of which carries a separate part-of-speech, as tokens. although a treebank for hebrew was already created by the mila institute @xcite , subsequently converted to dependencies @xcite and finally to the more recent universal dependencies framework (see @xcite , hence ud-htb), two major concerns motivate the current work in creating a new ud treebank for hebrew. the first is the age of the data: htb texts were taken from 1990-91 issues of a single newspaper, ha'aretz, as illustrated in (2), describing the introduction of computers to an office. this is one of only a dozen examples of the word 'computer' in the data, which predates mainstream web 1.0 times, mentions no cellular phones, the internet, or a variety of countries established after 1990, including the eu. previous work has shown that nlp systems retain strong lexical biases mirroring both period and author demographics @xcite , which in the case of htb reflect primarily israeli ha'aretz journalists from 1990. a second concern beyond the period and author demographics is genre. htb is a news corpus, and as such reflects formal journalistic writing, which focuses on describing prominent political events, sports news, and reported speech, usually in the past tense, but under-represents expository text, academic language, and colloquial spellings (which are much more variable in hebrew than in english). the importance of genre diversity in training data has often been noted @xcite , but without a second genre to test on, we simply do not know the extent to which fundamental hebrew nlp performance degrades outside of htb's language. in this paper, we attempt to broaden the range of data available for hebrew by creating and evaluating a new, freely available (cc-by-sa license), gold standard corpus using contemporary wikipedia data. although wikipedia's language is also relatively formal, it differs substantially from newspaper reporting, and is more contemporary than htb, covering a broad range of topics, while being available under an open license. 2 our main contributions are: ## previous work in terms of material, there is only one existing tb of modern hebrew prior to our work @xcite based on 1990-91 issues of the newspaper ha'aretz. however, there are multiple versions of this dataset, leading to some confusion. the original tb contained constituent trees, as well as wordinternal segmentation and treebank-specific pos tags. this dataset was converted into dependencies first by @xcite using a custom scheme, and later to an early version of ud by @xcite . the current ud htb, using an older version of ud v2 guidelines which became invalid in 2018, is described in @xcite , and is used below for evaluation using legacy tokenization. an updated version of this dataset released in this paper is based on the 2018 version. this work represents the second treebank ever produced for modern hebrew, and the first crossdomain nlp evaluation of dependency parsing for the language. we briefly survey the state-of-the-art in ud hebrew nlp in section 4. ## in-domain and cross-corpus nlp in this section we present a series of experiments on cross-corpus segmentation, tagging, lemmatization and parsing, which is made possible for the first time by the release of non-newswire treebank data. several systems exist for ud parsing of hebrew: in this section we compare two popular neural off the shelf, end-to-end ud systems: stanza @xcite and @xcite , the latter of which is the sota for hebrew parsing; and the current sota system for non-concatenative hebrew segmentation and tagging @xcite , which is the only one using a pre-trained hebrew transformer introduced in the same paper, aleph-bert, but which does not cover ud parsing based on the paper. 5 we release and describe our own system for these tasks, called hebpipe, 6 below, which is a pipeline system building on the best existing systems with some incremental improvements. segmentation all previous work on ud hebrew parsing has been limited by the use of nonconcatenative tokenization schemes. however with the conversion to the concatenative tokenization described in section 3.2, we are able to take advantage of high accuracy characterwise segmentation approaches using binary classification (each character either begins a new segment or not). the previous characterwise sota system, rftokenizer @xcite is based on lexical features, such as looking up possible pos tags of mwt substrings around a segmentation point, and an xgboost classifier, and has been applied to other morphologically rich languages, such as arabic and coptic @xcite ; however it does not utilize pretrained transformers, which do not generally encode character level information (aleph-bert is word-piece based). @xcite also mention that it is counterintuitive to use a word level model as input to character level tasks, but note that despite this their approach still achieves a high segmentation score. to improve the sota in segmentation, we combine the characterwise rftokenizer classifier with the transformer from @xcite operating at the mwt word level. we train an lstm using alephbert to predict whether each word contains prefixes requiring segmentation, suffixes, or both, thereby targeting an mwt-level property. we then feed these predictions to rftokenizer as a feature, an approach not previously applied to the problem to the best of our knowledge. to avoid overfitting, we reserve half of the training data and the development set for the alephbert lstm training, and the remaining half for the xgboost classifier, which is fed the features described in @xcite as well as the alephbert predictions (code attached to this paper). tagging for pos tagging and features we employ a simple lstm sequence tagger using the same pre-trained alephbert embeddings. we observe in development data that some tokens are ambiguous regarding part of speech depending on whether they are prefixes or suffixes: for example, the letter vav ⟨w⟩ can stand for the word 'and', pronounced [ve] and tagged cconj, at the beginning of a mwt, or the word 'him/his', a pronoun pronounced [o] at the end of a mwt. if tokens are tagged as a naive sequence, these are occasionally mistagged in ambiguous contexts, for example: if the first two tokens are spelled together ‫,)ביתו(‬ then the example is a predicative verbless sentence ('his house is a palace'), and the letter 'w' is a pronoun (pron). however if the last two tokens are spelled together ( ‫,)וארמו‬ then 'w' is a coordinating conjunction (cconj), meaning 'and', and the translation would be 'a house and a palace'. other errors can occur similarly, for example between an initial article ha-'the' spelled the same as the suffix feminine possessive -a 'her'. to make the model aware of such cases, we concatenate a dense embedding using bies notation to each token vector, indicating whether it begins a mwt (b), ends one (e), is inside one (i) or represents a single-token word (s). using concatenative tokenization, we observe a degradation of -0.32% in end-to-end tagging accuracy on htb by ablating this feature, which amounts to an error reduction of ∼12%. although this improvement is minor, it is not computationally complex, and we are hopeful that it could be a useful approach for other mrls, such as arabic. tagger predictions are also fed into stanza's lemmatizer to produce our lemmatization. parsing for dependency parsing we rely on the prevalent biaffine approach proposed by @xcite , using the implementation in diaparser @xcite , which adds intermediate transformer layer inputs to the transition classifier. we again rely on alephbert to represent input embeddings. we use the default hyperparameters for all components described above and do not perform hyperparameter optimization (see appendix for details). ## error analysis the discrepancy in nlp quality across corpora for different tasks, and especially lower ood scores below the segmentation level, leads us to suspect that while segmentation in the two corpora is largely mutually consistent (after conversion of the tokenization scheme), other annotation layers may have some issues. such issues could also explain why joint training does not outperform in-domain scores -improvements from having more training data could be cancelled out by inconsistencies. in order to better understand the most common errors, as the table shows, a substantial portion of errors is due to noun vs. propn confusions, which is not unusual in general (see @xcite ) but suspicious given that directions are flipped: testing on htb we see over-prediction for dependency relations, most confusions are due to obl vs. nmod, which are annotated consistently in the data and correspond to verbal and nominal prepositional modifiers, i.e. the well known challenge of pp attachment (see @xcite . manual inspection confirms that these are less worrying in terms of corpus compatibility. confusions involving the compound label are mainly due to errors in htb, which include common items such as brit ha-mo'atsot 'soviet union' (occasionally labeled nmod instead of compound), and often carry an additional annota-tion convuncertainlabel in the original data, indicating that this is an issue with the original conversion of the data to ud. we plan to address these and other inconsistencies in future releases of the data. ## conclusion in this paper we make available, describe and evaluate annotation quality in the first non-newswire treebank for hebrew, complementing the existing newswire data which is by now over 30 years old. we propose and implement a new, simpler tokenization scheme for hebrew, and release a revised version of the existing ud hebrew newswire corpus which follows the new scheme. our evaluation of nlp systems on the new tokenization indicate that it is somewhat easier for a range of tokenizers, at the cost of slightly lower scores for downstream morphosyntactic annotations. we also evaluate agreement on the complete ud annotation pipeline for hebrew for over 12k doubly annotated tokens, and build a novel infrastructure for error detection based on grew. as part of our evaluation of nlp quality on the new resources, we compared several off the shelf systems, whose approaches we combine and incrementally improve to produce a new system, hebpipe, which is released with this paper. our system achieves new sota results on most ud nlp tasks for the older htb dataset, including segmentation, lemmatization and dependency parsing, evaluated on both the new and older tokenization schemes. we also reported on the first cross-corpus parsing experiment results, which indicated that segmentation and tagging using our system generalize well and substantially better than previous systems, while full dependency parsing is still less reliable out of domain. error analysis indicates that some of the issues are caused by remaining compatibility problems across the corpora, which we plan to address in the future. in other future work, we are in the process of annotating data from further sources, including written language from blogs, social media and government websites, as well as spoken data from parliament proceedings and tv shows. to improve annotation, we hope to extend grewv to annotatormediated autocorrection of validation errors including choosing between multiple correction options, as well as machine-guided refinement of annotation guidelines, and guiding source selection for collecting new material. urls for iahltwiki, the revised ud htb and complete code for the nlp tools from section 4 will be included after review of this paper.