An Empirical Exploration of Local Ordering Pre-training for Structured Prediction (2020)

Abstract: Recently, pre-training contextualized encoders with language model (LM) objectives has been shown an effective semi-supervised method for structured prediction. In this work, we empirically explore an alternative pre-training method for contextualized encoders. Instead of predicting words in LMs, we “mask out” and predict word order information, with a local ordering strategy and word-selecting objectives. With evaluations on three typical structured prediction tasks (dependency parsing, POS tagging, and NER) over four languages (English, Finnish, Czech, and Italian), we show that our method is consistently beneficial. We further conduct detailed error analysis, including one that examines a specific type of parsing error where the head is misidentified. The results show that pre-trained contextual encoders can bring improvements in a structured way, suggesting that they may be able to capture higher-order patterns and feature combinations from unlabeled data.

## introduction recently, pre-trained contextualized encoders @xcite @xcite have been shown to be beneficial for nlp tasks, including structured prediction @xcite . most of the pre-training objectives are based on variants of language models (lm), that is, the model is trained to predict lexical items with partial inputs. masked language model (masklm) is a typical example, popularized by bert @xcite , which masks out lexical tokens in the input sequences and predicts their identities. since natural sentences contain not only lexical tokens but also their linearized word orders, it is a natural question if we can perform pre-training by "masking out" and recovering word order information. word order is an important method of grammatical encoding @xcite , and can play an important role in predicting basic sentence structures @xcite @xcite . recently, @xcite pre-train an explicit word reordering model and show that its contextualized representations improve dependency parsing. in this work, we explore a local ordering pretraining strategy with word-selection objectives. instead of completely discarding original word order information, we segment the input sentence into local bags of words and keep the ordering of these bags. inside each bag, we discard all the local word orders and train the model to recover them. furthermore, we simplify the training objectives: instead of training explicit word linearizers which require extra unidirectional decoders, we only ask the model to select original neighboring words. this scheme simplifies the pre-training procedure and enhances the encoder since it can take information from the whole sentence. a similar idea is explored in structbert @xcite , which adopts a word structural objective by shuffling and re-predicting randomly selected subsets of trigrams. our method is different in that we make local bags of words instead of shuffling and we adopt simpler and cheaper wordselection objectives. moreover, we focus on empirical experiments and error analysis on structured prediction tasks. we evaluate on three structured prediction tasks (dependency parsing, part-of-speech (pos) tagging, and named entity recognition (ner)) over four languages @xcite . the highlights of our findings are: ## local ordering pre-training word reordering or linearization itself is an interesting task, aiming to arrange a bag of words into a natural sentence @xcite @xcite . @xcite show that representations from an explicit reordering model can benefit dependency parsing. however, there may be two issues with an explicit reordering model for pre-training. firstly, the input is a bag of words without any positional information. this could discard too much information, leading to relatively large discrepancies between pre-training and fine-tuning. moreover, training explicit reordering models requires unidirectional decoders, which are only aware of contexts from one direction and cannot make full use of the bidirectional information at one time. to mitigate these issues, we explore a local ordering pre-training strategy with word-selection objectives. inspired by masklm, where only some of the tokens are masked out, we "mask out" par-tial ordering information by segmenting the input sentence into multiple local bags of words, and only discarding word orders inside each bag ( §2.1). moreover, we adopt simpler training objectives of selecting original neighboring words, which avoids the need of unidirectional decoders and focuses the pre-training on the encoder ( §2.2). ## conclusion in this work, we empirically explore an alternative pre-training strategy for contextualized encoders. instead of training variants of language models, we adopt a local word ordering strategy, which segments the inputs into local bags of words, together with order-based word-selection objectives. evaluated on typical structured prediction tasks, we show the effectiveness of this method. with further analysis on one typical structured task, we show that pre-trained encoders can bring improvements in a structured way. we hope this empirical work can shed some light and inspire future work on exploring how pre-trained contextualized encoders capture language structures. the related statistics. we adopt word-level inputs mainly to follow the conventions of the target tasks explored in this work and to compare with baseline models without pre-trained encoders. it will be interesting to explore other input schemes (such as sub-words as in bert) in future work, which is orthogonal to the main focus of this work. target tasks we explore three typical structured prediction tasks: dependency parsing, part-ofspeech (pos) tagging and named entity recognition (ner). for the tagging and parsing tasks, we utilize annotations from udv2.4 9 . specifically, we use the following treebanks: "english-ewt", "finnish-tdt", "czech-pdt" and "italian-isdt". for ner, we utilize various datasets, including conll03 10 @xcite for english, digitoday 11 (ruokolainen et al., 2019) for finnish, czech named entity corpus 12 @xcite for czech and evalita 2009 13 @xcite for italian. we only adopt simple settings for the ner tasks, specifically, ignoring nested annotations for finnish ner and considering supertypes for czech ner. for it-ner, we take the first 10k sentences as training set and the rest 1.2k as development set.