Low Resource Sequence Tagging using Sentence Reconstruction (2020)

Abstract: This work revisits the task of training sequence tagging models with limited resources using transfer learning. We investigate several proposed approaches introduced in recent works and suggest a new loss that relies on sentence reconstruction from normalized embeddings. Specifically, our method demonstrates how by adding a decoding layer for sentence reconstruction, we can improve the performance of various baselines. We show improved results on the CoNLL02 NER and UD 1.2 POS datasets and demonstrate the power of the method for transfer learning with low-resources achieving 0.6 F1 score in Dutch using only one sample from it.

## introduction the increased popularity of deep learning led to a giant leap in natural language processing (nlp). tasks such as neural machine translation @xcite , sentiment analysis @xcite and question answering @xcite achieved impressive results. a major limitation of deep learning is the need for huge amounts of training data. thus, when dealing with low resource datasets, transfer learning is a common solution. a popular approach in nlp is training a language model for getting a good context-based word representation. language models such as bert @xcite , roberta @xcite , elmo @xcite , and xlnet @xcite that are trained on very large corpora, are used by the community for different nlp tasks. this "transfer-learning" across tasks within the same language relies on fine-tuning a language model for a specific task @xcite . this work focuses on transfer learning between different languages. some approaches have been suggested for it. @xcite have proposed using joint training with a large dataset as a source and a small dataset as a target. @xcite have shown how by aligning sentence representations using an adversarial loss, they were able to transfer knowledge between two languages. contribution. this work analyzes the contribution of various techniques proposed for transfer learning between languages for the task of sequence tagging. in particular, we evaluate joint training and adversarial learning. moreover, we propose a novel regularization technique, namely, we add a reconstruction loss with ℓ 2 normalization. we show that the addition of this loss improves the performance of various sequence tagging tasks when doing transfer learning. our strategy shows promising results for training models without being language-specific, which saves expensive labeling time. an important characteristic of our technique is its ability to provide good tagging in "few-shot learning" @xcite . we achieve this result by adding to the small dataset, a larger corpus corresponding to another language. our proposed loss improves the transfer of information and thus the tagging accuracy. we demonstrate our approach on the conll02/03 and the universal dependency (ud) 1.2 datasets. ## related work solving sequence tagging tasks, such as named entity recognition (ner) or part of speech (pos), using statistical methods has been studied for more than two decades. early solutions used hidden markov models (hmms) @xcite , support-vector machines (svms) @xcite and conditional random fields (crf, @xcite , we focus on a more . this can be replaced by context-aware embeddings. the lstms are language-specific and are fed by the relevant embeddings per sample. we normalize the sentence representation for all sentences and the word representation as well. ## our approach this section describes our sentence reconstruction approach for improving low resource sequence tagging tasks. many successful sequence tagging network models are composed of an encoder-decoder structure. we suggest adding to them a new decoder branch comprised of a fully convolutional network (fcn) and an ℓ 2 loss term for reconstructing the word embeddings of the input sentence. @xcite , using sentence reconstruction (l2), using weight sharing based transfer learning (tl), using the adversarial loss and combining them all together. ## experiments we follow the experiments of @xcite to evaluate our approach for transfer learning between languages. we compare our proposed regularization to joint training and the adversarial loss. we start by evaluating the impact of each strategy alone, and then gradually combine the losses to each other. our source-target pairs are built of english and a selected target language (spanish, dutch or romanian). in ner, we test both directions of transfer learning, i.e english to spanish and spanish to english. in pos, english is always the source language. we focus on using word embeddings that are aligned across different languages, specifically "muse" @xcite . our motivation for choosing it is to leverage the word alignment, which makes the impact of the sentence alignment clearer. loss analysis. for understanding the impact of our approach, we test it with and without the other techniques for transfer learning between languages. we also compare to each of them being applied separately. ## conclusion this work demonstrates the power of sentence reconstruction for transferring knowledge from a rich dataset to a sparse one. it achieves competitive results with a relatively simple baseline. we also show its strength in few-shot and one-shot learning. we believe that using the proposed sentence ℓ 2 reconstruction may contribute as an auxiliary loss for other tasks. also, we have demonstrated our model with muse, since it provides word alignment across languages. yet, our approach can be applied also with other more recent language models that have stronger context-based embeddings.