Risk Minimization for Zero-shot Sequence Labeling (2021)

Abstract: Zero-shot sequence labeling aims to build a sequence labeler without human-annotated datasets. One straightforward approach is utilizing existing systems (source models) to generate pseudo-labeled datasets and train a target sequence labeler accordingly. However, due to the gap between the source and the target languages/domains, this approach may fail to recover the true labels. In this paper, we propose a novel unified framework for zero-shot sequence labeling with minimum risk training and design a new decomposable risk function that models the relations between the predicted labels from the source models and the true labels. By making the risk function trainable, we draw a connection between minimum risk training and latent variable model learning. We propose a unified learning algorithm based on the expectation maximization (EM) algorithm. We extensively evaluate our proposed approaches on cross-lingual/domain sequence labeling tasks over twenty-one datasets. The results show that our approaches outperform state-of-the-art baseline systems.

## introduction sequence labeling is an important task in natural language processing. it has many applications such as part-of-speech tagging (pos) @xcite and named entity recognition (ner) @xcite @xcite @xcite . approaches to sequence labeling are mostly based on supervised learning, which relies heavily on labeled data. however, the labeled data is generally expensive and hard to obtain (for lowresource languages/domains), which means that these supervised learning approaches fail in many cases. learning knowledge from imperfect predictions from other rich-resource sources (such as crosslingual, cross-domain transfer) @xcite @xcite is a feasible and efficient way to tackle the low-resource problem. it transfers knowledge from rich-resource languages/domains to low-resource ones. one typical approach to this problem is utilizing existing systems to provide predicted results for the zero-shot datasets. however, due to the gap between the source and the target languages/domains, this approach may fail to recover the true labels. several previous approaches try to alleviate this problem by relying heavily on cross-lingual information (e.g., parallel text @xcite ), labeled data in source languages @xcite , and prior domain knowledge @xcite for different kinds of zero-shot scenarios. however, these approaches are designed to be specific, and might not be generalizable to other kinds of settings where the required resources are expensive to obtain or not available due to data privacy @xcite . instead, we want a learning framework that can address the zero-shot learning problem in a unified perspective. in this work, we consider two widely explored settings in which we have access to: 1) the imperfect hard predictions @xcite ; 2) the imperfect soft predictions @xcite , produced by one or more source models on target unlabeled data , and propose two novel approaches. we start by introducing a novel approach based on the minimum risk training framework. we design a new decomposable risk function parameterized by a fixed matrix that models the relations between the noisy predictions from the source models and the true labels. we then make the matrix trainable, which leads to further expressiveness and connects minimum risk training to learning latent variable models. we propose a learning algorithm based on the em algorithm, which alternates between updating a posterior distribution and optimizing model parameters. to empirically evaluate our proposed approaches, we extensively conduct experiments on four sequence labeling tasks of twenty-one datasets. our two proposed approaches, especially the latent variable model, outperform several strong baselines. ## experiments we use the multilingual bert (mbert) as our word representations foot_1 as the sentence encoder. fol-lowing @xcite , the source model are previously trained on its corresponding training data. we use the bio scheme for conll and ontonotes ner tasks and aspect extraction. we run each model three times and report the average accuracy for the pos tagging task and f1-score for the other tasks. ## analysis we conduct the analysis on the multi-source setting with soft predictions from sources for its better performance. big data performance we experiment with our two models and the kd-re baseline on big target training data on the pos tagging task. we randomly select 100000 sentences (without labels) for the wikipedia-003 section of the ca language on the conll 2017 shared task @xcite . we randomly select 1000, 10000, and 100000 sentences to train these three approaches, evaluate on the ud test set for each of the three languages respectively, and show the results in figure 2 . it shows that our latent variable model outperforms the other two approaches over all the settings. though kd outperform mrt with less than 10000 sentences, but mrt has comparable result with enough unlabeled data. besides, with more unlabeled data used for training, each model further gains a considerable boost. comparison to direct gradient optimization our two proposed approaches can also be optimized directly by any gradient-based approach, such as the adamw optimizer @xcite . we use the two proposed approaches to compare the performance of the direct gradientbased training strategy and the em algorithm. we conduct the experiments on our two proposed approaches on conll ner task on the multi-source setting. we show the results in ## conclusion in this paper, we propose two approaches to the zero-shot sequence labeling problem. our mrt approach uses a fixed matrix to model the relations between the predicted labels from the source models and the true labels. our lvm approach uses trainable matrices to model these label relations. we extensively verify the effectiveness of our approaches on both single-source and multisource transfer over both cross-lingual and crossdomain sequence labeling problems. experiments show that mrt and lvm generally bring significant improvements over previous state-of-the-art approaches on twenty-one datasets.