Multi-View Cross-Lingual Structured Prediction with Minimum Supervision (2021)

Abstract: In structured prediction problems, cross-lingual transfer learning is an efficient way to train quality models for low-resource languages, and further improvement can be obtained by learning from multiple source languages. However, not all source models are created equal and some may hurt performance on the target language. Previous work has explored the similarity between source and target sentences as an approximate measure of strength for different source models. In this paper, we propose a multi-view framework, by leveraging a small number of labeled target sentences, to effectively combine multiple source models into an aggregated source view at different granularity levels (language, sentence, or sub-structure), and transfer it to a target view based on a task-specific model. By encouraging the two views to interact with each other, our framework can dynamically adjust the confidence level of each source model and improve the performance of both views during training. Experiments for three structured prediction tasks on sixteen data sets show that our framework achieves significant improvement over all existing approaches, including these with access to additional source language data.

## introduction structured prediction is the task of mapping input sentences to structured outputs. it is a fundamental task in natural language processing and has many applications, i.e., sequence labeling @xcite @xcite , dependency parsing @xcite @xcite and semantic role labeling @xcite @xcite . to achieve strong performance, structured prediction models mostly require manually labeled data that are costly to obtain in general. cross-lingual transfer learning @xcite @xcite @xcite recently attracted attention for tackling that problem, by transferring the knowledge from high-resource languages to low-resource ones. existing works can be categorized into two types: single-source transfer and multi-source transfer. the former is limited to transferring knowledge from one source language and generally results in inferior performance than the latter @xcite , especially when the target language is similar to multiple source language over various characteristics, i.e., domain, word order, capitalization, and script style. however, in practice, we are more likely to encounter the situation where some source languages are not as similar to the target language and may lead to worse performance @xcite ) (we provide an example in the appendix a). to tackle this challenging problem, most of the previous works do majority voting @xcite and truth inference on hard predictions of multiple sources @xcite . to better incorporate target language information, some recent works train a new model on the target unlabeled data with hard/soft predictions from multiple source models, such as mixture-of-experts model @xcite and knowledge distillation (kd) @xcite , and assign weights to multiple sources based on language similarity. however, these similaritybased approaches are heuristic-based, and cannot well learn the confidence level of multiple source models. in this paper, we propose to leverage a small number of labeled target data to selectively transfer the knowledge from multiple source models. in many real applications, we are generally easy to obtain a small number of target labeled data. these small amounts of data can reflect the diverse strength and weakness of different source models. concretely, the (small-size) labeled data can be utilized to learn the aggregation strategy of multiple source models or train a new task-specific model in the target language. both the aggregation model and target task-specific model can map the inputs to the structured outputs but there exists a tradeoff. the aggregation model generally has strong cross-lingual ability since source models are firstly well trained foot_0 , but has lower flexibility since source models are usually frozen. instead, the target taskspecific model tends to be more flexible and has strong capacity but has poor performance since the model is easily over-fitted on the small training sample. inspired by previous work on multi/cross-view learning @xcite @xcite , we regard the aggregation model (aggregated source view) and the target taskspecific model (target view) as two views since they both can map the input sentence to structured outputs. we propose a novel multi-view framework to achieve a good trade-off between the two views. to capture the diverse strength and weakness of multiple source models, we propose three approaches to obtain the aggregated source view from language/sentence/sub-structure level in a coarse-to-fine manner. by encouraging two views to influence each other, the proposed framework can dynamically learn the confidence level of multiple source models in three coarse-to-fine granularity and make the best use of the small number of labeled data, and make both views improved during training. benefited from the multi-view framework, our proposed approaches can leverage plenty of target unlabeled data to capture the useful target language information @xcite . the contributions of this work are: we extensively experiment on three structured prediction tasks, which are named entity recognition (ner), part-of-speech tagging (pos), and dependency parsing. our proposed approaches outperform several state-of-the-art approaches. ## methodology the left part of figure 1 depicts the proposed general framework. our framework contains two views, a target view which is a target structured predictor, and an aggregated source view based on multiple pre-trained source models. both views can map the input sentences to the structured outputs and have diverse statistical properties, and thus can provide complementary information to each other (learned by the consensus component). ## experiments we experiment on three structured prediction tasks: ner, pos tagging, and dependency parsing. following previous work @xcite , we conduct the experiments in a leaveone-out setting in which we hold out one language as the target language and the others as the source languages. to simulate the low-resources scenario, for each training set in a specific target language, we randomly select fifty sentences foot_3 with the gold annotations and discard the annotations of the remaining sentences to construct the training set. we randomly select six languages from universal dependencies treebanks (v2.2) foot_4 for dependency parsing and pos tagging tasks. we use the datasets from conll 2002 and conll 2003 shared tasks @xcite for ner tasks. we utilize the base cased multilingual bert @xcite as base model for all approaches. we run each approach five times and report the averaged accuracy for pos tagging, f1-score for ner, and unlabelled attachment score (uas) and labeled attachment score (las) for dependency parsing. more details can be found in the appendix b.1. ## conclusion we propose a novel multi-view framework to selectively transfer knowledge from multiple sources by utilizing a small amount of labeled dataset. experimental results show that our approaches achieve state-of-the-art performances on all tasks. moreover, even compared to approaches with extra resources like source language data, our substructure-level approach still shows significant improvements.