M eta TS : Meta Teacher-Student Network for Multilingual Sequence Labeling with Minimal Supervision (2021)

Abstract: Sequence labeling aims to predict a fine-grained sequence of labels for the text. However, such formulation hinders the effectiveness of supervised methods due to the lack of token-level annotated data. This is exacerbated when we meet a diverse range of languages. In this work, we explore multilingual sequence labeling with minimal supervision using a single unified model for multiple languages. Specifically, we propose a Meta Teacher-Student (MetaTS) Network, a novel meta learning method to alleviate data scarcity by leveraging large multilingual unlabeled data. Prior teacher-student frameworks of self-training rely on rigid teaching strategies, which may hardly produce high-quality pseudo-labels for consecutive and interdependent tokens. On the contrary, MetaTS allows the teacher to dynamically adapt its pseudo-annotation strategies by the student’s feedback on the generated pseudo-labeled data of each language and thus mitigate error propagation from noisy pseudo-labels. Extensive experiments on both public and real-world multilingual sequence labeling datasets empirically demonstrate the effectiveness of MetaTS.

## introduction sequence labeling or tagging is the task of detecting the boundary of all occurring entity mentions from unstructured text and classifying them into predefined types, such as named entity recognition (ner) @xcite , aspect-based sentiment analysis (absa) @xcite , etc. an entity mention should be a single word or a sequence of words that contain key information, such as a person, location, or institution. in the e-commerce search domain, we need to recognize product attributes from short queries, such as product type, brand, size, to better understand users' preferences and intents. [usb] 2. pseudo-labels (choice #1) [mixer with usb] 3. pseudo-labels (choice #2) [mixer] [with usb] despite recent advances in deep learning models for sequence labeling @xcite , they still rely on massive labeled data. nonetheless, the sequence labeling tasks usually lie in the low-data regime due to costly and laborintensive human annotation for token-level labels, especially for a variety of languages @xcite , as search engines or social networks usually cover a diverse range of countries and locales using different languages. in this paper, we attempt to explore a unified multilingual sequence labeling model with minimal supervision, where each language only has limited labeled data. the emergence of multilingual pre-trained language models (mplms) such as mbert @xcite and xlm @xcite have enabled breakthroughs on various multilingual nlp tasks. however, it has been recently noted that mplms are not data-efficient and typically require sufficient fine-tuning data for superior performance on downstream tasks. to mitigate data scarcity, semi-supervised learning (ssl) @xcite has been a promising paradigm that allows us to take advantage of large-scale unlabeled multilingual data. selftraining @xcite stands out among the ssl approaches, in which a teacher model produces pseudo-labels for unlabeled examples, and a student model learns from these examples with generated pseudo-labels. self-training has shown promising results in instance-level tasks, e.g., image classification @xcite . however, a major research challenge that dictates the success of self-training is the well-known confirmation bias problem @xcite , which results in progressive drifts on the noisy pseudo-labeled data provided by the teacher. this problem is more pronounced in sequence labeling @xcite , as complicated dependencies between tokens pose tremendous challenges towards the rigid teaching strategies, e.g., the fixed teacher @xcite or the periodically synchronizing teacher @xcite , to generate accurate pseudo-labels for consecutive and interdependent tokens. to encourage the teacher to generate better pseudo-labels for multilingual sequence labeling, we propose a novel meta teacher-student (metats) network, where the teacher learns dynamically and continuously from the student's feedback to adapt its teaching strategies, i.e., the pseudoannotation choices. concretely, given a language for each step, the student network will be updated based on the pseudo-labeled data produced by the teacher. to quantitatively measure how well the teacher generates these pseudo-labels at the current step, we will evaluate the difference between the student performance after the update using the pseudo-labeled data of the language and that before the update. the improvement or degradation of the student performance can be used as the feedback to meta-optimize the teacher network (a.k.a. learning to learn @xcite ). consider an example in overall, our contributions can be summarized as follows: (1) we explore a unified and effective multilingual sequence labeling setting with minimal supervision required; @xcite we propose a novel metats framework to alleviate the confirmation bias problem via learning from the student's feedback to generate better fine-grained pseudo-labels; ## experiment 4.1 datasets we consider the following three multilingual sequence labeling datasets for experiments, of which the statistics of the datasets are shown in ## conclusion the effectiveness of supervised methods for lowresource multilingual sequence labeling is limited due to data scarcity. to tackle this challenge, we propose a novel metats method to enhance the teacher-student framework of self-training, which leverages the student's feedback on multilingual token-level pseudo-labels to adjust the teacher's pseudo-annotation strategies. extensive evaluations on both the public academic benchmarks and the large-scale industrial dataset quantitatively and qualitatively demonstrate the effectiveness of metats. in the future, the proposed metats method can potentially be applied to multilingual natural language understanding (xlu) tasks @xcite and be generalized to multi-task learning @xcite problems.