Improving Low-Resource Named Entity Recognition using Joint Sentence and Token Labeling (2020)

Abstract: Exploiting sentence-level labels, which are easy to obtain, is one of the plausible methods to improve low-resource named entity recognition (NER), where token-level labels are costly to annotate. Current models for jointly learning sentence and token labeling are limited to binary classification. We present a joint model that supports multi-class classification and introduce a simple variant of self-attention that allows the model to learn scaling factors. Our model produces 3.78%, 4.20%, 2.08% improvements in F1 over the BiLSTM-CRF baseline on e-commerce product titles in three different low-resource languages: Vietnamese, Thai, and Indonesian, respectively.

## introduction neural named entity recognition (ner) has become a mainstream approach due to its superior performance @xcite @xcite @xcite . however, neural ner typically requires a large amount of manually labeled training data, which are not always available in low-resource languages. training neural ner with limited labeled data can be very challenging. in this paper, we consider bridging multi-task learning (mtl) @xcite and pretraining @xcite to leverage training signals of an auxiliary task that has a sufficiently large number of labeled data. researchers have investigated a wide variety of auxiliary tasks and resources to boost the performance of neural ner, e.g., training coarsegrained ner @xcite , fine-tuning bilingual word embeddings @xcite , applying language models @xcite , integrating part-of-speech (pos) tagging @xcite , using cross-lingual knowledge @xcite , and learning paraphrases @xcite . category: health_beauty title: combo gôm xịt tóc tigi bed head label: o b-product i-product e-product b-brand i-brand e-brand translation: combo hairspray tigi bed head ''… tigi bed head hairspray combo …'' category: electronics title: ốp lưng silicon dẻo hàn quốc label: b-product e-product s-material s-pattern o o translation: case silicon flexible korea ''… korean flexible silicon case …'' figure 1: examples of product titles with ner annotation in vietnamese. product categories are provided by sellers and can be used as sentence-level labels. while most of the previous studies have exploited token-level information from auxiliary tasks, a few of them have tried to use sentence-level information @xcite . our work is closely related to the joint labeling framework in @xcite . however, they only focused on binary classification, while we attempt to handle multi-class classification on both sentence and token levels. in this work, we focus on improving lowresource ner by exploiting large data, only having sentence-level labels. figure 1 shows examples of product titles on an e-commerce website in vietnamese. while the product titles with ner annotation done by our annotators are limited, those with product categories (e.g., electronics) labeled by sellers are abundant, which can be used to train a sentence-level classifier. 1 a key challenge is to pass useful training signals from the sentence-level classification to the token-level ner. our contributions are as follows. we present the joint sentence and token labeling framework that enables multi-class classification equipped with a pre-training strategy ( §2.1). we show that the current attention mechanisms can produce suboptimal ## proposed method figure 2 shows the architecture of our joint sentence and token labeling model. our model is based on hard parameter sharing @xcite in which the hidden layers are shared between two tasks. the task-specific layers include a conditional random field (crf) layer for ner and a linear layer for sentence classification. 2 unlike the standard mtl, which trains multiple tasks at once and expects the model to perform well on all tasks @xcite , the goal of our work is to improve the performance of the main task (ner) using the auxiliary task (sentence classification) for creating pre-trained representations and as a regularizer. ## conclusion we have shown that the proposed joint sentence and token labeling model is remarkably effective for low-resource ner in three different languages: vietnamese, thai, and indonesian. our model supports multi-class classification where the sentence and token labels can be weakly related, which indicates the potential of our model for many other real-world applications. using a larger amount of general domain texts to build pre-trained representations @xcite @xcite can complement with our model and is one of the directions that we plan to take in future work.