Embeddings of Label Components for Sequence Labeling: A Case Study of Fine-grained Named Entity Recognition (2020)

Abstract: In general, the labels used in sequence labeling consist of different types of elements. For example, IOB-format entity labels, such as B-Person and I-Person, can be decomposed into span (B and I) and type information (Person). However, while most sequence labeling models do not consider such label components, the shared components across labels, such as Person, can be beneficial for label prediction. In this work, we propose to integrate label component information as embeddings into models. Through experiments on English and Japanese fine-grained named entity recognition, we demonstrate that the proposed method improves performance, especially for instances with low-frequency labels.

## introduction sequence labeling is a problem in which a label is assigned to each word in an input sentence. in many label sets, each label consists of different types of elements. for example, iob-format entity labels @xcite , such as b-person and i-location, can be decomposed into span (e.g., b, i and o) and type information (e.g., person and location). also, morphological feature tags @xcite , such as @xmath0, can be decomposed into gender, number and other information. general sequence labeling models @xcite @xcite , however, do not consider such components. specifically, the probability that each word is assigned a label is computed on the basis of the inner product between word representation and label embedding (see equation 2 in section 2.1). here, the label embedding is associated with each label and independently trained without considering its components. this means that labels are treated as mutually exclusive. in fact, labels often share some components. consider the labels b-person and i-person. they share the component person, and injecting such component information into the label embeddings can improve the generalization performance. motivated by this, we propose a method that shares and learns the embeddings of label components (see details in section 2.2). specifically, we first decompose each label into its components. we then assign an embedding to each component and summarize the embeddings of all the components into one as a label embedding used in a model. this component-level operation enables the model to share information on the common components across label embeddings. to investigate the effectiveness of our method, we take the task of fine-grained named entity recognition (ner) as a case study. typically, in this task, a large number of entity-type labels are predefined in a hierarchical structure, and intermediate type labels can be used as label components, as well as leaf type labels and b/i-labels. in this sense, the fine-grained ner can be seen as a good example of the potential applications of the proposed method. furthermore, some entity labels occur more frequently than others. an interesting question is whether our method of label component sharing exhibits an improvement in recognizing entities of infrequent labels. in our experiments, we use the english and japanese ner corpora with the extended named entity hierarchy @xcite including 200 entity tags. to sum up, our main contributions are as follows: (i) we propose a method that shares and learns label component embeddings, and (ii) through experiments on english and japanese fine-grained ner, we demonstrate that the proposed method achieves better performance than a standard sequence labeling model, especially for instances with low-frequency labels. 2 methods ## related work sequence labeling has been widely studied and applied to many tasks, such as chunking @xcite , ner @xcite and semantic role labeling (srl) @xcite . in english fine-grained entity recognition, ling and weld (2012) created a standard fine-grained entity typing dataset with multi-class, multi-label annotations. @xcite developed a dataset for nested ner dataset. these datasets independently handle each label without considering label components. in japanese ner, @xcite combined word and character information to improve performance. @xcite reported that dictionary information improves the performance of finegrained ner. their methods do not consider label components and are orthogonal to our method. some existing studies take shared components (or information) across labels into account. in entity typing, @xcite and @xcite proposed to calculate entity label embeddings by considering a label hierarchical structure. while their method is limited to only a hierarchical structure, our method can be applied to any set of components and can be regarded as a general form of their method. in multi-label classification, @xcite assumed that the labels cooccurring in many instances are correlated with each other and share some common features, and proposed a method that learns a feature (label embedding) space where such co-occurring labels are close to each other. the work of @xcite is the closest to ours in terms of decomposing the features of labels. they regard an original label comprising a mixture of components as a set of multiple labels and made models that are able to exploit the multiple components to effectively learn in the srl task. ## conclusion we proposed a method that shares and learns the embeddings of label components. through experiments on english and japanese fine-grained ner, we demonstrated that our proposed method improves the performance, especially for instances with low-frequency labels. for future work, we envision to apply our method to other tasks and datasets and investigate the effectiveness. also, we plan to extend the simple label embedding calculation methods to more sophisticated ones.