MECT : M ulti-Metadata Embedding based Cross-Transformer for C hinese Named Entity Recognition (2021)

Abstract: Recently, word enhancement has become very popular for Chinese Named Entity Recognition (NER), reducing segmentation errors and increasing the semantic and boundary information of Chinese words. However, these methods tend to ignore the information of the Chinese character structure after integrating the lexical information. Chinese characters have evolved from pictographs since ancient times, and their structure often reflects more information about the characters. This paper presents a novel Multi-metadata Embedding based Cross-Transformer (MECT) to improve the performance of Chinese NER by fusing the structural information of Chinese characters. Specifically, we use multi-metadata embedding in a two-stream Transformer to integrate Chinese character features with the radical-level embedding. With the structural characteristics of Chinese characters, MECT can better capture the semantic information of Chinese characters for NER. The experimental results obtained on several well-known benchmarking datasets demonstrate the merits and superiority of the proposed MECT method.

## introduction named entity recognition (ner) plays an essential role in structuring of unstructured text. it is a sequence tagging task that extracts named entities from unstructured text. common categories of ner include names of people, places, organizations, time, quantity, currency, and some proper nouns. ner is the basis for many natural language processing (nlp) tasks such as event extraction @xcite , question answering @xcite , information retrieval @xcite , knowledge graph construction @xcite , etc. compared with english, there is no space between chinese characters as word delimiters. chinese word segmentation is mostly distinguished by readers through the semantic information of sentences, posing many difficulties to chinese ner @xcite . besides, the task also has many other challenges, such as complex combinations, entity nesting, and indefinite length @xcite . in english, different words may have the same root or affix that better represents the word's semantics. for example, physiology, psychology, sociology, technology and zoology contain the same suffix, '-logy', which helps identify the entity of a subject name. besides, according to the information of english words, root or affixes often determine general meanings @xcite . the root, such as 'ophthalmo-' (ophthalmology), 'esophage-' (esophagus) and 'epithelio-' (epithelium), can help human or machine to better recognize professional nouns in medicine. therefore, even the stateof-the-art methods, such as bert @xcite and gpt @xcite , trained on large-scale datasets, adopt this delicate word segmentation method for performance boost. ## related work the key of the proposed mect method is to use the radical information of chinese characters to enhance the chinese ner model. so we focus on the mainstream information enhancement methods in the literature. there are two main types of chinese ner enhancement methods, including lexical information fusion and glyph-structural information fusion. ## background the proposed method is based on the flat-lattice transformer (flat) model. thus, we first briefly introduce flat that improves the encoder structure of transformer by adding word lattice information, including semantic and position boundary information. these word lattices are obtained through dictionary matching. figure 1 shows the input and output of flat. it uses the relative position encoding transformed by head and tail position to fit the word's boundary information. the relative position encoding, r ij , is calculated as follows: ## the proposed mect method to better integrate the information of chinese character components, we use chinese character structure as another metadata and design a two-stream form of multi-metadata embedding network. the architecture of the proposed network is shown in figure 2a . the proposed method is based on the encoder structure of transformer and the flat method, in which we integrate the meaning and boundary information of chinese words. the proposed two-stream model uses a cross-transformer module similar to the self-attention structure to fuse the information of chinese character components. in our method, we also use the multi-modal collaborative attention method that is widely used in vision-language tasks @xcite . the difference is that we add a randomly initialized attention matrix to calculate the attention bias for the two types of metadata embedding. ## experimental results in this section, we evaluate the proposed mect method on four datasets. to make the experimental results more reasonable, we also set up two additional working methods for assessing the performance of radicals in a two-stream model. we use the span method to calculate f1-score (f1), precision (p), and recall (r) as the evaluation metrics. ## conclusion this paper presented a novel two-stream network, namely mect, for chinese ner. the proposed method uses multi-metadata embedding that fuses the information of radicals, characters and words through a cross-transformer network. additionally, random attention was used for further performance boost. experimental results obtained on four benchmarks demonstrate that the radical information of chinese characters can effectively improve the performance for chinese ner. the proposed mect method with the radical stream increases the complexity of a model. in the future, we will consider how to integrate the characters, words and radical information of chinese characters with a more efficient way in two-stream or multi-stream networks to improve the performance of chinese ner and extend it to other nlp tasks.