MFE - NER : Multi-feature Fusion Embedding for C hinese Named Entity Recognition (2024)

Abstract: “In Chinese Named Entity Recognition, character substitution is a complicated linguistic phe-nomenon. Some Chinese characters are quite similar as they share the same components or havesimilar pronunciations. People replace characters in a named entity with similar characters togenerate a new collocation but refer to the same object. As a result, it always leads to unrecog-nizable or mislabeling errors in the NER task. In this paper, we propose a lightweight method,MFE-NER, which fuses glyph and phonetic features to help pre-trained language models handlethe character substitution problem in the NER task with limited extra cost. Basically, in the glyphdomain, we disassemble Chinese characters into Five-Stroke components to represent structurefeatures. In the phonetic domain, an improved phonetic system is proposed in our work, makingit reasonable to describe phonetic similarity among Chinese characters. Experiments demon-strate that our method performs especially well in detecting character substitutions while slightlyimproving the overall performance of Chinese NER.”

## introduction recently, pre-trained language models have been widely used in natural language processing (nlp) @xcite , constantly refreshing the benchmarks of various nlp tasks @xcite . by applying the transformer structure, semantic features can be extracted more accurately. however, in the named entity recognition (ner) area, tricky problems still exist. most significantly, the character substitution problem severely affects the performance of ner models. to make things worse, this problem has become even more common these years, especially in social media, which leads to the dissemination of fake news @xcite . due to the particularity of chinese characters, there are multiple ways to replace original chinese characters in a word. characters with similar meanings, shapes, or pronunciations can be selected for character substitution. a simple example shown in figure 1 is a chinese word, which represents a famous place in shanghai. here, all three characters in the word are substituted by other characters with similar glyphs or pronunciations. after substitution, this word looks more like the name of a person rather than a place. in practice, it is extremely hard for those pre-trained language models to tackle this problem @xcite . currently, the tasks for pre-training chinese language models are mainly focused on the semantic domain, neglecting glyph and phonetic features @xcite . however, most character substitution cases exist in glyph and phonetic domains. at the same time, hot topics on social media are changing rapidly, creating new expressions or substitutions for original words every day, while it is technically impossible for pre-trained models to include all possible collocations. models that only saw the original collocations before will naturally suffer from out-of-vocabulary (oov) problems. in this paper, we propose a lightweight method, multi-feature fusion embedding for chinese named entity recognition (mfe-ner), which fuses extra glyph and phonetic features to detect possible substitution forms of named entities in chinese. on top of using pre-trained models to represent the semantic feature, we choose a structure-based encoding method, known as 'five-strokes', to introduce glyph features. in the phonetic domain, we propose 'trans-pinyin', which combines 'pinyin', a unique phonetic system, with international standard phonetic symbols. experiments on three general datasets and our special-designed dataset with character substitutions have illustrated that with limited extra cost, mfe-ner can not only help ner models handle the character substitution problem but also enhance the overall performance of ner models, which makes it especially suitable to be used in current language environments. to summarize, our major contributions are: ## related work after the stage of statistical machine learning algorithms, named entity recognition has stepped into the era of deep neural networks. researchers started to use recurrent neural network (rnn) @xcite to recognize named entities in sentences based on character embedding and word embedding like word2vec @xcite , solving the feature engineering problems that traditional statistical methods have. however, traditional rnns may have the problem of gradient explosion and can not capture the long dependency well. in this case, the bidirectional long short term memory (bi-lstm) network @xcite was then applied in chinese named entity recognition, which became one of the baseline models in the chinese ner task. these years, large-scale pre-trained language models based on transformer @xcite ) have shown their superiority in natural language processing tasks. the self-attention mechanism can better capture the long-distance dependency in sentences, and the parallel design is suitable for mass computing. bidirectional encoder representations from transformers (bert) (kenton and toutanova, 2019) has achieved great success in many branches of nlp. in the chinese named entity recognition field, these pre-trained models have greatly improved the recognition performance @xcite . however, the situation is more complicated in real-language environments. pre-trained language models do not guarantee the robustness of ner models. researchers have started to introduce prior knowledge to improve the generalization of ner models. @xcite introduces knowledge graphs to help the model acquire the medical vocabulary explicitly. meanwhile, in order to relieve character substitution problems and enhance the robustness of ner models, researchers have also paid attention to utilizing glyph and phonetic features of chinese characters. jiang yang and hongman wang suggested using the 'four-corner' code, a radical-based encoding method for chinese characters, to represent glyph features of chinese characters @xcite , showing the advantage of introducing glyph features in named entity recognition. however, the 'four-corner' code is not that expressive because it only works when chinese characters have the same radicals. tzu-ray su and hung-yi lee suggested using convolution auto encoder to build a bidirectional injection between images of chinese characters and pattern vectors @xcite . this method is brilliant but lacks certain supervision, which means that it's hard to explain the concrete meanings of the pattern vectors. fan yang, jianhu zhang et al introduced 'five-stroke' in ner models, which clearly improves the performance @xcite . recently, zijun @xcite chinesebert @xcite , fusing glyph and phonetic features to the pre-trained models. their work is impressive in fusing glyph and phonetic features as input for pre-trained models, but problems still exist. for example, using flattening character images is inefficient for adding huge costs to feature dimensions and enlarging possible negative influence on the model. as chinese characters have a limited number of components in structures, it will be more convenient and efficient to extract and denote their glyph patterns by certain component encoding methods. ## method our mfe-ner is a lightweight named entity recognition method fusing the glyph and phonetic feature embeddings for chinese character substitution, which is complementary to pre-trained language models in the representation of chinese characters. as shown in figure 2 , mfe-ner introduces an extra module, fusing glyph embedding with 'five-strokes' and phonetic embedding by 'tran-pinyin'. basically, for a chinese sentence s with length n, the sentence s is divided naturally to different chinese characters @xmath0, c 2 , c 3 , ..., c n . each character c i will be mapped to an embedding vector e i , which can be divided into the above three parts, e s figure 3: the 'five-strokes' representation of the chinese characters, 'pu3' (somewhere close to a river). 'five-strokes' divides the character into four character roots ordered by writing custom so that the structure similarity can be denoted. in this case, mfe-ner can better represent the inner distribution of chinese characters in embedding space. ## experiments we conduct experiments on our substitution dataset and three general datasets to verify our method from different perspectives. in order to verify whether our method has the ability to cope with the character substitution problem, we also built our own dataset. this specially designed dataset is collected from informal news reports and blogs. specifically, we construct the substitution dataset by manually labelling and replacing entities with known substitutions. then, we randomly change the characters in the entities with similar characters. meanwhile, the substitution dataset itself contains some sensitive materials that might violate media censorship. generally, we ensure that every entity in this dataset is processed and replaced by similar characters. in this case, the dataset is made of pairs of original entities and their character substitution forms. this dataset consists of 15780 sentences in total and is going to test our method in an extreme language environment. we also conduct experiments on three general ner datasets, chinese resume (zhang and yang, 2018), ontonote @xcite , and weibo @xcite . when introducing new features, it is important to make sure that the overall performance is not negatively affected. among three general datasets, chinese resume is mainly collected from resume materials. the named entities in it are mostly people's names, positions, and company names, which all have strong logic patterns. ontonote mainly selects corpus from official news reports, whose grammar is formal and vocabulary is quite common. different from the above three datasets, weibo is from social media, which includes many informal expressions. details of the above four datasets are described in ## results and discussions in this section, we first show why our method is lightweight and explain the improvement that mfe-ner achieved in recognizing substitution forms of named entities. then, we will analyse the overall performance enhancement by applying mfe-ner. here, the embeddings without glyph and phonetic features are named 'pure' embeddings. basically, we compare pure embeddings from pre-trained language models with those using mfe-ner to enhance the performance of the ner task. to control variables, we also introduce the ablation study to verify the separate contribution from glyph and phonetic features. ## conclusion nowadays, the informal language environment created by social media has deeply changed the way that people express their thoughts. using character substitution to generate new named entities to avoid social media filtering has become a common linguistic phenomenon that is a big challenge for ner. in this paper, we propose a lightweight method fusing the glyph and phonetic features for chinese named entity recognition (mfe-ner) to handle the chinese character substitution problem, which introduces 'five-strokes' to represent glyph features, making it more convenient to differentiate similar characters, and develops a system named 'trans-pinyin' to represent phonetic features of chinese characters, solving the problem of similar pronunciations in pinyin. experiments show that our model, mfe-ner, can assist pre-trained models and reduce the influence caused by character substitution.