Improving Neural Language Processing with Named Entities (2021)

Abstract: Pretraining-based neural network models have demonstrated state-of-the-art (SOTA) performances on natural language processing (NLP) tasks. The most frequently used sentence representation for neural-based NLP methods is a sequence of subwords that is different from the sentence representation of non-neural methods that are created using basic NLP technologies, such as part-of-speech (POS) tagging, named entity (NE) recognition, and parsing. Most neural-based NLP models receive only vectors encoded from a sequence of subwords obtained from an input text. However, basic NLP information, such as POS tags, NEs, parsing results, etc, cannot be obtained explicitly from only the large unlabeled text used in pretraining-based models. This paper explores use of NEs on two Japanese tasks; document classification and headline generation using Transformer-based models, to reveal the effectiveness of basic NLP information. The experimental results with eight basic NEs and approximately 200 extended NEs show that NEs improve accuracy although a large pretraining-based model trained using 70 GB text data was used.

## introduction in statistical nlp technologies, which were widely employed before neural-based nlp emerged, basic information, such as pos tags and nes were often used as features for document classification @xcite and other nlp tasks. however, since neural-based nlp technologies have emerged with state-of-the-art (sota) performances, basic nlp technologies, such as pos tagging and ne recognition are no longer used for obtaining features from given text. this is because most neural-based nlp methods attain higher accuracy using only sentence representation encoded by a pretraining model learned from the large scale unlabeled text. however, we think there are some rooms that basic information, such as pos tags and nes contribute to recent neural-based nlp even if large scale pretrained models such as bert @xcite and bart @xcite are used for obtaining sentence representations. one of the reasons is that pos tags and nes are usually obtained from outputs of analyzers trained from labeled training data created by human-beings for each specific purpose. for example, ne recognizers can identify single words or phrases with their classes, such as person, organization, and so on, which are not explicitly given from pretrained models. therefore, we think different kinds of information compared with pretraining ones are obtained from outputs of such nlp tools and we expect such information contributes to further improve accuracy. we propose the incorporation of basic nlp information to neural nlp architectures and evaluate their effectiveness on two japanese nlp tasks. in this paper, two ne categories, eight basic nes @xcite and approximately 200 extended nes @xcite , are considered. by combining ne information with a pretrained model, we train a model of each task. experimental results on document classification using the bert model and headline generation using the bart model, show that combining nes with a large pretrained model contributes to significantly improved accuracy. ## experimental design this paper investigates the effectiveness of ne information for sota nlp models. in order to investigate effectiveness of nes for sota nlp technologies, we conducted experiments on the two tasks. the first one is document classification for investigating the effectiveness on a classifcation task. the one is headline generation for investigating the effectiveness on a generation task. we use bert @xcite for document classification and bart @xcite for headline generation as pretraining models. to evaluate these tasks, we defined an architecture that uses ne class embeddings in addtion to subword embeddings given by one of the pretrained models, for each task. we refer to the architectures as bert ne and bart ne , where ne indicates the ne class definition. for the inputs of bert ne and bart ne , we recognize nes in texts with an ne recognizer. finally, using bert ne and bart ne , we evaluated the models with and without ne annotations on the two tasks for examining the effectiveness of ne annotations in each task. one expected effect is the impact of granularity of ne classes on the accuracy of each task. to investigate such an effect, we use the following japanese ne categories, described in section 5. furthermore, to evaluate the impact of ne recognition accuracy, we use the following two ne recognition methods. we use the following terms for the different settings. ## baseline models here, we use two pretraining architectures, a pretrained bert model for document classification and a pretrained bart model for headline generation. ## named entity-based models this section describes a bert-based document classification and bart-based headline generation using ne information. ## data sets this section introduces data sets for document classification and headline generation, created from the articles of mainichi newspaper foot_2 . ## experimental setting this section shows the preprocess of input texts, hyperparameters, evaluation metrics, and so on. ## experimental results we see from the results that nes contribute to improve the score of document classification task with transformer-based models. ## related work there are several variants of bert @xcite @xcite . additionally, recent studies for neural document summarization focus more on pretraining methods @xcite @xcite . al-though we used bert and bart, other methods can combine with the proposed method because the proposed method uses embeddings of nes from finetuning, which is left as future work. @xcite proposed an extractive summarization method that uses density of named entities to calculate the importance of a sentence. furthermore, they proposed an abstractive summarization method that concatenates one-hot representation of named entity categories with token embeddings. different from @xcite , we investigate the effectiveness of the use of nes in subword-based neural network models and different types of nes, i.e., bne and ene. additionally, we used embeddings to represent nes instead of one-hot vectors to obtain further representation. in addition to document summarization, some works use nes for improving neural machine translation, which uses similar architectures as neural document summarization. ugawa et al. (2018) @xcite , our proposed method does not insert ne tags into the sequence of words because the time and memory complexities of transformerbased models quadratically increase depending on the length of the sequence, which can be a significant problem in headline generation where a source document is long. additionally, because subword tokenization is not used, their motivation differs from ours that investigates the effectiveness of nes on subword-based neural networks. @xcite investigated the use of nes in non-neural network models on document classification. while they reported use of nes improve the accuracy of document classification, the contribution to subword-based neural network models was not investigated. @xcite compared the representation of nes for neural network-based models in document classification task. they reported that replacing tokens of named entities with special tokens representing ne categories does not improve the accuracy of document classification. our experiments showed that combining ne and token embeddings improved the accuracy of document classification. ## conclusion this paper explored the effectiveness of ne information in large-scale pretraining models. we evaluated the effectiveness of nes in document classification and headline generation tasks. the experimental results showed that ne information improved the accuracy of large-scale sota pretraining-based models. we incorporated ne information to pretrained models trained from subword sequences. for future work, we look to explore pretraining methods from subword sequences annotated with ne information from scratch.