Integration of Named Entity Recognition and Sentence Segmentation on A ncient C hinese based on Siku- BERT (2022)

Abstract: Sentence segmentation and named entity recognition are two significant tasks in ancient Chinese processing since punctuation and named entity information are important for further research on ancient classics. These two are sequence labeling tasks in essence so we can tag the labels of these two tasks for each token simultaneously. Our work is to evaluate whether such a unified way would be better than tagging the label of each task separately with a BERT-based model. The paper adopts a BERT-based model that was pre-trained on ancient Chinese text to conduct experiments on Zuozhuan text. The results show there is no difference between these two tagging approaches without concerning the type of entities and punctuation. The ablation experiments show that the punctuation token in the text is useful for NER tasks, and finer tagging sets such as differentiating the tokens that locate at the end of an entity and those are in the middle of an entity could offer a useful feature for NER while impact negatively sentences segmentation with unified tagging.

## introduction the chinese classics is the invaluable legacy for both public and academia. the study of ancient classic texts involves various disciplines such as religion, arts, literature, politics, etc. the public can learn about the diverse aspects of ancient china and enhance their knowledge of history; the ancient chinese texts provide evidence for research areas like the diachronic evolution of the chinese characters and so on. besides, handling languages like ancient chinese is also significant to multilanguage processing as the grammar, vocabulary and other linguistic categories are strongly different from those in other languages, including modern chinese. for example, "之" (zhi) was used to express the possessive in ancient chinese while "的" (de) replaces it to become the major word to express possessive in modern chinese. two related downstream tasks are sentence segmentation and named entity recognition since most ancient texts were not punctuated originally and the entity information would be the foundation for further research, only if we know the names of the official positions we could know the hierarchy of the administrative systems in ancient china. for the above two tasks, we can consider both of them as sequence labeling tasks (i.e. token classification tasks), which means the model would classify each token in the input sequence to a predesigned category. for the ner task, the model would identify which tokens would be components of a named entity, and identify the relative position to the entity. in other words, it would determine the span of the entity from the input sequence. in the sentence "as of now, twitter is still a publicly-traded company on the new york stock exchange.", there are two named entities: "twitter", we can tag it as "s" to represent this is an entity consists one single token, but an entity can also consist of several tokens, like "new york stock exchange". we can tag "new" in "new york stock exchange" as "b", which signifies that "new" is part of an entity and is the beginning of that entity; for sentence segmentation, we can tag the token which is followed by punctuation as a special tag and tag the rest as the other label. so in the previous instance, "now" and "exchange" can be tagged as special tokens to represent there is punctuation followed by these tokens. considering these two tasks as token classification makes it possible to combine them and tag each token for the two tasks simultaneously, which is efficient. the advancement of deep learning especially the development of bert-based models improved the performance of token classification tasks @xcite . some scholars have tried pre-trained the bert-based model on raw ancient chinese texts and fine-tuned them on the specific downstream tasks such as word segmen-tation, part-of-speech tagging, and so on, which verified the feasibility of applying the bert-based model to ancient chinese datasets (yu jingsong, 2019; @xcite . however, most current works just focus on one specific sub-task resulting in a lack of comparison between separate tagging and unified tagging. in our project, we want to answer the question: whether unified tagging on ancient chinese texts would be better than separate tagging in terms of ner and sentence segmentation when training with a bert-based model. we adopt the siku-bert model as our pre-trained model which was pre-trained on the unlabeled ancient chinese raw text and then fine-tuned on the zuozhuan, comparing the performance of the separated tagging scheme and the integrated tagging scheme. named entity recognition for ancient chinese: @xcite developed a deep neural network to generate word embeddings and conducted a named entity recognition task. the results showed that this model performed better than the state-of-theart crf model, arriving at the highest f-1 score of 92.80%. a bi-lstm-crf model was proposed and applied to the traditional chinese medicine patents' named entity recognition problems @xcite . the paper verified that context semantic information can be learned without feature engineering, and the performance was better than the baseline methods, arriving at an f-1 score of 94.48%. @xcite applied a bert-bi-lstm/idcnn-crf model to the ner task and performed better than the bi-lstm-crf benchmark model, which was 4.79% higher in f-1 score when cluener dataset was applied. a radical-level-based bi-lstm-crf model with a self-attention mechanism was employed @xcite , solving the problem of how to deal with hidden information due to the properties of chinese characters, and arriving at a f-1 score of 93.00% in ccks_2017 dataset and 86.34% in tp_cner dataset. our work is different from previous literature as we integrate the two sequence labeling tasks into one by merging labels and comparing the performance of the integrated tagging approach with the performance of the separated tagging approach. unified char-based tagging for ancient chinese: yu jiangde (2015) developed a max-entropy model with a unified character-based label set to perform word segmentation, part-of-speech tagging, and named entity recognition tasks. the experimental results showed that training with the unified char-based label set would be better than training in three separate turns. @xcite developed a bi-lstm-crf model to conduct word segmentation, part-of-speech tagging, and sentence segmentation tasks with a unified char-based label set, which also proved that labels integration with mixed corpus would get better performance. @xcite constructed a model unifying the word segmentation and part-of-speech (pos) tagging tasks and got the f-1 scores of 95.98% in the word segmentation task and 88.97% in the pos tagging task. compared to these works, although we also adopt the unified char-based tagging, our work adopts the bert-based model to perform on the ner and sentence segmentation tasks, which is expected to extract the features better and get better performance. hardware: all experiments are trained on a tesla v80 gpu. pre-processing on tagging scheme: as mentioned in the introduction part, we convert the ner task and sentence segmentation into sequence labeling tasks. to do that, we adopt the char-based labeling (ng and low, 2004) as the annotation scheme. that means, for the ner task, we would annotate each token (character) with one label, to signify whether this character is a part of an entity, and if so, what is the position of the token concerning the entity. we use "b" (beginning), "i" (internal), "s" (single token as an entity), and "o" (outside) to represent the position of the current character concerning an entity. for example, "twitter" and "new york stock exchange" are two named entities in the sentence "as of now, twitter is still a publicly-traded company on the new york stock exchange." since "new york stock exchange" is an entity, we would tag the sequence as "b i i i", the "b" corresponds to "new" and mirrors that it is the beginning of an entity, and "i" means the corresponding token inside an entity, "twitter" is a single token entity, so it is tagged as "s". other characters that are not a part of an entity are tagged as "o" (outside of the entity). an ancient chinese example is 九o 月o 晉b 惠i 公i 卒o 懷b 公i 立o (in september, jin hui gong died and huai gong inherited the throne.) "晉惠公"(jin hui gong)is a person's name, so it is tagged as b i i . since our purpose for the sentence segmentation task is exactly to find out the position where punctuation occurs, we wipe out all tokens that are punctuation in our experiments, otherwise, the model just needs to tag the special label for the token that is followed by punctuation to get a good result, but this is too easy for this task. we tag the character which is followed by punctuation as "p" and those are not with "l", so like the above sentence: as of now, twitter is still a publicly-traded company on the new york stock exchange . we will tag "now" and "exchange" as "p" and others as "l" since only these two precede punctuation. for our ancient chinese example: 九月, 晉惠公卒, 懷公立. we tag the sequence as followings: besides tagging these two tasks separately, we also merge the labels of each token for these two tasks into one so that we can train two tasks in one experiment. if we join the label for ner and sentence segmentation tasks with "-", the above example would be like so "晉 b-l" signifies that "晉" locates at the beginning of an entity but this character is not followed by punctuation. the purpose is to compare the performance of these two tagging approaches (one is tagging for specific tasks separately and the other is tagging for both two tasks by merging the label set) with the siku-bert model. training data: our training data is zuozhuan, a chronicle of general history records during the spring and autumn (770-476 bc) and warring states (475-221) periods in china. the number of tokens in the training data is 244,345, and including 25,005 entities, 33,775 punctuation. the split ratio for the training and validation sets is 8 : 2. hyperparameters setting: our main purpose is not the absolute score of each task but the relative difference between the two tagging approaches, thus we keep all settings the same across experiments. we set up 10 epochs for the training in total and the batch size for each epoch is 8, the learning rate is 0.001, and the scheduler step is 600. we adopt cross entropy as our loss function and adamw as the optimizer; the drop-out rate is 0.2. ## conclusions our experiment shows an inconsistent result to the previous research that also compared the unified tagging scheme with the separate tagging approach @xcite @xcite ; the further exploration we need to do is tagging the types of the entity and punctuation as well and count the performance based on different types of entities or punctuation, in addition, we want to try different model architectures and different pretrained models, we want to verify whether what we observe from our current experiments is specific for the model and architecture. our work pays attention to the ancient chinese data which is low-resource and being ignored. the shortage of annotated data and the relatively fewer application scenarios make it a minority field in nlp, which is required more research. the usage of advanced deep learning techniques for automatic sentence segmentation and named entity recognition of ancient chinese not only facilitate readers to read, but also can be of great significance to the arrangement of ancient books, and the intelligent application of ancient chinese.