An In-depth Study on Internal Structure of C hinese Words (2021)

Abstract: Unlike English letters, Chinese characters have rich and specific meanings. Usually, the meaning of a word can be derived from its constituent characters in some way. Several previous works on syntactic parsing propose to annotate shallow word-internal structures for better utilizing character-level information. This work proposes to model the deep internal structures of Chinese words as dependency trees with 11 labels for distinguishing syntactic relationships. First, based on newly compiled annotation guidelines, we manually annotate a word-internal structure treebank (WIST) consisting of over 30K multi-char words from Chinese Penn Treebank. To guarantee quality, each word is independently annotated by two annotators and inconsistencies are handled by a third senior annotator. Second, we present detailed and interesting analysis on WIST to reveal insights on Chinese word formation. Third, we propose word-internal structure parsing as a new task, and conduct benchmark experiments using a competitive dependency parser. Finally, we present two simple ways to encode word-internal structures, leading to promising gains on the sentence-level syntactic parsing task.

## introduction unlike english, chinese adopts a logographic writing system and contains tens of thousands of distinct characters. many characters, especially frequently used ones, have rich and specific meanings. however, words, instead of characters, are often considered as the basic unit in processing chinese texts. we believe the reason may be two-fold. first, usually a character may have many meanings and usages. word formation process greatly reduces such char-level ambiguity. second, by definition, figure 1 : three example words with internal structure under different annotation paradigms. "想(think of) 方(plan) 设(design) 法(method)" is a verb and means "find ways or means to do". "婚(marriage) 姻(marriage) 法(law)" is a noun. "法老" is phonetic transliteration of "pharaoh". the three words all contain the character "法" under different meanings. intuitively, modeling how multiple characters form a word, i.e., the word-formation process, allows us to more effectively represent the meaning of a word via composing the meanings of characters. this is especially helpful for handling rare words, considering that the vocabulary size of characters is much smaller than that of words. in fact, many nlp researchers have tried to utilize charlevel word-internal structures for better chinese understanding. most related to ours, previous studies on syntactic parsing have proposed to annotate word-internal structures to alleviate the data sparseness problem @xcite . however, their annotations mainly consider flat and shallow word-internal structure, as shown in figure 1-(a ) and (b). meanwhile, researchers try to make use of character information to learn better word embeddings @xcite . without explicitly capturing word-internal structures, these studies have to treat a word as a bag of characters. see section 2 for more discussion. this paper presents an in-depth study on charlevel internal structure of chinese words. we endeavour to address three questions. (1) what are the word-formation patterns for chinese words? (2) can we train a model to predict deep word-internal structures? (3) is modeling word-internal structures beneficial for word representation learning? for the first question, we propose to use labeled dependency trees to represent word-internal structures, and employ 11 labels to distinguish syntactic roles in word formation. we compile annotation guidelines following the famous textbook of @xcite on chinese syntax, and annotate a highquality word-internal structure treebank (wist), consisting of 30k words from penn chinese treebank (ctb) @xcite . we conduct detailed analysis on wist to gain insights on chinese wordformation patterns. for the second question, we propose wordinternal structure parsing as a new task, and present benchmark experimental results using a competitive open-source dependency parser. for the third question, we investigate two sim-ple ways to encode word-internal structure, i.e., labelcharlstm and labelgcn, and show that using the resulting word representation leads to promising gains on the dependency parsing task. we release wist at https://github.com/ suda-la/acl2021-wist , and also provide a demo to parse the internal structure of any input word. annotation of shallow internal structure of chinese words was first mentioned in @xcite , largely based on heuristic rules. @xcite ; @xcite found that many multi-char words could be divided into two subwords, i.e., root and affix. they annotated structures of about 19k words (35% of 54,214) in ctb6. their experiments showed that subword-level syntactic parsing is superior to word-level parsing. for the three words in figure 1 , their approach is only applicable to the second word, i.e., "婚姻/法". as an extension to @xcite , @xcite proposed char-level syntactic parsing by further dividing subwords into chars. as shown in figure 1-(a) , for each word, they annotated a binary hierarchical tree, using constituent labels to mark which child constituent is more syntactically important, i.e., left, right, or coordinate. in such way, they could convert a word-level constituent/dependency tree into a char-level one. similar to @xcite , @xcite annotated internal structure of synthesis (multi-morpheme) words with four relations, i.e., branching, coordinate, beginning and other parts of a single-morpheme word. in the dl era, three works have studied wordinternal structure. similarly to our work, @xcite internal structure. as shown in figure 1-(b) , for each multi-char word, they first annotate the partof-speech (pos) tag of each character, and then determine an unlabeled dependency tree, and finally use a pos tag triple as arc label, corresponding to the pos tags of the modifier/head characters and the whole word. however, we argue pos tag triples are only loosely related with word-formation patterns, not to mention the severe difficulty of annotating char-level pos tags in each word. recently, @xcite extended @xcite by using an extra label for marking singlemorpheme words, and annotated hierarchical internal structure of 53k words from a chinese-english machine translation (mt) dataset. @xcite annotated the internal structure of words with 4 dependency relations. in summary, we can see that most previous studies adopted quite shallow hierarchical structure. in contrast, this work presents a more in-depth investigation on internal structure of chinese words and employs 11 labels to distinguish different syntactic roles in word formation, as shown in figure 1-(c ). leveraging character information for better word representation. it has already become a standard way in many nlp tasks to obtain charaware word representation by applying lstm or cnn to the character sequence of a word, and concatenate it with word embedding as input, such as named entity recognition (chiu and nichols, 2016), dependency parsing @xcite , and constituent parsing @xcite . another research direction is to leverage charac-ter information to obtain better word embeddings. @xcite extended the cbow model and proposed to jointly learn character and word embeddings. based on @xcite , @xcite proposed to jointly learn embeddings of words, characters, and sub-characters. foot_2 however, both studies assume that characters contribute equally to the meaning of a word and directly average embeddings of all characters. to address this, @xcite extended @xcite and proposed a cross-lingual approach to distinguish contribution of characters for a word. the idea is to translate chinese words and characters into english words, and use similarities between corresponding english word embeddings for contribution measurement. instead of treating a word as a bag of characters, we experiment with two simple ways to obtain structure-aware word representations. meanwhile, enhancing their approach with explicit wordinternal structure could be also very interesting. utilizing word-internal structure. wordinternal structure have been explored in various nlp tasks. several works propose to learn wordinternal structure, word segmentation, pos tagging and parsing jointly @xcite @xcite , demonstrating the effectiveness of word-internal structure in helping downstream tasks. @xcite attempt to convert words into fine-grained subwords according to the internal structure of words for better dealing with unknown words during word segmentation. @xcite propose to integrate the representation of word-internal structure into the input of neural machine translation model, leading to improved translation performance. ## word-internal structure annotation in this section, we describe in detail the annotation process of wist. as shown in figure 1 -(c), we adopt dependency trees for representing word-internal structure. the reason is two-fold. first, word-formation process correlates with syntax in different ways depending on language type @xcite . such correlation is especially close for chinese due to its lack of morphological inflections. in particular, @xcite presented thorough investigation on chinese word formation mainly from a syntactic view. second, as a grammar formalism, dependency tree structure has been widely adopted for capturing sentence-level syntax due to its simplicity and flexibility in representing relations. meanwhile, its computational modeling is also developed quite well. annotation guidelines. after several months' survey, we have compiled systematic and detailed guidelines for word-internal structure annotation. our guidelines are mainly based on the famous textbook on chinese grammar of @xcite . we intensively studied all previous works on word-internal structure annotation, which are discussed in section 2. we also find that it is quite beneficial to be familiar with guidelines developed by previous annotation projects for chinese word segmentation @xcite . our guidelines contain 11 relations specifically designed to capture the internal dependency syntax for chinese words, as shown in with chinese syntax, and select 6 capable annotators with a lot of data annotation experience as expert annotators to handle inconsistent submissions. all the annotators (including expert annotators) were paid for their work . the salary is determined by both quantity and quality. besides, we give extra bonus to the annotators with high accuracy. the average salary of the annotators is 30 rmb per hour. all annotators are trained for several hours to be familiar with our guidelines and the usage of annotation tool. we apply strict double annotation in order to guarantee quality. each word is randomly assigned to two annotators. two identical submissions are directly used as the final answer. otherwise, a third expert annotator is asked to decide the final answer after analyzing the two inconsistent annotations. annotation tool. we build a browser-based annotation tool to support the annotation workflow and facilitate project management. given an annotation task, all its pos tags foot_3 of the focused word in ctb5 are presented to the annotator, in order to explore multiple internal structures for one word. in that case, the annotator can click a checkbox to inform us for further process. please note that the manually annotated pos tags in ctb5 are converted into universal dependencies (ud) foot_4 pos tags based on predefined mapping rules, since the original ctb5 pos tags are too fine-grained (33 tags) and difficult for annotators to understand. the interface also presents several example sentences to improve annotation efficiency. we strongly encourage annotators to look up difficult words or characters in electronic dictionaries. 5 data selection. following previous works, we select multi-char words from ctb5 for annotation. ## analysis on annotated wist in this section, we analyze the annotated wist from different aspects in order to gain more insights on chinese word-formation patterns. inter-annotator consistency. as discussed earlier, each word is labeled by two annotators, and inconsistent submissions are handled by a third senior annotator for obtaining a final answer. the averaged inter-annotator consistency ratio is 83.0 dependency-wise, i.e., the percent of characters receiving the same head and label from two annotators, and 75.8 word-wise, i.e., the percent of words receiving the same whole trees. if we do not consider labels, the unlabeled consistency ratios increase to 87.5 dependency-wise and 85.1 word-wise. although it may be a factor that most annotators are inexperienced in this new annotation task, such low consistency ratios indicate that annotating word-internal structure is quite challenging, especially when it comes to distinguishing syntactic roles. meanwhile, this also demonstrates the importance of strict double annotation, considering that nearly a quarter of words are inconsistent and require handling by senior annotators. annotation accuracy. we calculate annotation accuracy by comparing all submissions (as denominator) from annotators against the final answers in wist. please note that each word is double annotated. the overall dependency-wise accuracy for all annotators is 90.9, and word-wise is 86.9. if not considering labels, the overall unlabeled accuracy increases to 93.4 and 92.1, dependency-and word-wise respectively. the first major row in comparing unlabeled versus labeled accuracy, the gap is quite large. the extreme case is "pobj". annotators usually can correctly decide the head (84.5%), but very unlikely choose its true label "pobj" (48.2%). similarly, accuracy drops by 24.9 for "subj". we give more discussions on annotation difficulties below. label distribution. the third major row in besides the overall distribution, the third major row in we can see that nouns are mostly formed with "att" (33.8%) and "coo" (11.5%), whereas verbs are with "coo/obj/adv/cmp" in the descending order. proper nouns are evenly dominated by "frag" (29.6%) and "att" (28.4%). it is also obvious that proper nouns tend to be longer, consisting of 2.7 characters according to its "root" percentage. numerals are mainly composed via "att" (75.7%) and consist of 5.0 character on average. multiple structures for one word? many words have multiple meanings. then the question is: how many words really have multiple internal structures? as illustrated in section 3, we show all pos tags to annotators in order to obtain all internal structures of an ambiguous word. however, in annotated wist, we find there are only 103 such words with multiple internal structures, accounting for about 0.3% of all annotated words, and 2.7% of those having multiple pos tags. as a typical example, "制服" have two structures. as a verb, it means "subdue" and has "制(control) cmp --→ 服(tamely)". as a noun, it means "uniform" and has "制(regulated) att ← --服(cloth)". this low percentage reveals that most chinese words actually have very steady internal structure. they have multiple pos tags, mainly because they are used for different syntactic functions without morphological inflections, such as "发展" as verb ("develop") or noun ("development"). more on "frag". the "frag" label is designed to handle all words that have no internal structure due to the lack of semantic composition. from high-order structure distribution. to gain more insights on complex word-formation structure, we focus on all three-char words. we find that the root usually lies in the third character by 74.6%, and the percentage for the second and first characters is only 15.3 and 10.1 respectively. looking more closely, we find the following four dominated structures. difficulties in annotation. since it is difficult to capture the patterns on unlabeled-dependency inconsistencies, we focus on confusion patterns in label annotation. among all characters receiving the same head but different labels from two annotators, 20.1% correspond to "{att, adv}" confusion due to the ambiguity of the head character being a verb or a noun. the second confusion pattern is "{coo,frag}", with a proportion of 18.6, which are mainly from proper nouns. according to our guidelines, if the meaning of a proper noun is compounding, annotators have to annotate its real internal structures rather than using "frag". it is also very difficult to distinguish "obj" and "pobj", since the boundary between prepositions and verbs is vague in chinese. ## utilizing word-internal structure this section presents a preliminary study on utilizing word-internal structure, aiming to address the third question: is modeling word-internal structures beneficial for word representation learning? we use sentence-level dependency parsing as the focusing task @xcite , mainly considering resemblance in tree structure representation and close relatedness between the two tasks. given an input sentence w 0 w 1 ...w m , the goal of dependency parsing is to find an optimal dependency tree for the sentence. again, we adopt supar @xcite for implementation of biaffine parser @xcite as our basic parser. ## conclusions this paper presents a thorough study on internal structures of chinese words. first, we annotate a high-quality word-internal structure treebank covering over 30k words in ctb5, named as wist. second, we perform analysis on wist from different perspectives and draw many interesting findings on chinese word-formation patterns. third, we propose word-internal structure as a new task, and present benchmark results using a popular dependency parser. finally, we conduct preliminary experiments with two simple methods, i.e., la-belcharlstm and labelgcn, to encode wordinternal structure as extra word representation, and find promising performance gains on the sentencelevel dependency parsing task. analysis shows that the rich dependency labels adopted in wist play a key role, and word-internal structure is most beneficial for rare word representation.