Character Jacobian: Modeling C hinese Character Meanings with Deep Learning Model (2022)

Abstract: Compounding, a prevalent word-formation process, presents an interesting challenge for computational models. Indeed, the relations between compounds and their constituents are often complicated. It is particularly so in Chinese morphology, where each character is almost simultaneously bound and free when treated as a morpheme. To model such word-formation process, we propose the Notch (NOnlinear Transformation of CHaracter embeddings) model and the character Jacobians. The Notch model first learns the non-linear relations between the constituents and words, and the character Jacobians further describes the character’s role in each word. In a series of experiments, we show that the Notch model predicts the embeddings of the real words from their constituents but helps account for the behavioral data of the pseudowords. Moreover, we also demonstrated that character Jacobians reflect the characters’ meanings. Taken together, the Notch model and character Jacobians may provide a new perspective on studying the word-formation process and morphology with modern deep learning.

## introduction recent years have witnessed a growing interest in modeling the internal semantic dynamics of compounds. indeed, compounding is sometimes argued as a language universal, and it is claimed to be protolinguistic fossils which modern languages frequently elaborate. compounds are usually loosely defined as forming words with two independent words, such as blackboard and pineapple @xcite @xcite . it is apparent that the number of potential combinations is already enormous, even in this simplest form of the two-constituent compound. the productivity of compounding is particularly evident in chinese word-formation. due to the debating nature of chinese wordhood, there is a vague boundary between wordforming affix and bound root when treating a chinese character (字 zì) as a morpheme @xcite @xcite . there is considerable flexibility in chinese characters, where morphemes can be joined with one another either preceding or following them. for example, the following four words all start with the characters which are the ends of the previous one: 長老 zhǎng lǎo "elder", 老師 lǎo shī "teacher", 師範 shī fàn "teacher-training", 範圍 fàn wéi "area". the versatility of characters leads @xcite to describe chinese text as "a continuous parade of meaningful individual characters (morpheme)" (see @xcite for a complete introduction in chinese morphology). however, the productivity of chinese characters is hard to capture by a computational model. a traditional natural language pipeline starts with word segmentation, which removes the sublexical cues (characters) in the first step. even for the later distributional semantic model with sub-word information, such as fasttext @xcite , it cannot accommodate the different meanings carried by individual chinese characters. recent deep learning models, e.g., bert @xcite , provide contextualized embeddings of each token. but, as the final representations are mixed (hence contextualized), it is unclear how the embeddings could relate to the original character tokens. therefore, we propose the notch model (nonlinear transformation of character embeddings), and with which we derive character jacobians. we first train the notch model to learn the relationships between the constituent characters' and the whole word's meaning. next, we show that character jacobians derived from the model capture the different roles of characters in the whole word. the rest of this paper is organized as follows. first, in section 2, we briefly review the relationship between chinese characters and words and their complicated behaviors in chinese morphology. section 3 describes the notch model and shows that it not only predicts the real words but helps account for the behavioral data of pseudoword. finally, in 4, we explore the character jacobians and evaluate them on a chinese affixoid dataset. ## related works the vague boundary between affixation and compounding does not only occur in chinese word-formation. for example, @xcite points out the problem of neoclassical elements, such as the word bio-logy. while one would tend to treat bio-, and -logy as prefixes and suffixes, which would violate the basic assumption about the word structure; that is, there will be no root in this word. plag argues that these words, often called combining forms, should be best treated as compounds. studying compounds and their constituents poses interesting questions in computational modeling. as the meaning of the compounds may be free from its composing elements, determining the meaning of a newly encountered compound is thus difficult @xcite . one interesting attempt is to model the meanings of the constituents separated from their free-word counterparts @xcite . the difference between the as-constituent and free-word representations is called semantic shift. a linear model is then built to simultaneously estimate the semantic shifts of the constituents and their linear relations with the compounds by linear algebra. in chinese word formation, a character may play different morpho-semantic roles in different words, even if they are in the same position. therefore, it may not be straightforward to accommodate such versatility into a single linear transformation. yet, we could consider the relations between the constituents and compounds as a complex non-linear func-tion; the linear transformation is then a local approximation at that specific local neighborhood. here, we use the jacobian matrix to obtain the local linear approximation, which is previously used to construct a saliency map and understand network properties @xcite . however, before we compute and evaluate the jacobian, we should first build our non-linear function between constituent and word, that is, the notch model. ## the notch model the purpose of the notch model is two-fold. first, it should learn the semantic relationships between the variable-length character sequences (i.e., the constituents) and the whole words. as the semantics of constituents and words are both described by a semantic vector space, the model-learned relationship is essentially a function of r @xmath0, with n being the semantic space's dimension and k being the word length. second, the model also provides the jacobians with which we characterize the character's role in a given word. therefore, we first train the model to predict the whole word's embedding from the embeddings of its constituents. 1 ## character jacobian in this section, we extract and evaluate the character-level information in the notch model. being a model based on bert, the notch model learns the contextualized embeddings of each token in its input sequence. the characteristic has been applied to the word sense disambiguation task, where the model is successfully employed to create sense embeddings @xcite . therefore, an interesting question to ask is, how could we extract the characterlevel context-dependent information from the notch model? specifically, this character-level information should ideally differentiate the word contexts the character occurs. for example, the character 手 shǒu means "hand" as an independent word. however, the same character could occur at the start of the word and carries different meanings, such as 手氣 shǒu qì "luck" and 手臂 shǒu bì "arm". in addition, the meanings are also different when occurring at the end of the word: 歌手 gē shǒu "singer", or 分手 fēn shǒu "break up". that is, the character-level information should have different representations for the same character when it is used differently. there are at least three different approaches to extract character-level information from the notch model. the first one is to use the bert input embeddings directly. since the bert-base-chinese is a by-character tokenizer, the input embeddings act like character embeddings. however, this approach does not have access to other characters in the same word; therefore, it is not contextsensitive as we require. the second approach uses the token embeddings in the later layers of the bert model, with the advantage that the token embeddings would be contextsensitive after layers of transformation. yet, it also implies that the token embeddings have already incorporated or mixed the representations from other tokens. hence, it is not straightforward to attribute the token to the input character anymore. the third approach, which is simultaneously context-sensitive and character-specific, is through the jacobian matrix. ## conclusion in this paper, we present the notch model, from which we derive the character jacobians. in a series of experiments, we show that the model predicts the embeddings of the real words from their constituents and helps account for the behavioral data of the pseudowords. in addition, we also show that character jacobians capture characters' roles in the words, which reflect the meanings of the characters. the approach to study compounding by modeling the word embeddings and exploring their jacobians could also be applied to other languages. multilingual language models and word embeddings are readily available in the community. however, the most interesting question is how to study compounding if the language's writing system may introduce spaces in the compounds, such as the case in english. the commonly-used word embeddings only include entries with no interword spaces. that is, there will be entries for earth-quake, airport, but no entries for rush hour, coffee mug. moreover, it could be argued that the interword spaces bear significance in cognitive processing @xcite . it will be thus an interesting future work to systematically study compounding in this case. character jacobians open up new possibilities to study chinese characters or morphology with deep learning models. however, as the jacobian is an abstract mathematical object, other future works include further investigating its relation with morphological rules and lexical categories and how it connects to the distributional semantics.