G lyph2 V ec: Learning C hinese Out-of-Vocabulary Word Embedding from Glyphs (2020)

Abstract: Chinese NLP applications that rely on large text often contain huge amounts of vocabulary which are sparse in corpus. We show that characters’ written form, Glyphs , in ideographic languages could carry rich semantics. We present a multi-modal model, Glyph2Vec , to tackle Chinese out-of-vocabulary word embedding problem. Glyph2Vec extracts visual features from word glyphs to expand current word embedding space for out-of-vocabulary word embedding, without the need of accessing any corpus, which is useful for improving Chinese NLP systems, especially for low-resource scenarios. Experiments across different applications show the significant effectiveness of our model.

## introduction word embedding encoded semantic and syntactic information @xcite in lowdimensional space have served as useful features for various nlp applications but often require large-scale corpus with billions of tokens to train. a natural constraint of word embedding is that it is not practical to collect the entire vocabulary of any language with large enough frequency to train the embedding for every word, since some new words may appear in downstream tasks. a typical solution is to simply assign a specific unk embedding to all out-of-vocabulary (oov) words that do not appear in the training data. current solutions such as using subwords (e.g., characters) are mainly considering alphabetic languages (e.g., english and french) that are composed of small amount of characters. such techniques may not be sufficient for ideographic lan- * equally contribution. guages (e.g., chinese and japanese) in which a word is often composed with characters of a large amounts. an example is that traditional chinese includes about 17k distinct tokens. therefore, it could be expected to suffer from underfitting not only word embedding but also character embedding. even worse, words in ideographic languages are often composed of 2-3 characters only, unlike words in alphabetic languages are longer but with smaller types of characters. figure 1 provides the statistics in chinese sinica corpus. the visual structure (or glyph) of a chinese character contains rich semantics. a chinese charhttp://asbc.iis.sinica.edu.tw/indexreadme.htm acter is made up of several graphical components. figure 2 shows some examples that components in characters represent similar semantic or pronunciation. in addition to glyphs, we propose to use the high-quality features provided by cangjie input method to represent each character. cangjie is a popular chinese input method. similar to radicals, characters are composed of 24 basic graphical units. each unit is mapped to a corresponded letter key on a standard qwerty keyboard. building beyond character glyphs, one can intuitively guess the semantic of a word. recent work @xcite @xcite @xcite have shown benefits of the compositionality at character level or visual feature of chinese glyphs for some tasks. in this work, we suggest that in the oov scenario glyphs can be particularly useful. a key observation for solving oov problem matches the intuition of human generalization in chinese. when a chinese user reads an unseen word or a character, by decomposing the structure, graphical components such as radicals for a character often help chinese users understand the meaning and sometimes pronunciation of the character. we study a novel application that recovers chinese oov word embeddings from glyphs. our work is to answer a question : given the pretrained word embeddings, can we directly learn a mapping from word glyphs to their word embedding and generalize the mapping for the purpose of generating the embedding of oov words? we formulate it as a visual-to-text transfer learning problem and show that the visual structure of chinese characters is helpful in learning chinese oov embeddings. learning embedding for oovs to handle oov words, an approach is operating character level embeddings, then averages them into word embeddings @xcite . morphology-based approaches take advantage of meaningful linguistic substructures @xcite @xcite . morphology-based approaches often struggle with those vocabularies lacking linguistic substructures such as names and transliterations of foreign language, which often appears as oov words. in all the models above, just like word2vec @xcite ), the embeddings meed to learned by training over a large corpus. the most similar work is mimick model @xcite . by learning a character language generating model, guided by minimizing the distance between the output embedding of lstms and pre-trained word embeddings, mimick shows feasibility of generating oov word embedding from character compositions. however, mimick is mainly from the view of alphabetic languages that does not consider glyphs. chinese words often consist of short sequences composed of many kinds of tokens that are difficult for language model approaches to handle (see figure 1 ) and could suffer from under-fitting. ## our model: glyph2vec we formulate the task of learning oov embeddings as a transfer learning problem. formally, given a chinese vocabulary set v of size |v|, and a pre-trained embeddings matrix @xmath0, e i } |v| @xmath1. we aim to learn a mapping f : @xmath2, where f projects the input word to the d dimension embedding space such that f (w i ) ≈ e i . in testing, a word w t may be out of v, while the model is still obliged to predict the embedding e t with f (w t ). ## nearest neighbor examples we qualitatively analyze glyph2vec with nearest neighbor (nn) sanity check. we observe glyph2vec is able to model visual semantic by associating those characters that share related visual features since glyph2vec learns from the images of characters. for example, 鰻(eel) in 蛇鰻(snake-eel) shares the radicals of 魚(fish) with 石 鱸(haemulidae, fish name). 銠(rh) and 氯(cl) in 三氯化銠(rhcl3) associate some visual features relate to chemicals like 金 in 鈰(ce), 气 in 氟(f), 酉 in 酸(acid), and more. on the other hand, we observe some properties including composition (e.g., numbers) and character semantic that both glyph2vec and mimick can provide. (1) composition: composing characters that have very different meanhttps://sites.google.com/site/rmyeid/projects/polyglot ing after splitting them. for instance, 茲 尼 約 夫 is a transliteration of seleznev (russian name), for which every character is meaningless alone but a meaningful transliteration when combined. with character-level compositional model in glyph2vec, it could be retrieved given 克 羅 迪 歐(claudio, western name). moreover, glyph2vec preserves correct meaning of a character when attaching with the other characters. for example, 驟(abrupt)減(decrease) can retrieve 減 少(cut back) and 減低(reduce) properly when 減 (subtract) is associated to different characters. (2) character semantic: associating different characters with similar meaning. for example, 道(street) is related to 巷(lane) or 弄(alley) and they are retrieved by our model given 學府二道(xuefu 2nd street) as the oov word even though the characters look completely different. ## conclusion in this work, we propose a multi-modal framework that expand pre-trained embedding space to include oov words using character visual features such as cangjie feature and chinese character glyphs. we have demonstrated the effectiveness of glyph2vec on traditional chinese, and we believe glyph2vec can also be applied to other ideographic languages to handle oov words as well. for simplified chinese, we suggest users to first translate into traditional chinese since traditional characters have richer structures and probably more semantics can be extracted through glyph2vec.