Logographic Information Aids Learning Better Representations for Natural Language Inference (2022)

Abstract: Statistical language models conventionally implement representation learning based on the contextual distribution of words or other formal units, whereas any information related to the logographic features of written text are often ignored, assuming they should be retrieved relying on the cooccurence statistics. On the other hand, as language models become larger and require more data to learn reliable representations, such assumptions may start to fall back, especially under conditions of data sparsity. Many languages, including Chinese and Vietnamese, use logographic writing systems where surface forms are represented as a visual organization of smaller graphemic units, which often contain many semantic cues. In this paper, we present a novel study which explores the benefits of providing language models with logographic information in learning better semantic representations. We test our hypothesis in the natural language inference (NLI) task by evaluating the benefit of computing multi-modal representations that combine contextual information with glyph information. Our evaluation results in six languages with different typology and writing systems suggest significant benefits of using multi-modal embeddings in languages with logograhic systems, especially for words with less occurence statistics.

## introduction the essential idea in statistical language modeling is to represent the meaning of a word as a function of its context. the function, modeled via the conditional probability of observing a word in a given utterance, has most efficiently been approximated with a neural network based architecture @xcite @xcite . the outstanding performance of neural methods in language modeling and their recent development @xcite have them a preliminary component in various downstream nlp tasks. one of the main limitations in the formulation of language models lies however in the choice of ortographic units in calculating the contextual distribution, which is usually convenient in english and other languages using phonetic scripts. on the other hand, many languages rely on logographic writing systems, where surface forms are represented as a visual organization of smaller graphemic units and the word meaning can be changed through compositional variations of these units. although a direct segmentation of these units has been found quite challenging due to visual compositions in the final form of the grapheme, previous studies have found potential benefits of using visual information to aid nlp models in sentence representation @xcite @xcite . on the other hand, none of these studies have focused on isolating the effects of different linguistic features in relation to their correlation to visual features. as shown in figure 1 , logographic information often contain important features related to the word meaning. in this paper, we perform the first focused analysis to measure the significance of logographic features specifically to the semantic information encoded in token or character-level language representations by evaluating the performance of multimodal embeddings in the nli task. in particular, we aim to answer the following research questions: 1) how important may logographic information be to for an accurate representation of semantic information in word or character-level language units 2) whether the contribution of logographic information to semantic representations may depend on the language typology and writing system in order to answer these questions we implement a multi-modal representation learning model where each written text segment is representation as a combination of visual embeddings obtained from prominent convolutional neural network (cnn) based models @xcite @xcite , and contextual representations obtained from multilingual pre-trained language models @xcite . we evaluate the contribution of visual information to the performance in the nli task under few-shot learning settings in six languages with varying typology and writing systems: english, spanish, hindi, urdu, vietnamese and chinese. we also study the optimal representation granularity for semantic information by comparing word or character-level multi-modal representations in our experiments. in conclusion, we find that taking into account the visual information improves the performance in nli tasks especially in logographic languages like chinese and that the improvements are correlated with the factors that determine the quality of token representations, such as the occurence of the tokens in training data as well as language model capacity and hyperparameters. our findings suggest multi-modal processing is a promising direction, especially for processing languages where conditions of data sparsity may create fall backs in assumptions undertaken in statistical formulations. ## computing visual glyph embeddings our multi-modal embedding model is composed of two components: (i) the visual encoder, which computes embeddings based on the input images representing each text segment, and (ii) the pretrained language model providing the text-based embeddings. layer visual encoding model 1 spatial conv. (3,3) ! 32 2 relu 3 maxpool (2,2) 4 spatial conv. (3,3) ! 32 5 relu 6 maxpool (2,2) 7 spatial conv. (3,3) ! 32 8 relu 9 linear (800,128) 10 relu 11 linear (128,128) 12 relu acters) and then converted into images. sentences are split into 30 x 60 pixel word images using the jieba foot_0 tool. all graphemes are centralized to the middle of the image. ## results and discussion our experiment results are given in although the improvements highly correlate with the logograhic nature of the writing system, the fact that they apply to most languages, even urdu and hindi with phonetic alphabets, point to the suboptimal effects in tokenization or segmentation and their potential harms to correctly model the contextual distribution. we also see in high-resource language representations like english, our fusion method may be harmful to the downstream task, which we anticipate that could be resolved with higher amount of fine-tuning and development data. in light of all these considerations, the findings suggest that multi-modality is a promising direction for overcoming problems related to data sparsity, and eventually tokenization or segmentation-free language modeling. ## conclusion in this paper, we evaluated the benefits of using logographic information in language modeling by implementing a multi-modal representation learning model which combines contextual language representations with visual embeddings. our experiments in the nli task in six languages confirmed the benefits of logograhic information in obtaining more reliable semantic representations, especially under sparse learning settings. as future work we hope to contribute to the development of larger multilingual benchmarks to evaluate the effect of visual information on more languages and linguistic phenomena. our software and the experimental data will be available upon publication.