Coding Textual Inputs Boosts the Accuracy of Neural Networks (2020)

Abstract: Natural Language Processing (NLP) tasks are usually performed word by word on textual inputs. We can use arbitrary symbols to represent the linguistic meaning of a word and use these symbols as inputs. As “alternatives” to a text representation, we introduce Soundex, MetaPhone, NYSIIS, logogram to NLP, and develop fixed-output-length coding and its extension using Huffman coding. Each of those codings combines different character/digital sequences and constructs a new vocabulary based on codewords. We find that the integration of those codewords with text provides more reliable inputs to Neural-Network-based NLP systems through redundancy than text-alone inputs. Experiments demonstrate that our approach outperforms the state-of-the-art models on the application of machine translation, language modeling, and part-of-speech tagging. The source code is available at https://github.com/abdulrafae/coding_nmt .

## introduction we introduce novel coding schemes on the inputs of neural-network-based natural language processing (nn-nlp) that significantly boost the accuracy in three applications. the inputs of nn-nlp rely on observable forms of mental representations of linguistic expressions, and allow alternative designs. for example, both logographic kanji and syllabic kana represent japanese words, and emoticons and emojis can express sentiments. these showcase that alternative human language representation than text is possible and highlight a common belief of most linguists: the relationship between the mental representations and their phonological forms is highly arbitrary, even though a non-arbitrary (de saussure, 1916) mapping exists for some special cases, e.g., the bouba/kiki effect. in our work, we ask -are there alternative forms of mental representation in addition to text as we see in japanese and internet language to help language understanding in nn-nlp? to answer this question, we blend concepts from linguistic phonetics, grammatology, and the statistics of zipf law to find alternative language representations to text. more precisely, we code a textual word either naturally or artificially by exploring different facets of human languages, from phonetic and logogram codings to new coding constructions generalizable to all languages. natural codings inspire the finding of artificial codings, which in turn helps us understand and explain natural codings. all of our codings reinforce nlp inputs by reconstructing the character/symbol sequence of a word in various ways with a new alphabet. these variants and their "decomposition" are expressive because they contain insightful information about linguistic patterns in units smaller than words and even smaller than characters. for example, in the logogram wubi (that lists in a coded form the strokes caligraphing a chinese character), "众" (crowd) is coded as "www", which is made of three "人" (person, "w"), and "从" (follow, "ww") is a composition of two "人". a representation containing such granular details potentially reveals the semantic structure and linguistic meanings inside a word, thus enriching text and allowing a redundancy that ensures more reliable nlp inputs. now that we have put our previous question in context let us give an overview of how we incorporate coding schemes into an nlp framework in figure 1 . for an input sentence, we apply an alternative coding scheme word by word, then use byte-pair-encoding (bpe) to recombine these symbols (to shorten the input lengths), and finally perform embeddings (emd). in contrast to word embeddings that map words to real number vectors, our coding range is discrete. the coded sentence and its original textual input are then combined in three ways: concatenation, linear-interpolation at the encoder level, and multi-source encoding with or without bi-lstm, attention, and multi-head attention. the combined input is fed into nn-nlp models as a black-box to decode outputs. our approach is language-, task-, and system-independent and does not use any additional information besides our algorithms. we conduct experiments on three nlp applications and five languages, including (1) machine translation (mt) on english-german, german-english, english-french, french-english, and chinese-english; (2) language modeling (lm) on english; and (3) part-of-speech (pos) tagging on english. our approach significantly and consistently improves over state-of-the-art neural models: transformer, convs2s, xlm, and bi-lstm with attention mechanisms. in summary, our contribution mainly lies in the three consecutive folds: ## coding words we view each coding as a function γ that maps a textual word from @xmath0, a natural language vocabulary, into a codeword γ(x) ∈ v, a codeword vocabulary: for simplicity of exposition we will consider v to be the image of v under γ. each codeword γ(x) is a non-empty σ-string over the alphabet σ of this coding: γ(x) = σ 1 , σ 2 , σ 3 • • • σ l with code length l. σ + is an infinite set of all possible non-empty strings over σ, and v ⊆ σ + . as an example (albeit one which is practically not useful) consider the mapping of four english words to three binary codewords: @xmath1, "be", "or", "not"}, σ = {0, 1}, σ + = {0, 1, 00, 01, 10, 11, • • •}, @xmath2, 01, 11}, @xmath3, γ("to") = 00, γ("be") = 01, γ("or") = 11, γ("not") = 01, |v| = 4, and |v| = 3. to instantiate this function, we start by introducing several existing linguistically-motivated coding schemes (and later on we will extend this to new coding schemes we develop): the phonetic and logogram coding as surjective functions, where in particular |v| ≥ |v|; and the fixed-output-length and huffman coding as bijections, where |v| = |v|. in traditional coding theory, a compression code has to be injective in order to be uniquely decodable. in our work, we only care about the taskspecific prediction and not in decoding the original message. therefore, we relax the injective restriction on the codings to deviate a little from the standard typical coding theory applications for technical convenience. throughout this paper, we choose to name the function γ as "coding" (although sometimes it is also called "encoding") to distinguish from the encoder in the nn-nlp models. an overview of our coding schemes is illustrated in figure 2 . meta-phone p620 p620 … pars cala … prs klr … nysiis soundex text voc. purse parse … phonetic coding many-to-one @xmath4. the a … fixed-output-length coding one-to-one 100 221 … 110 1011 … the a … huffman coding one-to-one 1 0 … 0 1 code length: log b |v| wubi w ww … text voc. … logogram coding pinyin coding many-to-one pinyin ren zhong … figure 2: examples on different coding schemes. in contrast to pinyin only applies to chinese, the logogram coding wubi and its variant apply to japanese kanji and chinese. furthermore, phonetic codings, including metaphone, soundex, and nysiis, cover most western languages. finally, the artificial codings, i.e., the fixed-output-length and huffman coding, can be applied to any language. phonetic and logogram codings are many-to-one mappings, while fixed-output-length and huffman coding are one-to-one mappings. ## related work previous important work investigated the role of auxiliary information to nlp tasks, such as polysemous word embedding structures by @xcite , factored models by @xcite , and feature compilation by @xcite . we emphasize that we do not use any additional information besides our algorithms. @xcite ; @xcite applied explicit phonological rules or constraints to tasks such as word segmentation. in neural networks, we can implicitly learn from phonetic data and leave the networks to discover hidden phonetic features through end-to-end training opt specific nlp tasks, instead of applying hand-coded constraints. closely related, but independent to our work, is the character-based mt, such as the work of @xcite and @xcite , among many others. we go beyond text level representations and look for novel representations for decompositions, sometimes even smaller than characters. different from the inspiring work that uses pinyin @xcite , skip-ngram @xcite , and huffman on source/target @xcite , our study aims to improve nn-nlp including nmt overall rather than only eliminating unknown words, introducing six new codings into nlp in addition to pinyin and text. importantly, our artificial codings apply on all languages. moreover, we achieve experimental improvements overall. @xcite added pinyin embedding to robustify nmt against homophone noises. they described that it was unknown why pinyin also improved predictions on the clean test. this is a very interesting work, and we explain this phenomenon through our theory that the multi-channel coding offers an ensemble of the code words and the text, making the communication more reliable. ## conclusion in this paper, we conduct a comprehensive study on how to code textual inputs from multiple linguistically-motivated perspectives and how to integrate alternative language representations into nn-nlp systems. we propose to use soundex, nysiis, metaphone, logogram, fixed-outputlength, and huffman codings into nlp and describe how to combine them in state-of-the-art nn architectures, such as transformer, convs2s, bi-lstm with attentions. our paradigm is general for any language and adaptable to various models. we conduct extensive experiments on five languages over six tasks. our approach appears to be very useful and achieves up to 20.77%, 20%, and 15.79% relative improvements on state-of-the-art models of mt, lm, and pos, respectively.