On-Device Text Representations Robust To Misspellings via Projections (2021)

Abstract: Recently, there has been a strong interest in developing natural language applications that live on personal devices such as mobile phones, watches and IoT with the objective to preserve user privacy and have low memory. Advances in Locality-Sensitive Hashing (LSH)-based projection networks have demonstrated state-of-the-art performance in various classification tasks without explicit word (or word-piece) embedding lookup tables by computing on-the-fly text representations. In this paper, we show that the projection based neural classifiers are inherently robust to misspellings and perturbations of the input text. We empirically demonstrate that the LSH projection based classifiers are more robust to common misspellings compared to BiLSTMs (with both word-piece & word-only tokenization) and fine-tuned BERT based methods. When subject to misspelling attacks, LSH projection based classifiers had a small average accuracy drop of 2.94% across multiple classifications tasks, while the fine-tuned BERT model accuracy had a significant drop of 11.44%.

## introduction at the core of natural language processing (nlp) neural models are pre-trained word embeddings like word2vec @xcite , glove @xcite and elmo @xcite . they help initialize the neural models, lead to faster convergence and have improved performance for numerous application such as question answering @xcite , summarization @xcite , sentiment analysis @xcite . while word embeddings are powerful in unlimited constraints such as computation power * work done during internship at google † work done while at google ai and compute resources, it becomes challenging to deploy them to on-device due to their huge size. figure 1 : one-hot word embedding look-up vectors vs linear combination of lsh projection based vectors @xcite representing the same word. this led to interesting research by @xcite , who showed that word embeddings can be replaced with lightweight binary locality-sensitive hashing (lsh) based projections learned on-the-fly. the projection approach surmounts the need to store any embedding matrices, since the projections are dynamically computed. this further enables user privacy by performing inference directly on device without sending user data (e.g., personal information) to the server. the embedding memory size is reduced from o(v ) to o(k), where v is the token vocabulary size and @xmath0, is the binary lsh projection size. the projection representations can operate on either word or character level, and can be used to represent a sentence or a word depending on the nlp application. for instance, recently the projection sequence networks (proseqo) (kozareva and ravi, 2019) used bilstms over word-level projection representations to represent long sentences and achieved close to state-ofthe-art results in both short and long text classification tasks with varying amounts of supervision and vocabulary sizes. despite being successful, there are no existing systematic research efforts focusing on evaluating the capabilities of the lsh based projection for text representations. to that end, we empirically analyze the effectiveness and robustness of the lsh projection approach for text representation by conducting two types of studies in this paper. overall, our studies showcase the robustness of lsh projection representations and resistance to misspellings. due to their effectiveness, we believe that in the future, text representations using lsh projections can go beyond memory constrained settings and even be exploited in large scale models like transformers @xcite . ## binary lsh projections for text representations the dependency on vocabulary size v , is one of the primary reasons for the huge memory footprint of embedding matrices. it is common to represent a token, x by one-hot representation, y(x) ∈ [0, 1] v and a distributed representation of the token is obtained by multiplying the one-hot representation with the embedding matrix, one way to remove the dependency on the vocabulary size is to learn a smaller matrix, w @xmath1, as shown in figure 2 . for instance, 300-dimensional glove embeddings, w v @xcite with 400k vocabulary size @xmath2.2 mb for @xmath3. instead of learning a unique vector for each token in the vocabulary, we can think of the columns of this w k matrix as a set of basis vectors and each token can be represented as a linear combination of basis vectors in w k as in figure 1 . we select the basis vectors from w k for each token with a fixed k-bit binary vector instead of a v -bit one-hot vector. the lsh projection function, p (figure 3 ) @xcite used in sgnn @xcite and proseqo (kozareva and ravi, 2019) does exactly this as it dynamically generates a fixed binary projection representation, p(x) ∈ [0, 1] k for any token, x by extracting morphological input features like char (or token) n-gram & skip-gram features, parts of speech tags etc. from x and a modified locality-sensitive hashing (lsh) based transformation, l as in as for the projection based classifiers, the proseqo model (kozareva and ravi, 2019) runs a bil-stm over word-level binary lsh projection representations to predict the correct classes, while the sgnn model @xcite computes a binary lsh projection representation for the entire input text, followed by a 2-layer mlp and a softmax layer on top of it for class prediction. sgnn was designed for short text, while proseqo is also suitable for long text classification tasks. there have been a number of research efforts @xcite @xcite to improve the robustness of neural classifiers to misspelling attacks and other text transformations. recently, @xcite observe that fine-tuned bert and bilstm based models are very brittle (for e.g., accuracy drops from 90.3% to 45.8% in the sst @xcite classification task) to adversarial misspelling attacks. contrary to intuition, they observe that word-piece and character-level models are more susceptible to spelling attacks compared to the word-level models. the lsh projection operation, p, is a function of n-grams (and skip-grams) of the input text x and usually the fraction of n-grams affected by spelling attacks tend to be minimal resulting in insignificant changes to the projection representation, p(x). therefore, we hypothesize that the projection based models like proseqo, sgnn, etc. are inherently robust to commonly occurring spelling attacks. in the following sections, we investigate the robustness of projection based classifiers by subjecting them to common misspellings, followed by an analysis of changes in the binary lsh projections of input text under such transformations. ## effect of misspellings on text classification we study the robustness of two types of projection based models -proseqo and sgnn. on the other hand, we fine-tune the pretrained bert-base model (with word-piece tokenization) @xcite and train two-layer bilstms (with both word-only and word-piece tokenization) for comparable accuracies with respect to the projection based models for a fair comparison. by word-only tokenization, we mean that models encode input words using a lookup table for each word. in our setup, we test the robustness of the neural classifiers by subjecting the corresponding test sets to common misspellings and omissions. we consider the following perturbation operations: randomly dropping, inserting, and swapping internal characters within words of the input sentences @xcite foot_0 . we decide to perturb each word in a sentence with a fixed probability, p perturb . following @xcite , we fix the projection dimension to @xmath4. ## perturbation analysis apart from the classification experiments, we also directly analyze the changes in the binary lsh projection representations by subjecting input text to different types and amount of perturbations. to that end, we take a large corpus enwik9 foot_1 (vocabulary size of 500k and 129m words) to analyze the average hamming distance between lsh projections of the words in the corpus. next, we compute the average changes in the projection representations by subjecting them to the character perturbations from section 3. also from ## conclusion in this work, we perform a detailed study analyzing the robustness of recent lsh-based projection