Consonant is all you need: a compact representation of E nglish text for efficient NLP (2023)

Abstract: In natural language processing (NLP), the representation of text plays a crucial role in various tasks such as language modeling, sentiment analysis, and machine translation. The standard approach is to represent text in the same way as we, as humans, read and write. In this paper, we propose a novel approach to represent text with only consonants which presents a compact representation of English text that offers improved efficiency without sacrificing performance. We exploit the fact that consonants are more discriminative than vowels and by representing text using consonants, we can significantly reduce the overall memory and compute footprint required for storing and processing textual data. We present two alternative representations: ‘consonants-only’, where we completely remove the vowels from the text, and ‘masked-vowels’, where we mask all the vowels into one special symbol. To evaluate our approaches, we conducted experiments on various NLP tasks, including text classification, part-of-speech (POS) tagging, named-entity recognition (NER), and neural machine translation (NMT), in addition to language modeling. Our results demonstrate that the proposed consonant-based representation achieves comparable performance compared to the standard text representation while requiring significantly fewer computational resources. Furthermore, we show that our representation can be seamlessly integrated with existing NLP models and frameworks, providing a practical solution for efficient text processing. Last but not the least, we present a technique to retrieve the vowel information from our processed text representation keeping in mind the need to reproduce text in human readable form in some NLP applications.

## introduction in the field of natural language processing (nlp), text representation and tokenization are closely intertwined and have significant implications for various aspects of an nlp task, such as the vocabulary size, embedding size, and out-of-vocabulary (oov) rates. the standard approach to text representation in english involves including both consonants and vowels, mirroring the way humans read and write. however, this representation may not be an optimal representation for english nlp. in this paper, we propose a novel approach to english text representation in nlp, inspired by the abjad writing system, where only consonants are considered as characters and vowels can be inferred @xcite ). our objective is to develop a compact and efficient representation of english text that not only reduces computational demands but also impacts key aspects of nlp tasks. by focusing solely on consonants, we aim to achieve a significant reduction in vocabulary size, resulting in smaller embedding layers and lower oov rates. we present two alternative representations: the "consonants-only" approach, where vowels are entirely removed from the text, and the "maskedvowels" approach, where all the vowels are masked to a special symbol. by leveraging the inherent discriminative nature of consonants, which carry more linguistic information compared to vowels, we can substantially decrease the memory and compute footprint associated with storing and processing textual data. to evaluate the effectiveness of our proposed consonant-based representations, we conducted experiments on various nlp tasks, including text classification, pos tagging, namedentity recognition (ner), neural machine translation (nmt), and language modeling. our results demonstrate that our approach not only achieves comparable performance to the standard text representation but also leads to significant reductions in vocabulary size, resulting in smaller embedding layers and lower oov rates. these findings highlight the practical benefits of our approach in terms of memory efficiency and improved handling of unseen words. furthermore, we address the need for human readability in certain nlp applications, such as machine translation and question answering, by introducing a technique to retrieve the vowel information from our consonants-only text representations. this ensures that our approach remains practical and applicable in scenarios where reproducing text in its original form is necessary. the contributions of the paper can be summarized as below: the rest of the paper is organized as follows: in section 2 we present related works. in section 3 we present our text representations and the different tasks we conducted to compare the effectiveness of our approach. in section 4, we present the experiments conducted, the results and the discussions on the results. in section 5, we present the conclusions and discuss some possible future works. finally, in section 6, we present some of the limitations of our work. ## related work to the best of our knowledge, we did not find any work in the literature that introduces consonantbased text representations for nlp tasks. however, one of the prominent advantages of using these representations is the reduction in vocabulary size. word-based embedding layer is usually a large layer due to the size of the vocabulary. for that reason, it has a high potential to suffer from overfitting. to address this issue, regularization techniques like embedding dropout can be applied @xcite . on the other side, techniques such as dimensionality reduction @xcite quantization @xcite and distillation networks @xcite can be also be used. although these methods are effective in addressing the issue of embedding overfitting, they do not address the issue by manipulating the vocabulary before embedding them. it would be an interesting to explore the model performance where these methods are combined with vocabulary reduction techniques. switching focus back to the issue of vocabulary size and reduction techniques, methods such as bagof-words and tf-idf were employed, traditionally. however, these approaches suffered from the issue of out-of-vocabulary (oov) words and representing the words with huge sparse embeddings. while these methods excelled in modeling word relations, the oov problem limited their use, particularly in natural language generation tasks. additionally, they also faced the challenge of vocabulary explosion, where a large number of vocabulary items needed to be embedded, most of which had very few occurrences in the text (cf. zipf's law @xcite ). to overcome the limitations of sparse embeddings, dense-embedding representations, such as word2vec @xcite , were proposed that were more efficient and effective as compared to spare representations. on the other hand, using characters as tokens do not suffer from the aforementioned issue. moreover, tokenizing text as characters leads to a very small vocabulary size and no oov issues. however, they do result in longer sequences, which can introduce bottlenecks such as vanishing and exploding gradients in sequence modeling models @xcite . character-based approaches are known to generate meaningless new words. nonetheless, a promising approach is to first input characters into cnns to extract features be-fore feeding them into an rnn layer, as this has been shown to yield good results @xcite @xcite . this character-level approach has also been incorporated into transformer-based architectures @xcite . subword tokenization techniques were introduced as a trade-off between characters and words as tokens. these methods split a given word into chunks of characters called subwords. numerous techniques have been proposed to achieve optimal subword splits for a given text. the main motivation behind this approach is to represent the text with a smaller vocabulary size of subwords without compromising performance on a given task. some of these methods are language-specific, while others are data-driven @xcite . data-driven methods are prevalent in recent language models such as the bert family @xcite , gpt family @xcite , and other large models. examples of these methods include sentencepiece @xcite , wordpiece @xcite , and unigramlm @xcite . most of these methods are based on byte-pair-encoding (bpe) data compression methods @xcite . another direction to mitigate the issues associated with tokenization is to utilize visual representation @xcite . this approach has demonstrated robustness against noise and spelling mistakes. however, it is not widely adopted and may not be sufficiently competitive at its current stage of development @xcite . ## methodology we introduce two variants of consonant-based representations for english text. the first representation involves removing all vowel letters (a, e, i, o, and u), regardless of their case, from the text, resulting in a consonants-only representation. this approach may exclude complete words that consist solely of vowels, such as the determiner 'a' and pronoun 'i'. however, such vowel-only words are rare in english. it is worth noting that certain nlp tasks require preserving the total number of words in the input sequence in the output sequence, as is the case with sequence labeling tasks such as pos tagging. for these tasks, this representation might be less suitable. this leads us to propose our second representation, where we mask all vowels with the symbol '#'. figure 1 illustrates an example of an english statement written in standard english as well as using our introduced representations. ## results and discussion this section presents the results of the experiments conducted in this study. we begin by presenting the text analysis on the text corpora, followed by a discussion of the performance of the proposed representations compared to standard text in the context of language modeling. next, we compare the performance of the proposed representations with standard english text across various nlp tasks. additionally, we report the results on retrieval of the standard text representation from the proposed vowels-free representations. finally, we conclude this section by providing further comparisons between these representations. ## conclusion and future work in this paper, we introduced two novel text representations based on english consonants. the first representation completely removes vowels from the input sequence, while the second representation masks vowels to a unique symbol. we conducted a comprehensive analysis of these representations by comparing their corpus statistics, including vocabulary and token sizes, text entropy, as well as the language model perplexity. additionally, we evaluated the performance of these representations on various nlp tasks and compared the results with those obtained using the standard english text representation. the experimental results demonstrated that the proposed representations achieved competitive performance without significant sacrifices. notably, the models based on these representations exhibited smaller model sizes and a reduced number of parameters. this indicates the potential for more efficient training and deployment of models using these representations. in future work, our focus will be on optimizing the architectures specifically for the proposed representations to achieve superior performances compared to the standard english text. we aim to evaluate the representations on a broader range of nlp tasks, such as text summarization and question answering, to assess their generalization abilities. further investigation will be conducted to improve vowel retrieval.