E mp L ite: A Lightweight Sequence Labeling Model for Emphasis Selection of Short Texts (2020)

Abstract: Word emphasis in textual content aims at conveying the desired intention by changing the size, color, typeface, style (bold, italic, etc.), and other typographical features. The emphasized words are extremely helpful in drawing the readers’ attention to specific information that the authors wish to emphasize. However, performing such emphasis using a soft keyboard for social media interactions is time-consuming and has an associated learning curve. In this paper, we propose a novel approach to automate the emphasis word detection on short written texts. To the best of our knowledge, this work presents the first lightweight deep learning approach for smartphone deployment of emphasis selection. Experimental results show that our approach achieves comparable accuracy at a much lower model size than existing models. Our best lightweight model has a memory footprint of 2.82 MB with a matching score of 0.716 on SemEval-2020 public benchmark dataset.

## introduction emphasizing words or phrases is commonly performed to drive a point strongly and/or to highlight the key terms and phrases. while speaking, speakers can use tone, pitch, pause, repetition, etc. to highlight the core of a speech in the minds of an audience. similarly, while writing or messaging, authors can emphasize the words by customizing the formatting like typeface, font size, bold, italic, font color, etc. as illustrated in figure 1 . with the explosion of social media and messaging platforms, word emphasis has become more critical in engaging readers' attention and conveying the author's message in the shortest possible time. emphasis selection of text has recently emerged as a focus of research interest in natural language processing (nlp). the goal of emphasis selection is to automate the identification of words or phrases that bring clarity and convey the desired meaning. automatic emphasis selection can help in better graphic designing and presentation applications, as well as can enable voice assistants and digital avatars to realize expressive text-to-speech (tts) synthesis. high-quality emphasis selection models can enable automatic design assistance for creating flyers, posters and accelerate the workflow of design programs such as adobe spark @xcite , microsoft pow-erpoint, etc. these emphasis selection models can also empower digital avatars like samsung @xcite to achieve humanlike tts systems. understanding emphasis selection is also crucial for many downstream applications in nlp tasks including text summarization, text categorization, information retrieval, and opinion mining. in the current work, we propose a novel lightweight neural architecture for automatic emphasis selection in short texts, which can perform inference in a low-resource constrained environment like a smartphone. our proposed architecture achieves near-sota performance, with as low as 0.6% of its model size. ## related work prior work in nlp literature towards identifying important words or phrases have focused widely on keyword or key-phrase extraction. considerable progress has been made in keyword or key-phrase extraction systems for long documents such as news articles, scientific publications, etc. @xcite . the core operation procedure of these systems is to extract the nouns and noun phrases. to achieve these, researchers have used statistical co-occurrence @xcite , svm @xcite , crf @xcite , graph-based extraction @xcite , etc. recent efforts have even expanded the idea from a set of documents to social big data (kim, 2020). however, in the context of short texts like text messages, headlines, or quotes, keyword extraction systems often mislabel most nouns as important without considering the essence of the text, thus performing poorly at the task. emphasis selection aims to overcome this by scoring words which properly capture the essence of a text by focusing on subtle cues of emotions, clarifications, and words that capture readers' attention, as seen in thus, while keyword extraction systems are not suitable for short text content, emphasis selection systems perform much better at such tasks. however, most existing architectures of the latter are not light-weight, and thus, not suitable for on-device inferencing on lowresource devices. this motivates us to propose emplite, which (a) outperforms keyword extraction systems by using emphasis selection for use with short texts, and (b) differs from existing emphasis selection architectures by ensuring a very light-weight model for efficient on-device inferencing on mobile devices. our decisions towards achieving low model size include using a subset of glove @xcite word embeddings, thus, reducing embedding size from 347.1 mb to 2.5 mb, which we discuss in section 4.1. ## data collection we use the officially released semeval-2020 dataset (ritual-uh, 2020), which is the combination of spark dataset @xcite and wisdom quotes dataset (quotes, 2020). the dataset consists of 3,134 samples labeled for token-level emphasis by multiple annotators. there are 7,550 tokens with fewer than 10 words in a sample and they are randomly divided into training (70%), development (10%), and test (20%) sets by the organizers. ## system overview we begin with a basic model and enhance that model with contextual information (in the form of pre-trained embeddings, char-level embeddings, parts of speech tag concatenation, etc.). we describe the key components (layers) of our final emplite neural network architecture, as illustrated in figure 2 . ## experimental settings & results we attempt numerous small changes to our model to enhance the performance. we choose the hyperparameters to optimize accuracy while maintaining a small model size. as the proposed solution is for mobile devices, we have also captured a system-specific metric, the model size in mb. we use the tensorflow framework @xcite for building the models. the total number of trainable parameters vary in the range of 21,574 to 238,620 for all the model results reported in we can also infer that using 50-dimensional glove embeddings and setting it as trainable improves the overall matching score. this is because we are utilizing language semantic knowledge acquired from the pre-trained embeddings and then fine-tuning it for our task. however, there is an increase in model size due to more number of trainable parameters. furthermore, we observe marginal gains in the matching score by adding pos tag as a feature followed by an attention layer as these help in a better identification of prominent keywords based on the context. figure 4 presents the emphasis heatmap for some examples from the test set using our final model. in ## conclusion modeling lightweight neural models that can run on low-resource devices can greatly enhance the end-user experience. in this work, we propose a novel, lightweight emplite model for text emphasis selection that can run on edge devices such as smartphones for choosing prominent words from short, informal text. we approach the emphasis selection problem as a sequence labeling task and multiple experiments have shown consistent improvement in the accuracy. our experimental results show the impact of the attention layer and of using pos as an additional feature in boosting the matching score. our best performing model achieves an overall matching score of 0.716 with a size of 2.82 mb, proving its effectiveness to run on low-resource edge devices. future work includes increasing the vocabulary with commonly used words in english and exploring thin versions of bert like distil-bert @xcite , mobilebert @xcite , and tinybert @xcite for modeling emphasis.