The Overlooked Repetitive Lengthening Form in Sentiment Analysis (2024)

Abstract: Individuals engaging in online communication frequently express personal opinions with informal styles (e.g., memes and emojis). While Language Models (LMs) with informal communications have been widely discussed, a unique and emphatic style, the Repetitive Lengthening Form (RLF), has been overlooked for years. In this paper, we explore answers to two research questions: 1) Is RLF important for SA? 2) Can LMs understand RLF? Inspired by previous linguistic research, we curate **Lengthening**, the first multi-domain dataset with 850k samples focused on RLF for sentiment analysis. Moreover, we introduce **Explnstruct**, a two-stage Explainable Instruction Tuning framework aimed at improving both the performance and explainability of LLMs for RLF. We further propose a novel unified approach to quantify LMs’ understanding of informal expressions. We show that RLF sentences are expressive expressions and can serve as signatures of document-level sentiment. Additionally, RLF has potential value for online content analysis. Our comprehensive results show that fine-tuned Pre-trained Language Models (PLMs) can surpass zero-shot GPT-4 in performance but not in explanation for RLF. Finally, we show ExpInstruct can improve the open-sourced LLMs to match zero-shot GPT-4 in performance and explainability for RLF with limited samples. Code and sample data are available at https://github.com/Tom-Owl/OverlookedRLF

## introduction informal styles are prevalent on social media platforms, where people use nuanced expressions to share opinions and emotions personally and engagingly @xcite @xcite . previous research has explored various informal styles such as meme @xcite , emoji @xcite @xcite , slogan @xcite and abbreviation @xcite . however, it remains a challenge for language models (lms) to understand the sentiment in nuanced and subtle linguistic expressions, which require a deep contextual and cultural understanding @xcite . this work delves into one specific informal expression -repeated lengthening form (rlf), which refers to the linguistic phenomenon where additional characters are added to the standard spelling of a word to enhance or alter its conveyed meaning @xcite . we further generalize the concept of rlf and divide it into two types: repetitive letters (e.g., 'loooove') and repetitive punctuations (e.g., 'love!!!!'). our study finds that an average of 5.8% documents possess rlf among 4 public datasets and 5 domains ( lms consist of pre-trained language models (plms) @xcite such as roberta @xcite , t5 @xcite , and gpt-2 @xcite , and large language models (llms), including @xcite and llama2 @xcite . lms exhibit impressive performance on many nlp tasks @xcite . however, we know little about the boundaries of performance and explainability of lms for rlf. such linguistic features, common in daily communication, have yet to be thoroughly investigated @xcite @xcite @xcite . the lack of a specialized dataset for rlf impedes us from evaluating and improving lms to learn the nuanced communications in real-world and online social media content. this research gap raises two research questions: 1) is rlf important for sa? 2) can lms understand rlf? in this study, we aim to evaluate and improve the performance and explainability of plms and llms for rlf. ## related work repetitive lengthening form (rlf) the study by @xcite shows that rlf is a written emulation of nonverbal spoken cues in user-generated content and proposes a method to reduce stretched words to their root words. @xcite highlights the importance of accurately interpreting rlf for sa and employing rlf to augment sentiment dictionary @xcite . a large-scale dataset lengthening with rlf for sa. we use the dataset to evaluate and improve the performance and explainability of the up-to-date plms and llms for rlf. instruction tuning is a paradigm that fine-tunes language models on multiple tasks with instructioninput-output pairs to improve performance and generalize to unseen tasks @xcite . emerging work explores instruction tuning to tasks such as text editing @xcite , information extraction @xcite and classification @xcite . we extend this line of work by instruct-tuning llms for sa with rlf. we focus on a novel task of predicting the documentlevel sentiment label using only one sentence. honovich et al. (2023); yin et al. (2023) use in-context learning strategies to prompt llms to automatically generate data for instruction tuning. lampinen et al. (2022); zhang et al. (2023c); zhou et al. (2023) instruct llms with emphasis on explainability. inspired by those works, we prompt gpt-4 with chain of thought (cot) @xcite to generate word importance scores for rlf sentences. using these scores along with the ground truth document-level sentiment labels, we automatically generate prompts for explainable instruction tuning of llms. ## the lengthening dataset we introduce the lengthening dataset in this section. we need a few definitions first: rlf sentence is a sentence with one or more rlf words, rlf document is a document (e.g., user review) with at least one rlf sentence. we present an overview of data statistics for our dataset and present in table1. more detailed examples from our dataset can be found in ## method we introduce expinstruct, a two-stage instruction tuning framework aimed to improve both performance and explainability of llms for rlf. expinstruct first prompts gpt-4 for wis scores and then finetunes llms with instructions for wis and sa. we further propose a unified approach to evaluate the comprehension level of lms for rlf. formally, consider (x, rlf, y) to be a single tuple in our dataset d, where x is an rlf sentence with an rlf word rlf and y is the document-level instruction for wis 1. decompose the sentence into words, preserving punctuation (for example, 'love!!!!', 'gooood.'). 2. assign word important scores (1-5) based on conveyed sentiment. 3. organize results as a list of tuples (word, score) keeping word order like [(w0, s0), (w1, s1)....]. note: directly and only return expected output. now input: {input} (a) (b) instruction for sa give the input sentence a sentiment label (1: positive, 0:negative). note: directly and only return 1 or 0 now input: {input} <@xmath0:{input}[/inst]{ouput} prompt template for expinstruct (c) sentiment label. we denote a transformer-based model by f . 4.1 expinstruct prompt design for explanation we prompt gpt-4 with cot to generate word importance scores (wis) to reflect word-level understanding of input sentence x as shown in figure 2(a). the cot consists of 3 sequential reasoning steps. 1) sentence decomposition: segment sentences into words and keep punctuation marks with a few-shot strategy. 2) word importance scoring: assign sentiment importance scores (1-5) to each word, which can reflect llms' understanding of word-level sentiment. 3) structured output: we specify the structured output format for subsequent analysis. instruction template expinstruct has two tasks with the same prompt template as shown in figure 2(c). this is achieved by adding three placeholders: {task instruction}, {input}, and {output}. the first task is instruction for wis: the task instruction is shown in figure 2(a), with the input as rlf sentence x and the output is the structured output generated by gpt-4 for wis. the second task is instruction for sa: the task instruction is shown in figure 2(b); the input is the rlf sentence x and the output is the document-level sentiment label y. ## experimental setup in this section, we introduce baseline models, experimental design and implementation details. baseline models to explores the boundaries of performance and explainability for sota plms and llms for rlf, we choose 3 fine-tuned 3 plms for sa task with backbone models as roberta (large), gpt-2 (medium) and t5 (base). these models have comparable parameter scales and represent encoder-only, decoder-only, and encoderdecoder architectures. for llms, we use gpt-4 via the openai api. moreover, we select llama2 (13b-chat-hf) for instruct-tuning because it's opensourced with auto-regressive architecture like gpt-4. and the scalable parameter size of llama2 (13b) allows inference and fine-tuning with lora @xcite on a single gpu. more details of these baseline models and api usage are provided in the appendix b.1 implementation we fine-tune 3 plms on the entire lengthening dataset. the batch sizes are set to 64 for roberta and t5, and 32 for gpt-2 to optimize gpu memory usage. we sampled 3,000 instances from lengthening as a subset dataset for experiments with llms with stratified sampling method based on domains, where we used 1.6k instances for training and others for validation and testing. this sample size aligns with previous studies @xcite @xcite . more details of data split see appendix a.2. we instruct-tune llama2 with the our expinstruct framework with lora and supervised finetuning (sft) @xcite to lower computational costs. we set the temperature at 0.2 for all llms. all models are trained for a maximum of 5 epochs with k-fold cross-validation strategy (@xmath1. in each iteration one fold serves as the test set while the remaining data is split into train/val with 4/1 ratio. it takes one week to conduct all experiments with a single rtx a6000 gpu with 50 gb of memory. ## analysis explanation reliability we ask the annotators to evaluate the reliability of the wis generated by the zero-shot gpt-4 and the four fine-tuned lms (roberta, gpt-2, t5, and llama2) with criteria of 1: agree, 0: disagree. the final reliability score for each sample was determined by averaging scores among annotators. we report a moderate overall iaa score of 0.44 and show detailed result in