Refining App Reviews: Dataset, Methodology, and Evaluation (2024)

Abstract: With the growing number of mobile users, app development has become increasingly lucrative. Reviews on platforms such as Google Play and Apple App Store provide valuable insights to developers, highlighting bugs, suggesting new features, and offering feedback. However, many reviews contain typos, spelling errors, grammar mistakes, and complex sentences, hindering efficient interpretation and slowing down app improvement processes. To tackle this, we introduce RARE (Repository for App review REfinement), a benchmark dataset of 10,000 annotated pairs of original and refined reviews from 10 mobile applications. These reviews were collaboratively refined by humans and large language models (LLMs). We also conducted an evaluation of eight state-of-the-art LLMs for automated review refinement. The top-performing model (Flan-T5) was further used to refine an additional 10,000 reviews, contributing to RARE as a silver corpus.

## introduction the mobile app landscape has seen immense growth, with millions of apps providing essential services @xcite . app stores host hundreds of millions of reviews @xcite , but only a fraction offer truly informative insights @xcite . user feedback is crucial for developers, offering insights into experiences, bugs, and feature suggestions @xcite . however, reviews often contain informal language, mixed sentiments, and varied expressions, complicating manual analysis. for example, consider the following review from spotify "love this app! but it crashes all the time. super frustrating! fix it plz!". this review combines positive feedback with criticism. in addition to this, reviews often include typos, grammatical errors, non-english words, slangs, app-specific jargons and subjective phrases such as "kinda get boring" and "super-addictive". consider another example, from netflix app review: "i love netflix but it's genuinely making me angry that i can't make my brightness higher bc the app is in control of my brightness panel. so i'm at lunch sitting outside and i can't see the screen cuz i can't make the brightness higher because for some reason netflix is in control. it's frustrating for sure". this review clearly expresses user frustration, yet is riddled with informal language("cuz"), irrelevant details ("so i'm at lunch") , non-standard abbreviations ("bc") and unclear statements ("bc the app is in control of my brightness panel"), making it difficult for app developers to manually analyze the core concerns of the user. refining these reviews is essential for enhancing app functionality and improving the overall user experience. while transformer-based language models have excelled in refining natural language text for various downstream tasks such as enhancing code readability @xcite ) and question refinement @xcite , app review refinement remains an unexplored area. we introduce rare (repository for app review refinement), a new benchmark dataset containing 10,000 annotated pairs of original and refined app reviews from 10 different mobile applications. these reviews were generated using state-of-the-art llms and the expertise of experienced software engineers. we identified five prevalent issues in app reviews and the necessary operations to rectify them. using prompt engineering @xcite , we designed six prompts to guide gpt-3.5-turbo @xcite in refining the reviews. the best-performing prompt was used to generate 10,000 refined reviews, which were then reviewed and corrected by five software engineers with over five years of experience in app development. this formed the gold-standard corpus for rare. these original and refined review pairs from the gold-standard corpus were used to fine-tune state-of-the-art llms for app review refinement, including bart @xcite , flan-t5 @xcite , pegasus @xcite , llama-2 @xcite , falcon @xcite , mistral @xcite , @xcite , and gemma @xcite . we evaluated these models using human evaluation metrics and standard automatic metrics including, system output against references and against the input sentence (sari) @xcite , bertscore precision (bp) @xcite , flesch-kincaid grade level fkgl @xcite , flesch-kincaid reading ease (fkre) @xcite , and average length (len) @xcite . flan-t5 emerged as the most effective model based on both human and automatic metrics. we then used flan-t5 to automatically refine 10,000 additional reviews, creating silver-standard corpus for rare. refined reviews provide several advantages for the app development community: a) improved user feedback analysis-refined reviews offer developers clearer insights into user sentiments, facilitating better-informed decisions regarding feature enhancements, bug fixes, and user experience improvements. b) standardization-the refinement process helps standardize the analysis of app reviews. our preliminary experiments (reported in section 4.2) indicate that refined reviews yield better results in classifying reviews into bug reports, feature requests, and user experience compared to raw reviews. c) efficient resource allocation-by clarifying user sentiments and common issues, refined reviews enable development teams to allocate resources more effectively, enhancing overall productivity. the key contributions of our paper are: ## related work significant efforts have been made on refining natural language text outputs, including summarization @xcite , where extracted sentences are refined by omitting unimportant words or phrases before summary generation; content planning @xcite , that devises an iterative refinement algorithm to improve incorrectness and incoherence of generated content; questions refinement @xcite , aimed to refine questions by improving readability; and so on ((hua and wang, 2020); @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ). these works predominantly utilize llms to refine text. however, llms often face challenges when handling complex text. this difficulty is especially evident in tasks with multifaceted objectives or tasks with hard-to-define goals, such as enhancing program readability @xcite . in the domains of text simplification and lexical normalization, significant progress has been made, from early rule-based methods @xcite to statistical models ( @xcite ; @xcite ; @xcite ; @xcite ; @xcite ). the introduction of transformer-based models such as bert and gpt has advanced the field, achieving top results in domains such as medical, legal, clinical, news, and wikipedia texts ( @xcite ; @xcite ; @xcite ; @xcite ). we acknowledge that simplification and refinement are related concepts and can overlap in some cases, but their goals are different. while simplification aims to make text easier to understand, refinement in our context focuses on making reviews more actionable for developers by ensuring clarity, removing irrelevant details, and maintaining technical accuracy. the works reported above primarily focus on refining outputs based on a single objective. in contrast, our task of refining app reviews encompasses multiple facets, including ensuring grammatical accuracy, rephrasing, removing irrelevant words and information, rearranging words and information, and modifying sentences. these tasks necessitate careful handling due to their nuanced and diverse requirements. furthermore, none of the existing literature specifically addresses app review refinement. to the best of our knowledge, our work represents the first effort in the area of app review refinement. ## rare: a new benchmark dataset due to the absence of a ground truth for automated app review refinement, we created the rare (repository for app review refinement) dataset. rare includes 10,000 annotated pairs of raw and refined reviews as the gold corpus, and an additional 10,000 reviews refined by the bestperforming model as the silver corpus. this dataset benchmarks llms and other machine learning models, aiding future research in automated app review refinement. figure 1 provides an overview of our dataset collection, analysis, and refinement process. ## conclusions and future work in this work, we introduce rare, a benchmark for app review refinement. rare comprises a corpus of 10,000 annotated reviews, collaboratively refined by humans and llms sourced from 10 different application domains, constituting the gold corpus. additionally, it includes a set of 10,000 automatically refined reviews, forming the silver corpus. we evaluated eight stateof-the-art models and determined that flan-t5 is the best-performing model for app review refinement. the complete rare benchmark and code are included in the supplementary material, establishing rare as a benchmark in text refinement for app development. future work may focus on two directions: first, extracting non-functional requirements from app reviews and assessing how app review refinement enhances this process compared to using raw reviews; second, summarizing the extracted requirements from these app reviews.