Enhancing Editorial Tasks: A Case Study on Rewriting Customer Help Page Contents Using Large Language Models (2024)

Abstract: In this paper, we investigate the use of large language models (LLMs) to enhance the editorial process of rewriting customer help pages. We introduce a German-language dataset comprising Frequently Asked Question-Answer pairs, presenting both raw drafts and their revisions by professional editors. On this dataset, we evaluate the performance of four large language models (LLM) through diverse prompts tailored for the rewriting task. We conduct automatic evaluations of content and text quality using ROUGE, BERTScore, and ChatGPT. Furthermore, we let professional editors assess the helpfulness of automatically generated FAQ revisions for editorial enhancement. Our findings indicate that LLMs can produce FAQ reformulations beneficial to the editorial process. We observe minimal performance discrepancies among LLMs for this task, and our survey on helpfulness underscores the subjective nature of editors’ perspectives on editorial refinement.

## introduction in this paper, we evaluate the suitability of large language models to support the editorial process of customer help pages. the continuous evolution of natural language processing (nlp) technologies, particularly exemplified by advanced models like gpt-4 (team, 2023), presents exciting prospects for content management across various sectors. one area where these models hold promise is in the maintenance and enhancement of customer help pages, which serve as vital resources for addressing user queries and concerns related to products or services. the editorial workflow for customer help pages necessitates precision, clarity, and relevance to ensure users can efficiently locate solutions. tradi-tionally, this workflow involves manual content creation, review, and updates by human editors. however, managing the volume of content and keeping information current pose significant challenges. large language models offer a compelling opportunity to enhance and expedite these editorial processes, potentially boosting efficiency and responsiveness to user needs. our objective is to explore practical applications of large language models in supporting essential editorial tasks for customer help pages. we will investigate how these models can contribute to content creation and quality control. by evaluating the advantages and constraints of incorporating such models into the editorial workflow, we aim to provide insights into their feasibility and effectiveness within customer support operations. this evaluation is essential for understanding how large language models can impact the scalability and responsiveness of customer help services in the digital era. the main contributions of this paper are: ## related work the application of llms for rewriting texts covers a variety of text generation tasks, such as summarizing @xcite ), text simplification @xcite , style transfer @xcite or query rewriting @xcite . the evaluation datasets often cover only one of those tasks, however multi-purpose benchmarks have started emerged in recent years. @xcite created editeval, an instruction-based suite that leverages high-quality existing and new datasets to automatically assess editing capabilities, including enhancing text fluency and clarity, as well as rewriting to simplify, neutralize, or update content. it covers various text types such as wikipedia articles, wikinews, news articles, and scientific publications from arxiv. the benchmark is provided with results of baselines, which use greedy decoding and do not perform any task-specific fine-tuning or in-context learning. the authors evaluate various llms using zeroshot prompting. the evaluation reveals that most baseline models lag behind the supervised stateof-the-art, especially in tasks like neutralizing and updating information. the analysis also indicates that commonly used metrics for editing tasks do not always correlate well, and optimizing for the highest-performing prompts does not necessarily ensure robustness across different models. @xcite created a benchmark open-rewriteeval by collecting human-generated text rewrites with natural language instructions. the benchmark is designed for testing cross-sentence rewrite of various types, such as text formality, expansion, conciseness, paraphrasing, tone and style transfer. the authors also developed rewritelm, an instruction-tuned large language model designed for cross-sentence text rewriting. the model undergoes supervised fine-tuning and reinforcement learning (rl). for instruction tuning, edits from wikipedia are extracted and filtered, and the associated edit summary of the revision is used as a proxy for the instructions. additionally, to diversify the dataset a synthetic set of instructions is generated using chain-of-thought prompting and post-processing. the authors tested rewritelm on editeval and openrewriteeval and compared the results against a set of models, including various palm variants, llama, alpaca, gpt-3, insgpt. @xcite addresses the problem of impracticality of large language models for the rewriting task on mobile-device due to models size. the authors recognize that developing a smaller, effective language model for text rewriting is challenging due to the need to balance size with maintaining capabilities, which requires expensive data collection. to tackle the challenge, a new instruction tuning method for mobile text rewriting models is introduced, generating high-quality training data without human labeling. a heuristic reinforcement learning framework improves performance without preference data. for the assesment of mobile text rewriting tasks a benchmark messagerewriteeval is introduced. empirical tests show the on-device model outperforms current state-of-the-art models while being much smaller. ## text generation pipeline in the following we describe the text generation pipeline using various llms and prompt types. ## conclusion our study explores the effectiveness of large language models in supporting the editorial process of rewriting customer help pages. we introduce a dataset containing frequently asked question-answer pairs, comprising raw drafts and their revisions by professional editors. through various prompts tailored for the rewriting task, we evaluate the performance of four llms. using rouge, bertscore, and chatgpt, we conduct automatic assessments of content and text quality. additionally, we design an evaluation of the helpfulness of automatically generated faq revisions for editorial work, conducted by professional editors. our findings demonstrate that llms can generate helpful faq reformulations for the editorial process. however, minimal performance differences were observed among llms for this task, and our survey on helpfulness highlights the subjective nature of editors' perspectives on editorial refinement. in our future work, we aim to explore additional editorial tasks, such as rephrasing texts to align with the editorial style guide or generating "metatexts" (teaser headlines, teaser texts, titles) for advisory articles.