PSST : A Benchmark for Evaluation-driven Text Public-Speaking Style Transfer (2024)

Abstract: Language style is necessary for AI systems to accurately understand and generate diverse human language. However, previous text style transfer primarily focused on sentence-level data-driven approaches, limiting exploration of potential problems in large language models (LLMs) and the ability to meet complex application needs. To overcome these limitations, we introduce a novel task called Public-Speaking Style Transfer (PSST), which aims to simulate humans to transform passage-level, official texts into a public-speaking style. Grounded in the analysis of real-world data from a linguistic perspective, we decompose public-speaking style into key sub-styles to pose challenges and quantify the style modeling capability of LLMs. For such intricate text style transfer, we further propose a fine-grained evaluation framework to analyze the characteristics and identify the problems of stylized texts. Comprehensive experiments suggest that current LLMs struggle to generate public speaking texts that align with human preferences, primarily due to excessive stylization and loss of semantic information. We will release our data, code, and model upon acceptance.

## introduction text style transfer (tst) is crucial in natural language processing (nlp), focusing on modifying text style while retaining the original content's information (hu et al., 2022; jin et al., 2022). by modeling complex human styles, including personality, habits, and mindset (jin et al., 2022; geroda et al., 2023), ai models can further accurately understand and generate diverse human languages for user-centric applications such as role-playing (wang et al., 2023) and digital personas (clarke, 1994; morande and amini, 2023). mixing any combination of prescription drugs, over-the-counter drugs, illicit drugs and alcohol can be unpredictable and dangerous… --dangers of mixing drugs ## public-speaking style transfer in this section, we introduce the text public-speaking style transfer (psst) task. initially, we provide a precise definition of psst and describe the source dataset we constructed for this task. furthermore, leveraging real-world public-speaking style data, we decompose the abstract publicspeaking style into essential sub-styles, which are introduced in detail in section 3.3. ## fine-grained evaluation system the evaluation of psst primarily focuses on style strength and semantic preservation. for style strength evaluation of long text, we propose a finegrained evaluation framework in section 4.1. for semantic preservation, we introduce a qa-based approach in section 4.2, which focuses on key details and logical structure. it is noteworthy that the evaluation framework we propose is scalable. it is applicable across various scenarios, particularly those involving complex and abstract styles. ## conclusion in this paper, we introduce public-speaking style transfer, which requires llms to transform a formal and long text into a public-speaking style. by analyzing real-world data from a linguistic perspective, decompose the intricate text style into key sub-styles. we propose a fine-grained evaluation framework that enables an evaluation-driven approach to continuously assess and enhance the language style modeling capabilities of llms. for style strength evaluation, we distill the llm's ability to assess the style elements of sentences into a smaller model for each sub-style. we introduce two metrics: passage-level style strength score and style strength distribution, allowing for a detailed analysis of long stylized texts. for semantic preservation, we propose a qa-based method that focuses on key information and logical structure. by observing changes in the qa model's accuracy, we identify and analyze the llm's ability to retain semantics. our experiments reveal that current llms exhibit issues such as over-stylization, uneven style strength distribution, and severe semantic degradation in long and complicated language style modeling. these findings highlight the shortcomings in llms' language style modeling capabilities and underscore the substantial room for improvement.