N ews B ench: A Systematic Evaluation Framework for Assessing Editorial Capabilities of Large Language Models in C hinese Journalism (2024)

Abstract: We present NewsBench, a novel evaluation framework to systematically assess the capabilities of Large Language Models (LLMs) for editorial capabilities in Chinese journalism. Our constructed benchmark dataset is focused on four facets of writing proficiency and six facets of safety adherence, and it comprises manually and carefully designed 1,267 test samples in the types of multiple choice questions and short answer questions for five editorial tasks in 24 news domains. To measure performances, we propose different GPT-4 based automatic evaluation protocols to assess LLM generations for short answer questions in terms of writing proficiency and safety adherence, and both are validated by the high correlations with human evaluations. Based on the systematic evaluation framework, we conduct a comprehensive analysis of eleven popular LLMs which can handle Chinese. The experimental results highlight GPT-4 and ERNIE Bot as top performers, yet reveal a relative deficiency in journalistic safety adherence in creative writing tasks. Our findings also underscore the need for enhanced ethical guidance in machine-generated journalistic content, marking a step forward in aligning LLMs with journalistic standards and safety considerations. The evaluation framework and experimental results are expected to provide an in-depth understanding of the editorial capabilities of LLMs and speed up the development of LLMs in journalism.

## introduction the increasing availability of large language models (llms) with application programming interfaces (apis), like openai's chatgpt, has further accelerated the adoption of the llm technology across a variety of application scenarios. however, while llms offer significant benefits to natural language processing (nlp), their nondeterministic and black-box nature has sparked discussions and concerns about ensuring the responsible and ethical utilization of this advanced technology @xcite . although general safety evaluation benchmarks @xcite @xcite and safeguard measures @xcite , including the openai moderation api designed to prevent toxic and harmful content, have been proposed and some put in place, there is a need for specialized benchmarks tailored to the unique rules, responsibilities, and styles of various professional domains and scenarios @xcite . in journalism, the significant role it plays in informing the general public and its potential to influence public perception demands a higher and more specific ethical and safety standard. there are an increasing number of llms being applied in chinese journalism to complete editorial tasks, such as headline generation, summarization, continuation writing, expansion writing and refinement. despite considerable discussions among the academia and industry on comprehending, regulating, and mitigating the risks associated with llms in journalism @xcite @xcite , there is a notable absence of a standardized benchmark or systematic evaluation framework that assess the alignment of llms with journalistic ethics and safety standard and integrates them with common journalistic editorial tasks. drawing on discussions about ai safety in journalism @xcite , this paper introduces newsbench, a systematic evaluation framework which is focused on assessing the editorial capabilities of llms for not only journalistic writing proficiency but also safety adherence. for journalistic writing proficiency, we focus on language fluency, logical coherence, style alignment, and instruction fulfilment, while for safety adherence we consider six facets including civil language, bias and discrimination, personal privacy, social harm, journalistic ethics, and illegal activities. we construct the benchmark dataset with 1,267 test samples in the types of multiple choice and short answer questions in five editorial tasks including headline generation, summarization, continuation of writing, expansion of writing, and style refinement from 24 news domains. additionally, newsbench incorporates two automatic evaluation protocols for assessing llm generations for short answer questions in terms of writing proficiency and safety adherence. utilizing this comprehensive framework, we have evaluated eleven popular llms which can handle chinese, providing insights into their performance across a diverse range of journalistic tasks and safety considerations. the main contributions of the paper are as follows: ## related work prominent media outlets, such as the bbc (british broadcasting corporation) (arguedas and simon, 2023), have incorporated llms into their news production workflow, undertaking tasks such as summarization, headline generation, translation, and writing style refinement @xcite , to speed up and refine their editorial process. alongside this evolving technological integration, there is an increasing focus on ensuring the safety of ai in journalism and embedding ethical and professional journalistic values within these technologies @xcite . the core concept of professional journalism ethic emphasize the responsible exercise of the freedom to publish. 2 this concept is expanded into four key principles by the society of professional journalists: seeking truth, minimizing harm, acting independently, and being accountable. efforts have been undertaken to recommend guidelines that align llms usages with media ethics and safety standards @xcite . @xcite compiles an exhaustive list of risks associated with the use of llms in journalism covering three major categories including editorial risks, legal and regulatory risks, and societal risks. there are an increasing number of benchmarks for assessing the performance and safety of llms @xcite @xcite . however, these benchmarks primarily target general scenarios, while news media operations necessitate adherence to specific professional ethics, styling requirements, and safety standards, and bear a greater social responsibility. zagorulko (2023) are pioneering in evaluating llms generated outputs against evaluation specific to journalism, such as topicality, reliability, balance of opinion, and accuracy of information. however, since the framework requires human evaluation, it cannot be efficiently applied to newly emerging llms. despite these recent advancements, there remains a gap in benchmarks specifically tailored to evaluate the journalistic writing @xcite and safety @xcite of llms within the context of specific news editorial tasks. ## the evaluation framework the evaluation framework is designed to conduct a comprehensive and automated assessment of llms for editorial capabilities with a particular focus on journalistic writing proficiency (jwp) and safety adherence (sa) (section 3.1). evaluations are in types of short answer questions (saqs) and multiple choice questions (mcqs) (section 3.2). to aggregate and compare performances of llms, we develop four types of evaluations: jwp on multiple choice questions (jwp-mcqs), jwp on short answer questions (jwp-saqs), sa on multiple choice questions (sa-mcqs), sa on short answer questions (sa-saqs). figure 1 illustrates the comprehensive workflow of our evaluation framework. specifically, for each llm to be evaluated, the framework first takes the four types of evaluations to elicit corresponding answers for mcqs and saqs. to get the final performances of the llm on the four types of evaluations, we adopt accuracy as the evaluation metric for mcqs while implementing gpt-4 based evaluation metrics for saqs. ## benchmark dataset construction llms are increasingly adopted as editorial assistants in the editorial phase of the publishing workflow in journalism @xcite . based on existing studies @xcite and consulting with professional journalists, llms are usually used on five editorial tasks: headline generation (head), summarization (summ), continuation of writing (cont), expansion of writing (expa), and style refinement (refi). with a focus on these five editorial tasks, the benchmark dataset is manually constructed by human experts and it comprises test samples for these editorial tasks, spanning across various news domains. in our benchmark dataset, each test sample is assigned with a specific editorial task. ## evaluation protocols for short answer questions the evaluation framework utilizes the constructed test samples to automate the reference-free evaluation of llms, focusing on their proficiency in general journalistic writing and their adherence to safety standards. while the evaluation of multiple choice questions can be straightforwardly conducted by calculating accuracy, assessing responses for short answer questions necessitates a more complicated approach. drawing on recent advancements in natural language generation evaluation methodologies, which have employed llms for the automated assessment of fine-grained attributes @xcite , we have developed two distinctive automatic evaluation protocols. these protocols are designed to assess general journalistic writing proficiency and safety standard compliance, respectively, using llms such as gpt-4 (openai, 2024). additionally, we have performed human validation to ensure the effectiveness of our evaluation protocols and their alignment with human judgments. ## conclusion we develop the evaluation framework, newsbench and it marks a significant advancement in the automatic evaluation of editorial capabilities of llms in chinese journalism. our benchmark dataset consists of 1,267 meticulously constructed testing samples that cover five editorial tasks, seven evaluation facets (including six safety adherence facets and the other four evaluation facets for journalistic writing proficiency), two question types (multiple choice and short answer questions), and spans 24 news domains. we also propose and implement two evaluation protocols for safety adherence and journalistic writing proficiency. our extensive comparison experiments on eleven llms with newsbench identify gpt-4 and ernie bot as the standout models in most evaluations. however, it also reveals a notable weakness across all llms in maintaining journalistic ethics during creative writing tasks such as writing expansion and continuation. this insight underscores the necessity for future improvements in the ethical adherence of llms within automated content generation.