Harnessing the power of LLM s: Evaluating human- AI text co-creation through the lens of news headline generation (2023)

Abstract: To explore how humans can best leverage LLMs for writing and how interacting with these models affects feelings of ownership and trust in the writing process, we compared common human-AI interaction types (e.g., guiding system, selecting from system outputs, post-editing outputs) in the context of LLM-assisted news headline generation. While LLMs alone can generate satisfactory news headlines, on average, human control is needed to fix undesirable model outputs. Of the interaction methods, guiding and selecting model output added the most benefit with the lowest cost (in time and effort). Further, AI assistance did not harm participants’ perception of control compared to freeform editing.

## introduction recent advancements in large language models (llms), including the generative pre-trained transformer (gpt) series @xcite , have shattered the previous ceiling of human-like text generation @xcite . this has led to a paradigm shift in nlp tasks, with task-agnostic llms surpassing the performance of other state-of-the-art task-specific models @xcite . however, llm-backed systems are not without their flaws, often suffering from hallucinations, bias, or occasional generation of inappropriate content, such as toxic, discriminatory, or misleading information @xcite . human-ai text co-creation-or writing with ai assistance-allows some control over the generation process and the opportunity to overcome some llm deficiencies. co-creation approaches have shown tremendous promise in areas such as summarization @xcite and creative writing @xcite @xcite . yet, the lack of accountability of ai @xcite shifts the burden of responsibility to humans when mistakes arise in the text co-creation process. there are several methods of human-ai interaction for writing, which vary in terms of effort and control afforded by each @xcite . for example, humans might simply select from alternative model outputs or have the flexibility to revise them (e.g., post-editing). recent research by @xcite explored the dynamics of interaction between crowdworkers and large language models (llms) within the scenario of composing short texts. extending from their work, we evaluate human-ai interaction in the context of news headline creation, focusing on three of the interaction types identified by @xcite -guidance, selection, and post-editing. our study advances @xcite wizard-of-oz framework by using an implemented ai system, providing more realistic insights into human-ai co-creation for news headline generation. we explore three main aspects of co-creation with llms, through the lens of headline generation: which type of assistance is most effective for helping humans write high-quality headlines (rq1)? which type of assistance can reduce perceived effort for such task (rq2)? and, how does interacting with these models affect trust and feelings of ownership of the produced headlines (rq3)? in a controlled experiment, 40 participants wrote news headlines-either manually or assisted by one of three variants of an llm-powered headline writing system: (1) selection, (2) guidance + selection, and (3) guidance + selection + post-editing. to explore the balance between control and effort in tasks involving human-ai interaction (figure 1 ), we examined the different interaction types in combination rather than separately. next, 20 evaluators rated the headlines generated by five methodsmanual, ai-only, and the three assisted conditions. while llms alone generated the highest quality headlines on average, they were not perfect, and human control is needed to overcome errors. the simpler interactive condition guidance + selec- @xcite . selecting from model outputs (selection) alone provides less control (but also is easier) than when adding additional interactions in the form of guiding the model (guidance) and post-editing the outputs. this work contributes: (1) a systematic evaluation of three variants of human-ai interaction for headline generation, (2) design guidelines for builders of these systems, and (3) a dataset foot_0 of 840 human-rated news headlines-written by humans, ai, or the combination, which could be used as an evaluation set for further leveraged with reinforcement learning, e.g., rlhf @xcite ), to adapt llms for news headline generation tasks. ## related work our study draws upon a convergence of prior research on news headline generation and evaluation, and human-ai interactions for text summarization. ## assessing human-ai co-creation of news headlines we compared three variants of human-ai cocreation for news headlines against fully manual and fully automatic approaches. specifically, we explored three research questions (rqs): rq1: which type of assistance is most effective for helping humans write high-quality headlines? rq2: which type of assistance can reduce perceived effort for such task? rq3: how does interacting with these models affect trust and feelings of ownership of the produced headlines? we chose news headline creation as it is a familiar task-many participants would already have some intuition about what constitutes a compelling (or not) news headline-of manageable complexity-both in task and evaluation, compared to free-form creative writing or long-form summarization (e.g., authoring paper abstracts). ## results we compared headline quality (rq1), task effort (rq2) and perceived control and trust (rq3) between the different headline writing conditions. ## discussion our study explored how diverse forms of ai assistance for news headline writing influence headline quality (rq1), task effort (rq2), and perceived trust and control (rq3). we interpret our results for each of the rqs and discuss further implications in the following. ## conclusion we explore human-ai text co-creation through the context of news headline generation: how can humans best harness the power of llms in writing tasks, in terms of both quality and efficiency? and, how does interacting with these models affect trust and feelings of ownership of the produced text? 40 participants with editing backgrounds wrote news headlines-either manually or assisted by a variant of our llm-powered headline generation systemutilizing some combination of three common interaction types (guiding model outputs, selecting from model outputs, and post-editing). following this, 20 expert evaluators rated the headlines generated by five methods (human-only, ai-only, and the three assisted conditions). while the llm on its own can generate satisfactory news headlines on average, human intervention is necessary to correct undesirable model outputs. among the interaction methods, guiding model output provided the most benefit with the least cost (in terms of time and effort). furthermore, ai assistance did not diminish participants' perception of control compared to freeform editing. finally, while we focus on a simpler, low-stakes text generation scenario, this work lays the foundation for future research in more complex areas of human-ai co-creation.