Summary Cycles: Exploring the Impact of Prompt Engineering on Large Language Models’ Interaction with Interaction Log Information (2023)

Abstract: With the aim of improving work efficiency, we examine how Large Language Models (LLMs) can better support the handoff of information by summarizing user interactions in collaborative intelligence analysis communication. We experiment with interaction logs, or a record of user interactions with a system. Inspired by chain-of-thought prompting, we describe a technique to avoid API token limits with recursive summarization requests. We then apply ChatGPT over multiple iterations to extract named entities, topics, and summaries, combined with interaction sequence sentences, to generate summaries of critical events and results of analysis sessions. We quantitatively evaluate the generated summaries against human-generated ones using common accuracy metrics (e.g., ROUGE-L, BLEU, BLEURT, and TER). We also report qualitative trends and the factuality of the output. We find that manipulating the audience feature or providing single-shot examples minimally influences the model’s accuracy. While our methodology successfully summarizes interaction logs, the lack of significant results raises questions about prompt engineering and summarization effectiveness generally. We call on explainable artificial intelligence research to better understand how terms and their placement may change LLM outputs, striving for more consistent prompt engineering guidelines.

## related work the nlp field has seen public attention this year from the widespread adoption and use of generative pre-trained models @xcite . in this work, we explore how llms can support analytic provenance research, especially when paired with prompt engineering approaches. ## experimental procedure to better understand the expectations and effects of using large language models for summarization of interaction logs, we conduct a handful of experiments, starting with the collection of user feedback from a qualitative study. from this pilot study, we then conduct a series of nlp prompting experiments to compare differences in how the addition of examples and audience types influence model output summaries. throughout these experiments, we use the openai chat completions api with the "gpt-3.5-turbo foot_0 " model as the llm for our approach (brown et al., 2020). ## results our goal with this work is to demonstrate the simplicity of a recursive summarization technique for communicating user interaction logs. overall, the generated summaries are promising and may offer a realistic possibility for generating sufficient support for report generation with human refinement. in this section, we offer a handful of observations. ## discussions and future work in this work, we explore the factors of audience and example inclusion as a demonstration of applying prompt engineering to generate work summaries in the intelligence domain. while we have not found other evidence of a methodology where the proposed pipeline consults a large language model, the pre-processing steps taken on the dataset documents are inspired by the chain-of-thought prompting strategies. we use a series of prompts to extract information from documents and segment an interaction log to build up a complete summary prompt and discuss the results. our independent variables are derived from our pilot study, where users identify essential elements of a work summary. yet, we do not see strong effects on baseline summary factuality or accuracy when adjusting the audience or the inclusion of examples. instead, in our testing, we observe different important factors. we observe differences in summary lengths when we included contextual information twice. therefore we use two different kinds of ground truth (i.e., baseline and additional) to account for this. this leads us to think about how specific wording in the prompt messages may noticeably impact the focus of the output. novel methods may emerge that afford the direct manipulation of prompt wording. for example, it would be interesting to investigate how opposite terms, antonymic to the adjectives used in our study, may impact the model's attention. additionally, ab-lation studies that target the specific adjectives we use may offer fascinating insights into which terms make the biggest difference. regardless of the technique employed, studies exploring the influence of individual terms do not, to our knowledge, have consistent summarization evaluation criteria, thus calling attention to a need for more established evaluation methods. finally, corresponding to the chain-of-thought nature of the work presented, there are obvious future directions that could consider how the prompting process could involve human users to adjust and modify the prompt in real time. it would be helpful to have domain experts rank the summaries and use these rankings to fine-tune the prompting process. additionally, giving users interface controls that manipulate the generated prompt by using prompt engineering guidelines could be imagined for future exploration into model behaviors. it is also interesting to consider the downstream tasks from a work summary and how different generation methods are perceived and may influence future work by human users. ultimately, in this work, we observe the feasibility of generating human-sounding summaries of work from user interaction logs, but they tend to list steps completed without a hierarchical structure that captures the concepts that are most important or structures the content to flow like a story. perhaps future work could explore how additional analysis layers, prompt engineering interfaces, or human feedback may help summaries acquire a sense of structured storytelling. ## conclusion by harnessing llms, researchers can enhance transparency, reproducibility, and collaboration, improving problem-solving communication. in this work, we showcase the potential of chatgpt to generate work summaries from data analysis interaction logs and the associated document contexts. by manipulating prompt engineering features, we investigate the impact of different prompts on the llm's output in the intelligence domain. we develop a recursive prompt reduction method to handle token limitations and evaluated prompt examples and audience types, both quantitatively and qualitatively. while we show the potential llms have for automating work summaries from provenance information, we find few consistent impacts of these factors on summary accuracy. instead, we recognize that more reliable prompt engineering guidelines will be helpful when developing more sophisticated tools to analyze provenance information and control generated output. as has become a common discussion within the research community @xcite , the need to better understand these models and their impact on society is critical. while what we demonstrate shows promise for productivity increases, there are tradeoffs that come from automation that will impact how we individually engage with society. therefore, we complete this work with a discussion of the various limitations of what we proposed and the ethical considerations of llm usage in workplace cohesion tasks.