HELP ME THINK : A Simple Prompting Strategy for Non-experts to Create Customized Content with Models (2023)

Abstract: Controlling the text generated by language models and customizing the content has been a long-standing challenge. Existing prompting techniques proposed in pursuit of providing control are task-specific and lack generality; this provides overwhelming choices for non-expert users to find a suitable method for their task. The effort associated with those techniques, such as in writing examples, explanations, instructions, etc. further limits their adoption among non-expert users. In this paper, we propose a simple prompting strategy Help Me Think where we encourage largelanguage models (such as GPT3 and ChatGPT) to help non-expert users by asking a set of relevant questions and leveraging user answers to execute the task. We demonstrate the efficacy of our technique Help Me Think on a variety of tasks. Specifically, we focus on tasks that are hard for average humans and require significant thinking to perform. We hope our work will encourage the development of unconventional ways to harness the power of large language models.

## introduction large language models (llm) like gpt3, chat-gpt @xcite and palm @xcite have excelled in many nlp tasks, however creating customized content in the form of long text generation using these models is a challenge (figure 1 ), as (1) models have not been reliable in following a list of instructions, (2) correcting model's output post generation by providing negative instructions (e.g. don't do this) in the form of dialogue has also not worked consistently. more importantly, significant effort is required for non-expert users to write instructions containing the important ingredients of a task; for example, in order to create instructions for models to write ˚work done while interning at microsoft research. the bio of a user, the user needs to think and provide various personal information. travel & event plan generation are similar to such tasks where a non-expert user has to think and find out various information necessary to make a plan, such as 'number of attendees', 'venue selection', 'budget', 'special arrangements', etc. we introduce help me think: a simple prompting strategy for non-experts to write customized content with models. on a broader level, the goal of help me think is similar to make-a-scene @xcite ; the application area, however, is vastly different as we focus on diverse applications that do not involve images and are purely based on text data. help me think involves prompting models to ask relevant questions that reduce the thinking burden of users in identifying key information specific to that task. this application is in contrast to the dominant application of llms where various prompting techniques have been developed to enable llms in answering a question @xcite . we hypothesize that knowing the right question to ask is often more important than answering it, especially in the context of personalized content generation. we experiment with six customized content generation tasks foot_0 : (1) bio generation, (2) travel plan generation, (3) dialogue generation, (4) poem generation, (5) event summary generation, and (6) story generation. we prompt gpt3 and collect 68 questions corresponding to these tasks. we find that 100% of the generated questions are valid and relevant. we use crowd-sourcing to collect answers; we leverage these question-answer pairs to prompt gpt3 again and get task-specific outputs e.g. bio, event plan, story, etc. with the help me think prompting, we construct a dataset of " 2k question-answer pairs with 180 task specific outputs spanning over the 6 tasks. we develop a questionnaire-based human evaluation scheme for crowdworkers to evaluate the quality of model generations: (1) questions and (2) task-specific outputs. our results show that 100% of task-specific outputs generated by gpt3 are valid (e.g. valid bio) and " 94% of them do not contain any extra and irrelevant information. moreover, we observe that, in " 83% of cases, gpt3 corrects typos/grammatical issues/invalid answers present in the crowdworkers' answers. gpt3 also adds appropriate context and generates coherent sentences for " 99% of cases. in 70% of cases, gpt3 performs accurate knowledge transfer foot_1 by transferring all information from input question-answer pairs to task-specific outputs. we hope the help me think prompting and our focus on tasks hard for average humans will encourage the development of unconventional ways to harness the power of llms and bring more attention to the under-explored tasks. ## help me think in this section, we introduce the help me think prompting. we first present the algorithm behind help me think. next, we describe the prompts i am an expert @xmath0. i will ask some questions to collect information and then i will use the information to @xmath1 question: figure 2 : prompt given to the model to generate question. @xmath2 and @xmath3 are 'bio generator' and 'generate a bio for you' for the bio generation task. they vary across tasks. prompt: figure 2 illustrates the prompt given to the model that generates a question specific to the task of interest. figure 3 shows that model generates a question and an answer, in response to the prompt. figure 4 shows how we generate multiple questions in this process. figure 5 shows where users fill in their answers. figure 6 shows the prompt that is attached with the questionanswer pairs which finally gives rise to the modelgenerated task-specific output. ## experiments we have conducted experiments with help me think on a diverse set of novel tasks. in this section, we describe these tasks and provide the i am an expert @xmath4. i will ask some questions to collect information and then i will use the information to @xmath5 question: <model generates @xmath10: <-> figure 3 : model generation in response to the prompt (figure 2 ). model also generates an answer along with the question, @xmath11. i am an expert @xmath6. i will ask some questions to collect information and then i will use the information to @xmath7 question: <model generated @xmath12: <-> question: <model generated @xmath13: <-> ... details of the data collection and our evaluation setup. ## results we report the task-wise performance of gpt3 and analyze its variation across different aspects of evaluation. insights: we observe that 100% of the questions generated by gpt3 are valid and relevant to the task ( extension to other tasks: we apply help me think on 57 additional tasks (appendix h) and find that help me think is effective in generating valid and relevant questions. this shows the generalization of help me think beyond the six tasks we have analyzed (section 3.1). qualitative comparison to conversational language models: help me think can also be used for customized content creation with conversational models such as chatgpt and bard. here the "role" parameter can be used as part of the prompts, specifically the prompts in figure 2 -6 can now be reframed from first person to second person since these models are designed to interact and respond to instructions. for example, the prompt in figure 2 can now be "you are an expert @xmath8. you will ask some questions to collect information and then you will use the information to @xmath9." in our qualitative analysis without help me think 7 , conversational language models does not show consistent question answering behavior and hallucination is still a major problem. help me think fixes this issue along with the knowledge cut off issue of some conversational language models because it collects facts from the user during the interaction. the instruction following capability of conversational language models is better than conventional language models, but the number of instructions needed to correct/customize its generations will probably be higher in regular multiturn conversations compared to help-me-think. we view help-me-think as a model agnostic approach that can be used and adopted with any of the state of the art models. ## conclusion we introduce help me think to help non-expert users prompt models for the relatively underexplored customized content generation tasks. we demonstrate the efficacy of help me think on 6 different tasks. our results show that (1) questions generated by gpt3 are valid and relevant, and (2) task-specific outputs generated by gpt3 are valid, relevant, robust, coherent, and have significant knowledge absorption. we hope this will bring more attention to the development of unconventional applications of llms in helping humans perform the tasks that are hard for non-expert users, in contrast to the conventional tasks like questionanswering where models chase human baseline.