Machine-in-the-Loop Rewriting for Creative Image Captioning (2022)

Abstract: Machine-in-the-loop writing aims to build models that assist humans to accomplish their writing tasks more effectively. Prior work has found that providing users a machine-written draft or sentence-level continuations has limited success since the generated text tends to deviate from users’ intention. To allow the user to retain control over the content, we train a rewriting model that, when prompted, modifies specified spans of text within the user’s original draft to introduce descriptive and figurative elements in the text. We evaluate the model on its ability to collaborate with humans on the task of creative image captioning. On a user study through Amazon Mechanical Turk, our model is rated to be more helpful by users than a baseline infilling language model. In addition, third-party evaluation shows that users write more descriptive and figurative captions when collaborating with our model compared to completing the task alone. However, the improvement is not uniform across user groups: the model is more helpful to skilled users, which risks widening the gap between skilled and novice users, highlighting a need for careful, user-centric evaluation of interactive systems.

## introduction creative writing tasks are challenging for humans because of their open-ended nature. prior work shows that exposing authors to a collaborator that provides independent suggestions can spark new ideas @xcite . this has motivated a line of work in machine-in-the-loop writing @xcite @xcite where a human collaborates with a model to complete a writing task. however, recent work has shown that providing humans a draft generated by a machine is not very helpful because it may diverge from the direction envisioned by the author @xcite . as a result, very little machinegenerated text is ultimately retained @xcite . in this work, we aim to provide a form of interaction that gives human authors more control over the content while also assisting them to better express their own ideas @xcite . we focus on the setting where authors have a clear writing outline but would benefit from suggestions on wording or framing. to allow authors to control the content, we develop a machine-in-the-loop system called creative rewriting assistant (cra) which either rewrites a span of text or infills between two pieces of text when requested (figure 1 ). cra is a sequence-to-sequence model, building upon recent advances in controllable text generation @xcite @xcite ) and text infilling @xcite @xcite . we train the cra model on a pseudo-parallel corpus of sentence pairs-a generic sentence and a more descriptive or figurative alternative (section 3.1). we evaluate the model on the task of creative image captioning (section 2). users that collaborate with cra report that it is significantly more helpful than a baseline infilling language model (section 5.1). additionally, through a controlled experiment, we find that, on average, users writing with cra produce more creative captions than those writing without assistance, highlighting the endto-end benefit of our machine-in-the-loop setup (section 5.3). in particular, users writing with the model produce captions with a more diverse vocabulary. to understand how the system impacts different users, we analyze the user-model interaction logs (section 6) and find that the machine-in-the-loop setup is more helpful to skilled writers because they tend to request targeted suggestions for shorter spans of text while giving the model sufficient context. this highlights a need to study the impact of study of interactive systems on different user groups as these become more ubiquitous in assisting content generation given that this technology could result in an even wider gap in performance between different users. ## system overview creative image captioning to evaluate our system, ideally, we would use tasks like poem or story writing. however, it is challenging to control the content for a fair comparison of different systems on such open-ended tasks. therefore, we evaluate on a proxy task, creative image captioning @xcite , where the user is asked to write an expressive caption (a figurative or descriptive one as opposed to a literal one) for a given image. the user is given access to a model that provides editing suggestions while they are working on the task. our goal is to study if collaborating with the model makes them more effective at completing the task. note that our model does not use the image when generating the suggestions, which is analogous to real use cases where the model does not have access to the author's global writing plan but instead provide improvements based on the local textual context. ## experimental design we train the cra model using the scheme laid out in section 3.1 and deploy it in the machine-in-theloop setup detailed in section 2 in order to answer the following research questions. user experience when we study collaborative writing, the key stakeholder is the user so we evaluate if users writing in tandem with cra find the model helpful. to answer this, we run a user study and compare suggestions obtained from cra and a baseline bart model in the machine-in-the-loop setup (section 5.1). once cra is deployed with real users, we would like to adapt it to users' preferences inferred from the observed interactions. hence we also compare a user-adapted model (per section 3.2) to the previously deployed cra (section 5.2). ## experiments user study we hire users on amazon mechanical turk to perform the creative image captioning task. a screenshot of our user interface and the details about worker remuneration are provided in appendix b. the plan for our user study was approved by the institutional review board of our university. each user is presented with an image and asked to write a caption that is as figurative and/or descriptive as possible with at least 100 characters. the images were randomly sampled from the figurative subset of the déjà captions dataset @xcite , where the gold caption contains literary elements like metaphors and hyperbole. we ask users to request suggestions from the model at least twice while they are writing; however, they are free to ignore the suggestions. users are instructed to use square brackets (as seen in figure 1 ) to mark spans to be rewritten and underscore to indicate blanks to be infilled. they can edit the text with the model iteratively until they are satisfied with the caption. once users submit the final caption, they are asked to complete a survey to rate the helpfulness and grammaticality of the assistant as well as their satisfaction with the final caption. the full task instruction is provided in appendix b. ## related work and discussion collaborative writing. our work builds on existing literature on collaborative writing. early approaches @xcite that provide text suggestions to users in the creative domain were retrievalbased. creative help @xcite retrieved sentence-level suggestions at locations specified by a user from a large corpus of stories. a follow-up study @xcite found that grammaticality and the presence of noun phrases in the text were indicative of helpful suggestions. we observe similar trends in section 6 and appendix e.2. more recently, collaborative systems have incorporated text generation models for assistance. @xcite evaluated a machine-in-the-loop setting on the tasks of story and slogan writing. they tested one system that generates sentence-level continuations for story writing and another one that generates a slogan from a given set of keywords, and found that solo-writing was a very competitive baseline. akoury et al. (2020) gave human writers a machinegenerated draft for storytelling and observed that writers tend to retain only a fraction of the generated text. @xcite frames collaborative writing as a conversation between a human and a dialog system leveraging large language models. our work is closest to @xcite , which demonstrated that a collaborative rewriting system helps non-native english speakers revise drafts of research papers. we focus on the more challenging domain of creative writing where users are more selective of the suggestions they accept. in addition, we study how the assistant helps with the creating writing process in an end-to-end manner, whereas @xcite focus on editing a given draft. editing models. transformer models have shown to be good at editing text to change the style @xcite , debias text @xcite , post-edit translations @xcite and simplify text @xcite . @xcite train a model to generate metaphors employing a pseudo-parallel corpus of metaphoric sentences and corresponding literal sentences similar to how we use the sources of creative text. additionally, infilling literature @xcite @xcite has shown that we can train models to fill in blanks. we incorporate editing models to collaborative writing which adapts to human feedback. ## conclusions and future work in this paper, we develop a creative rewriting assistant that is able to effectively assist users to complete the task of creative image captioning. our machine-in-the-loop rewriting setup allows human users to control the content of their writing while utilizing the strengths of text generation models. our model is found to be more useful for skilled users, so it remains to be explored how to better assist novice writers. one direction is to explore generating text from keywords because these users might need help with planning the global content and structure of their writing. additionally, the most common error mode we see amongst the rejected suggestions is content drift, so another challenge is to balance faithfulness to the author's content with creativity in text generation.