IGA : An Intent-Guided Authoring Assistant (2021)

Abstract: While large-scale pretrained language models have significantly improved writing assistance functionalities such as autocomplete, more complex and controllable writing assistants have yet to be explored. We leverage advances in language modeling to build an interactive writing assistant that generates and rephrases text according to fine-grained author specifications. Users provide input to our Intent-Guided Assistant (IGA) in the form of text interspersed with tags that correspond to specific rhetorical directives (e.g., adding description or contrast, or rephrasing a particular sentence). We fine-tune a language model on a dataset heuristically-labeled with author intent, which allows IGA to fill in these tags with generated text that users can subsequently edit to their liking. A series of automatic and crowdsourced evaluations confirm the quality of IGA’s generated outputs, while a small-scale user study demonstrates author preference for IGA over baseline methods in a creative writing task. We release our dataset, code, and demo to spur further research into AI-assisted writing.

## introduction writing can be both an exhilarating, creative experience as well as a frustrating slog. can recent advances in neural language modeling help improve the human writing experience? pretrained transformer language models @xcite have improved writing aids such as email "autocomplete" @xcite , while commercial tools such as grammarly and microsoft editor can rewrite full sentences to increase clarity. 1 few existing writing assistants provide support for the underlying cognitive process of writing @xcite . in this paper, we explore more advanced writing assistance functions: figure 1 : general concept overview of our intent-guided authoring assistant iga. given context, by specifying different writing intents, the system generates output satisfying the intent. in addition to wellformed sentence fragments, keywords can also be part of user input, serving as arguments for the intents, and are preserved in the output. we train iga in supervised fashion by creating a large multi-domain dataset in which spans corresponding to particular directives are replaced with a single tag: for the input "it was @xmath0, the ground-truth completion could be "it was raining , the trees were swaying and the wind was oppressive." to build our dataset, we use heuristics based on lexical and syntactic choice to isolate spans corresponding to each directive. for the above example, we extract the first simple declarative clause, then highlight the span that contains words (such as "oppressive") in a large list of adjectives, and finally extract keywords such as "trees" using a keyword extractor. at inference time, our model can flexibly take any tag as input: given "it was @xmath1, for example, our model inserts a contrastive clause to produce "it was raining but still the trees were not wet". to evaluate the effectiveness and usability of our ai-assisted writing paradigm, we design iga to be interactive, in the spirit of human-ai coauthoring. in addition to automatic and crowdsourced evaluations that demonstrate iga's output quality, we perform a user study in which participants make use of our system for creative storytelling (section 6). our results show most users prefer writing with assistance from iga compared to writing from scratch or with a non-controllable infilling model. our contributions are as follows: ## intent-guided assistant iga extends text infilling models with finegrained rhetorical control. specifically, we build on the infilling language model (ilm) of donahue et al. (2020), which fine-tunes an off-theshelf language model such as gpt-2 on a dataset of text with masked spans. to continue with our running example, the input to this model is the sequence "it was @xmath2, and a target output is "and getting harder to see the". at inference time, the blanks are substituted with the words predicted by the lm and combined with the input in a post-processing step to generate the final output: "it was raining and getting harder to see the trees". building on this framework, we fine-tune an off-the-shelf gpt-2 medium model 3 on our dataset (described in section 4) created for generating text conditioned on author intent. instead of replacing spans with a @xmath3, we replace spans with more fine-grained tags corresponding to rhetorical directives. for fine-tuning, we concatenate a tag-replaced sentence with the ground-truth spans that should be infilled using a special separator @xmath4, as in @xcite . if multiple tags are used 3 this model has 431m parameters. in the input, the multiple ground-truth spans following @xmath5. at inference time, the model is fed a tag-replaced sentence and @xmath6, from which it generates span(s) to infill the input tags. in a post-processing step, we replace the tags with the generated answer spans. figure 2 summarizes the fine-tuning process (left) and the inference process (right). during inference time, we use top-k sampling @xcite with k fixed to 40. to sum up, unlike the standard gpt-2 model which only supports strict left-to-right generation, both ilm and iga are capable of text-infilling. moreover, iga has finer control over author writing intents than ilm, which further narrows down the output generation space. although providing gpt-2 and ilm with specific discourse markers may result in output conforming to certain intents, they are less flexible than iga, which can sample from multiple discourse cues. the novelty of iga lies not in its model architecture but the way in which we supervise it to enable controlled fine-grained text infilling. we construct the fine-tuning dataset primarily by heuris- tag system input and output para input: <@xmath7. <@xmath8: the pandemic has brought severe economic, social and political effects that have seriously affected many countries. bio input: oria, <@xmath9, mentions that technically only humans can cry in response to emotional state. output: oria, a psychologist specializing in the study of human emotion, mentions that technically only humans can cry in response to emotional state. cause input: this is a really good @xmath10. output: this is a really good book because the plot is always so well written and never predictable. effect input: janet had suffered a traumatic brain injury in a car accident in 1988, <effect>. output: janet had suffered a traumatic brain injury in a car accident in 1988, thus had no memory of who she was or what happened. cntra input: the castle was built in 1865, <@xmath11. output: the castle was built in 1865, though a full-scale modern restoration has been underway for the past four years. descp input: <@xmath12. output: there are individual and social beliefs that should lead us to be skeptical of the facts and the wrong. idiom input: this report only shows @xmath13, as many details can only be uncovered if you sign the document. output: this report only shows the tip of the iceberg , as many details can only be uncovered if you sign the document. choosing a collection of tags: before we start collecting data, we conduct an internal survey with potential users of our system to determine what writing assistance functions they would most benefit from. we surveyed nine nlp researchers about their opinions on the ideal functionality of an authoring assistant. after removing simple functions such as generating synonyms, antonyms, adjectives, and adverbs, which are already implemented in existing writing assistant tools, we condense the most requested writing intents into seven tags (described in detail below; examples provided in effect: as a conjugate writing intent of cause, effect is used when one needs to describe the result or consequence. it usually cooccurs with the words/phrases such as 'therefore' and 'as a result'. similar to cause, we manually select 15 discourse markers that signify effect, mine sentences from newsroom, and highlight spans of interest using the same process as before. specifically, we extract declarative clauses based on the constituent labels returned from the parser. then, we decide the intent according to the starting markers of those clauses. inside each clause, we run yake, an unsupervised keyword extraction algorithm, to extract keywords for the clause. all markers are included in appendix b. cntra: comparison is a commonly used writing technique that encompasses two separate intents: concession and contrast. concession refers to the unexpectedness of a consequence @xcite : words/phrases such as "although" and "even though" raise an expectation curbed by the rest of the sentence. contrast is often confused with concession, but its markers include "by comparison", "in contrast", etc. we manually select 31 concession markers and six comparison markers, and mine the newsroom corpus with these to obtain labeled data. ## evaluation against references as an initial comparison of iga and ilm, we evaluate the generated outputs of each model against reference completions from our dataset, both automatically and through a crowdsourced evaluation. we acknowledge that this type of evaluation (especially using automatic metrics) is limited for open-ended generation tasks like ours @xcite @xcite , which is why we also conduct an in-depth user study in section 6. while results of these evaluations cannot reflect how practical iga can be used as an authoring assistant, they do indicate that iga is more constrained than ilm and produces output that better fulfills the writing intents. ## user study due to the limitations of the previous evaluations, we launch a user study in the same spirit as @xcite to understand the interactive behavior of real users. we measure whether human authors benefit from ai-assisted writing, and whether they prefer intent-guided generation to the uncontrolled ilm model. we design an interactive web demo that allows participants to write with the help of each model, logging their behavior (e.g., queries to the model, edits made on generated text) and self-reported feedback. our interactive demo is inspired by markup 8 we employ workers with an approval rate higher than 96% and total approved hits greater than 1000. each rater is rewarded for $0.1 per hit. 9 the best fleiss κ (fleiss, 1971) across all tags was only slightly over 0.2, which indicates slight agreement @xcite . language editors such as overleaf foot_1 or lyx foot_2 ; a screenshot of the interface is shown in figure 3 . in the textbox to the left, users type sentences with any of our supported tags. after clicking generate, the model's output will be displayed on the right hand side foot_3 . by default, three samples are shown to the user, although they can increase the number of samples if they wish. after a user selects a sample, it is appended to the existing text in the top-right textbox, which contains all of the text the user has already written. users can then edit the sample (or completely delete it) and write continuations directly into the top-right textbox. this process repeats every time the user decides to use a tag to obtain model-generated text. on the backend, the input fed to our iga model is the concatenation of content in the main textbox (i.e., context) and the input in the assistant box, truncated at 300 tokens. ## limitations although our user study demonstrates that subjects prefer iga over competing models, it has many limitations. first, nlp researchers are not the right group to ideate the set of writing intents, and in the user study, computer science graduate students are not representative enough as the target users. a more ideal setup is to conduct both the ideation of intents and user study with expert users, preferably english students or teachers. this sort of study could be done on platforms like upwork. to validate the usefulness of the existing intents in iga, we also need to conduct interviews with writing professionals and inquire about new prospective intents for future development. ## conclusion in this paper, we introduce a new approach to interactive human-ai co-authoring by means of an intent-guided authoring assistant (iga). our model is able to infill around author-provided keywords, sentence fragments, and rhetorical instructions with fluent and coherent text. we conduct a small-scale user study which shows that our method has advantages over baseline methods on a creative writing task.