Enhancing AI Assisted Writing with One-Shot Implicit Negative Feedback (2024)

Abstract: AI-mediated communication enables users to communicate more quickly and efficiently. Various systems have been proposed such as smart reply and AI-assisted writing. Yet, the heterogeneity of the forms of inputs and architectures often renders it challenging to combine insights from user behaviour in one system to improve performance in another. In this work, we consider the case where the user does not select any of the suggested replies from a smart reply system, and how this can be used as one-shot implicit negative feedback to enhance the accuracy of an AI writing model. We introduce Nifty, an approach that uses classifier guidance to controllably integrate implicit user feedback into the text generation process. Empirically, we find up to 34% improvement in Rouge-L, 89% improvement in generating the correct intent, and an 86% win-rate according to human evaluators compared to a vanilla AI writing system on the MultiWOZ and Schema-Guided Dialog datasets. The code is available at https://github.com/BenjaminTowle/NIFTY.

## introduction the average worker reportedly spends around 23% of their time on reading and answering emails @xcite . to alleviate this burden, there is a growing demand for ai-mediated communication systems to draft and potentially fully automate replies for users. these facilitate faster communication by providing suggestions at different stages of the conversational pipeline. various modes of interaction exist, each with differing trade-offs: smart reply systems -such as in gmail @xcite or outlook @xcite ) -offer a lowlatency solution to dealing with simple requests, using a retrieval-based model to present canned suggested replies to the user which can be clicked instead of requiring manual typing. ai writing models -such as used by jasper or grammarly -figure 1 : example of how an agent may utilise either a smart reply system or an ai writing system to speed up communication with a customer. our approach, nifty, uses implicit negative feedback from the rejected suggestions to improve the ai writer's prediction. yet, the heterogeneity of the forms of interaction and architectures often renders it non-obvious how the information from one system can be leveraged to improve another: e.g., a smart reply system is only useful when the user clicks one of the suggestions. in practice however, there is often too much uncertainty surrounding the user's intent for any of the suggestions to be relevant (figure 1 ) @xcite . due to reading the suggestions, this increases the user's cognitive load, with no additional payoff. this contrasts with positive feedback, when selecting a suggestion reduces user typing time. to address this problem, we conduct a pilot study into how one-shot implicit negative feedback can be used to integrate and improve the performance between two heterogeneous ai-mediated communication systems. in particular, we consider how feedback when a user clicks none of a smart reply system's suggestions can improve a downstream ai writing model at run-time. we concentrate on the one-shot setting, which has the advantage of enabling a single shared model for all users, as well as being more challenging due to the limited amount of interaction information per user. future work may extend this to greater degrees of user personalisation, although this is currently out of scope due to lack of data access. in this paper, we introduce negative implicit feedback from smart reply ("nifty"). nifty employs classifier guidance @xcite to controllably integrate one-shot implicit negative feedback into the generation process at run-time. in particular, given an unconditional ai writing model, we condition the model on a desired attribute c, via an application of bayes' rule, using a classifier trained to predict c. we explore two possible settings for c: an intent-based approach that conditions on the most likely next intent not represented in the suggestions and a user actionbased approach that conditions directly on the user rejecting the suggestions. overall our method affords several key advantages: (i) it keeps the smart reply and ai writing systems decoupled, allowing it to be readily integrated into existing systems which can be optimised by separate teams; (ii) it enables additional forms of negative feedback to be introduced in the future, e.g., lingering on a suggestion without clicking it @xcite , via a linear combination with a separate classifier; (iii) incorporating implicit negative feedback into the ai writing process reduces the cost to the user of being presented with irrelevant suggested replies. empirically, by evaluating on two publicly available datasets, we find up to 34% improvement in rouge-l and 89% improvement in generating the correct intent compared to a vanilla ai writing system, and an 86% win-rate according to human evaluators. in summary, our key contributions are: (1) we introduce the framework of implicit negative feedback to the smart reply and ai writing tasks; (2) we develop and open-source an approach that uses classifier guidance, considering both intent-based and user action-based attributes for conditioning; ## related work early ai-mediated communication centred around smart reply systems, which provided canned replies to a user message @xcite . work then expanded to various forms of heterogeneous interaction including ai writing systems, such as autocompletion @xcite , alternative word suggestions @xcite , or conditioning on: rough drafts @xcite , abbreviated sentences @xcite , or user provided intents @xcite . while these methods all require explicit user effort, our approach requires no interactive effort from the user, by leveraging implicit negative feedback. various types of implicit feedback have been explored in previous works, such as user lingering time @xcite , skipping content @xcite , or conversation length @xcite . to the best of our knowledge, our work is the first to apply this to the smart reply setting. finally, our work builds upon the growing literature on controllable text generation. here, early work focused on conditioning generation on particular control 'codes' @xcite . more recently, work focuses on classifier guidance, in which a separate classifier guides token-by-token generation @xcite @xcite . note that our focus is not to introduce a novel algorithm for classifier guidance, but rather to demonstrate how it can be combined with implicit negative feedback to solve a problem in ai-mediated communication -namely -the lack of easy integration between different modes of interaction. therefore, as new techniques emerge from this field they may be used to enhance our own method. ## method filtering with user simulator we run our user simulator using the dataset @xmath0, r, s)} i @xmath1, where m is the incoming message, r is the groundtruth reply, and @xmath2, ...s k } is the set of reply suggestions obtained from a black-box smart reply system. we are concerned with the scenario in which the user clicks none of these suggestions. to represent this concretely, let z represent the set of all possible intents, z s be the set of intents as-signed to suggestions in s, and z be the intent of the ground-truth response r -e.g., 'yes i can! table for 1?' corresponds to the booking-inform intent. we simulate the user by having the user reject all suggestions when z ̸ ∈ z s , i.e. none of the suggestions contain the intent of the ground-truth reply. by filtering according to this criterion, we obtain dataset d ′ which is used for downstream evaluation. note that both our generator and classifier are trained on the full version of our dataset d. smart reply the smart reply system uses a vector retrieval model @xcite , trained to jointly embed messages and replies into a shared latent space. at run-time, it retrieves the top k nearest neighbours as suggested replies. following convention @xcite , we set @xmath3. we choose this straightforward approach in order to demonstrate that even an outof-the-box smart reply system can provide useful implicit negative feedback, without requiring any bespoke alterations. 1 generator following previous approaches @xcite , the ai writing model is a transformer trained to generate tokens autoregressively. given a message m, the probability of generating reply r can be factorised as: this model is trained only on the (message, reply) pairs from d, without requiring access to the smart reply system, or any implicit user feedback, using negative log likelihood. classifier guidance allows an unconditional text generation model p θ (r t |m, @xmath4, by performing a classification step over possible next tokens p φ (c|m, @xmath5. these can be combined through bayesian decomposition @xcite . note, the classifier predicts whether the attribute will be obtained by the time the response is completed, not whether the attribute is currently present. by operating at the token level, rather than over completed generations (i.e. reranking), the model is able to explore a larger search space. for our purposes, we consider two possible approaches for what attribute to condition the generator on. first, we condition on the desired intent of the response. in particular, we train a classifier to predict the final intent of the reply, given only the reply prefix @xmath6. at run-time, we then condition on the most likely intent that is not present in the suggestions: ## experiment datasets we evaluate our method on two taskoriented datasets covering various domains: multi-woz v2.2 @xcite and schema-guided dialog (sgd) @xcite . both have the advantage of being annotated with intents, and also feature the professional style of conversations that many ai writing applications aim to facilitate. we treat replies containing multiple intents as unique intents in their own right as it is unclear that a user would accept a suggestion that only partially contained the desired intents. as we are concerned with the assisted writing setting, rather than creating a consumer-facing task-oriented chatbot, we evaluate using standard ai writing metrics, rather than success-based task-oriented metrics. see appendix a for dataset statistics. metrics we employ both automatic and human evaluation. rouge-l, which has been used previously in ai writing tasks @xcite , measures the longest common subsequence within the prediction, capturing surface-level overlap with the ground-truth. however, there are often many ways of expressing the same meaning that lack termoverlap. therefore, inspired by previous evaluation work on smart reply systems @xcite we complement this metric with r@1, which measures the proportion of generated responses that contain the correct intent. as the responses are generated on-the-fly, we detect intents using a distilbert-based classifier trained to map utterances to the corresponding intent for each dataset (a separate classifier to the one used in nifty), which we then compare to the ground-truth intent. we conduct human evaluation using amazon mechanical turk. as the purpose of the system is to reduce uncertainty about the user's intended reply, we apply a pairwise setup, in which the evaluator is asked to select which candidate reply is most similar to the target reply. for each model and dataset, we randomly select 100 data points from the test set and assign 3 annotators to each data point. we define a 'win' as when a system is achieves a majority vote for a given data point. overall, the procedure was carried out by 268 unique annotators, across 400 data points. see appendix c for further details. baselines we compare our approach to: the standard baseline ai writing model -i.e. without any classifier guidance; a reranker approach which reranks the final output beams, without any tokenlevel reranking, by selecting the beam with the highest score for the desired intent / action; an unlikelihood decoding approach that downweights the probability for terms that occur in the rejected intents, encouraging the model to generate one of the non-rejected intents @xcite ; a rules-based approach in which multiple candidate multiwoz sgd model α r-l r@1 r-l r@1 t5 nifty intent 0.5 29.4 27.8 24.5 53.1 1.0 29.0 28.5 24.2 53.2 2.0 27.9 ## conclusion in this work we introduce nifty, an ai writing system that uses classifier guidance to account for implicit negative user feedback from an upstream smart reply system. empirically, we find up to 34% improvement in relevance and 89% improvement in generating the correct intent compared to a vanilla ai writing system. future work may explore applying these techniques to real-life data from deployed systems and / or may be extended to a broader range of types of feedback beyond click data from smart reply systems.