T uring A dvice: A Generative and Dynamic Evaluation of Language Use (2021)

Abstract: We propose TuringAdvice, a new challenge task and dataset for language understanding models. Given a written situation that a real person is currently facing, a model must generate helpful advice in natural language. Our evaluation framework tests a fundamental aspect of human language understanding: our ability to use language to resolve open-ended situations by communicating with each other. Empirical results show that today’s models struggle at TuringAdvice, even multibillion parameter models finetuned on 600k in-domain training examples. The best model, T5, writes advice that is at least as helpful as human-written advice in only 14% of cases; a much larger non-finetunable GPT3 model does even worse at 4%. This low performance reveals language understanding errors that are hard to spot outside of a generative setting, showing much room for progress.

## introduction language models today are getting ever-larger, and are being trained on ever-increasing quantities of text. for an immense compute cost, these models like t5 @xcite and @xcite show gains on a variety of standard nlp benchmarks -often even outperforming humans. yet, when a giant model like t5 generates language, we observe clear gaps between machinelevel and human-level language understandingeven after it has been finetuned for the task at hand. consider figure 1 , in which a woman asks for advice. she is assigned to dissect an animal for her class project, but has extreme anxiety about dead animals -and her teacher refused to give her another assignment. humans can respond with helpful advice, reflecting our unique ability of realworld language use: to communicate and tackle open-ended issues. the helpful advice in this ex-t5 i'd send a short email to the next higher-up authority figure, ideally a counselor. be forthright; it's the best approach when selfadvocating as a student. go to your teacher and say "i'm asking you to do a project that requires me to see dead animals. this is a dealbreaker." if she doesn't concede, tell your principal about your trauma. i have to do a dissection for my high school class, but i'm distressed by dead animals. last time we dissected an animal in class, i had a panic attack. i asked my teacher for another assignment, but she refused. i don't want to play a 'victim' card, but i don't know what to do. help! ## real world language use we propose to evaluate machines by their success at real-world language use: using language to communicate with a human, in response to a naturally occurring situation, in order to achieve a desired outcome. this is how educators often measure (human) language understanding of a second language -by how well the learner can use the language (council of europe, 2001). our approach is also inspired by wittgenstein's notion of semantics, that "meaning is use:" language is grounded in our desire to make sense of one another and cooperate to meet our needs @xcite . as machines do not have humanlike needs or desires, we propose to evaluate machines' success at a task by how well it serves a human who is interested in the outcome. for example, if a machine orders food on my behalf, then i can evaluate it based on whether i enjoy the dish it ordered. though this requires careful task selection in order to make things feasible for current models, as we will show in section 3, it results in a powerful and reliable human evaluation. 4858 2.1 related work 2.1.1 pragmatics in nlp our evaluation relates to pragmatics in nlp, where communication is modeled also through listeners and speakers @xcite . one approach is to introduce a communication game, with an explicit objective. for example, @xcite study a blocks world where humans give commands to a block-placing machine. the machine is then graded on accuracy. our proposed evaluation instead covers complex everyday scenarios faced by a human, where the objective is to help them as much as possible. pragmatics can also be studied through machinemachine communication; e.g., through emergent language @xcite . recent work uses pretrained question-answering models to evaluate summarization models @xcite @xcite . however, ensuring that machines communicate in standard english is difficult, as there is usually a more efficient machine-language coding scheme for the task @xcite . ## turingadvice: a new challenge for natural language understanding as a case study of our framework, we introduce turingadvice, a new challenge task for ai systems to test language understanding. the format is simple: given a situation expressed in natural language, a machine must respond with helpful advice. to pass the challenge, machine-written advice must be at least as helpful to the advice-seeker as humanwritten advice, in aggregate. we focus on advice for a few reasons. first, advice-giving is both an important and an everyday task. people ask for and give advice in settings as diverse as relationship advice and tech support @xcite . thus, we as humans have inherent familiarity with the task, and what it means for advice to be helpful -making it easy to evaluate, as we later show empirically. moreover, because there are many internet communities devoted to advice-giving, training data is plentiful. second, the framework of advice-giving allows us to study subtasks such as reading comprehension and natural language inference (section 5.3); we argue both of these are needed to consistently give good advice. learning to recognize advice has recently been studied as an nlp task on its own (govindarajan et al., 2020), though we are not aware of past work in learning to generate advice. ## experimental results on redditadvice in this section, we report results from one round of dynamic evaluation on redditadvice. we evaluate the following strong nlp models and baselines: a. rule-based: a templated system to give legal, relationship, or life advice. the system first chooses randomly empathetic sentence from ten choices, for example "i'm sorry you're facing this." it then chooses a random piece of advice that is loosely related to the situation's topic; we infer this from the subreddit the situation was posted on. for example, for figure 4: improvement (in absolute percentage %) between pairs of models, along with statistical significance from a paired t-test. the improvement of t5-11b over smaller models like grover-mega is highly statistically significant (10% gap, pă.01), while being far worse than human performance. our evaluation thus meaningfully grades varying levels of performance. legaladvice the model might write "i'd suggest getting a lawyer immediately." b. tf-idf retrieval: for a new situation, we compute its tf-idf bag-of-word vector and use it to retrieve the most similar situation from the training set. we then reply with the top-scoring advice for that situation. c. grover-mega @xcite : a left-toright transformer model with 1.5 billion parameters. grover was pretrained on news articles with multiple fields, perhaps making it a good fit for our task, with multiple fields of context (like the subreddit, date, and title). our situation-advice pairs are often quite long, so we adapt grover for length; pretraining it on sequences of up to 1536 characters. d. t5 @xcite : a sequence-tosequence model with a bidirectional encoder and a left-to-right generator, with 11 billion parameters. t5 was trained on a large dataset of cleaned web text. at the time of writing, t5 is the top-scoring model on the glue and superglue benchmarks @xcite , scoring above human performance on glue and near human-performance on superglue. e. gpt3 (brown et al., 2020): a left-to-right transformer model with 175 billion parameters. gpt3 must be "prompted" to generate advice since it has not been released for finetuning. we cannot provide few-shot examples in the prompt due to the length of situation-advice pairs; we instead mimic the formatting of a website quoting from reddit (appendix b.5). last, to quantify the measurement error of our evaluation, we additionally evaluate: f. the second-highest rated reddit advice for each situation. we send this advice through the same pipeline as machine-written advice. we finetune all models (except gpt3) and generate using nucleus sampling @xcite ; more details in appendix b. in our study, we exclude purely bidirectional models, such as bert @xcite . while these models can be made to generate text, these generations are usually worse than those of left-toright models @xcite . t5 also tends to outperform them, even on discriminative tasks. ## analysis and discussion so far, we have shown that we are able to reliably evaluate models in our dynamic setup, and that doing so results in model performance that is significantly lower than human performance. to break down what this gap in performance means, we show a qualitative example in figure 5 . a user is asking for online legal advice about being stuck at work for their entire 4pm-midnight shiftwith no eating allowed due to covid-19. the toprated reddit advice understands this situation and then offers advice, suggesting the advice-seeker organize alongside other workers -as "new jersey doesn't require employers to give any meal breaks." machine advice seems to misunderstand the issue. t5 asks if there is "a reason that you can't leave the property," even though this reason is stated in the situation. gpt3's advice is selfcontradictory; it also makes something up about a "restaurant" in the building. ## conclusion; ethical considerations we introduced new methodology for evaluating language tasks, reducing the gap between benchmarks and the real world. we also introduced a new challenge for the community, turingadvice, with an accompanying dataset and dynamic leaderboard. yet, if our field is to progress towards nlp models that 'understand natural language,' we should be cognizant of the impact that such technology 8 might have on society. in this paper, we presented a sketch of nlp models helping people who need advice on sensitive topics, which could be a measurable goal for the field. at the same time, we do not claim that our approach is a panacea. there are almost certainly better non-technical solutions to ensure mentorship and legal advice for all @xcite . moreover, there are significant dual-use risks with models that understand language @xcite . our evaluation measures some risks of generative models -such as the tendency to generate toxic language -but more work in this area is needed. and legaladvice. b. we skip 'update' posts, in which a user refers to an older situation that they posted, and 'meta' posts, in which subreddit rules are discussed. c. we skip any post that has an html link, since today's models (presumably) would not be able to visit such a link. d. we skip any post with a score of less than 20. e. we do our best to clean the text of the post. many posts include valid situations, but are then edited to include updates that took place afterwards, in response to advice that was given. these are typically delimited by dashed lines, and the word edit or update. the examples in redditadvice are significantly longer, representing highly complex situations. for a retrieved situation, we do the following to extract valid advice: a. given a post that contains a valid situation, we order the comments from highest-to-lowest scoring. we perform the following checks to determine if we can extract valid advice. once we find valid advice, we will stop iterating. b. we skip any comment that was posted by a moderator, the reddit user who posted the original situation, or that was edited. c. we skip any comment with a score of less than 20. d. we skip any comment that contains fewer than 32 spacy tokens. e. one corner case is highly-scoring advice comments that refer implicitly to others. for instance, a comment might say 'you should listen to the other commenters and...' these references make sense inside a reddit post, however, they are somewhat nonsensical when we pull the comment out of context. we thus skip any comment that seems to refer to others. once we retrieve a situation, that has at least one piece of valid advice, we are done -and we move on to the next situation. we loop over the topscoring 1000 posts in total, and randomly select 200 valid situations from this pool.