D yna S ent: A Dynamic Benchmark for Sentiment Analysis (2021)

Abstract: We introduce DynaSent (‘Dynamic Sentiment’), a new English-language benchmark task for ternary (positive/negative/neutral) sentiment analysis. DynaSent combines naturally occurring sentences with sentences created using the open-source Dynabench Platform, which facilities human-and-model-in-the-loop dataset creation. DynaSent has a total of 121,634 sentences, each validated by five crowdworkers, and its development and test splits are designed to produce chance performance for even the best models we have been able to develop; when future models solve this task, we will use them to create DynaSent version 2, continuing the dynamic evolution of this benchmark. Here, we report on the dataset creation effort, focusing on the steps we took to increase quality and reduce artifacts. We also present evidence that DynaSent’s Neutral category is more coherent than the comparable category in other benchmarks, and we motivate training models from scratch for each round over successive fine-tuning.

## introduction sentiment analysis is an early success story for nlp, in both a technical and an industrial sense. it has, however, entered into a more challenging phase for research and technology development: while present-day models achieve outstanding results on all available benchmark tasks, they still fall short when deployed as part of real-world systems @xcite @xcite and display a range of clear shortcomings @xcite @xcite @xcite . in this paper, we seek to address the gap between benchmark results and actual utility by introduc-figure 1 : the dynasent dataset creation process. the human validation task is the same for both rounds; five responses are obtained for each sentence. on dynabench, we explore conditions with and without prompt sentences that workers can edit to achieve their goal. figure 1 summarizes our method, which incorporates both naturally occurring sentences and sentences created by crowdworkers with the goal of fooling a top-performing model. the starting point is model 0, which is trained on standard sentiment benchmarks and used to find challenging sentences in existing data. these sentences are fed into a human validation task, leading to the round 1 dataset. next, we train model 1 on round 1 in addition to publicly available datasets. in round 2, this model runs live on the dynabench platform for humanand-model-in-the-loop dataset creation; foot_1 crowdworkers try to construct examples that fool model 1. these examples are human-validated, which results in the round 2 dataset. taken together, rounds 1 and 2 have 121,634 sentences, each with five human validation labels. thus, with only two rounds collected, dynasent is already a substantial new resource for sentiment analysis. in addition to contributing dynasent, we seek to address a pressing concern for any dataset collection method in which workers are asked to construct original sentences: human creativity has intrinsic limits. individual workers will happen upon specific strategies and repeat them, and this will lead to dataset artifacts. these artifacts will certainly reduce the value of the dataset, and they are likely to perpetuate and amplify social biases. we explore two methods for mitigating these dangers. first, by harvesting naturally occurring examples for round 1, we tap into a wider population than we can via crowdsourcing, and we bring in sentences that were created for naturalistic reasons, rather than the more artificial goals present during crowdsourcing. second, for the dynabench cases created in round 2, we employ a 'prompt' setting, in which crowdworkers are asked to modify a naturally occurring example rather than writing one from scratch. we compare these sentences with those created without a prompt, and we find that the prompt-derived sentences are more like naturally occurring sentences in length and lexical diversity. of course, fundamental sources of bias remain -we seek to identify these in the datasheet @xcite distributed with our datasetbut we argue that these steps help, and can inform crowdsourcing efforts in general. as noted above, dynasent presently uses the labels positive, negative, and neutral. this is a minimal expansion of the usual binary (positive/negative) sentiment task, but a crucial one, as it avoids the false presupposition that all texts convey binary sentiment. we chose this version of the problem to show that even basic sentiment analysis poses substantial challenges for our field. we find that the neutral category is especially difficult. while it is common to synthesize such a category from middle-scale product and service reviews, we use an independent validation of the stanford sentiment treebank @xcite dev set to argue that this tends to blur neutrality together with mixed sentiment and uncertain sentiment (section 5.2). dynasent can help tease these phenomena apart, since it already has a large number of neutral examples and a large number of examples displaying substantial variation in validation. finally, we argue that the variable nature of the neutral category is an obstacle to fine-tuning (section 5.3), which favors our strategy of training models from scratch for each round. ## related work sentiment analysis was one of the first natural language understanding tasks to be revolutionized by data-driven methods. rather than trying to survey the field (see @xcite @xcite , we focus on the benchmark tasks that have emerged in this space, and then seek to situate these benchmarks with respect to challenge (adversarial) datasets and crowdsourcing methods. ## round 1: naturally occurring sentences we now begin to describe our method for constructing dynasent (figure 1 ). the current section focuses on model 0 and round 1, and section 4 explains how these feed into model 1 and round 2. ## round 2: dynabench in round 2, we leverage dynabench to begin creating a new dynamic sentiment benchmark. ## discussion we now address a range of issues that our methods raise but that we have so far deferred in the interest of succinctly reporting on the methods themselves. ## conclusion we presented dynasent, as the first stage in an ongoing effort to create a dynamic benchmark for sentiment analysis. to date, the best future-looking model 2 we have developed achieves 83.1 f1 on round 1 and 70.8 f1 on round 2 while maintaining good performance on our external benchmarks. appendix e provides details on this model and others, and the dynabench platform offers a detailed and up-to-date leaderboard. we hope and expect that the community will find models that solve both rounds. that will be our cue to launch another round of data collection to fool those models and push the field of sentiment forward by another step.