  We introduce DynaSent ('Dynamic Sentiment'), a new English-language benchmark
task for ternary (positive/negative/neutral) sentiment analysis. DynaSent
combines naturally occurring sentences with sentences created using the
open-source Dynabench Platform, which facilities human-and-model-in-the-loop
dataset creation. DynaSent has a total of 121,634 sentences, each validated by
five crowdworkers, and its development and test splits are designed to produce
chance performance for even the best models we have been able to develop; when
future models solve this task, we will use them to create DynaSent version 2,
continuing the dynamic evolution of this benchmark. Here, we report on the
dataset creation effort, focusing on the steps we took to increase quality and
reduce artifacts. We also present evidence that DynaSent's Neutral category is
more coherent than the comparable category in other benchmarks, and we motivate
training models from scratch for each round over successive fine-tuning.
