Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks (2022)

Abstract: We introduce Dynatask: an open source system for setting up custom NLP tasks that aims to greatly lower the technical knowledge and effort required for hosting and evaluating state-of-the-art NLP models, as well as for conducting model in the loop data collection with crowdworkers. Dynatask is integrated with Dynabench, a research platform for rethinking benchmarking in AI that facilitates human and model in the loop data collection and evaluation. To create a task, users only need to write a short task configuration file from which the relevant web interfaces and model hosting infrastructure are automatically generated. The system is available at https://dynabench.org/ and the full library can be found at https://github.com/facebookresearch/dynabench .

## introduction data is the backbone of nlp research. one of the most fruitful approaches for making progress on nlp tasks has historically been benchmarking. benchmarking is where the community adopts a high quality dataset for a particular task and tests various models against it to determine which is best. the process of benchmarking requires the effort of a large number of researchers, who collect and clean data, train and evaluate models, and work to understand model weaknesses. this process is iterative: once models perform very highly on the currently accepted community benchmark, another is created to push progress further. taken as a whole, the benchmarking process is both notoriously difficult and expensive. this is due to a variety of facts: the community is a loose conglomeration of researchers with different areas of expertise, there is ever increasing need for larger datasets @xcite , and the ai community has historically under-valued @xcite and under-invested in data collection and best practices @xcite @xcite . to make matters worse, in recent years, benchmarks have been saturating with increasing speed. taking the trends from the greater ai community into account, it took mnist @xcite , switchboard @xcite , and im-agenet @xcite several years to saturate, and newer benchmarks such as squad @xcite , glue @xcite , and super-glue @xcite about a year. because of this, data-centric approaches are gaining more attention @xcite @xcite @xcite . this trend is clear evidence of the urgency of finding a sustainable and data-centric way to support the full benchmarking ecosystem, from end-to-end, in a way that causes the least amount of friction for anyone who wants to use it. in this paper, we introduce our answer to these issues: an easy-to-use, open source system that integrates the creation of benchmark datasets for any task, the selection of appropriate metrics, and the evaluation of models while natively supporting revisions to the benchmark as models saturate the original version. we share a unified library that enables these functionalities for the dynabench platform @xcite . ## background dynabench was proposed as an open-source and community-driven platform to host dynamic benchmarks. the existing dynabench tasks avoid saturation by leveraging crowdworkers who continually interact with state-of-the-art models. crowdworkers either write examples that fool existing models @xcite , or collaborate with generative models to increase example diversity @xcite . each task is administered by one or more task owners from the research community who col- a task is an essential concept in understanding our work. on dynabench, a distinct task is a particular relationship between inputs and outputs. 1 inputs and outputs are framed within some prespecified format. for example, natural language inference is a task on dynabench. the input format is two strings and the output format is a classification label. the relationship between the inputs and outputs is defined by what humans would do when loosely instructed to treat the input strings as a context (sometimes called the "premise") and a hypothesis, and return a label for whether they think the hypothesis is entailed by the context. mnli @xcite , snli @xcite , and anli @xcite ) can be viewed as different datasets that instantiate the same task. @xcite takes a similar view. ## dynatask before the introduction of dynatask, adding a new task required close collaboration between task owners and the dynabench team, and extensive software contributions to the dynabench codebase. this paper presents a system that enables dynabench to scale up to more tasks, including into multimodal and multilingual domains, without such requirements. now, a task owner can create their own task page on dynabench with a short task config file. the config file is used to automatically generate crowdworker data collection interfaces, as well as the model and dataset hosting/evaluating infrastructure. the data collection interfaces and hosting overlay existing services such as amazon mechanical turk, foot_1 which provide a workforce and payment mechanisms, but do not provide crowdworker interfaces for dynamic model-in-the-loop data collection or their corresponding backends. in fact, local installations of dynabench can be run on mechanical turk. overall, a dynabench task owner can set up and host: crowdworker data collection: task owners can configure interfaces for data collection. models-inthe-loop can be optionally added, so crowdworkers can receive real-time model responses from their data (figure 1 ). crowdworker data validation: task owners can configure interfaces for crowdworkers to label collected examples as correct or incorrect. (figure 1 ). dynamic dataset metrics: metrics on the crowdworker data are computed, such as verified model error rate (vmer) @xcite . crowdworker example leaderboards are displayed. a train file leaderboard: task owners can enable users to upload training data files for the automatic creation, training, and evaluation of models in our evaluation cloud. a dynamic and interactive model leaderboard @xcite : task owners can configure a leaderboard, selecting from a variety of metrics to determine model performance. owners can also upload new datasets, which triggers automatic evaluation for all of the user-uploaded models. every leaderboard model can be interacted with in real-time. see figure 2 for an example. a model upload pipeline: once a new task goes live on dynabench, our command line tool foot_2 allows anyone to create a handler script and upload models by following a few command line instructions. after models are uploaded, they are dockerized and deployed automatically. models can be viewed on the leaderboard and put in-the-loop with crowdworkers for data collection. ## decentralized evaluation-as-a-service most task owners currently use the centralized dynabench evaluation and model deployment server. with dynatask, however, we offer a decentralized evaluation feature that will increase the platform's flexibility even further. with this feature, task owners can set up a dynabench model deployment and evaluation server or select an existing one. to set up a new server, an owner only needs to follow our documentation, creating an aws account and installing some dynabench code along the way. distributed hosting of model building and evaluation enables dynatask to scale: no one organization needs to fund hosting for all of the models on dynabench, and every owner of a model deployment and evaluation server can flexibly upload or take down models to suit their budget. it is also designed with re-usability in mind: several tasks can share the same evaluation servers. task owners do not need to do any setup if they have permission to use an existing evaluation server. most of the following projects (besides image labelling and open domain qa) were added to dynabench before the introduction of dynatask, which took months of coding in every case. with dynatask, they can all be implemented in minutes. ## conclusion we introduced dynatask, a collection of open source features in the dynabench platform that empowers anyone to create and own a task on dynabench with only a short config file. dynabench started as an nlp project with only four englishonly tasks. since then, dynatask has helped researchers produce several datasets and host competitions, expanding scalably into multimodal and multilingual domains with owners from various corners of the ai community. dynatask offers the functionalities of dynabench to the broader research community by allowing them to easily create and host new ai tasks on the platform: it provides a one-stop shop for constructing datasets with or without models in the loop, hosting challenges and competitions, investigating the effects of models in the loop, characterizing distributional shift and continual learning, exploring annotator efficiency and expertise, and improving model robustness through collaboration with humans. finally, dynabench is an open source, community-driven effort. anyone who wants to add a new input/output type, a new metric, or any other new feature, need only submit a pull request. we hope that our our work can help enable new exciting scientific progress in data-centric ai research in general and dynamic (adversarial) data collection in particular.