Targeting the Benchmark: On Methodology in Current Natural Language Processing Research (2021)

Abstract: It has become a common pattern in our field: One group introduces a language task, exemplified by a dataset, which they argue is challenging enough to serve as a benchmark. They also provide a baseline model for it, which then soon is improved upon by other groups. Often, research efforts then move on, and the pattern repeats itself. What is typically left implicit is the argumentation for why this constitutes progress, and progress towards what. In this paper, we try to step back for a moment from this pattern and work out possible argumentations and their parts.

## introduction the goal of any field of research is to make progress towards answering its foundational questions. to do so, a methodology is required that guides attempts at providing or improving answer proposals. in natural language processing, the object of study is human language, and any methodology for doing research in this field will need to have some contact with examples of this object. this contact has become more and more direct in the past decades, with samples of language becoming more directly the material from which proposals (in the form of statistical models) are derived. recent years have seen an increase in the collection of samples specifically for the purpose of creating benchmarks, against which progress in devising models can be measured. it is this function of benchmarking, and its role in a progress-oriented methodology, that this paper aims to investigate. figure 1 illustrates the basic structure of a benchmarking methodology: a language task is devised that is a) restricted enough to be managable with current methods, and b) deemed challenging for the capabilities that it involves. @xcite for this task, a dataset 1 this figure is from (schlangen, 2019), of which this is a shorter version developed in a somewhat different direction. is collected, often via crowd sourcing, on which in turn models are trained and compared, using evaluation metrics defined together with the task. what can we learn by following such a methodology? let's look at the components first and then at ways in which this methodology is, might, and perhaps should be used. in computing, a benchmark is "a problem that has been designed to evaluate the performance of a system [which] is subjected to a known workload and the performance of the system against this workload is measured. typically the purpose is to compare the measured performance with that of other systems that have been subject to the same benchmark test." (butterfield et al., 2016). the use of this term in nlp is related: here, benchmark tasks are also specifically designed for evaluation; however, an important difference is that what is being evaluated is not a full system that has a separate main purpose, but rather an algorithm that is instantiated on the benchmark itself. i will discuss the consequences of this below. this kind of evaluation of learning algorithm has a long tradition in the field of machine learning research. @xcite in this field, a new algorithm would normally be tested on a large collection of datasets, possibly ranging from classifications of flowers to classification of credit records, with no assumption of any internal connection between the datasets. again, nlp is different here, as all datasets represent facets of the same underlying phenomenon, language use. i will argue that these two differences (life outside of benchmarking, and internal connection between tasks) are important, but understudied. but first we look at the notion of a language task in some more detail. ## conclusions in this short paper, i have discussed the methodology of using language tasks to drive research on models of language competence. i have argued that the success of this approach hinges on how well progress on one task can be translated into progress on other tasks. while some steps have been taken in this direction, current work still appears to mostly focus on isolated tasks (or groups of tasks). overcoming this, in my opinion, will require more explicit considerations about how tasks and capabilities are connected, and how the set of capabilities is structured-to ensure movement is not only uphill, but rather up the right hill (bender and koller, 2020), and it indeed is a single hill. for this, a (re-)connection with the fields that study the composition of language competence-linguistics and cognitive and developemental psychologyseems advisable (if only to disagree explicitly). as a positive proposal, i suggest that a focus should be put on assembling a curriculum of tasks, organised in a complexity and inclusion hierarchy, and that the benchmarking target should be the developmental trajectory on this. working this out in detail i must leave for future work.