The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling (2023)

Abstract: Incremental dialogue model components produce a sequence of output prefixes based on incoming input. Mistakes can occur due to local ambiguities or to wrong hypotheses, making the ability to revise past outputs a desirable property that can be governed by a policy. In this work, we formalise and characterise edits and revisions in incremental sequence labelling and propose metrics to evaluate revision policies. We then apply our methodology to profile the incremental behaviour of three Transformer-based encoders in various tasks, paving the road for better revision policies.

## introduction since the dawn of wikipedia, users have made 1.7 × 10 9 edits to its pages. its most revised entry contains 56,713 revisions, all documented in the page history. 1 in such an active community, conflicts inevitably occur. editors can begin competing to override each other's contributions, causing dysfunctional edit warrings. 2 to help regulate the environment, an editing policy is in force, aiming at making edits constructive and improving quality. 3 edits, revisions and policies are key concepts in incremental processing, where a model must rely on partial input to generate partial output. incrementality can help optimise reactivity, naturalness, quality and realism in interactive settings @xcite . this is particularly relevant in dialogue models whose nlu components need to operate on incoming input, e.g. while performing ner, slot filling or disfluency detection, or doing simultaneous translation. local ambiguities in the linguistic input and transient mistakes by the model can result in wrong partial hypotheses, so that the ability to revise, by editing previous outputs, is desirable (kahardipraja 1 according to wikimedia statistics and wiki special. 2 https://en.wikipedia.org/wiki/wikipedia:edit_warring 3 https://en.wikipedia.org/wiki/wikipedia:editing_policy et al., 2023). beyond monitoring the occurrence of edits, it is also beneficial to have a policy regulating when and which revisions should be made, reducing the occurrence of undesirable edits. existing literature using consolidated incremental evaluation metrics falls short in capturing relevant nuances of the incremental behaviour in terms of revisions. in this work, we propose an evaluation methodology for revision policies in incremental sequence labelling. a constructed example is shown in figure 1 , with revisions indicated in the right column. specifically, our contributions to address the identified evaluation gap are: a formalisation of revision policy in incremental sequence labelling, characterising types of edits and of revisions ( §4.1-4.2); a proposal of specialised evaluation metrics for revision policies, accompanied by a discussion on the desired behaviour of incremental processors ( §4.4-4.5); and a demonstration of our methodology with an analysis of the revision policy in three sequence labelling transformer-based models ( §5). 4 2 motivation incremental natural language processing foot_1 has time at front line, being pivotal for interactive settings. at each time step, models must operate on partial input to deliver partial output, but sometimes previous decisions have to be revised. for example, at time step 4 in figure 1 , the labels for the input tokens 2 and 3 were edited into new states. with regard to revisions, at least three types of incremental processors exist, as summarised in they keep an internal state that is updated and used to extend the output at each time step, but cannot revise previous outputs. models that depend on the availability of full sentences at once can be "incrementalised" with the restart-incremental paradigm @xcite , causing revisions to occur via recomputations. 6 cutting-edge nlp models currently rely on transformers @xcite , which are nonincremental. using them in a restart-incremental fashion requires recomputing from scratch at every time step, which we hereby name the naive recomputation policy. it is a very expensive policy because, for a sequence of n tokens, the complexity is n @xmath0.e. the n-th square pyramidal number). besides, this naive approach wastes computational budget, because not all recomputations cause revisions. the results reported by @xcite , for example, show that only around 25% of the recomputations actually changed the output prefix. the disadvantages of the naive policy can be alleviated by a smarter policy that cuts down the number of time steps with recomputations. still, beyond deciding when to recompute, a revision policy par excellence should directly guide the more specific decision of when (and what) to actually revise, and must be evaluated accordingly. ## related literature revisability is in the nature of incremental processing: hypothesis revision is a necessary operation to correct mistakes and build up a high-quality final output @xcite . still, there is a trade-off between requiring that later modules handle a processor's revisions and buying stability by reducing some of its incrementality, which makes the concept of hypothesis stability very relevant @xcite . @xcite argue that performing revisions should not take as long as the initial processing, so as to retain the advantages of incremental processing. they propose two strategies: allowing revisions only within a fixed window or limiting their types. empirically determining how often a model changes the output is an aspect of their analysis we also rely on. the restart-incremental paradigm was investigated for transformer-based sequence labelling by @xcite and @xcite ; recently, adaptive policies were proposed to reduce the computational load @xcite . @xcite and @xcite explored adaptation strategies to use transformers for incremental disfluency detection. in simultaneous translation, where policies are a central concept @xcite , the restartincremental approach is in use and revisions are studied @xcite . sequence labelling is a staple of various incremental linguistic tasks possibly used in dialogue systems, like srl @xcite , postagging @xcite , dialogue act segmentation @xcite , disfluency detection @xcite and dependency parsing @xcite . ## evaluation methodology in this section, we present our evaluation methodology for incremental sequence labelling with a focus on revisions. after formalising the task, we characterise revisions and edits, define policies and revision-oriented metrics and discuss the ideal behaviour of incremental sequence labelling models. ## architecture profiling we now apply our methodology to profile the revision policy behaviour of three models: the reference restart-incremental transformer and the two tapir variations, which have a recomputation policy, proposed by @xcite . we evaluate them on three sequence labelling tasks: slot filling @xcite , pos tagging @xcite and ner @xcite , using the final output as gold standard. 14 note that the same profiling can be applied to any model with the ability of performing revisions on any sequence labelling task. quantitative assessment the cumulative distributions of the fraction of time steps with revisions per sentence, shown in figure 5 , illustrate that the policy reduces the number of revisions per sentence: 50% or less of the sentences have no revisions in the naive policy, which makes all recomputation effort be used to perform only an addition, while tapir's policy caused more sentences to not trigger revisions. ## conclusion in this work, we have argued that the importance of a solid evaluation framework for revision policies