Switching Contexts: Transportability Measures for NLP (2021)

Abstract: This paper explores the topic of transportability, as a sub-area of generalisability. By proposing the utilisation of metrics based on well-established statistics, we are able to estimate the change in performance of NLP models in new contexts. Defining a new measure for transportability may allow for better estimation of NLP system performance in new domains, and is crucial when assessing the performance of NLP systems in new tasks and domains. Through several instances of increasing complexity, we demonstrate how lightweight domain similarity measures can be used as estimators for the transportability in NLP applications. The proposed transportability measures are evaluated in the context of Named Entity Recognition and Natural Language Inference tasks.

## introduction the empirical evaluation of the quality of nlp models under a specific task is a fundamental part of the scientific method of the nlp community. however, commonly, many proposed models are found to perform well in the specific context in which they are evaluated and state-of-the-art claims are usually found not transportable to similar but different settings. the current evaluation metrics may only indicate which algorithm or setup performs best: they are unable to estimate performance in a new context, to demonstrate internal validity, or to verify causality. to offset this, statistical significance testing is sometimes applied in conjunction with performance measures (e.g. f1-score, bleu) to attempt to establish validity. however, statistical significance testing has been shown to be lacking. @xcite reviewed nlp papers from acl17 and tacl17 and found that only a third of these papers use significance * equal contribution testing. further, many papers did not specify the type of test used, and some even employed an inappropriate statistical test. performance is measured in nlp tasks primarily through f1 score or task-specific metrics such as bleu. the limited scope of these as performance evaluation techniques has been shown to have issues. @xcite highlights the data selection bias in nlp system performance. @xcite show issues of using standard splits, as opposed to random splits. we support their statement that "practitioners who wish to firmly establish that a new system is truly state-of-the-art augment their evaluations with bonferroni-corrected random split hypothesis testing". in an nli task, using snli and multinli datasets with a set of different models, it has been shown that permutations of training data leads to substantial changes in performance @xcite . further, the lack of transportability for nlp tasks has been raised by specialists in applied domains. for example, healthcare experts have expressed their frustration in the limitations of algorithms built in research settings for practical applications @xcite and the reduction of performance "outside of their development frame" @xcite . more generally, "machine learning researchers have noted current systems lack the ability to recognize or react to new circumstances they have not been specifically programmed or trained for" @xcite . the advantages of "more transportable" approaches, such as bert, in terms of their performance in multiple different domains, is currently not expressed (other than the prevalence of such architectures across a range of state-of-the-art tasks and domains). to support analysis and investigation into the insight that could be gained by examination of these properties, we suggest metrics and a method for measuring the transportability of models to new domains. this has immediate relevance for domain experts, wishing to implement existing solutions on novel datasets, as well as for nlp researchers wishing to assemble new dataset, design new models, or evaluate approaches. to support this, we propose feature gradient, and show it to have promise as a way to gain lexical or semantic insight into factors influencing the performance of different architectures in new domains. this differs from data complexity, being a comparative measure between two datasets. we aim to start a conversation about evaluation of systems in a broader setting, and to encourage the creation and utilisation of new datasets. this paper focuses on the design and evaluation of a lightweight transportability measure in the context of the empirical evaluation of nlp models. a further aim is to provide a category of measures which can be used to estimate the stability of the performance of a system across different domains. an initial transportability measure is built by formalising properties of performance stability and variation under a statistical framework. the proposed model is evaluated in the context of named entity recognition tasks (ner) and natural language inference (nli) tasks across different domains. our contribution is to present a measure that evaluates the transportability and robustness of an nlp model, to evaluate domain similarity measures to understand and anticipate the transportability of an nlp model, and to compare state of the art models across different datasets for ner and nli. ## conclusion we have presented a model of transportability for nlp tasks, together with metrics to allow for the quantification in the change in performance. we have shown that the proposed transportability measure allows for direct comparison of nlp systems' performance in new contexts. further, we demonstrated domain similiarity as a measure to model corpus and domain complexity, and predict nlp system performance in unseen domains. this paper lays the foundations for further work in more complex transportability measures and estimation of nlp system performance in new contexts.