T ext F lint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing (2021)

Abstract: TextFlint is a multilingual robustness evaluation toolkit for NLP tasks that incorporates universal text transformation, task-specific transformation, adversarial attack, subpopulation, and their combinations to provide comprehensive robustness analyses. This enables practitioners to automatically evaluate their models from various aspects or to customize their evaluations as desired with just a few lines of code. TextFlint also generates complete analytical reports as well as targeted augmented data to address the shortcomings of the model in terms of its robustness. To guarantee acceptability, all the text transformations are linguistically based and all the transformed data selected (up to 100,000 texts) scored highly under human evaluation. To validate the utility, we performed large-scale empirical evaluations (over 67,000) on state-of-the-art deep learning models, classic supervised methods, and real-world systems. The toolkit is already available at https://github.com/textflint with all the evaluation results demonstrated at textflint.io.

## introduction the detection of model robustness has been attracting increasing attention in recent years, given that deep neural networks (dnns) of high accuracy can still be vulnerable to carefully crafted adversarial examples @xcite , distribution shift @xcite , data transformation @xcite , and shortcut learning @xcite . existing approaches to textual robustness evaluation focus on slightly modifying the input data, which maintains the original meaning and results in a different prediction. however, these methods often concentrate on either universal or ## textflint framework textflint is designed to be flexible enough to allow practitioners to configure the workflow while providing appropriate abstractions to alleviate the concerns of the low-level implementation. according to its pipeline architecture, textflint can be organized into three blocks, as shown in figure 2 : (a) input layer, which prepares the necessary information for sample generation; (b) generation layer, which applies generation functions to each sample; and (c) reporter layer, which analyzes the evaluation results and generates a robustness report. ## usage using textflint to verify the robustness of a specific model is as simple as running the following command: moreover, it also supports the configuration of subpopulation and adversarial attack. for more details about parameter configuration, please move to https://github.com/textflint/textflint . based on the design of the decoupling sample generation and model verification, textflint can be used inside another nlp project with just a few lines of code. from textflint import engine @xmath0.json' @xmath1.run(data_path, config) textflint is also available for use through our web demo, displayed in figure 4 , which is available at https://www.textflint.io/demo . case studies of usage due its user-friendly design philosophy, textflint shows its practicality in real applications. we summarize three occasions in which model robustness evaluation is challenging: case 1: general evaluation for users who want to evaluate the robustness of nlp models in a general way, textflint supports generating massive and comprehensive transformed samples with just one command. by default, textflint performs all single transformations on the original dataset to form the corresponding transformed datasets, and the performance of the target models is tested on these datasets. the evaluation report provides a comparative view of model performance on datasets before and after certain types of transformations, which supports model weakness analyses and guides particular improvements. for example, take bert base @xcite as the target model to verify its robustness on the conll2003 dataset @xcite , whose robustness report is shown in figure 5 . the performance of bert base decreases significantly in some morphology transformations, such as ocr, keyboard, typos, and spelling error. to combat these errors of input texts and improve the robustness of the model, we suggest that placing a word correction model @xcite before bert would be beneficial. case 2: customized evaluation for users who want to test model performance on specific aspects, they demand a customized transformed dataset of certain transformations or their combinations. in textflint, this could be achieved by modifying config, which determines the configuration of textflint in generation. config specifies the transformations being performed on the given dataset. it can be modified manually or generated automatically. by modifying the configuration, users could decide to generate multiple transformed samples on each original data sample, validate samples by semantics, preprocess samples with certain processors, and more. case 3: target model improvement for users who want to improve the robustness of target models, they may work hard to inspect the weakness of a model with less alternative support. to tackle the issue, we believe a diagnostic report revealing the influence of comprehensive aspects on model performance will provide concrete suggestions on model improvement. by using textflint and applying a transformed dataset to target models, the transformations corresponding to significant performance decline in the evaluation report will provide guidance for improvements to the target models. moreover, textflint supports adversarial training on target models with a largescale transformed dataset, and the change of performance will also be reported to display performance gain due to adversarial training. ## benchmarking existing models with textflint to verify the quality of transformation, we conducted human evaluation on the original and transformed texts under all of the mentioned transformations. specifically, we considered two metrics in human evaluation: plausibility and gram- from the numerous evaluations and comparisons conducted by textflint, we have a thorough view of existing nlp systems and discovered underlying patterns about model robustness. as for the absa task ( ## related work our work is related to many existing open-source tools and works in different areas. robustness evaluation many tools include evaluation methods for robustness, including nlpaug @xcite , errudite @xcite , allennlp interpret @xcite , which are only applicable to limited parts of robustness evaluations, while textflint supports comprehensive evaluation methods, e.g., subpopulation, adversarial attacks, transformations, and so on. besides the common transformation methods like synonym substitution and typos, various task-specific transformations are tailored for each of the 12 nlp tasks, which is peculiar to textflint. moreover, we are the first to provide linguistic support for the transformations, the designs for which were inspired by linguistics and have been proved plausible and readable by human annotators. several tools also exist concerning robustness, which are similar to our work @xcite @xcite and include a wide range of evaluation methods. however, these tools only focus on general generalization evaluations and lack quality evaluations on generated texts or only support automatic quality constraints. textflint supports both general and task-specific evaluations and guarantees the acceptability of each transformation method with human evaluations. in addition, textflint provides a standard report that can be displayed with visualization and tabulation. more importantly, all of the tools and modules are encapsulated within a unified framework, which completely differs from robustness gym @xcite , a simple aggregation of apis of various tools including checklist @xcite and @xcite . in addition, all of the transformations can be realized automatically by a simple modification to the configuration in textflint, while manually defined patterns are required by some of the functions in robustness gym. ## conclusion we introduced textflint, a unified multilingual robustness evaluation toolkit that incorporates universal text transformation, task-specific transformation, adversarial attack, subpopulation, and their combinations to provide comprehensive robustness analyses. textflint enables practitioners to evaluate their models with just a few lines of code and then obtain complete analytical reports. additionally, we also performed large-scale empirical evaluations on state-of-the-art deep learning models, classic supervised methods, and realworld systems, with all the experimental results reported on our website. almost all models showed significant performance degradation, indicating the urgency and necessity of including robustness in nlp model evaluations.