Azimuth: Systematic Error Analysis for Text Classification (2022)

Abstract: We present Azimuth, an open-source and easy-to-use tool to perform error analysis for text classification. Compared to other stages of the ML development cycle, such as model training and hyper-parameter tuning, the process and tooling for the error analysis stage are less mature. However, this stage is critical for the development of reliable and trustworthy AI systems. To make error analysis more systematic, we propose an approach comprising dataset analysis and model quality assessment, which Azimuth facilitates. We aim to help AI practitioners discover and address areas where the model does not generalize by leveraging and integrating a range of ML techniques, such as saliency maps, similarity, uncertainty, and behavioral analyses, all in one tool. Our code and documentation are available at github.com/servicenow/azimuth.

## introduction as academic and research labs push the boundaries of artificial intelligence, more and more enterprises are including nlp models 1 in their real-world systems. this is exciting yet risky due to the complexity of current deep learning models and their increasing social impact. whereas in traditional software development, engineers have methods to trace errors to code, it is challenging to isolate sources of error in ai systems. nlp models can suffer from poor linguistic capabilities @xcite , hallucinations @xcite , learning spurious correlations via annotation artifacts @xcite , or amplifying social biases @xcite . in addition to the adverse effects that these problems can cause to both individuals and society, deploying problematic models can lead to legal and financial penalties (burt, 1 while models are part of pipelines that can include preand post-processors, we use the term models for simplicity. 2021). even when poorly functioning models have no adverse social effects, they can still degrade trust, leading to problems with user adoption @xcite . while it may be unreasonable to expect a perfect model, ai practitioners should communicate existing limitations so that stakeholders can make informed decisions about deployment, risk mitigation, and allocation of resources for further improvements @xcite . however, the current common error analysis practices may not be sufficiently thorough to provide this visibility, limiting our capacity to build safe, trustworthy ai systems. as part of the ml development cycle, practitioners choose appropriate metrics based on business requirements. these metrics should assess whether nlp models have acquired the linguistic skills to perform the specified task. while metrics are useful to rapidly compare different models, they are only the beginning of quality assessment as they do not expose failure modes, i.e., types of input where the model fails, and do not readily indicate what can be improved. furthermore, relying on metrics alone can be harmful, as they can overestimate quality and robustness while hiding unintended biases @xcite . to discover and address these limitations, er-ror analysis is crucial. unfortunately, compared to other stages of the ml development cycle, the error analysis stage is less mature. while various tools are commonly used to train and tune neural networks 2 , there is less convergence and adoption of both standards and tools to analyze errors. analysis is typically conducted using custom scripts, spreadsheets, and jupyter notebooks, guided by a practitioner's intuition, which may help uncover some problems while missing others. large evaluation sets make per-example analysis of incorrect predictions time-consuming and tedious. because this error analysis process is often informal, ad hoc, and cumbersome, practitioners often skip or rush this stage, focusing solely on high-level metrics. this hinders traceability and accountability, and introduces risks, especially for models deployed in production. to alleviate these issues, we contribute the following: ## systematic error analysis the goal of error analysis is to understand where and why a model succeeds or fails, to better inform both model improvement and deployment decisions. we propose a process grouped into two categories: (1) dataset analysis and (2) model quality assessment, as illustrated in fig. 2 . dataset analysis involves assessing and improving the data used to train and evaluate the model, while quality assessment focuses on model behavior. this process should be iterative, as analyzing model predictions can help identify and fix dataset problems, and dataset findings can help explain and reduce model errors. a limitation of our approach is that it does not currently assess whether models are learning a task ethically. while some of the techniques we refer to below can be used for this purpose, building ethical nlp systems is outside the scope of this paper. ## azimuth, an open-source tool azimuth was developed as an internal tool at ser-vicenow and open sourced in april 2022. it facilitates our proposed approach to systematize error analysis of ml systems. while it is currently tailored for text classification, it could be extended to support other use cases. the tool was built by a cross-collaborative team of scientists, engineers, and designers, with a human-centered approach that focuses on the ai practitioner's needs. before launching azimuth, the user defines in a configuration file the dataset splits and pipelines to load and analyze. in azimuth, pipelines refer to the ml model as well as any pre-processing and post-processing steps. the tool is built on top of the hugging face (hf) datasets library @xcite and easily interfaces with hf pipelines @xcite . for flexibility, any python function can be defined as a pipeline. if pipelines are unavailable, azimuth can still be used, although with limited features, by reading predictions from a file. the tool can also be used for dataset analysis without a pipeline. ## related work there exist other tools that can help practitioners evaluate their nlp systems beyond observing metrics. broadly speaking, these solutions are implemented as component libraries, standalone applications, or some combination thereof. small components have a lower investment of effort to get started but may require more technical expertise to use. while they can produce results quickly, they fail to address the problem of ad-hoc processes and may lead to a "paradox of choice" @xcite . examples include the check-list python package (ribeiro et al., 2020) that can be used for behavioral testing and the allennlp interpret toolkit @xcite that computes gradient-based saliency maps. on the other hand, standalone applications may require more effort to set up and it may not be obvious how to integrate them into the ml development process. their benefits are that, once setup, less technical practitioners can use them and their usage can be standardized. crosscheck @xcite and robustness gym @xcite are used as jupyter widgets, while errudite @xcite and language interpretability tool (lit) @xcite exist both as standalone applications and as widgets in jupyter notebooks. currently, azimuth is a standalone application that requires low setup effort and can be integrated into the ml development cycle as proposed in section 2. at the same time, we provide the benefits of existing third-party component libraries, added as features in an easy-to-use interface. we built azimuth to assist in performing comprehensive error analysis using a single tool by including functionality found elsewhere: filtering and analysis of behavior on subpopulations (crosscheck, errudite, lit, robustness gym), input variations such as counterfactual error analysis, robustness testing (errudite, lit, robustness gym), model comparison (cross-check, lit), and explainability techniques (lit, allennlp interpret). ## conclusion we propose a systematic approach to error analysis as an iterative workflow between dataset analysis and model quality assessment. we contribute azimuth to the nlp community in order to facilitate this approach. future work includes expanding our capabilities, exposing potential ethical concerns in the data or ml models, and extending azimuth to cover other tasks and domains. we welcome open-source contributions.