ferret: a Framework for Benchmarking Explainers on Transformers (2023)

Abstract: As Transformers are increasingly relied upon to solve complex NLP problems, there is an increased need for their decisions to be humanly interpretable. While several explainable AI (XAI) techniques for interpreting the outputs of transformer-based models have been proposed, there is still a lack of easy access to using and comparing them. We introduce ferret, a Python library to simplify the use and comparisons of XAI methods on transformer-based classifiers. With ferret, users can visualize and compare transformers-based models output explanations using state-of-the-art XAI methods on any free-text or existing XAI corpora. Moreover, users can also evaluate ad-hoc XAI metrics to select the most faithful and plausible explanations. To align with the recently consolidated process of sharing and using transformers-based models from Hugging Face, ferret interfaces directly with its Python library. In this paper, we showcase ferret to benchmark XAI methods used on transformers for sentiment analysis and hate speech detection. We show how specific methods provide consistently better explanations and are preferable in the context of transformer models.

## introduction transformers have revolutionized nlp applications in recent years due to their strong performance on various tasks; their black-box nature remains an obstacle for practitioners who need explanations about why specific predictions were made and what features drove them. the development of explainable ai (xai) techniques on several nlp tasks @xcite has helped bridge this gap by providing insight into the inner workings of transformers and helping users gain trust in their decisions. several xai approaches have been proposed in the literature @xcite @xcite , also tailored to transformer models @xcite @xcite . despite the importance of making xai methods accessible to nlp experts and practitioners through practical tools, there is still a lack of accessibility for transformer models. xai for transformers is mainly scattered and hard to operationalize. methods come with independent implementations or framework-specific libraries that do not allow either evaluation or cross-method comparison. further, existing implementations are not integrated with widespread transformers libraries (e.g., hugging face's transformers @xcite ). the lack of standardization and weak interoperability leaves practitioners with unsolved questions, such as choosing the best method given a task and a model @xcite . we introduce ferret (framework for bench-marking explainers on transformers), an opensource python library that drastically simplifies the use and comparison of xai methods on transformers. the library stems from vertical scientific contributions and focused engineering efforts. on the one hand, ferret provides the first-of-its-kind api (see figure 1 ) to use and compare explanation methods along the established criteria of faithfulness and plausibility @xcite . on the other hand, it integrates seamlessly with transformers @xcite , making it an easy add-on to existing transformer-based pipelines and nlp tasks. ferret permits to run four state-of-the-art xai methods, compute six ad-hoc xai evaluation metrics, and easily load four existing interpretability datasets. further, it offers abstract interfaces to foster future integration of methods, metrics, and datasets. we showcase ferret on sentiment analysis and hate speech detection case studies. faithfulness and plausibility metrics highlight shap (lundberg and lee, 2017) as the most consistent explainer on single-and multiple-samples scenarios. contributions. we release ferret, the first-of-itskind benchmarking framework for interpretability tightly integrated with hugging face's transformers library. we release our code and documentation, foot_0 an interactive demo, foot_1 and a video tutorial. 3 2 library design ferret builds on four core principles. ## case studies we showcase ferret in two real-world tasks, focusing on benchmarking explainers on individual samples or across multiple instances. in the following, we describe how ferret highlights the best explainers in sentiment analysis and hate speech detection tasks. our running examples use an xlm-roberta model fine-tuned for sentiment analysis @xcite and a bert model fine-tuned for hate speech detection @xcite . ## related work this section provides a review of tools and libraries that offer a subset of the ferret's functionalities, namely the option to use multiple xai methods and datasets, evaluation api, transformer-readiness, and built-in visualization. tools for post-hoc xai. toolkits for post-hoc interpretability offer built-in methods to explain model prediction, typically through a code interface. ferret builds on and extends this idea to a unified framework to generate explanations, evaluate and compare them, with support to several xai datasets. moreover, ferret's explainers are integrated with transformers's @xcite principles and conventions. pytorch's captum @xcite ) is a generic python library supporting many interpretability methods. however, the library lacks integration with the hugging face hub and offers no evaluation procedures. allennlp interpret (wallace et al., 2019b) provides interpretability methods based on gradients and adversarial attacks for al-lennlp models @xcite . we borrow the modular and extensible design and extend it to a wider set of explainers. transformers-interpret 6 leverages captum to explain transformer models, but it supports only a limited number of methods. thermostat @xcite exposes pre-computed feature attribution scores through the hugging-face hub but no features oriented to implement or evaluate xai. we support the thermostat as a third-party add-on and let users test and benchmark pre-computed explanations. unlike our study, inseq @xcite focuses on post-hoc interpretability for sequence generation models. although researchers can use the library to add interpretability evaluations to their models, the toolkit lacks built-in evaluation metrics. other related approaches enable global (rather than local) explainability @xcite , or explanation interfaces for non-transformers models on non-nlp tasks @xcite . other approaches study model behavior at the subgroup level @xcite @xcite , focusing more on model evaluation and robustness rather than its interpretation. visualization. most studies that develop visualization tools to investigate the relationships among the input, the model, and the output focus either on specific nlp models -nlpvis @xcite , seq2seq-vis @xcite , or explainers 6 https://github.com/cdpierse/ transformers-interpret -bertviz (vig, 2019), eli5 foot_5 . lit @xcite streamlines exploration and analysis in different models. however, it acts mainly as a graphical browser interface. ferret provides a python interface easy to integrate with pre-existing pipelines. evaluation. although prior works introduced diagnostic properties for xai techniques, evaluating them in practice remains challenging. studies either concentrate on specific model architectures @xcite @xcite , individual datasets @xcite , or a single group of explainability methods @xcite . hence, providing a generally applicable and automated tool for choosing the most suitable method is crucial. to this end, @xcite present a comparative study of xai techniques in three application tasks and model architectures. to the best of our knowledge, we are the first to present a userfriendly python interface to interpret, visualize and empirically evaluate models directly from the hugging face hub across several metrics. we extend previous work from @xcite , who developed a benchmark for evaluating rationales on nlp models called eraser by offering a unified interface for evaluation and visual comparison of the explanations at the instance-and dataset-level. closer to ferret, the openxai framework @xcite enables a systematic evaluation of feature attribution explanation, integrating multiple explainers and xai structured datasets. openxai supports tabular datasets while we focus on textual data and nlp models. ## conclusions we introduced ferret, a novel python framework to easily access xai techniques on transformer models. with ferret, users can explain using stateof-the-art post-hoc explainability techniques, evaluate explanations on several metrics for faithfulness and plausibility, and easily interact with datasets annotated with human rationales. we built ferret with modularity and abstraction in mind to facilitate future extensions and contributions from the community (see appendix b for an overview of the ongoing development). as future work, we envision off-the-shelf support for new nlp tasks and scenarios. building on the classification setup presented in this paper, we plan to add support to more nlp tasks that can be framed as classification, such as mask filling prediction, natural language inference, zero-shot text classification, next sentence prediction, token classification, and multiple-choice qa. one further direction would be improving ferret's interoperability with new libraries, e.g., inseq @xcite for xai on text generation tasks and models.