Inseq: An Interpretability Toolkit for Sequence Generation Models (2023)

Abstract: Past work in natural language processing interpretability focused mainly on popular classification tasks while largely overlooking generation settings, partly due to a lack of dedicated tools. In this work, we introduce Inseq, a Python library to democratize access to interpretability analyses of sequence generation models. Inseq enables intuitive and optimized extraction of models’ internal information and feature importance scores for popular decoder-only and encoder-decoder Transformers architectures. We showcase its potential by adopting it to highlight gender biases in machine translation models and locate factual knowledge inside GPT-2. Thanks to its extensible interface supporting cutting-edge techniques such as contrastive feature attribution, Inseq can drive future advances in explainable natural language generation, centralizing good practices and enabling fair and reproducible model evaluations.

## introduction recent years saw an increase in studies and tools aimed at improving our behavioral or mechanistic understanding of neural language models (belinkov and glass, 2019). in particular, feature attribution methods became widely adopted to quantify the importance of input tokens in relation to models' inner processing and final predictions @xcite . many studies applied such techniques to modern deep learning architectures, including transformers @xcite , leveraging gradients @xcite , attention patterns @xcite and input perturbations @xcite to quantify input importance, often leading to controversial outcomes in terms of faithfulness, plausibility and overall usefulness of such explanations (adebayo 1 library: https://github.com/inseq-team/inseq documentation: https://inseq.readthedocs.io this paper describes the inseq v0.4.0 release on pypi. in this work, we introduce inseq, a python library to democratize access to interpretability analyses of generative language models. inseq centralizes access to a broad set of feature attribution methods, sourced in part from the captum @xcite ) framework, enabling a fair comparison of different techniques for all sequence-tosequence and decoder-only models in the popular transformers library @xcite . thanks to its intuitive interface, users can easily integrate interpretability analyses into sequence generation experiments with just 3 lines of code (figure 2 ). nevertheless, inseq is also highly flexible, including cutting-edge attribution methods with built-in post-processing features ( § 4.1), supporting customizable attribution targets and enabling constrained decoding of arbitrary sequences ( § 4.2). in terms of usability, inseq greatly simplifies access to local and global explanations with built-in support for a command line interface (cli), optimized batching enabling dataset-wide attribution, and various methods to visualize, serialize and reload attribution outcomes and generated sequences ( § 4.3). ultimately, inseq's aims to make sequence models first-class citizens in interpretability research and drive future advances in interpretability for generative applications. tools for nlp interpretability although many post-hoc interpretability libraries were released recently, only a few support sequential feature attribution. notably, lit @xcite , a structured framework for analyzing models across modalities, and ecco @xcite , a library specialized in interactive visualizations of model internals. lit is an all-in-one gui-based tool to analyze model behaviors on entire datasets. however, the library does not provide out-of-the-box support for transformers models, requiring the definition of custom wrappers to ensure compatibility. moreover, it has a steep learning curve due to its advanced ui, which might be inconvenient when working on a small amount of examples. all these factors limit lit usability for researchers working with custom models, needing access to extracted scores, or being less familiar with interpretability research. on the other hand, ecco is closer to our work, being based on transformers and having started to support encoder-decoder models concurrently with inseq development. despite a marginal overlap in their functionalities, the two libraries provide orthogonal benefits: inseq's flexible interface makes it especially suitable for methodical quantitative analyses involving repeated evaluations, while ecco excels in qualitative analyses aimed at visualizing model internals. other popular tools such as eraser (deyoung et al., 2020), thermostat @xcite , transformersinterpret @xcite and ferret @xcite do not support sequence models. ## design inseq combines sequence models sourced from transformers @xcite and attribution methods mainly sourced from captum @xcite . while only text-based tasks are currently supported, the library's modular design 3 would enable the inclusion of other modeling frameworks (e.g. fairseq (ott et al., 2019)) and modalities (e.g. speech) without requiring substantial redesign. optional dependencies include datasets @xcite and rich 4 . ## conclusion we introduced inseq, an easy-to-use but versatile toolkit for interpreting sequence generation models. with many libraries focused on the study of classification models, inseq is the first tool explicitly aimed at analyzing systems for tasks such as machine translation, code synthesis, and dialogue generation. researchers can easily add interpretability evaluations to their studies using our library to identify unwanted biases and interesting phenomena in their models' predictions. we plan to provide continued support and explore developments for inseq, 13 to provide simple and centralized access to a comprehensive set of thoroughly-tested implementations for the interpretability community. in conclusion, we believe that inseq has the potential to drive real progress in explainable language generation by accelerating the development of new analysis techniques, and we encourage members of this research field to join our development efforts. contributions are financed by the nwo as part of the project "the biased reality of online media -using stereotypes to make media manipulation visible" (406.di.19.059).