Original: M ULTI V IZ : T OWARDS V ISUALIZING AND U NDERSTANDING M ULTIMODAL M ODELS
Revised: M ULTI V IZ : T OWARDS V ISUALIZING AND U NDERSTANDING M ULTIMODAL M ODELS

Original: A BSTRACT
Revised: A BSTRACT

Original: The promise of multimodal models for real-world applications has inspired research in visualizing and understanding their internal mechanics with the end goal of empowering stakeholders to visualize model behavior, perform model debugging, and promote trust in machine learning models.
Revised: The promise of multimodal models for real-world applications has inspired research in visualizing and understanding their internal mechanics with the end goal of empowering stakeholders to visualize model behavior, perform model debugging, and promote trust in machine learning models.

Original: However, modern multimodal models are typically black-box neural networks, which makes it challenging to understand their internal mechanics.
Revised: However, modern multimodal models are typically black-box neural networks, which makes it challenging to understand their internal mechanics.

Original: How can we visualize the internal modeling of multimodal interactions in these models?
Revised: How can we visualize the internal modeling of multimodal interactions in these models?

Original: Our paper aims to fill this gap by proposing M ULTI V IZ , a method for analyzing the behavior of multimodal models by scaffolding the problem of interpretability into 4 stages: (1) unimodal importance : how each modality contributes towards downstream modeling and prediction, (2) cross-modal interactions : how different modalities relate with each other, (3) multimodal representations : how unimodal and cross-modal interactions are represented in decision-level features, and (4) multimodal prediction : how decision-level features are composed to make a prediction.
Revised: Our paper aims to fill this gap by proposing M ULTI V IZ , a method for analyzing the behavior of multimodal models by scaffolding the problem of interpretability into 4 stages: (1) unimodal importance : how each modality contributes towards downstream modeling and prediction, (2) cross-modal interactions : how different modalities relate with each other, (3) multimodal representations : how unimodal and cross-modal interactions are represented in decision-level features, and (4) multimodal prediction : how decision-level features are composed to make a prediction.

Original: M ULTI V IZ is designed to operate on diverse modalities, models, tasks, and research areas.
Revised: M ULTI V IZ is designed to operate on diverse modalities, models, tasks, and research areas.

Original: Through experiments ontrained models across 6 real-world tasks, we show that the complementary stages in M ULTI V IZ together enable users to (1) simulate model predictions, (2) assign interpretable concepts to features, (3) perform error analysis on model misclassifications, and (4) use insights from error analysis to debug models.
Revised: Through experiments ontrained models across 6 real-world tasks, we show that the complementary stages in M ULTI V IZ together enable users to (1) simulate model predictions, (2) assign interpretable concepts to features, (3) perform error analysis on model misclassifications, and (4) use insights from error analysis to debug models.

Original: M ULTI V IZ is publicly available, will be regularly updated with new interpretation tools and metrics
Revised: M ULTI V IZ is publicly available, will be regularly updated with new interpretation tools and metrics

Original: and welcomes inputs from the community.
Revised: and welcomes inputs from the community.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The recent promise of multimodal models that integrate information from heterogeneous sources of data has led to their proliferation in numerous real-world settings such as multimedia (Naphade et al., 2006), affective computing (Poria et al., 2017), robotics (Lee et al., 2019), and healthcare (Xu et al., 2019).
Revised: The recent promise of multimodal models that integrate information from heterogeneous sources of data has led to their proliferation in numerous real-world settings such as multimedia (Naphade et al., 2006), affective computing (Poria et al., 2017), robotics (Lee et al., 2019), and healthcare (Xu et al., 2019).

Original: Subsequently, their impact towards real-world applications has inspired recent research in visualizing and understanding their internal mechanics (Chandrasekaran et al., 2018;
Revised: Subsequently, their impact towards real-world applications has inspired recent research in visualizing and understanding their internal mechanics (Liang et al., 2022; Goyal et al., 2016;

Original: Goyal et al., 2016; Park et al., 2018) as a step towards accurately benchmarking their limitations for more reliable deployment (Hendricks et al., 2018; Jabri et al., 2016).
Revised: Park et al., 2018) as a step towards accurately benchmarking their limitations for more reliable deployment (Hendricks et al., 2018; Jabri et al., 2016).

Original: However, modern parameterizations of multimodal models are typically black-box neural networks, such as pretrained transformers (Li et al., 2019; Lu et al., 2019).
Revised: However, modern parameterizations of multimodal models are typically black-box neural networks, such as pretrained transformers (Li et al., 2019; Lu et al., 2019).

Original: How can we visualize and understand the internal modeling of multimodal information and interactions in these models?
Revised: How can we visualize and understand the internal modeling of multimodal information and interactions in these models?

Original: As a step in interpreting multimodal models, this paper introduces an analysis and visualization method called M ULTI V IZ (see Figure 1).
Revised: As a step in interpreting multimodal models, this paper introduces an analysis and visualization method called M ULTI V IZ (see Figure 1).

Original: To tackle the challenges of visualizing model behavior, we scaffold the problem of interpretability into 4 stages: (1) unimodal importance : identifying the contributions of each modality towards downstream modeling and prediction, (2) cross-modal interactions : uncovering the various ways in which different modalities can relate with each other and the types of new information possibly discovered as a result of these relationships, (3) multimodal representations : how unimodal and cross-modal interactions are represented in decision-level features, and (4) multimodal prediction : how decision-level features are composed to make a prediction for a given task.
Revised: To tackle the challenges of visualizing model behavior, we scaffold the problem of interpretability into 4 stages: (1) unimodal importance : identifying the contributions of each modality towards downstream modeling and prediction, (2) cross-modal interactions : uncovering the various ways in which different modalities can relate with each other and the types of new information possibly discovered as a result of these relationships, (3) multimodal representations : how unimodal and cross-modal interactions are represented in decision-level features, and (4) multimodal prediction : how decision-level features are composed to make a prediction for a given task.

Original: In addition to including current approaches for unimodal importance (Goyal et al., 2016;
Revised: In addition to including current approaches for unimodal importance (Goyal et al., 2016;

Original: Merrick and Taly, 2020; Ribeiro et al., 2016) and cross-modal interactions (Hessel and Lee, 2020;
Revised: Merrick and Taly, 2020; Ribeiro et al., 2016) and cross-modal interactions (Hessel and Lee, 2020;

Original: Lyu et al., 2022), we additionally propose new methods for interpreting cross-modal interactions, multimodal representations, and prediction to complete these stages in M ULTI V IZ .
Revised: Lyu et al., 2022), we additionally propose new methods for interpreting cross-modal interactions, multimodal representations, and prediction to complete these stages in M ULTI V IZ .

Original: By viewing multimodal interpretability through the lens of these 4 stages, M ULTI V IZ contributes a modular and human-in-the-loop visualization toolkit for the community to visualize popular multimodal
Revised: By viewing multimodal interpretability through the lens of these 4 stages, M ULTI V IZ contributes a modular and human-in-the-loop visualization toolkit for the community to visualize popular multimodal

Original: [Figure]
Revised: [Figure]

Original: RQ1: What is a taxonomy of multimodal phenomenon that we should aim to interpret?
Revised: RQ1: What is a taxonomy of multimodal phenomenon that we should aim to interpret?

Original: [Figure]
Revised: [Figure]

Original: is the building
Revised: is the building

Original: [Figure]
Revised: [Figure]

Original: modal
Revised: modal

Original: [Figure]
Revised: [Figure]

Original: “color”
Revised: “color”

Original: [Figure]
Revised: [Figure]

Original: datasets and models as well as compare with other interpretation perspectives, and for stakeholders to understand multimodal models in their research domains.
Revised: datasets and models as well as compare with other interpretation perspectives, and for stakeholders to understand multimodal models in their research domains.

Original: M ULTI V IZ is designed to support many modality inputs while also operating on diverse modalities, models, tasks, and research areas.
Revised: M ULTI V IZ is designed to support many modality inputs while also operating on diverse modalities, models, tasks, and research areas.

Original: Through experiments on 6 real-world multimodal tasks (spanning fusion, retrieval, and question-answering), 6 modalities, and 8 models, we show that M ULTI V IZ helps users gain a deeper understanding of model behavior as measured via a proxy task of model simulation.
Revised: Through experiments on 6 real-world multimodal tasks (spanning fusion, retrieval, and question-answering), 6 modalities, and 8 models, we show that M ULTI V IZ helps users gain a deeper understanding of model behavior as measured via a proxy task of model simulation.

Original: We further demonstrate that M ULTI V IZ helps human users assign interpretable language concepts to previously uninterpretable features and perform error analysis on model misclassifications.
Revised: We further demonstrate that M ULTI V IZ helps human users assign interpretable language concepts to previously uninterpretable features and perform error analysis on model misclassifications.

Original: Finally, using takeaways from error analysis, we present a case study of human-in-the-loop model debugging.
Revised: Finally, using takeaways from error analysis, we present a case study of human-in-the-loop model debugging.

Original: Overall, M ULTI V IZ provides a practical toolkit for interpreting multimodal models for human understanding and debugging.
Revised: Overall, M ULTI V IZ provides a practical toolkit for interpreting multimodal models for human understanding and debugging.

Original: M ULTI V IZ datasets, models, and code are at https: //github.com/pliang279/MultiViz .
Revised: M ULTI V IZ datasets, models, and code are at https: //github.com/pliang279/MultiViz .

Original:  M ULTI V IZ : V ISUALIZING AND U NDERSTANDING M ULTIMODAL M ODELS
Revised:  M ULTI V IZ : V ISUALIZING AND U NDERSTANDING M ULTIMODAL M ODELS

Original: This section presents M ULTI V IZ , our proposed analysis framework for analyzing the behavior of multimodal models.
Revised: This section presents M ULTI V IZ , our proposed analysis framework for analyzing the behavior of multimodal models.

Original: As a general setup, we assume multimodal datasets take the form
Revised: As a general setup, we assume multimodal datasets take the form

Original: D = { ( x 1 , x 2 , y ) ni
Revised: D = { ( x 1 , x 2 , y ) ni

Original: =1 } = { ( x (1)1 , x (2)1 , ..., x (1)2 , x (2)2 , ..., y ) ni
Revised: =1 } = { ( x (1)1 , x (2)1 , ..., x (1)2 , x (2)2 , ..., y ) ni

Original: =1 } , with boldface x denoting the entire modality, each x 1 , x 2 indicating modality atoms (i.e., fine-grained sub-parts of modalities that we would like to analyze, such as individual words in a sentence, object regions in an image, or time-steps in time-series data), and y denoting the label.
Revised: =1 } , with boldface x denoting the entire modality, each x 1 , x 2 indicating modality atoms (i.e., fine-grained sub-parts of modalities that we would like to analyze, such as individual words in a sentence, object regions in an image, or time-steps in time-series data), and y denoting the label.

Original: These datasets enable us to train a multimodal model
Revised: These datasets enable us to train a multimodal model

Original: ˆ y = f
Revised: ˆ y = f

Original: ( x 1 , x 2 ; θ ) which we are interested in visualizing.
Revised: ( x 1 , x 2 ; θ ) which we are interested in visualizing.

Original: Modern parameterizations of multimodal models f are typically black-box neural networks, such as multimodal transformers (Hendricks et al., 2021;
Revised: Modern parameterizations of multimodal models f are typically black-box neural networks, such as multimodal transformers (Hendricks et al., 2021;

Original: Tsai et al., 2019) and pretrained models (Li et al., 2019; Lu et al., 2019).
Revised: Tsai et al., 2019) and pretrained models (Li et al., 2019; Lu et al., 2019).

Original: How can we visualize and understand the internal modeling of multimodal information and interactions in these models?
Revised: How can we visualize and understand the internal modeling of multimodal information and interactions in these models?

Original: Having an accurate understanding of their decisionmaking process would enable us to benchmark their opportunities and limitations for more reliable real-world deployment.
Revised: Having an accurate understanding of their decisionmaking process would enable us to benchmark their opportunities and limitations for more reliable real-world deployment.

Original: However, interpreting f is difficult.
Revised: However, interpreting f is difficult.

Original: In many multimodal problems, it is useful to first scaffold the problem of interpreting f into several intermediate stages from low-level unimodal inputs to high-level predictions, spanning unimodal importance , cross-modal interactions , multimodal representations , and multimodal prediction .
Revised: In many multimodal problems, it is useful to first scaffold the problem of interpreting f into several intermediate stages from low-level unimodal inputs to high-level predictions, spanning unimodal importance , cross-modal interactions , multimodal representations , and multimodal prediction .

Original: Each of these stages provides complementary information on the decision-making process (see Figure 1).
Revised: Each of these stages provides complementary information on the decision-making process (see Figure 1).

Original: We now describe each step in detail and propose methods to analyze each step.
Revised: We now describe each step in detail and propose methods to analyze each step.

Original: 2. U NIMODAL IMPORTANCE (U)
Revised: 2. U NIMODAL IMPORTANCE (U)

Original: Unimodal importance aims to understand the contributions of each modality towards modeling and prediction.
Revised: Unimodal importance aims to understand the contributions of each modality towards modeling and prediction.

Original: It builds upon ideas of gradients (Simonyan et al., 2013;
Revised: It builds upon ideas of gradients (Simonyan et al., 2013;

Original: Baehrens et al., 2010;
Revised: Baehrens et al., 2010;

Original: Erhan
Revised: Erhan

Original: [Figure]
Revised: [Figure]

Original: VQA 2.
Revised: VQA 2.

Original: MM-IMDb CLEVR
Revised: MM-IMDb CLEVR

Original: Flickr-30k
Revised: Flickr-30k

Original: [Figure]
Revised: [Figure]

Original: one
Revised: one

Original: [Figure]
Revised: [Figure]

Original: white,
Revised: white,

Original: [Figure]
Revised: [Figure]

Original: a sidewalk.
Revised: a sidewalk.

Original: CMU-MOSEI
Revised: CMU-MOSEI

Original: Why am I spending my money watching this?
Revised: Why am I spending my money watching this?

Original: (sigh)
Revised: (sigh)

Original: I think I was more sad .... et al., 2009) and feature attributions (e.g., LIME (Ribeiro et al., 2016), Shapley values (Merrick and Taly, 2020)).
Revised: I think I was more sad .... et al., 2009) and feature attributions (e.g., LIME (Ribeiro et al., 2016), Shapley values (Merrick and Taly, 2020)).

Original: We implement unimodal feature attribution methods as a module U NI ( f θ , y, x ) taking in a trained model f θ , an output/feature y which analysis is performed with respect to, and the modality of interest x .
Revised: We implement unimodal feature attribution methods as a module U NI ( f θ , y, x ) taking in a trained model f θ , an output/feature y which analysis is performed with respect to, and the modality of interest x .

Original: U NI returns importance weights across atoms x of modality x .
Revised: U NI returns importance weights across atoms x of modality x .

Original: 2. C ROSS - MODAL INTERACTIONS (C)
Revised: 2. C ROSS - MODAL INTERACTIONS (C)

Original: Cross-modal interactions describe various ways in which atoms from different modalities can relate with each other and the types of new information possibly discovered as a result of these relationships.
Revised: Cross-modal interactions describe various ways in which atoms from different modalities can relate with each other and the types of new information possibly discovered as a result of these relationships.

Original: Recent work (Hessel and Lee, 2020; Lyu et al., 2022) has formalized a definition of cross-modal interactions by building upon literature in statistical non-additive interactions:
Revised: Recent work (Hessel and Lee, 2020; Lyu et al., 2022) has formalized a definition of cross-modal interactions by building upon literature in statistical non-additive interactions:

Original: Definition 1 (Statistical Non-Additive Interaction (Friedman and Popescu, 2008; Sorokina et al., 2008; Tsang et al., 2018; 2019)).
Revised: Definition 1 (Statistical Non-Additive Interaction (Friedman and Popescu, 2008; Sorokina et al., 2008; Tsang et al., 2018; 2019)).

Original: A function f learns a feature interaction
Revised: A function f learns a feature interaction

Original: I between 2 unimodal atoms x 1 and x 2 if and only if f cannot be decomposed into a sum of unimodal subfunctions g 1 , gsuch that f ( x 1 , x 2 ) =
Revised: I between 2 unimodal atoms x 1 and x 2 if and only if f cannot be decomposed into a sum of unimodal subfunctions g 1 , gsuch that f ( x 1 , x 2 ) =

Original: g 1 ( x 1 ) + g 2 ( x 2 ) .
Revised: g 1 ( x 1 ) + g 2 ( x 2 ) .

Original: This definition of non-additive interactions is general enough to include different ways that interactions can happen, including multiplicative interactions from complementary views of the data (i.e., an interaction term x 1 W x 2 (Jayakumar et al., 2020)), or cooperative interactions from equivalent views (i.e., an interaction term majority ( f ( x 1 ) , f ( x 2 ))
Revised: This definition of non-additive interactions is general enough to include different ways that interactions can happen, including multiplicative interactions from complementary views of the data (i.e., an interaction term x 1 W x 2 (Jayakumar et al., 2020)), or cooperative interactions from equivalent views (i.e., an interaction term majority ( f ( x 1 ) , f ( x 2 ))

Original: (Ding and Tibshirani, 2021)).
Revised: (Ding and Tibshirani, 2021)).

Original: Using this definition, M ULTI V
Revised: Using this definition, M ULTI V

Original: IZ first includes two recently proposed methods for understanding cross-modal interactions: EMAP (Hessel and Lee, 2020) decomposes f ( x 1 , x 2 )
Revised: IZ first includes two recently proposed methods for understanding cross-modal interactions: EMAP (Hessel and Lee, 2020) decomposes f ( x 1 , x 2 )

Original: =
Revised: =

Original: g 1 ( x 1 ) + g 2 ( x 2 ) + g 12 ( x 1 , x 2 ) into strictly unimodal representations g 1 , g 2 , and cross-modal representation g 12 = f − E x 1 ( f ) − E x 2 ( f ) + E x 1 ,x 2 ( f ) to quantify the degree of global cross-modal interactions across an entire dataset.
Revised: g 1 ( x 1 ) + g 2 ( x 2 ) + g 12 ( x 1 , x 2 ) into strictly unimodal representations g 1 , g 2 , and cross-modal representation g 12 = f − E x 1 ( f ) − E x 2 ( f ) + E x 1 ,x 2 ( f ) to quantify the degree of global cross-modal interactions across an entire dataset.

Original: DIME (Lyu et al., 2022) further extends EMAP using feature visualization on each disentangled representation locally (per datapoint).
Revised: DIME (Lyu et al., 2022) further extends EMAP using feature visualization on each disentangled representation locally (per datapoint).

Original: However, these approaches require approximating expectations over modality subsets, which may not scale beyond 2 modalities.
Revised: However, these approaches require approximating expectations over modality subsets, which may not scale beyond 2 modalities.

Original: To fill this gap, we propose an efficient approach for visualizing these cross-modal interactions by observing that the following gradient definition directly follows from Definition 1:
Revised: To fill this gap, we propose an efficient approach for visualizing these cross-modal interactions by observing that the following gradient definition directly follows from Definition 1:

Original: Definition 2 (Gradient definition of statistical non-additive interaction).
Revised: Definition 2 (Gradient definition of statistical non-additive interaction).

Original: A function f exhibits non-additive interactions among 2 unimodal atoms x 1 and x 2 if E x 1 ,x 2 (cid:104) ∂ 2 f ( x 1 ,x 2 ) ∂x 1 ∂x 2 (cid:105) 2 > 0 .
Revised: A function f exhibits non-additive interactions among 2 unimodal atoms x 1 and x 2 if E x 1 ,x 2 (cid:104) ∂ 2 f ( x 1 ,x 2 ) ∂x 1 ∂x 2 (cid:105) 2 > 0 .

Original: Taking a second-order gradient of f zeros out the unimodal terms g 1 ( x 1 ) and g 2 ( x 2 ) and isolates the interaction g 12 ( x 1 , x 2 ) .
Revised: Taking a second-order gradient of f zeros out the unimodal terms g 1 ( x 1 ) and g 2 ( x 2 ) and isolates the interaction g 12 ( x 1 , x 2 ) .

Original: Theoretically, second-order gradients are necessary and sufficient to recover cross-modal interactions: purely additive models will have strictly 0 second-order gradients so
Revised: Theoretically, second-order gradients are necessary and sufficient to recover cross-modal interactions: purely additive models will have strictly 0 second-order gradients so

Original: E x 1 ,x 2 (cid:104) ∂ 2 f ( x 1 ,x 2 ) ∂x 1 ∂x 2 (cid:105) 2 = 0 , and any non-linear interaction term g 12 ( x 1 , x 2 ) has non-zero second-order gradients since g cannot be a constant or unimodal function, so
Revised: E x 1 ,x 2 (cid:104) ∂ 2 f ( x 1 ,x 2 ) ∂x 1 ∂x 2 (cid:105) 2 = 0 , and any non-linear interaction term g 12 ( x 1 , x 2 ) has non-zero second-order gradients since g cannot be a constant or unimodal function, so

Original: E x 1 ,x 2 (cid:104) ∂ 2 f ( x 1 ,x 2 ) ∂x 1 ∂x 2 (cid:105) 2 > 0 .
Revised: E x 1 ,x 2 (cid:104) ∂ 2 f ( x 1 ,x 2 ) ∂x 1 ∂x 2 (cid:105) 2 > 0 .

Original: Definition 2 inspires us to extend first-order gradient and perturbation-based approaches (Han et al., 2020; Ribeiro et al., 2016;
Revised: Definition 2 inspires us to extend first-order gradient and perturbation-based approaches (Han et al., 2020; Ribeiro et al., 2016;

Original: Yosinski et al., 2015) to the second order.
Revised: Yosinski et al., 2015) to the second order.

Original: Our implementation first computes a gradient of f with respect to a modality atom which the user is interested in querying cross-modal interactions for (e.g., x 1 = birds ), which results in a vector ∇ 1 = ∂f∂x 1 of the same dimension as x 1 (i.e., token embedding dimension).
Revised: Our implementation first computes a gradient of f with respect to a modality atom which the user is interested in querying cross-modal interactions for (e.g., x 1 = birds ), which results in a vector ∇ 1 = ∂f∂x 1 of the same dimension as x 1 (i.e., token embedding dimension).

Original: We aggregate the vector components of ∇via summation to produce a single scalar ∥∇ 1 ∥ , before taking a second-order gradient with respect to all atoms of the second modality x 2 ∈ x 2 (e.g., all image pixels), which results in a vector ∇ 

Revised: We aggregate the vector components of ∇via summation to produce a single scalar ∥∇ 1 ∥ , before taking a second-order gradient with respect to all atoms of the second modality x 2 ∈ x 2 (e.g., all image pixels), which results in a vector ∇

Original: = (cid:20) ∂
Revised: = (cid:20) ∂

Original: 2 f ∂x 1 ∂x (1)2 , ..., ∂ 2 f ∂x 1 ∂x ( | x 2 | )(cid:21) of the same dimension as x 2 (i.e., total number of pixels).
Revised: 2 f ∂x 1 ∂x (1)2 , ..., ∂ 2 f ∂x 1 ∂x ( | x 2 | )(cid:21) of the same dimension as x 2 (i.e., total number of pixels).

Original: Each scalar entry in ∇ 12 highlights atoms x 2 that have non-linear interactions with the original atom x 1 , and we choose the x 2 ’s with the largest magnitude of interactions with x 1 (i.e., which highlights the birds in the image, see Figure 2 for examples on real datasets).
Revised: Each scalar entry in ∇ 12 highlights atoms x 2 that have non-linear interactions with the original atom x 1 , and we choose the x 2 ’s with the largest magnitude of interactions with x 1 (i.e., which highlights the birds in the image, see Figure 2 for examples on real datasets).

Original: We implement a general module
Revised: We implement a general module

Original: CM ( f θ , y, x 1 , x 2 ) for cross-modal visualizations, taking in a trained model f θ , an output/feature y , the first modality’s atom of interest x 1 , and the entire second modality of interest x 2 , before returning importance weights across atoms x 2 of modality x 2 (see details in Appendix A.2).
Revised: CM ( f θ , y, x 1 , x 2 ) for cross-modal visualizations, taking in a trained model f θ , an output/feature y , the first modality’s atom of interest x 1 , and the entire second modality of interest x 2 , before returning importance weights across atoms x 2 of modality x 2 (see details in Appendix A.2).

Original: 2. M ULTIMODAL REPRESENTATIONS
Revised: 2. M ULTIMODAL REPRESENTATIONS

Original: Given these highlighted unimodal and cross-modal interactions at the input level, the next stage aims to understand how these interactions are represented at the feature representation level.
Revised: Given these highlighted unimodal and cross-modal interactions at the input level, the next stage aims to understand how these interactions are represented at the feature representation level.

Original: Specifically, given a trained multimodal model f , define the matrix M z ∈ R N × d as the penultimate layer of f representing (uninterpretable) deep feature representations implicitly containing information from both unimodal and cross-modal interactions.
Revised: Specifically, given a trained multimodal model f , define the matrix M z ∈ R N × d as the penultimate layer of f representing (uninterpretable) deep feature representations implicitly containing information from both unimodal and cross-modal interactions.

Original: For the i th datapoint, z = M z ( i ) collects a set of individual feature representations z 1 , z 2 , ..., z d ∈ R .
Revised: For the i th datapoint, z = M z ( i ) collects a set of individual feature representations z 1 , z 2 , ..., z d ∈ R .

Original: We aim to interpret these feature representations through both local and global analysis (see Figure 1 (right) for an example):
Revised: We aim to interpret these feature representations through both local and global analysis (see Figure 1 (right) for an example):

Original: Local representation analysis (R ℓ ) informs the user on parts of the original datapoint that activate feature z j .
Revised: Local representation analysis (R ℓ ) informs the user on parts of the original datapoint that activate feature z j .

Original: To do so, we run unimodal and cross-modal visualization methods with respect to feature z j
Revised: To do so, we run unimodal and cross-modal visualization methods with respect to feature z j

Original: (i.e., U NI ( f θ , z j , x ) , CM ( f θ , z j , x 1 , x 2 ) ) in order to explain the input unimodal and cross-modal interactions represented in feature z j .
Revised: (i.e., U NI ( f θ , z j , x ) , CM ( f θ , z j , x 1 , x 2 ) ) in order to explain the input unimodal and cross-modal interactions represented in feature z j .

Original: Local analysis is useful in explaining model predictions on the original datapoint by studying the input regions activating feature z j .
Revised: Local analysis is useful in explaining model predictions on the original datapoint by studying the input regions activating feature z j .

Original: Global representation analysis (R g ) provides the user with the top k datapoints
Revised: Global representation analysis (R g ) provides the user with the top k datapoints

Original: D k
Revised: D k

Original: ( z j )
Revised: ( z j )

Original: = { ( x 1 , x 2 , y )
Revised: = { ( x 1 , x 2 , y )

Original: ki =(1) } that also maximally activate feature z j .
Revised: ki =(1) } that also maximally activate feature z j .

Original: By further unimodal and cross-modal visualizations on datapoints in D k ( z j ) , global analysis is especially useful in helping humans assign interpretable language concepts to each feature by looking at similarly activated input regions across datapoints (e.g., the concept of color in Figure 1, right).
Revised: By further unimodal and cross-modal visualizations on datapoints in D k ( z j ) , global analysis is especially useful in helping humans assign interpretable language concepts to each feature by looking at similarly activated input regions across datapoints (e.g., the concept of color in Figure 1, right).

Original: Global analysis can also help to find related datapoints the model also struggles with for error analysis.
Revised: Global analysis can also help to find related datapoints the model also struggles with for error analysis.

Original: 2. M ULTIMODAL PREDICTION (P)
Revised: 2. M ULTIMODAL PREDICTION (P)

Original: Finally, the prediction step takes the set of feature representations z 1 , z 2 , ..., z d and composes them to form higher-level abstract concepts suitable for a task.
Revised: Finally, the prediction step takes the set of feature representations z 1 , z 2 , ..., z d and composes them to form higher-level abstract concepts suitable for a task.

Original: We approximate the prediction process with a linear combination of penultimate layer features by integrating a sparse linear prediction model with neural network features (Wong et al., 2021).
Revised: We approximate the prediction process with a linear combination of penultimate layer features by integrating a sparse linear prediction model with neural network features (Wong et al., 2021).

Original: Given the penultimate layer M z ∈ R N × d , we fit a linear model E ( Y | X = x ) =
Revised: Given the penultimate layer M z ∈ R N × d , we fit a linear model E ( Y | X = x ) =

Original: M ⊤ z β (bias β 0 omitted for simplicity) and solve for sparsity using:
Revised: M ⊤ z β (bias β 0 omitted for simplicity) and solve for sparsity using:

Original: [Equation]
Revised: [Equation]

Original: The resulting understanding starts from the set of learned weights with the highest non-zero coefficients β top = { β (1) , β (2) , ... } and corresponding ranked features z top =
Revised: The resulting understanding starts from the set of learned weights with the highest non-zero coefficients β top = { β (1) , β (2) , ... } and corresponding ranked features z top =

Original: { z (1) , z (2) , ... } .
Revised: { z (1) , z (2) , ... } .

Original: β top tells the user how features z top are composed to make a prediction, and z top can then be visualized with respect to unimodal and cross-modal interactions using the representation stage (Section 2.3).
Revised: β top tells the user how features z top are composed to make a prediction, and z top can then be visualized with respect to unimodal and cross-modal interactions using the representation stage (Section 2.3).

Original: 2. P UTTING EVERYTHING TOGETHER
Revised: 2. P UTTING EVERYTHING TOGETHER

Original: We summarize these proposed approaches for understanding each step of the multimodal process and show the overall M ULTI V IZ user interface in Figure 3.
Revised: We summarize these proposed approaches for understanding each step of the multimodal process and show the overall M ULTI V IZ user interface in Figure 3.

Original: This interactive API enables users to choose multimodal datasets and models and be presented with a set of visualizations at each stage, with an overview page for general unimodal importance, cross-modal interactions, and prediction weights, as well as a feature page for local and global analysis of user-selected features (see Appendix B for more algorithm and user interface details).
Revised: This interactive API enables users to choose multimodal datasets and models and be presented with a set of visualizations at each stage, with an overview page for general unimodal importance, cross-modal interactions, and prediction weights, as well as a feature page for local and global analysis of user-selected features (see Appendix B for more algorithm and user interface details).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our experiments are designed to verify the usefulness and complementarity of the 4 M ULTI V IZ stages.
Revised: Our experiments are designed to verify the usefulness and complementarity of the 4 M ULTI V IZ stages.

Original: We start with a model simulation experiment to test the utility of each stage towards overall model understanding (Section 3.1).
Revised: We start with a model simulation experiment to test the utility of each stage towards overall model understanding (Section 3.1).

Original: We then dive deeper into the individual stages by testing how well
Revised: We then dive deeper into the individual stages by testing how well

Original: M ULTI V IZ enables representation interpretation (Section 3.2) and error analysis (Section 3.3), before presenting a case study of model debugging from error analysis insights (Section 3.4).
Revised: M ULTI V IZ enables representation interpretation (Section 3.2) and error analysis (Section 3.3), before presenting a case study of model debugging from error analysis insights (Section 3.4).

Original: We showcase the following selected experiments and defer results on other datasets to Appendix D.
Revised: We showcase the following selected experiments and defer results on other datasets to Appendix D.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Setup : We use a large suite of datasets from MultiBench (Liang et al., 2021a) which span real-world fusion (Zadeh et al., 2018;
Revised: Setup : We use a large suite of datasets from MultiBench (Liang et al., 2021a) which span real-world fusion (Zadeh et al., 2018;

Original: Arevalo et al., 2017; Johnson et al., 2016), retrieval (Plummer et al., 2015), and QA (Johnson et al., 2017; Goyal et al., 2017) tasks.
Revised: Arevalo et al., 2017; Johnson et al., 2016), retrieval (Plummer et al., 2015), and QA (Johnson et al., 2017; Goyal et al., 2017) tasks.

Original: For each dataset, we test a corresponding state-of-the-art model: M UL T (Tsai et al., 2019), LRTF (Liu et al., 2018), LF (Baltrušaitis et al., 2018),
Revised: For each dataset, we test a corresponding state-of-the-art model: M UL T (Tsai et al., 2019), LRTF (Liu et al., 2018), LF (Baltrušaitis et al., 2018),

Original: V I LT (Kim et al., 2021), CLIP (Radford et al., 2021), CNN-LSTM-SA (Johnson et al., 2017), MDETR (Kamath et al., 2021), and LXMERT (Tan and Bansal, 2019).
Revised: V I LT (Kim et al., 2021), CLIP (Radford et al., 2021), CNN-LSTM-SA (Johnson et al., 2017), MDETR (Kamath et al., 2021), and LXMERT (Tan and Bansal, 2019).

Original: These cover models both pretrained and trained from scratch.
Revised: These cover models both pretrained and trained from scratch.

Original: We summarize all 6 datasets and 8 models tested in Table 1, and provide implementation details in Appendix C and user study details in Appendix D.
Revised: We summarize all 6 datasets and 8 models tested in Table 1, and provide implementation details in Appendix C and user study details in Appendix D.

Original: 3. M ODEL SIMULATION
Revised: 3. M ODEL SIMULATION

Original: We first design a model simulation experiment to determine if M ULTI V IZ helps users of multimodal models gain a deeper understanding of model behavior.
Revised: We first design a model simulation experiment to determine if M ULTI V IZ helps users of multimodal models gain a deeper understanding of model behavior.

Original: If M ULTI V IZ indeed generates humanunderstandable explanations, humans should be able to accurately simulate model predictions given these explanations only, as measured by correctness with respect to actual model predictions and annotator agreement (Krippendorff’s alpha (Krippendorff, 2011)).
Revised: If M ULTI V IZ indeed generates humanunderstandable explanations, humans should be able to accurately simulate model predictions given these explanations only, as measured by correctness with respect to actual model predictions and annotator agreement (Krippendorff’s alpha (Krippendorff, 2011)).

Original: To investigate the utility of each stage in M ULTI V IZ , we design a human study to see how accurately 21 humans users ( 3 users for each of the following 7 local ablation settings) can simulate model predictions:
Revised: To investigate the utility of each stage in M ULTI V IZ , we design a human study to see how accurately 21 humans users ( 3 users for each of the following 7 local ablation settings) can simulate model predictions:

Original: (1) U : Users are only shown the unimodal importance (U) of each modality towards label y .
Revised: (1) U : Users are only shown the unimodal importance (U) of each modality towards label y .

Original: (2) U + C : Users are also shown cross-modal interactions (C) highlighted towards label y .
Revised: (2) U + C : Users are also shown cross-modal interactions (C) highlighted towards label y .

Original: (3) U + C + R ℓ : Users are also shown local analysis (R ℓ ) of unimodal and cross-modal interactions of top features z top =
Revised: (3) U + C + R ℓ : Users are also shown local analysis (R ℓ ) of unimodal and cross-modal interactions of top features z top =

Original: { z (1) , z (2) , ... } maximally activating label y .
Revised: { z (1) , z (2) , ... } maximally activating label y .

Original: (4) U + C + R ℓ + R g : Users are additionally shown global analysis (R g ) through similar datapoints that also maximally activate top features z top for label y .
Revised: (4) U + C + R ℓ + R g : Users are additionally shown global analysis (R g ) through similar datapoints that also maximally activate top features z top for label y .

Original: (5) M ULTI V
Revised: (5) M ULTI V

Original: IZ
Revised: IZ

Original: (U + C + R ℓ + R g + P) :
Revised: (U + C + R ℓ + R g + P) :

Original: The entire M ULTI V IZ method by further including visualizations of the final prediction (P) stage: sorting top ranked feature neurons z top =
Revised: The entire M ULTI V IZ method by further including visualizations of the final prediction (P) stage: sorting top ranked feature neurons z top =

Original: { z (1) , z (2) , ... } with respect to their coefficients β top = { β (1) , β (2) , ... } and showing these coefficients to the user.
Revised: { z (1) , z (2) , ... } with respect to their coefficients β top = { β (1) , β (2) , ... } and showing these coefficients to the user.

Original: [Table]
Revised: [Table]

Original: Using 20 datapoints per setting
Revised: Using 20 datapoints per setting

Original: [Table]
Revised: [Table]

Original: these experiments with 15 users on 3 datasets and 3 models involve 35 total hours of users interacting with M ULTI V IZ , which is a significantly larger-scale study of model simulation compared to prior work (Aflalo et al., 2022;
Revised: these experiments with 15 users on 3 datasets and 3 models involve 35 total hours of users interacting with M ULTI V IZ , which is a significantly larger-scale study of model simulation compared to prior work (Aflalo et al., 2022;

Original: Lyu et al., 2022; Wang et al., 2021).
Revised: Lyu et al., 2022; Wang et al., 2021).

Original: Quantitative results : We show these results in Table 2 and find that having access to all stages in
Revised: Quantitative results : We show these results in Table 2 and find that having access to all stages in

Original: M ULTI V IZ leads to significantly highest accuracy of model simulation on VQA 2.0, along with lowest variance and most consistent agreement between annotators.
Revised: M ULTI V IZ leads to significantly highest accuracy of model simulation on VQA 2.0, along with lowest variance and most consistent agreement between annotators.

Original: On fusion tasks with MMIMD B and CMU-MOSEI, we also find that including each visualization stage consistently leads to higher correctness and agreement, despite the fact that fusion models may not require cross-modal interactions to solve the task (Hessel and Lee, 2020).
Revised: On fusion tasks with MMIMD B and CMU-MOSEI, we also find that including each visualization stage consistently leads to higher correctness and agreement, despite the fact that fusion models may not require cross-modal interactions to solve the task (Hessel and Lee, 2020).

Original: More importantly, humans are able to simulate model predictions, regardless of whether the model made the correct prediction or not.
Revised: More importantly, humans are able to simulate model predictions, regardless of whether the model made the correct prediction or not.

Original: To test additional intermediate ablations, we conducted user studies on (6) R ℓ + P (local analysis on final-layer features along with their prediction weights) and (7) R g + P
Revised: To test additional intermediate ablations, we conducted user studies on (6) R ℓ + P (local analysis on final-layer features along with their prediction weights) and (7) R g + P

Original: (global analysis on final-layer features along with their prediction weights), to ablate the effect of overall analysis ( U and C ) and feature analysis ( R ℓ or R g in isolation).
Revised: (global analysis on final-layer features along with their prediction weights), to ablate the effect of overall analysis ( U and C ) and feature analysis ( R ℓ or R g in isolation).

Original: R ℓ + P results in an accuracy of 51 .
Revised: R ℓ + P results in an accuracy of 51 .

Original: 7 ± 12 .
Revised: 7 ± 12 .

Original: 6 with 0 .agreement, while R g + P gives 71 .
Revised: 6 with 0 .agreement, while R g + P gives 71 .

Original: 7 ± 7 . 6 with 0 .
Revised: 7 ± 7 . 6 with 0 .

Original: 53 agreement.
Revised: 53 agreement.

Original: Indeed, these underperform as compared to including overall analysis ( U and C ) and feature analysis ( R ℓ + R g ).
Revised: Indeed, these underperform as compared to including overall analysis ( U and C ) and feature analysis ( R ℓ + R g ).

Original: Finally, we also scaled to 100 datapoints on VQA 2.0, representing upwards of 10 hours of user interaction (for the full M ULTI V IZ setting), and obtain an overall correctness of 80% , reliably within the range of model simulation using 20 points ( 81 .
Revised: Finally, we also scaled to 100 datapoints on VQA 2.0, representing upwards of 10 hours of user interaction (for the full M ULTI V IZ setting), and obtain an overall correctness of 80% , reliably within the range of model simulation using 20 points ( 81 .

Original: 7 ± 2 . 9 ).
Revised: 7 ± 2 . 9 ).

Original: Therefore, the sample size of 20 points that makes all experiments feasible is still a reliable sample.
Revised: Therefore, the sample size of 20 points that makes all experiments feasible is still a reliable sample.

Original: We also conducted qualitative interviews to determine what users found useful in M ULTI V
Revised: We also conducted qualitative interviews to determine what users found useful in M ULTI V

Original: IZ : (1) Users reported that they found local and global representation analysis particularly useful: global analysis with other datapoints that also maximally activate feature representations were important for identifying similar concepts and assigning them to multimodal features.
Revised: IZ : (1) Users reported that they found local and global representation analysis particularly useful: global analysis with other datapoints that also maximally activate feature representations were important for identifying similar concepts and assigning them to multimodal features.

Original: (2) Between Overview ( U + C ) and Feature ( R ℓ + R g + P ) visualizations, users found Feature visualizations more useful in 31 .
Revised: (2) Between Overview ( U + C ) and Feature ( R ℓ + R g + P ) visualizations, users found Feature visualizations more useful in 31 .

Original: 7% , and 80 .
Revised: 7% , and 80 .

Original: % of the time under settings (3), (4), and (5) respectively, and found Overview more useful in the remaining points.
Revised: % of the time under settings (3), (4), and (5) respectively, and found Overview more useful in the remaining points.

Original: This means that for each stage, there exists a significant fraction of data points where that stage is most needed.
Revised: This means that for each stage, there exists a significant fraction of data points where that stage is most needed.

Original: (3) While it may be possible to determine the prediction of the model with a subset of stages, having more stages that confirm the same prediction makes them a lot more confident about their prediction, which is quantitatively substantiated by the higher accuracy, lower variance, and higher agreement in human predictions.
Revised: (3) While it may be possible to determine the prediction of the model with a subset of stages, having more stages that confirm the same prediction makes them a lot more confident about their prediction, which is quantitatively substantiated by the higher accuracy, lower variance, and higher agreement in human predictions.

Original: We also include additional experiments in Appendix D.1.
Revised: We also include additional experiments in Appendix D.1.

Original: 3. R EPRESENTATION INTERPRETATION
Revised: 3. R EPRESENTATION INTERPRETATION

Original: We now take a deeper look to check that M ULTI V IZ generates accurate explanations of multimodal representations.
Revised: We now take a deeper look to check that M ULTI V IZ generates accurate explanations of multimodal representations.

Original: Using local and global representation visualizations, can humans consistently assign interpretable concepts in natural language to previously uninterpretable features?
Revised: Using local and global representation visualizations, can humans consistently assign interpretable concepts in natural language to previously uninterpretable features?

Original: We study this question by tasking 15 human users ( 5 users for each of the following 3 settings) to assign concepts to each feature z when given access to visualizations of (1) R ℓ (local analysis of unimodal and cross-modal interactions in z ), (2) R ℓ + R g (no viz) (including global analysis through similar datapoints that also maximally activate feature z ), and (3) R ℓ + R g (adding highlighted unimodal and
Revised: We study this question by tasking 15 human users ( 5 users for each of the following 3 settings) to assign concepts to each feature z when given access to visualizations of (1) R ℓ (local analysis of unimodal and cross-modal interactions in z ), (2) R ℓ + R g (no viz) (including global analysis through similar datapoints that also maximally activate feature z ), and (3) R ℓ + R g (adding highlighted unimodal and

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: capture image-only, language-only, and multimodal concepts.
Revised: capture image-only, language-only, and multimodal concepts.

Original: cross-modal interactions of global datapoints).
Revised: cross-modal interactions of global datapoints).

Original: Using 20 datapoints per setting, these experiments with 15 users involve roughly 10 total hours of users interacting with M ULTI V IZ .
Revised: Using 20 datapoints per setting, these experiments with 15 users involve roughly 10 total hours of users interacting with M ULTI V IZ .

Original: Quantitative results : Since there are no ground-truth labels for feature concepts, we rely on annotator confidence ( 1 - 5 scale) and annotator agreement (Krippendorff, 2011) as a proxy for accuracy.
Revised: Quantitative results : Since there are no ground-truth labels for feature concepts, we rely on annotator confidence ( 1 - 5 scale) and annotator agreement (Krippendorff, 2011) as a proxy for accuracy.

Original: From
Revised: From

Original: Table 3 (left), we find that having access to both local and global visualizations are crucial towards interpreting multimodal features, as measured by higher confidence with low variance in confidence, as well as higher agreement among users.
Revised: Table 3 (left), we find that having access to both local and global visualizations are crucial towards interpreting multimodal features, as measured by higher confidence with low variance in confidence, as well as higher agreement among users.

Original: Qualitative interviews : We show examples of human-assigned concepts in Figure 4 (more in
Revised: Qualitative interviews : We show examples of human-assigned concepts in Figure 4 (more in

Original: Appendix D.3).
Revised: Appendix D.3).

Original: Note that the 3 images in each box of Figure 4 (even without feature highlighting) does constitute a visualization generated by M ULTI V IZ , as they belong to data instances that maximize the value of the feature neuron (i.e. R g in stage 3 multimodal representations).
Revised: Note that the 3 images in each box of Figure 4 (even without feature highlighting) does constitute a visualization generated by M ULTI V IZ , as they belong to data instances that maximize the value of the feature neuron (i.e. R g in stage 3 multimodal representations).

Original: Without M ULTI V IZ , it would not be possible to perform feature interpretation without combing through the entire dataset.
Revised: Without M ULTI V IZ , it would not be possible to perform feature interpretation without combing through the entire dataset.

Original: Participants also noted that feature visualizations make the decision a lot more confident if its highlights match the concept.
Revised: Participants also noted that feature visualizations make the decision a lot more confident if its highlights match the concept.

Original: Taking as example Figure 4 top left, the visualizations serve to highlight what the model’s feature neuron is learning (i.e., highlighting the person holding sports equipment), rather than what category of datapoint it is.
Revised: Taking as example Figure 4 top left, the visualizations serve to highlight what the model’s feature neuron is learning (i.e., highlighting the person holding sports equipment), rather than what category of datapoint it is.

Original: If the visualization was different, such as highlighting the ground, then users would have to conclude that the feature neuron is capturing ‘ outdoor ground ’ rather than ‘ sports equipment ’.
Revised: If the visualization was different, such as highlighting the ground, then users would have to conclude that the feature neuron is capturing ‘ outdoor ground ’ rather than ‘ sports equipment ’.

Original: Similarly, for text highlights (Figure 4 top right), without using M ULTI V IZ to highlight ‘ counter ’, ‘ countertop ’, and ‘ wall ’, along with the image crossmodal interactions corresponding to these entities, one would not be able to deduce that the feature asks about material - it could also represent ‘ what ’ questions, or ‘ household objects ’, and so on.
Revised: Similarly, for text highlights (Figure 4 top right), without using M ULTI V IZ to highlight ‘ counter ’, ‘ countertop ’, and ‘ wall ’, along with the image crossmodal interactions corresponding to these entities, one would not be able to deduce that the feature asks about material - it could also represent ‘ what ’ questions, or ‘ household objects ’, and so on.

Original: Therefore, these conclusions can only be reliably deduced with all MultiViz stages.
Revised: Therefore, these conclusions can only be reliably deduced with all MultiViz stages.

Original: 3. E RROR ANALYSIS
Revised: 3. E RROR ANALYSIS

Original: We further examine a case study of error analysis on trained models.
Revised: We further examine a case study of error analysis on trained models.

Original: We task 10 human users ( 5 users for each of the following 2 settings) to use M ULTI V IZ and highlight the errors that a multimodal model exhibits by categorizing these errors into one of 3 stages: failures in (1) unimodal perception, (2) capturing cross-modal interaction, and (3) prediction with perceived unimodal and cross-modal information.
Revised: We task 10 human users ( 5 users for each of the following 2 settings) to use M ULTI V IZ and highlight the errors that a multimodal model exhibits by categorizing these errors into one of 3 stages: failures in (1) unimodal perception, (2) capturing cross-modal interaction, and (3) prediction with perceived unimodal and cross-modal information.

Original: Again, we rely on annotator confidence ( 1 - 5 scale) and agreement due to lack of
Revised: Again, we rely on annotator confidence ( 1 - 5 scale) and agreement due to lack of

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: the model’s failure to identify color (especially blue).
Revised: the model’s failure to identify color (especially blue).

Original: Targeted localization of the error to this specific stage (prediction) and representation concept (blue) via M ULTI V IZ enabled us to identify a bug in the popular Hugging Face LXMERT repository.
Revised: Targeted localization of the error to this specific stage (prediction) and representation concept (blue) via M ULTI V IZ enabled us to identify a bug in the popular Hugging Face LXMERT repository.

Original: ground-truth error categorization, and compare (1) M ULTI V IZ with (2) No viz , a baseline that does not provide any model visualizations to the user.
Revised: ground-truth error categorization, and compare (1) M ULTI V IZ with (2) No viz , a baseline that does not provide any model visualizations to the user.

Original: Using 20 datapoints per setting, these experiments with 10 users on 2 datasets and 2 models involve roughly 15 total hours of users interacting with
Revised: Using 20 datapoints per setting, these experiments with 10 users on 2 datasets and 2 models involve roughly 15 total hours of users interacting with

Original: M ULTI V IZ .
Revised: M ULTI V IZ .

Original: From Table 3 (right), we find that M ULTI V IZ enables humans to consistently categorize model errors into one of 3 stages.
Revised: From Table 3 (right), we find that M ULTI V IZ enables humans to consistently categorize model errors into one of 3 stages.

Original: We show examples that human annotators classified into unimodal perception, cross-modal interaction, and prediction errors in Figure 5 (more in Appendix D.4).
Revised: We show examples that human annotators classified into unimodal perception, cross-modal interaction, and prediction errors in Figure 5 (more in Appendix D.4).

Original: 3. A CASE STUDY IN MODEL DEBUGGING
Revised: 3. A CASE STUDY IN MODEL DEBUGGING

Original: Following error analysis, we take a deeper investigation into one of the errors on a pretrained
Revised: Following error analysis, we take a deeper investigation into one of the errors on a pretrained

Original: LXMERT model fine-tuned on VQA 2.0.
Revised: LXMERT model fine-tuned on VQA 2.0.

Original: Specifically, we first found the top 5 penultimate-layer neurons that are most activated on erroneous datapoints.
Revised: Specifically, we first found the top 5 penultimate-layer neurons that are most activated on erroneous datapoints.

Original: Inspecting these neurons carefully through
Revised: Inspecting these neurons carefully through

Original: M ULTI V IZ local and global representation analysis
Revised: M ULTI V IZ local and global representation analysis

Original: , human annotators found that 2 of the 5 neurons were consistently related to questions asking about color, which highlighted the model’s failure to identify color correctly (especially blue).
Revised: , human annotators found that 2 of the 5 neurons were consistently related to questions asking about color, which highlighted the model’s failure to identify color correctly (especially blue).

Original: The model has an accuracy of only 5 . 5% amongst all blue-related points (i.e., either have blue as correct answer or predicted answer), and these failures account for 8 .
Revised: The model has an accuracy of only 5 . 5% amongst all blue-related points (i.e., either have blue as correct answer or predicted answer), and these failures account for 8 .

Original: 8% of all model errors.
Revised: 8% of all model errors.

Original: We show examples of such datapoints and their M ULTI V IZ visualizations in Figure 6.
Revised: We show examples of such datapoints and their M ULTI V IZ visualizations in Figure 6.

Original: Observe that the model is often able to capture unimodal and cross-modal interactions perfectly, but fails to identify color at prediction.
Revised: Observe that the model is often able to capture unimodal and cross-modal interactions perfectly, but fails to identify color at prediction.

Original: Curious as to the source of this error, we looked deeper into the source code for the entire pipeline of LXMERT, including that of its image encoder, Faster R-CNN (Ren et al., 2015) 1 .
Revised: Curious as to the source of this error, we looked deeper into the source code for the entire pipeline of LXMERT, including that of its image encoder, Faster R-CNN (Ren et al., 2015) 1 .

Original: We in fact uncovered a bug in data preprocessing for Faster R-CNN in the popular Hugging Face repository that swapped the image data storage format from RGB to BGR formats responsible for these errors.
Revised: We in fact uncovered a bug in data preprocessing for Faster R-CNN in the popular Hugging Face repository that swapped the image data storage format from RGB to BGR formats responsible for these errors.

Original: This presents a concrete use case of M ULTI V IZ : through visualizing each stage, we were able to (1) isolate the source of the bug (at prediction and not unimodal perception or cross-modal interactions), and (2) use representation analysis to localize the bug to the specific color concept.
Revised: This presents a concrete use case of M ULTI V IZ : through visualizing each stage, we were able to (1) isolate the source of the bug (at prediction and not unimodal perception or cross-modal interactions), and (2) use representation analysis to localize the bug to the specific color concept.

Original: In Appendix D.5, we further detail our initial attempt at tackling this error by using M ULTI V IZ analysis to select additional targeted datapoints in an active learning scenario, which proved to be much more effective (higher improvement with fewer data) as compared to baselines that add data randomly or via uncertainty sampling (Lewis and Catlett, 1994), which may be of independent interest.
Revised: In Appendix D.5, we further detail our initial attempt at tackling this error by using M ULTI V IZ analysis to select additional targeted datapoints in an active learning scenario, which proved to be much more effective (higher improvement with fewer data) as compared to baselines that add data randomly or via uncertainty sampling (Lewis and Catlett, 1994), which may be of independent interest.

Original: 3. A DDITIONAL EXPERIMENTS AND TAKEAWAYS MESSAGES
Revised: 3. A DDITIONAL EXPERIMENTS AND TAKEAWAYS MESSAGES

Original: New models : We included results on V
Revised: New models : We included results on V

Original: I LT (Kim et al., 2021), CLIP (Radford et al., 2021), and
Revised: I LT (Kim et al., 2021), CLIP (Radford et al., 2021), and

Original: MDETR (Kamath et al., 2021) in Appendix D.2, showing that M ULTI V IZ is a general approach that can be quickly applied to new models.
Revised: MDETR (Kamath et al., 2021) in Appendix D.2, showing that M ULTI V IZ is a general approach that can be quickly applied to new models.

Original: We also study the correlation between performance and crossmodal interactions across several older and recent models, and find that the ability to capture crossmodal alignment, as judged by M ULTI V IZ , correlates strongly with final task performance.
Revised: We also study the correlation between performance and crossmodal interactions across several older and recent models, and find that the ability to capture crossmodal alignment, as judged by M ULTI V IZ , correlates strongly with final task performance.

Original: Sanity checks : In Appendix A.5, we show that M ULTI V IZ passes the data randomization and model randomization sanity checks for interpretation approaches (Adebayo et al., 2018).
Revised: Sanity checks : In Appendix A.5, we show that M ULTI V IZ passes the data randomization and model randomization sanity checks for interpretation approaches (Adebayo et al., 2018).

Original: Intermediate-layer features : In Appendix B.3, we show that M ULTI V IZ can be extended to visualize any intermediate layer, not just the final layer of multimodal models.
Revised: Intermediate-layer features : In Appendix B.3, we show that M ULTI V IZ can be extended to visualize any intermediate layer, not just the final layer of multimodal models.

Original: We showcase a few examples of R ℓ and R g on intermediate-layer neurons and discuss several tradeoffs: while they reveal new visualization opportunities, they run the risk of overwhelming the user with the number of images they have to see multiplied by d L ( d : dimension of each layer, L : number of layers).
Revised: We showcase a few examples of R ℓ and R g on intermediate-layer neurons and discuss several tradeoffs: while they reveal new visualization opportunities, they run the risk of overwhelming the user with the number of images they have to see multiplied by d L ( d : dimension of each layer, L : number of layers).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Interpretable ML aims to further our understanding and trust of ML models, enable model debugging, and use these insights for joint decision-making between stakeholders and AI (Chen et al., 2022;
Revised: Interpretable ML aims to further our understanding and trust of ML models, enable model debugging, and use these insights for joint decision-making between stakeholders and AI (Chen et al., 2022;

Original: Gilpin et al., 2018).
Revised: Gilpin et al., 2018).

Original: Interpretable ML is a critical area of research straddling machine learning (Adebayo et al., 2018), language (Tenney et al., 2020), vision (Simonyan et al., 2013), and HCI (Chuang et al., 2012).
Revised: Interpretable ML is a critical area of research straddling machine learning (Adebayo et al., 2018), language (Tenney et al., 2020), vision (Simonyan et al., 2013), and HCI (Chuang et al., 2012).

Original: We categorize related work in interpreting multimodal models into:
Revised: We categorize related work in interpreting multimodal models into:

Original: Unimodal importance : Several approaches have focused on building interpretable components for unimodal importance through soft (Park et al., 2018) and hard attention mechanisms (Chen et al., 2017).
Revised: Unimodal importance : Several approaches have focused on building interpretable components for unimodal importance through soft (Park et al., 2018) and hard attention mechanisms (Chen et al., 2017).

Original: When aiming to explain black-box multimodal models, related work rely primarily on gradient-based visualizations (Simonyan et al., 2013;
Revised: When aiming to explain black-box multimodal models, related work rely primarily on gradient-based visualizations (Simonyan et al., 2013;

Original: Baehrens et al., 2010; Erhan et al., 2009) and feature attributions (e.g., LIME (Ribeiro et al., 2016), Shapley values (Merrick and Taly, 2020)) to highlight regions of the image which the model attends to.
Revised: Baehrens et al., 2010; Erhan et al., 2009) and feature attributions (e.g., LIME (Ribeiro et al., 2016), Shapley values (Merrick and Taly, 2020)) to highlight regions of the image which the model attends to.

Original: Cross-modal interactions : Recent work investigates the activation patterns of pretrained transformers (Cao et al., 2020; Li et al., 2020), performs diagnostic experiments through specially curated inputs (Frank et al., 2021; Krojer et al., 2022; Parcalabescu et al., 2021;
Revised: Cross-modal interactions : Recent work investigates the activation patterns of pretrained transformers (Cao et al., 2020; Li et al., 2020), performs diagnostic experiments through specially curated inputs (Frank et al., 2021; Krojer et al., 2022; Parcalabescu et al., 2021;

Original: Thrush et al., 2022), or trains auxiliary explanation modules (Kanehira et al., 2019; Park et al., 2018).
Revised: Thrush et al., 2022), or trains auxiliary explanation modules (Kanehira et al., 2019; Park et al., 2018).

Original: Particularly related to our work is EMAP (Hessel and Lee, 2020) for disentangling the effects of unimodal (additive) contributions from cross-modal interactions in multimodal tasks, as well as M2Lens (Wang et al., 2021), an interactive visual analytics system to visualize multimodal models for sentiment analysis through both unimodal and cross-modal contributions.
Revised: Particularly related to our work is EMAP (Hessel and Lee, 2020) for disentangling the effects of unimodal (additive) contributions from cross-modal interactions in multimodal tasks, as well as M2Lens (Wang et al., 2021), an interactive visual analytics system to visualize multimodal models for sentiment analysis through both unimodal and cross-modal contributions.

Original: Multimodal representation and prediction : Existing approaches have used language syntax (e.g., the question in VQA) for compositionality into higher-level features (Amizadeh et al., 2020; Andreas et al., 2016; Vedantam et al., 2019).
Revised: Multimodal representation and prediction : Existing approaches have used language syntax (e.g., the question in VQA) for compositionality into higher-level features (Amizadeh et al., 2020; Andreas et al., 2016; Vedantam et al., 2019).

Original: Similarly, logical statements have been integrated with neural networks for interpretable logical reasoning (Gokhale et al., 2020; Suzuki et al., 2019).
Revised: Similarly, logical statements have been integrated with neural networks for interpretable logical reasoning (Gokhale et al., 2020; Suzuki et al., 2019).

Original: However, these are typically restricted to certain modalities or tasks.
Revised: However, these are typically restricted to certain modalities or tasks.

Original: Finally, visualizations have also uncovered several biases in models and datasets (e.g., unimodal biases in VQA questions (Anand et al., 2018;
Revised: Finally, visualizations have also uncovered several biases in models and datasets (e.g., unimodal biases in VQA questions (Anand et al., 2018;

Original: Cadene et al., 2019) or gender biases in image captioning (Hendricks et al., 2018)).
Revised: Cadene et al., 2019) or gender biases in image captioning (Hendricks et al., 2018)).

Original: We believe that
Revised: We believe that

Original: M ULTI V IZ will enable the identification of biases across a wider range of modalities and tasks.
Revised: M ULTI V IZ will enable the identification of biases across a wider range of modalities and tasks.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposes M ULTI V IZ for analyzing and visualizing multimodal models.
Revised: This paper proposes M ULTI V IZ for analyzing and visualizing multimodal models.

Original: M ULTI V IZ scaffolds the interpretation problem into unimodal importance, cross-modal interactions, multimodal representations, and multimodal prediction, before providing existing and newly proposed analysis tools in each stage.
Revised: M ULTI V IZ scaffolds the interpretation problem into unimodal importance, cross-modal interactions, multimodal representations, and multimodal prediction, before providing existing and newly proposed analysis tools in each stage.

Original: M ULTI V IZ is designed to be modular (encompassing existing analysis tools and encouraging research towards understudied stages), general (supporting diverse modalities, models, and tasks), and human-in-the-loop (providing a visualization tool for human model interpretation, error analysis, and debugging), qualities which we strive to upkeep by ensuring its public access and regular updates from community feedback.
Revised: M ULTI V IZ is designed to be modular (encompassing existing analysis tools and encouraging research towards understudied stages), general (supporting diverse modalities, models, and tasks), and human-in-the-loop (providing a visualization tool for human model interpretation, error analysis, and debugging), qualities which we strive to upkeep by ensuring its public access and regular updates from community feedback.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: Multimodal data and models are ubiquitous in a range of real-world applications.
Revised: Multimodal data and models are ubiquitous in a range of real-world applications.

Original: M ULTI V IZ is our attempt at a standardized and modular framework for visualizing these multimodal models.
Revised: M ULTI V IZ is our attempt at a standardized and modular framework for visualizing these multimodal models.

Original: While we believe these tools can help stakeholders gain a deeper understanding and trust of multimodal models as a step towards reliable real-world deployment, we believe that special care must be taken in the following regard to ensure that these tools are reliably interpreted:
Revised: While we believe these tools can help stakeholders gain a deeper understanding and trust of multimodal models as a step towards reliable real-world deployment, we believe that special care must be taken in the following regard to ensure that these tools are reliably interpreted:

Original: Reliability of visualizations : There has
Revised: Reliability of visualizations : There has

Original: been
Revised: been

Original: recent work examining the reliability
Revised: recent work examining the reliability

Original: of
Revised: of

Original: model interpretability methods for real-world practitioners (Pruthi et al., 2020; Srinivas and Fleuret, 2020).
Revised: model interpretability methods for real-world practitioners (Pruthi et al., 2020; Srinivas and Fleuret, 2020).

Original: Lipton (2018) examines the motivations underlying interest in interpretability, finding them to be diverse and occasionally discordant.
Revised: Lipton (2018) examines the motivations underlying interest in interpretability, finding them to be diverse and occasionally discordant.

Original: Krishna et al.
Revised: Krishna et al.

Original: (2022) find that state-of-the-art explanation methods may disagree in terms of the explanations they output.
Revised: (2022) find that state-of-the-art explanation methods may disagree in terms of the explanations they output.

Original: Chandrasekaran et al.
Revised: Chandrasekaran et al.

Original: (2018) further conclude that existing explanations on VQA model do not actually make its responses and failures more predictable to a human.
Revised: (2018) further conclude that existing explanations on VQA model do not actually make its responses and failures more predictable to a human.

Original: We refer the reader to Chen et al.
Revised: We refer the reader to Chen et al.

Original: for a critique on the disconnect between technical objectives targeted by interpretable ML research and the high-level goals stated as consumers’ use cases, as well as Bhatt et al.
Revised: for a critique on the disconnect between technical objectives targeted by interpretable ML research and the high-level goals stated as consumers’ use cases, as well as Bhatt et al.

Original: for an analysis of how interpretable and explainable ML tools can be used in real-world deployment.
Revised: for an analysis of how interpretable and explainable ML tools can be used in real-world deployment.

Original: Humanin-the-loop interpretation and evaluation could be a promising direction towards connecting technical solutions with real-world stakeholders, while also offering users an interactive medium to incorporate feedback in multimodal models.
Revised: Humanin-the-loop interpretation and evaluation could be a promising direction towards connecting technical solutions with real-world stakeholders, while also offering users an interactive medium to incorporate feedback in multimodal models.

Original: Pitfalls of gradient-based interpretation : We are aware of the limitations underlying gradientbased interpretation of black-box models (Lipton, 2018; Srinivas and Fleuret, 2020) with issues surrounding their faithfulness and usefulness.
Revised: Pitfalls of gradient-based interpretation : We are aware of the limitations underlying gradientbased interpretation of black-box models (Lipton, 2018; Srinivas and Fleuret, 2020) with issues surrounding their faithfulness and usefulness.

Original: Future work should examine the opportunities and risks of gradient-based approaches, particularly in the context of cross-modal interactions.
Revised: Future work should examine the opportunities and risks of gradient-based approaches, particularly in the context of cross-modal interactions.

Original: The role of cross-modal interactions : There has been work showing that certain multimodal tasks do not need models to pick up cross-modal interactions to achieve good performance (Hessel and Lee, 2020).
Revised: The role of cross-modal interactions : There has been work showing that certain multimodal tasks do not need models to pick up cross-modal interactions to achieve good performance (Hessel and Lee, 2020).

Original: Indeed, for tasks like cross-modal retrieval, simply learning one interaction between a word and its corresponding image region is enough for typical datasets.
Revised: Indeed, for tasks like cross-modal retrieval, simply learning one interaction between a word and its corresponding image region is enough for typical datasets.

Original: This makes interpretation of cross-modal interactions difficult, since even well-performing models may not need to pick up all cross-modal interactions.
Revised: This makes interpretation of cross-modal interactions difficult, since even well-performing models may not need to pick up all cross-modal interactions.

Original: User studies : Based on direct communication with our institution’s IRB office, this line of user-study research is aligned with similar annotation studies at our institution that are exempt from IRB.
Revised: User studies : Based on direct communication with our institution’s IRB office, this line of user-study research is aligned with similar annotation studies at our institution that are exempt from IRB.

Original: The information obtained during our study is recorded in such a manner that the identity of the human subjects cannot readily be ascertained, directly or through identifiers linked to the subjects.
Revised: The information obtained during our study is recorded in such a manner that the identity of the human subjects cannot readily be ascertained, directly or through identifiers linked to the subjects.

Original: We do not collect any identifiable information from annotators.
Revised: We do not collect any identifiable information from annotators.

Original: Usability : While we tried to be comprehensive in providing visualizations to the user, more information beyond a certain point is probably not useful and may overwhelm the user.
Revised: Usability : While we tried to be comprehensive in providing visualizations to the user, more information beyond a certain point is probably not useful and may overwhelm the user.

Original: We plan to work closely with HCI researchers to rethink usability and design of our proposed interpretation tools through careful user studies.
Revised: We plan to work closely with HCI researchers to rethink usability and design of our proposed interpretation tools through careful user studies.

Original: M ULTI V IZ will also welcome feedback from the public to improve its usability.
Revised: M ULTI V IZ will also welcome feedback from the public to improve its usability.

Original: Beyond M ULTI V IZ stages : While we believe that many multimodal problems can benefit from breaking them down into our proposed interpretation stages, we also acknowledge that certain problems may not benefit from this perspective.
Revised: Beyond M ULTI V IZ stages : While we believe that many multimodal problems can benefit from breaking them down into our proposed interpretation stages, we also acknowledge that certain problems may not benefit from this perspective.

Original: For example, problems in multimodal translation (mapping from one modality to another, such as image captioning) will not involve prediction layers and instead require new stages to interpret the generation process, and problems in crossmodal transfer will also require new stages to interpret knowledge transfer.
Revised: For example, problems in multimodal translation (mapping from one modality to another, such as image captioning) will not involve prediction layers and instead require new stages to interpret the generation process, and problems in crossmodal transfer will also require new stages to interpret knowledge transfer.

Original: In Appendix E, we include more details on new datasets we plan to add to M ULTI V IZ to enable the study of new multimodal interpretability problems, and other interpretation tools we plan to add.
Revised: In Appendix E, we include more details on new datasets we plan to add to M ULTI V IZ to enable the study of new multimodal interpretability problems, and other interpretation tools we plan to add.

Original: Evaluating interpretability : Progress towards interpretability is challenging to evaluate (Chan et al., 2022;
Revised: Evaluating interpretability : Progress towards interpretability is challenging to evaluate (Chan et al., 2022;

Original: Dasgupta et al., 2022; Jacovi and Goldberg, 2020; Shah et al., 2021; Srinivas and Fleuret, 2020).
Revised: Dasgupta et al., 2022; Jacovi and Goldberg, 2020; Shah et al., 2021; Srinivas and Fleuret, 2020).

Original: Model interpretability (1) is highly subjective across different population subgroups (Arora et al., 2021; Krishna et al., 2022), (2) requires high-
Revised: Model interpretability (1) is highly subjective across different population subgroups (Arora et al., 2021; Krishna et al., 2022), (2) requires high-

Original: dimensional model outputs as opposed to low-dimensional prediction objectives (Park et al., 2018), and (3) has desiderata that change across research fields, populations, and time (Murdoch et al., 2019).
Revised: dimensional model outputs as opposed to low-dimensional prediction objectives (Park et al., 2018), and (3) has desiderata that change across research fields, populations, and time (Murdoch et al., 2019).

Original: We plan to continuously expand M ULTI V IZ through community inputs for new interpretation methods in each stage and metrics to evaluate interpretability methods (see Appendix E for details).
Revised: We plan to continuously expand M ULTI V IZ through community inputs for new interpretation methods in each stage and metrics to evaluate interpretability methods (see Appendix E for details).

Original: Some metrics we have in mind include those for measuring faithfulness, as proposed in recent work (Chan et al., 2022; Dasgupta et al., 2022; Jacovi and Goldberg, 2020; Madsen et al., 2021;
Revised: Some metrics we have in mind include those for measuring faithfulness, as proposed in recent work (Chan et al., 2022; Dasgupta et al., 2022; Jacovi and Goldberg, 2020; Madsen et al., 2021;

Original: Shah et al., 2021; Srinivas and Fleuret, 2020; Wu and Mooney, 2019).
Revised: Shah et al., 2021; Srinivas and Fleuret, 2020; Wu and Mooney, 2019).

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: Our code, datasets, and documentation are released at https://github.com/ pliang279/MultiViz .
Revised: Our code, datasets, and documentation are released at https://github.com/ pliang279/MultiViz .

Original: This link also includes human-in-the-loop evaluation scripts and instructions on running M ULTI V IZ for new datasets and models.
Revised: This link also includes human-in-the-loop evaluation scripts and instructions on running M ULTI V IZ for new datasets and models.

Original: Details on the M ULTI V IZ visualization approaches are provided in Appendix A. 3.
Revised: Details on the M ULTI V IZ visualization approaches are provided in Appendix A. 3.

Original: Details on the M ULTI V IZ website, sample webpages with visualizations, code structure, and sample tutorials are provided in Appendix B. 4.
Revised: Details on the M ULTI V IZ website, sample webpages with visualizations, code structure, and sample tutorials are provided in Appendix B. 4.

Original: Dataset collection and preprocessing details are provided in Appendix C.
Revised: Dataset collection and preprocessing details are provided in Appendix C.

Original: We provide documentation for M ULTI V IZ in the form of datasheets for datasets (Gebru et al., 2018).
Revised: We provide documentation for M ULTI V IZ in the form of datasheets for datasets (Gebru et al., 2018).

Original: Experimental details, including all details on user studies and evaluation, are provided in Appendix D.
Revised: Experimental details, including all details on user studies and evaluation, are provided in Appendix D.

Original: A CKNOWLEDGEMENTS
Revised: A CKNOWLEDGEMENTS

Original: This material is based upon work partially supported by the National Science Foundation (Awards #1722822 and #1750439), National Institutes of Health (Awards #R01MH125740, #R01MH096951, and #U01MH116925), and BMW of North America.
Revised: This material is based upon work partially supported by the National Science Foundation (Awards #1722822 and #1750439), National Institutes of Health (Awards #R01MH125740, #R01MH096951, and #U01MH116925), and BMW of North America.

Original: PPL is partially supported by a Facebook
Revised: PPL is partially supported by a Facebook

Original: PhD Fellowship and a Carnegie Mellon University’s Center for Machine Learning and Health
Revised: PhD Fellowship and a Carnegie Mellon University’s Center for Machine Learning and Health

Original: Fellowship.
Revised: Fellowship.

Original: RS is supported in part by ONR award N000141812861 and DSTA.
Revised: RS is supported in part by ONR award N000141812861 and DSTA.

Original: Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, National Institutes of
Revised: Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, National Institutes of

Original: Health, Facebook, Carnegie Mellon University’s Center for Machine Learning and Health, Office of Naval Research, or DSTA, and no official endorsement should be inferred.
Revised: Health, Facebook, Carnegie Mellon University’s Center for Machine Learning and Health, Office of Naval Research, or DSTA, and no official endorsement should be inferred.

Original: We are extremely grateful to Ben Eysenbach, Martin Ma, Chaitanya Ahuja, Volkan Cirik, Peter Wu, Amir Zadeh, Alex
Revised: We are extremely grateful to Ben Eysenbach, Martin Ma, Chaitanya Ahuja, Volkan Cirik, Peter Wu, Amir Zadeh, Alex

Original: Wilf, Victoria Lin, Dong Won Lee, Torsten Wortwein, and Tiffany Min for helpful discussions and feedback on initial versions of this paper.
Revised: Wilf, Victoria Lin, Dong Won Lee, Torsten Wortwein, and Tiffany Min for helpful discussions and feedback on initial versions of this paper.

Original: Finally, we would also like to acknowledge NVIDIA’s GPU support.
Revised: Finally, we would also like to acknowledge NVIDIA’s GPU support.

Original: A PPENDIX C ONTENTS A Analysis Details
Revised: A PPENDIX C ONTENTS A Analysis Details

Original: A.1 Unimodal importance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.1 Unimodal importance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.
Revised: A.

Original: 2 Cross-modal interactions
Revised: 2 Cross-modal interactions

Original: A.
Revised: A.

Original: Multimodal
Revised: Multimodal

Original: representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.
Revised: A.

Original: 4 Multimodal
Revised: 4 Multimodal

Original: prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.
Revised: A.

Original: 5 Sanity checks for saliency
Revised: 5 Sanity checks for saliency

Original: maps . .
Revised: maps . .

Original: B M ULTI V IZ Visualization Tool
Revised: B M ULTI V IZ Visualization Tool

Original: B.
Revised: B.

Original: M
Revised: M

Original: ULTI V IZ code
Revised: ULTI V IZ code

Original: framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.
Revised: B.

Original: 2 The M ULTI V IZ website
Revised: 2 The M ULTI V IZ website

Original: B
Revised: B

Original: .3 Intermediate layer representation visualization
Revised: .3 Intermediate layer representation visualization

Original: C Datasets and Models
Revised: C Datasets and Models

Original: C.
Revised: C.

Original: 1 Datasets
Revised: 1 Datasets

Original: in
Revised: in

Original: M ULTI
Revised: M ULTI

Original: V IZ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: V IZ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.
Revised: C.

Original: Multimodal fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Multimodal fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C
Revised: C

Original: Multimodal
Revised: Multimodal

Original: retrieval . . .
Revised: retrieval . . .

Original: C.1.
Revised: C.1.

Original: Multimodal question answering . .
Revised: Multimodal question answering . .

Original: D Additional Experiments and Details
Revised: D Additional Experiments and Details

Original: D.1 Model simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.1 Model simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.1.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.1.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.1.2 VQA 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.1.2 VQA 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.1.3 MM-IMDb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.1.3 MM-IMDb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.1.4 CMU-MOSEI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.1.4 CMU-MOSEI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.2 Cross-modal interactions . .
Revised: D.2 Cross-modal interactions . .

Original: D.2.1 CLEVR . .
Revised: D.2.1 CLEVR . .

Original: D.2.2 F LICKR -30 K . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.2.2 F LICKR -30 K . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.3 Representation interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.3 Representation interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.3.1 VQA 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.3.1 VQA 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.3.2 CLEVR . .
Revised: D.3.2 CLEVR . .

Original: D.3.3 MM-IMD B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.3.3 MM-IMD B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.4 Error analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.4 Error analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.4.

Revised: D.4.

Original: Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.4.2 VQA 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.4.2 VQA 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.4.3 CLEVR . .
Revised: D.4.3 CLEVR . .

Original: D.5 Model debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.5 Model debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.5.1 VQA 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.5.1 VQA 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.6 Summary of takeaway messages . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.6 Summary of takeaway messages . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E Limitations and Future Directions
Revised: E Limitations and Future Directions

Original: E.1 Limitations . .
Revised: E.1 Limitations . .

Original: E.2 New classes of interpretation tools . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: E.2 New classes of interpretation tools . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E.2.
Revised: E.2.

Original: Cross-modal interactions . .
Revised: Cross-modal interactions . .

Original: E
Revised: E

Original: [Table]
Revised: [Table]

Original: Multimodal prediction . .
Revised: Multimodal prediction . .

Original: E
Revised: E

Original: [Table]
Revised: [Table]

Original: 3 Evaluating interpretability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: 3 Evaluating interpretability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E
Revised: E

Original: [Table]
Revised: [Table]

Original: stakeholders . .
Revised: stakeholders . .

Original: A A NALYSIS D ETAILS A. U NIMODAL IMPORTANCE
Revised: A A NALYSIS D ETAILS A. U NIMODAL IMPORTANCE

Original: Unimodal importance aims to understand the contributions of each modality towards modeling and prediction.
Revised: Unimodal importance aims to understand the contributions of each modality towards modeling and prediction.

Original: It builds upon ideas of gradient-based visualizations (e.g., Gradient Simonyan et al.
Revised: It builds upon ideas of gradient-based visualizations (e.g., Gradient Simonyan et al.

Original: Baehrens et al.
Revised: Baehrens et al.

Original: ; Erhan et al.
Revised: ; Erhan et al.

Original: and feature attributions (e.g., LIME (Han et al., 2020;
Revised: and feature attributions (e.g., LIME (Han et al., 2020;

Original: Ribeiro et al., 2016;
Revised: Ribeiro et al., 2016;

Original: Yosinski et al., 2015), Shapley values (Merrick and Taly, 2020; Rodríguez-Pérez and Bajorath, 2020; Sundararajan and Najmi, 2020)).
Revised: Yosinski et al., 2015), Shapley values (Merrick and Taly, 2020; Rodríguez-Pérez and Bajorath, 2020; Sundararajan and Najmi, 2020)).

Original: Taking LIME
Revised: Taking LIME

Original: (Ribeiro et al., 2016)
Revised: (Ribeiro et al., 2016)

Original: for an example, given model f , we would like to return weights over each of the x 1 and x 2 ’s such that important modalities are accurately weighted.
Revised: for an example, given model f , we would like to return weights over each of the x 1 and x 2 ’s such that important modalities are accurately weighted.

Original: LIME perturbs the set of x 1 and x 2 ’s, observes how model predictions change, and fits a local linear model with respect to that datapoint.
Revised: LIME perturbs the set of x 1 and x 2 ’s, observes how model predictions change, and fits a local linear model with respect to that datapoint.

Original: The areas with the highest positive weights are presented as the important ones.
Revised: The areas with the highest positive weights are presented as the important ones.

Original: Other feature attribution and visualization approaches, such as Gradient -based
Revised: Other feature attribution and visualization approaches, such as Gradient -based

Original: (Chandrasekaran et al., 2018;
Revised: (Chandrasekaran et al., 2018;

Original: Selvaraju et al., 2017) or Shapley values (Merrick and Taly, 2020; Rodríguez-Pérez and
Revised: Selvaraju et al., 2017) or Shapley values (Merrick and Taly, 2020; Rodríguez-Pérez and

Original: Bajorath, 2020; Sundararajan and Najmi, 2020), work similarly (Yosinski et al., 2015).
Revised: Bajorath, 2020; Sundararajan and Najmi, 2020), work similarly (Yosinski et al., 2015).

Original: We implement unimodal feature attribution methods as a module U NI ( f θ , y, x ) taking in a trained model f θ , an output/feature y which analysis is performed with respect to, and the modality of interest x .
Revised: We implement unimodal feature attribution methods as a module U NI ( f θ , y, x ) taking in a trained model f θ , an output/feature y which analysis is performed with respect to, and the modality of interest x .

Original: U NI returns importance weights across atoms x of modality x .
Revised: U NI returns importance weights across atoms x of modality x .

Original: A. C ROSS - MODAL INTERACTIONS
Revised: A. C ROSS - MODAL INTERACTIONS

Original: Cross-modal interactions describe ways in which atoms from different modalities can relate with each other and the types of new information possibly discovered as a result of these relationships.
Revised: Cross-modal interactions describe ways in which atoms from different modalities can relate with each other and the types of new information possibly discovered as a result of these relationships.

Original: M ULTI V IZ includes two recent methods for understanding cross-modal interactions:
Revised: M ULTI V IZ includes two recent methods for understanding cross-modal interactions:

Original: EMAP (Hessel and Lee, 2020) decomposes f ( x x 2 ) =
Revised: EMAP (Hessel and Lee, 2020) decomposes f ( x x 2 ) =

Original: g 1 ( x
Revised: g 1 ( x

Original: ) + g 2 (
Revised: ) + g 2 (

Original: + g
Revised: + g

Original: ) into strictly to captured by model.
Revised: ) into strictly to captured by model.

Original: DIME (Lyu et al., 2022) further extends EMAP by designing an efficient method for feature visualization on each disentangled representation locally (per datapoint).
Revised: DIME (Lyu et al., 2022) further extends EMAP by designing an efficient method for feature visualization on each disentangled representation locally (per datapoint).

Original: Higher-order Gradient is our proposed method for efficiently quantifying the presence of crossmodal interactions.
Revised: Higher-order Gradient is our proposed method for efficiently quantifying the presence of crossmodal interactions.

Original: Based on the gradient definition of statistical non-additive interaction (Friedman and Popescu, 2008; Tsang et al., 2019), a function f exhibits non-additive interactions amongunimodal atoms x 1 and x 2 if (cid:104) ∂ 2 f ( x 1 ,x 2 ) ∂x 1 ∂x 2 (cid:105) 2 > 0 .
Revised: Based on the gradient definition of statistical non-additive interaction (Friedman and Popescu, 2008; Tsang et al., 2019), a function f exhibits non-additive interactions amongunimodal atoms x 1 and x 2 if (cid:104) ∂ 2 f ( x 1 ,x 2 ) ∂x 1 ∂x 2 (cid:105) 2 > 0 .

Original: Writing the multimodal model f as f ( x 1 , x 2 )
Revised: Writing the multimodal model f as f ( x 1 , x 2 )

Original: =
Revised: =

Original: g 1 ( x 1 ) + g 2 ( x 2 ) + g 12 ( x 1 , x 2 ) , we can isolate the effect of g 12 ( x 1 , x 2 ) by taking a second-order gradient of f with respect to x 1 and x 2 so the g 1 ( x 1 ) and g 2 ( x 2 ) terms becomes zero.
Revised: g 1 ( x 1 ) + g 2 ( x 2 ) + g 12 ( x 1 , x 2 ) , we can isolate the effect of g 12 ( x 1 , x 2 ) by taking a second-order gradient of f with respect to x 1 and x 2 so the g 1 ( x 1 ) and g 2 ( x 2 ) terms becomes zero.

Original: Theoretically, second-order gradients are necessary and sufficient to recover cross-modal interactions: purely additive models will have strictly 0 second-order gradient information, and any non-linear interaction term g 12 ( x 1 , x 2 ) must have strictly non-zero second-order gradient information.
Revised: Theoretically, second-order gradients are necessary and sufficient to recover cross-modal interactions: purely additive models will have strictly 0 second-order gradient information, and any non-linear interaction term g 12 ( x 1 , x 2 ) must have strictly non-zero second-order gradient information.

Original: Definition 2 inspires us to extend first-order gradient and perturbation-based approaches (Han et al., 2020; Ribeiro et al., 2016;
Revised: Definition 2 inspires us to extend first-order gradient and perturbation-based approaches (Han et al., 2020; Ribeiro et al., 2016;

Original: Yosinski et al., 2015) to the second order.
Revised: Yosinski et al., 2015) to the second order.

Original: Our implementation first computes a gradient of f with respect to one input modality atom (e.g., x 1 = birds ), which results in a vector ∇ 1 = ∂f∂x 1 of the same dimension as x 1 (i.e., token embedding dimension).
Revised: Our implementation first computes a gradient of f with respect to one input modality atom (e.g., x 1 = birds ), which results in a vector ∇ 1 = ∂f∂x 1 of the same dimension as x 1 (i.e., token embedding dimension).

Original: We aggregate the vector components of ∇ 1 via summation to produce a single scalar ∥∇ 1 ∥ , before taking a secondorder gradient with respect to all atoms of the second modality x 2 ∈ x 2 (e.g., all image pixels), which results in a vector ∇ 

Revised: We aggregate the vector components of ∇ 1 via summation to produce a single scalar ∥∇ 1 ∥ , before taking a secondorder gradient with respect to all atoms of the second modality x 2 ∈ x 2 (e.g., all image pixels), which results in a vector ∇

Original: = (cid:20) ∂
Revised: = (cid:20) ∂

Original: 2 f ∂x 1 ∂x (1)2 , ..., ∂ 2 f ∂x 1 ∂x ( | x 2 | )(cid:21) of the same dimension as | x 2 | (i.e., total number of pixels).
Revised: 2 f ∂x 1 ∂x (1)2 , ..., ∂ 2 f ∂x 1 ∂x ( | x 2 | )(cid:21) of the same dimension as | x 2 | (i.e., total number of pixels).

Original: Each scalar entry in ∇ 12 highlights atoms x 2 that have non-linear interactions with the original atom x 1 (e.g., only the birds in the image, see Figure 2 for examples on real datasets).
Revised: Each scalar entry in ∇ 12 highlights atoms x 2 that have non-linear interactions with the original atom x 1 (e.g., only the birds in the image, see Figure 2 for examples on real datasets).

Original: We implement a general module CM ( f θ , y, x 1 , x 2 ) for cross-modal visualizations, taking in a trained model f θ , an output/feature y , the first modality’s atom of interest x 1 , and the entire second modality of interest x 2 .
Revised: We implement a general module CM ( f θ , y, x 1 , x 2 ) for cross-modal visualizations, taking in a trained model f θ , an output/feature y , the first modality’s atom of interest x 1 , and the entire second modality of interest x 2 .

Original: CM returns importance weights across atoms x 2 of modality x 2 , and can build on top of any first-order unimodal attribution method (i.e., gradient visualization (Erhan et al., 2009),
Revised: CM returns importance weights across atoms x 2 of modality x 2 , and can build on top of any first-order unimodal attribution method (i.e., gradient visualization (Erhan et al., 2009),

Original: LIME (Ribeiro et al., 2016), or Shapley values (Merrick and Taly, 2020), see Appendix A.2).
Revised: LIME (Ribeiro et al., 2016), or Shapley values (Merrick and Taly, 2020), see Appendix A.2).

Original: We plan to make several approximations: only estimating single instances ( x 1 , x 2 ) at a time which avoids the expectation, and computing the magnitude w ( x 1 , x 2 ) =
Revised: We plan to make several approximations: only estimating single instances ( x 1 , x 2 ) at a time which avoids the expectation, and computing the magnitude w ( x 1 , x 2 ) =

Original: (cid:16) ∂f ( x ) ∂x 1 ∂x 2 (cid:17) 2 as a measure of cross-modal interaction strength.
Revised: (cid:16) ∂f ( x ) ∂x 1 ∂x 2 (cid:17) 2 as a measure of cross-modal interaction strength.

Original: Specifically, given a model f , we first take a gradient of f with respect to an input word (e.g., x 1 = dog ), before taking a second-order gradient with respect to all input image pixels x 2 , which should result in only the dog in the image being highlighted (see
Revised: Specifically, given a model f , we first take a gradient of f with respect to an input word (e.g., x 1 = dog ), before taking a second-order gradient with respect to all input image pixels x 2 , which should result in only the dog in the image being highlighted (see

Original: Figure 2 for examples on real datasets).
Revised: Figure 2 for examples on real datasets).

Original: We implement a general module CM ( f θ , y, x 1 , x 2 ) for cross-modal visualizations, taking in a trained model f θ , an output/feature y , the first modality’s atom of interest x 1 , and the entire second modality of interest x 2 .
Revised: We implement a general module CM ( f θ , y, x 1 , x 2 ) for cross-modal visualizations, taking in a trained model f θ , an output/feature y , the first modality’s atom of interest x 1 , and the entire second modality of interest x 2 .

Original: CM returns importance weights across atoms x 2 of modality x 2 , and can build on top of any first-order unimodal attribution method, such as gradient visualization (Goyal et al., 2016),
Revised: CM returns importance weights across atoms x 2 of modality x 2 , and can build on top of any first-order unimodal attribution method, such as gradient visualization (Goyal et al., 2016),

Original: LIME (Ribeiro et al., 2016), or Shapley values (Merrick and Taly, 2020).
Revised: LIME (Ribeiro et al., 2016), or Shapley values (Merrick and Taly, 2020).

Original: A. M ULTIMODAL REPRESENTATIONS
Revised: A. M ULTIMODAL REPRESENTATIONS

Original: Given these highlighted unimodal and cross-modal interactions at the input level, the next stage aims to understand how these interactions are represented at the feature representation level.
Revised: Given these highlighted unimodal and cross-modal interactions at the input level, the next stage aims to understand how these interactions are represented at the feature representation level.

Original: Specifically, given a trained multimodal model f , define the matrix M z ∈ R N × d as the penultimate layer of f representing (uninterpretable) deep feature representations implicitly containing information from both unimodal and cross-modal interactions.
Revised: Specifically, given a trained multimodal model f , define the matrix M z ∈ R N × d as the penultimate layer of f representing (uninterpretable) deep feature representations implicitly containing information from both unimodal and cross-modal interactions.

Original: For the i th datapoint, z = M z ( i ) collects a set of individual feature representations z 1 , z 2 , ..., z d ∈ R .
Revised: For the i th datapoint, z = M z ( i ) collects a set of individual feature representations z 1 , z 2 , ..., z d ∈ R .

Original: We aim to interpret these feature representations through both local and global analysis (see Figure ??
Revised: We aim to interpret these feature representations through both local and global analysis (see Figure ??

Original: for an example):
Revised: for an example):

Original: Local representation analysis (R ℓ ) informs the user on parts of the original datapoint that activate feature z j .
Revised: Local representation analysis (R ℓ ) informs the user on parts of the original datapoint that activate feature z j .

Original: To do so, we run unimodal and cross-modal visualization methods with respect to feature z j
Revised: To do so, we run unimodal and cross-modal visualization methods with respect to feature z j

Original: (i.e., U NI ( f θ , z j , x ) , CM ( f θ , z j , x 1 , x 2 ) ) in order to explain the input unimodal and cross-modal interactions represented in feature z j .
Revised: (i.e., U NI ( f θ , z j , x ) , CM ( f θ , z j , x 1 , x 2 ) ) in order to explain the input unimodal and cross-modal interactions represented in feature z j .

Original: Local analysis is useful in explaining model predictions on the original datapoint by studying the input regions activating feature z j .
Revised: Local analysis is useful in explaining model predictions on the original datapoint by studying the input regions activating feature z j .

Original: Global representation analysis (R g ) provides the user with the top k datapoints
Revised: Global representation analysis (R g ) provides the user with the top k datapoints

Original: D k
Revised: D k

Original: ( z j )
Revised: ( z j )

Original: = { ( x 1 , x 2 , y )
Revised: = { ( x 1 , x 2 , y )

Original: ki =(1) } that also maximally activate feature z j .
Revised: ki =(1) } that also maximally activate feature z j .

Original: By further unimodal and cross-modal visualizations on datapoints in D k ( z j ) , global analysis is especially useful in helping humans assign interpretable language concepts to each feature by looking at similarly activated input regions across datapoints (e.g., the concept of color in Figure ?? ).
Revised: By further unimodal and cross-modal visualizations on datapoints in D k ( z j ) , global analysis is especially useful in helping humans assign interpretable language concepts to each feature by looking at similarly activated input regions across datapoints (e.g., the concept of color in Figure ?? ).

Original: Global analysis can also help to find related datapoints the model also struggles with for error analysis.
Revised: Global analysis can also help to find related datapoints the model also struggles with for error analysis.

Original: A. M ULTIMODAL PREDICTION
Revised: A. M ULTIMODAL PREDICTION

Original: Finally, the prediction step takes the set of feature representations z 1 , z 2 , ..., z d and composes them to form higher-level abstract concepts suitable for a task.
Revised: Finally, the prediction step takes the set of feature representations z 1 , z 2 , ..., z d and composes them to form higher-level abstract concepts suitable for a task.

Original: We approximate the prediction process with a linear combination of penultimate layer features by integrating a sparse linear prediction model with neural network features (Wong et al., 2021).
Revised: We approximate the prediction process with a linear combination of penultimate layer features by integrating a sparse linear prediction model with neural network features (Wong et al., 2021).

Original: Given the penultimate layer M z ∈ R N × d , we fit a linear model E ( Y | X = x ) =
Revised: Given the penultimate layer M z ∈ R N × d , we fit a linear model E ( Y | X = x ) =

Original: M ⊤ z β (bias β 0 omitted for simplicity) and solve for sparsity using:
Revised: M ⊤ z β (bias β 0 omitted for simplicity) and solve for sparsity using:

Original: [Equation]
Revised: [Equation]

Original: The resulting understanding starts from the set of learned weights with the highest non-zero coefficients β top = { β (1) , β (2) , ... } and corresponding ranked features z top =
Revised: The resulting understanding starts from the set of learned weights with the highest non-zero coefficients β top = { β (1) , β (2) , ... } and corresponding ranked features z top =

Original: { z (1) , z (2) , ... } .
Revised: { z (1) , z (2) , ... } .

Original: β top tells the user how features z top are composed to make a prediction, and z top can then be visualized with respect to unimodal and cross-modal interactions using the representation stage.
Revised: β top tells the user how features z top are composed to make a prediction, and z top can then be visualized with respect to unimodal and cross-modal interactions using the representation stage.

Original: A. S ANITY CHECKS FOR SALIENCY MAPS
Revised: A. S ANITY CHECKS FOR SALIENCY MAPS

Original: According to Adebayo et al.
Revised: According to Adebayo et al.

Original: (2018), a visualization/interpretation method should be rejected if it admits invariance over either data or model, i.e. transformation of data or model does not change the output of the method.
Revised: (2018), a visualization/interpretation method should be rejected if it admits invariance over either data or model, i.e. transformation of data or model does not change the output of the method.

Original: We perform a similar sanity check on M ULTI V IZ :
Revised: We perform a similar sanity check on M ULTI V IZ :

Original: Data randomization test : M ULTI V IZ does not admit data invariance, as MultiViz visualizations on the same model varies between different data points and labels (see visualization examples in
Revised: Data randomization test : M ULTI V IZ does not admit data invariance, as MultiViz visualizations on the same model varies between different data points and labels (see visualization examples in

Original: Figure 4, 5, and 6).
Revised: Figure 4, 5, and 6).

Original: The visualizations reliably capture unique input regions, related datapoints, feature concepts, and errors specific to each data point.
Revised: The visualizations reliably capture unique input regions, related datapoints, feature concepts, and errors specific to each data point.

Original: [Table]
Revised: [Table]

Original: Given: Dataset D = { ( x 1 , x 2 , y ) ni =1 } = { ( x (1)1 , ..., x (1)2 , ..., y ) ni =1 } and trained model f θ .
Revised: Given: Dataset D = { ( x 1 , x 2 , y ) ni =1 } = { ( x (1)1 , ..., x (1)2 , ..., y ) ni =1 } and trained model f θ .

Original: Given: Unimodal and cross-modal visualization subroutines U NI
Revised: Given: Unimodal and cross-modal visualization subroutines U NI

Original: ( f θ , y, x ) , CM ( f θ , y, x 1 , x 2 ) .
Revised: ( f θ , y, x ) , CM ( f θ , y, x 1 , x 2 ) .

Original: Obtain deep features M z = f θ ( x 1 , x 2 ) for datapoint of interest.
Revised: Obtain deep features M z = f θ ( x 1 , x 2 ) for datapoint of interest.

Original: Fit sparse linear model:
Revised: Fit sparse linear model:

Original: f θ sparse =
Revised: f θ sparse =

Original: M ⊤ z ˆ β by solving equation (2).
Revised: M ⊤ z ˆ β by solving equation (2).

Original: Obtain predictions ˆ y =
Revised: Obtain predictions ˆ y =

Original: f θ sparse ( x 1 , x 2 ) and ranked features z top with largest coefficients.
Revised: f θ sparse ( x 1 , x 2 ) and ranked features z top with largest coefficients.

Original: Visualize overall unimodal importance wrt
Revised: Visualize overall unimodal importance wrt

Original: ˆ y : U 

Revised: ˆ y : U

Original: =
Revised: =

Original: U NI ( f θ , ˆ y, x 1 ) , U 2 =
Revised: U NI ( f θ , ˆ y, x 1 ) , U 2 =

Original: U NI ( f θ , ˆ y, x 2 ) .
Revised: U NI ( f θ , ˆ y, x 2 ) .

Original: Visualize overall cross-modal interactions wrt
Revised: Visualize overall cross-modal interactions wrt

Original: ˆ y : C = CM ( f θ , ˆ y, x 1 , x 2 ) , CM ( f θ , ˆ y, x 2 , x 1 ) .
Revised: ˆ y : C = CM ( f θ , ˆ y, x 1 , x 2 ) , CM ( f θ , ˆ y, x 2 , x 1 ) .

Original: for each top feature z in z top do
Revised: for each top feature z in z top do

Original: # local analysis for original datapoint
Revised: # local analysis for original datapoint

Original: Visualize unimodal and cross-modal interactions of original datapoint wrt z .
Revised: Visualize unimodal and cross-modal interactions of original datapoint wrt z .

Original: global analysis across similar datapoints
Revised: global analysis across similar datapoints

Original: Obtain top k datapoints that also maximally activate z .
Revised: Obtain top k datapoints that also maximally activate z .

Original: for each new top k datapoint ( x 1 , x 2 , y ) do
Revised: for each new top k datapoint ( x 1 , x 2 , y ) do

Original: Visualize unimodal and cross-modal interactions of new datapoint wrt z .
Revised: Visualize unimodal and cross-modal interactions of new datapoint wrt z .

Original: Model randomization test : In Appendix D.2, we demonstrated that M ULTI V IZ produces different results for two different models on the same data for both CLEVR question answering and Flickr-30K retrieval: M ULTI V IZ enables us to explain differences in performance across 2 models based on the accuracy of cross-modal interactions each model captures, so M ULTI V IZ passes the model randomization test.
Revised: Model randomization test : In Appendix D.2, we demonstrated that M ULTI V IZ produces different results for two different models on the same data for both CLEVR question answering and Flickr-30K retrieval: M ULTI V IZ enables us to explain differences in performance across 2 models based on the accuracy of cross-modal interactions each model captures, so M ULTI V IZ passes the model randomization test.

Original: Therefore, our methods do not admit data or model invariance and passes the sanity checks from Adebayo et al.
Revised: Therefore, our methods do not admit data or model invariance and passes the sanity checks from Adebayo et al.

Original: B M ULTI V IZ V ISUALIZATION T OOL
Revised: B M ULTI V IZ V ISUALIZATION T OOL

Original: We summarize these proposed approaches for understanding each step of the multimodal process in Table 4, and show the overall pipeline in Algorithm 1 and Figure 1.
Revised: We summarize these proposed approaches for understanding each step of the multimodal process in Table 4, and show the overall pipeline in Algorithm 1 and Figure 1.

Original: To enable human studies,
Revised: To enable human studies,

Original: M ULTI V IZ provides an interactive API where users can choose multimodal datasets and models and be presented with a set of visualizations at each stage.
Revised: M ULTI V IZ provides an interactive API where users can choose multimodal datasets and models and be presented with a set of visualizations at each stage.

Original: In this section, we will include both introductions to our code framework that enables easy application of analysis visualization methods to datasets and models, and also present the M ULTI V IZ website that showcases some examples of visualizations generated for each stage on different datasets and models.
Revised: In this section, we will include both introductions to our code framework that enables easy application of analysis visualization methods to datasets and models, and also present the M ULTI V IZ website that showcases some examples of visualizations generated for each stage on different datasets and models.

Original: B. M ULTI V IZ CODE FRAMEWORK
Revised: B. M ULTI V IZ CODE FRAMEWORK

Original: One additional major contribution of our works is that we designed a code framework in Python for easy analysis, interpretation and visualization of models on multimodal datasets with only a few lines of code.
Revised: One additional major contribution of our works is that we designed a code framework in Python for easy analysis, interpretation and visualization of models on multimodal datasets with only a few lines of code.

Original: The framework is modularized and extendable to new datasets, models and visualization methods.
Revised: The framework is modularized and extendable to new datasets, models and visualization methods.

Original: Figure 7 is an illustration of the main modules of the code framework:
Revised: Figure 7 is an illustration of the main modules of the code framework:

Original: • Within the datasets module, we include scripts for retrieving information directly from the dataset, including getting specific data points from a split, getting the ground truth labels, label-id-to
Revised: • Within the datasets module, we include scripts for retrieving information directly from the dataset, including getting specific data points from a split, getting the ground truth labels, label-id-to

Original: -answer and answer-to-label-id mappings, etc.
Revised: -answer and answer-to-label-id mappings, etc.

Original: Some dataset scripts also
Revised: Some dataset scripts also

Original: VQA
Revised: VQA

Original: [Figure]
Revised: [Figure]

Original: CLEVR
Revised: CLEVR

Original: CMU-MOSEI
Revised: CMU-MOSEI

Original: MIMIC
Revised: MIMIC

Original: Flickr-30k MM-IMDb
Revised: Flickr-30k MM-IMDb

Original: Datasets
Revised: Datasets

Original: Models
Revised: Models

Original: LXMERT
Revised: LXMERT

Original: ViLT
Revised: ViLT

Original: CLIP
Revised: CLIP

Original: CNN-LSTM-SA
Revised: CNN-LSTM-SA

Original: MulT
Revised: MulT

Original: LRTF
Revised: LRTF

Original: Analysis methods
Revised: Analysis methods

Original: LIME
Revised: LIME

Original: [Figure]
Revised: [Figure]

Original: DIME
Revised: DIME

Original: Sparse linear model
Revised: Sparse linear model

Original: Visualization tools
Revised: Visualization tools

Original: Sparse linear model visualizer
Revised: Sparse linear model visualizer

Original: [Figure]
Revised: [Figure]

Original: supports generating visualizations for data points (for example, the script for VQA supports generating pictures that contain both the image and the question).
Revised: supports generating visualizations for data points (for example, the script for VQA supports generating pictures that contain both the image and the question).

Original: • Within the models module, we write a wrapper for every supported model that inherits a common parent class called analysismodel , which defines a set of functionalities commonly used in various analysis methods.
Revised: • Within the models module, we write a wrapper for every supported model that inherits a common parent class called analysismodel , which defines a set of functionalities commonly used in various analysis methods.

Original: The functions in analysismodel include forward (just making a prediction on a specific data point), forwardbatch (forward but on multiple points in a batch), getgrad (compute gradient, if applicable), getprelinear (getting representation features), and many others.
Revised: The functions in analysismodel include forward (just making a prediction on a specific data point), forwardbatch (forward but on multiple points in a batch), getgrad (compute gradient, if applicable), getprelinear (getting representation features), and many others.

Original: This design allows the same analysis script to work on vastly different models, as long as the models are wrapped by a class that shares these functionalities.
Revised: This design allows the same analysis script to work on vastly different models, as long as the models are wrapped by a class that shares these functionalities.

Original: • Within the analysis module, we have scripts that can take in arbitrary data point and a model class (that inherits analysismodel ) and perform various analysis methods such as LIME, DIME, EMAP, Sparse Linear Model, etc.
Revised: • Within the analysis module, we have scripts that can take in arbitrary data point and a model class (that inherits analysismodel ) and perform various analysis methods such as LIME, DIME, EMAP, Sparse Linear Model, etc.

Original: These scripts generate the outputs in numerical format without visualizations, and users can choose to visualize them in arbitrary ways.
Revised: These scripts generate the outputs in numerical format without visualizations, and users can choose to visualize them in arbitrary ways.

Original: • Within the visualizations module, we have scripts that provide tools to visualize the analysis results from the analysis module.
Revised: • Within the visualizations module, we have scripts that provide tools to visualize the analysis results from the analysis module.

Original: In Algorithm 2, we showcase an example of running LIME, DIME, Sparse Linear Model and representation feature analysis (local and global), thus covering all stages of M ULTI V IZ .
Revised: In Algorithm 2, we showcase an example of running LIME, DIME, Sparse Linear Model and representation feature analysis (local and global), thus covering all stages of M ULTI V IZ .

Original: As you can see, the code is actually very short (without the comments) for running this many analysis and visualizations.
Revised: As you can see, the code is actually very short (without the comments) for running this many analysis and visualizations.

Original: Our code framework is also easily extendible to support new datasets, models and analysis/visualization methods, by writing and adding scripts to the datasets/models/analysis/visualizations modules respectively.
Revised: Our code framework is also easily extendible to support new datasets, models and analysis/visualization methods, by writing and adding scripts to the datasets/models/analysis/visualizations modules respectively.

Original: B. T HE M ULTI V IZ WEBSITE
Revised: B. T HE M ULTI V IZ WEBSITE

Original: We also created a visualization website accompanying M ULTI V IZ which organizes visualizations of all stages on a particular datapoint of specific dataset-model pairs.
Revised: We also created a visualization website accompanying M ULTI V IZ which organizes visualizations of all stages on a particular datapoint of specific dataset-model pairs.

Original: The URL link of the webpage is available at https://github.com/pliang279/MultiViz .
Revised: The URL link of the webpage is available at https://github.com/pliang279/MultiViz .

Original: Figure 8 is one example webpage for a data point in VQA.
Revised: Figure 8 is one example webpage for a data point in VQA.

Original: On the left there is a control panel that allows users to switch between different datasets and instances (i.e., data points), and then below the two boxes shows all information about the data point (image and question in the case of VQA) and also the ground truth ("GT") label and the predicted ("Pred") label.
Revised: On the left there is a control panel that allows users to switch between different datasets and instances (i.e., data points), and then below the two boxes shows all information about the data point (image and question in the case of VQA) and also the ground truth ("GT") label and the predicted ("Pred") label.

Original: On the right side, we have a graph showing a simplified version of the Sparse Linear Model: we only show the top 5 features with the
Revised: On the right side, we have a graph showing a simplified version of the Sparse Linear Model: we only show the top 5 features with the

Original: [Table]
Revised: [Table]

Original: (explanations
Revised: (explanations

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ,0,‘textdimeunimodal.png’)visualizelime(explanations[1],‘text’,
Revised: ,0,‘textdimeunimodal.png’)visualizelime(explanations[1],‘text’,

Original: [Table]
Revised: [Table]

Original: ,‘textdimemultimodal.png’)
Revised: ,‘textdimemultimodal.png’)

Original: # train
Revised: # train

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ,‘trainfeats
Revised: ,‘trainfeats

Original: [Table]
Revised: [Table]

Original: ,‘valfeats
Revised: ,‘valfeats

Original: [Table]
Revised: [Table]

Original: ,‘valfeats
Revised: ,‘valfeats

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ,‘sparseplot
Revised: ,‘sparseplot

Original: [Table] [Table]
Revised: [Table] [Table]

Original: and
Revised: and

Original: [Table]
Revised: [Table]

Original: getseqdata
Revised: getseqdata

Original: [Table]
Revised: [Table]

Original: # local analysis analyzepointandvisualizeall(params,instance,analysismodel,predlabel,‘tmp/local’,‘local’)#globalanalysis analyzefeaturesandvisualizeall(params,instance,sampledata,analysismodel,predlabel,‘tmp/global’,‘global’)
Revised: # local analysis analyzepointandvisualizeall(params,instance,analysismodel,predlabel,‘tmp/local’,‘local’)#globalanalysis analyzefeaturesandvisualizeall(params,instance,sampledata,analysismodel,predlabel,‘tmp/global’,‘global’)

Original: highest weights for each label (the weights are shown as numbers on the lines).
Revised: highest weights for each label (the weights are shown as numbers on the lines).

Original: Note that we will show both correct and predicted labels in the graph (so if the model got the answer wrong, there will be two labels shown under "classes" as shown in Figure 10, and clicking on each label will navigate to a webpage that shows visualizations with respect to that specific label).
Revised: Note that we will show both correct and predicted labels in the graph (so if the model got the answer wrong, there will be two labels shown under "classes" as shown in Figure 10, and clicking on each label will navigate to a webpage that shows visualizations with respect to that specific label).

Original: In the middle tab titled Main
Revised: In the middle tab titled Main

Original: View , we show the visualizations from U and C stages.
Revised: View , we show the visualizations from U and C stages.

Original: In the case of VQA we present unimodal
Revised: In the case of VQA we present unimodal

Original: LIME as U stage visualization (first column under Main View ) and DIME as C stage visualization (second and third column under Main View ).
Revised: LIME as U stage visualization (first column under Main View ) and DIME as C stage visualization (second and third column under Main View ).

Original: We call this webpage the Overview webpage.
Revised: We call this webpage the Overview webpage.

Original: For each of the top five representation features shown within the graph, the user can access R ℓ and R g visualizations of each feature by clicking on the circle in the graph representing that feature and the user will see a feature webpage like Figure 9.
Revised: For each of the top five representation features shown within the graph, the user can access R ℓ and R g visualizations of each feature by clicking on the circle in the graph representing that feature and the user will see a feature webpage like Figure 9.

Original: Under Main View , we include local analysis visualizations (unimodal lime with respect to the feature in the case of VQA) on the top and then global analysis visualizations on the bottom.
Revised: Under Main View , we include local analysis visualizations (unimodal lime with respect to the feature in the case of VQA) on the top and then global analysis visualizations on the bottom.

Original: To return to the Overview page, the user can just press the label circle under "classes" in the graph on the right again.
Revised: To return to the Overview page, the user can just press the label circle under "classes" in the graph on the right again.

Original: We also show additional example webpages: MM-IMDb (Figure 11 and Figure 12, with first order gradient for U stage, second order gradient for C stage), CMU-MOSEI (Figure 13 and Figure 14, with first order gradient for U stage, second order gradient for C stage) and MIMIC (Figure 15, with first order gradient for U stage).
Revised: We also show additional example webpages: MM-IMDb (Figure 11 and Figure 12, with first order gradient for U stage, second order gradient for C stage), CMU-MOSEI (Figure 13 and Figure 14, with first order gradient for U stage, second order gradient for C stage) and MIMIC (Figure 15, with first order gradient for U stage).

Original: Note that we only ran U stage for MIMIC LF model because its cross-modal interactions are negligible (second order gradients are all zero) and there are too few representation features to do sparse linear models.
Revised: Note that we only ran U stage for MIMIC LF model because its cross-modal interactions are negligible (second order gradients are all zero) and there are too few representation features to do sparse linear models.

Original: We have also used modified versions of these webpages to conduct all our experiments with human annotators.
Revised: We have also used modified versions of these webpages to conduct all our experiments with human annotators.

Original: See Appendix D for details.
Revised: See Appendix D for details.

Original: B. I NTERMEDIATE LAYER REPRESENTATION VISUALIZATION
Revised: B. I NTERMEDIATE LAYER REPRESENTATION VISUALIZATION

Original: Our codebase is designed such that the user may specify any layer in a model as the representation and run R ℓ and R g analysis on neurons in that layer.
Revised: Our codebase is designed such that the user may specify any layer in a model as the representation and run R ℓ and R g analysis on neurons in that layer.

Original: We showcase a few examples of R ℓ and R g on neurons on the third-last layer on LXMERT model on the VQA dataset in Figure 16.
Revised: We showcase a few examples of R ℓ and R g on neurons on the third-last layer on LXMERT model on the VQA dataset in Figure 16.

Original: The reason we choose to use the second last layer in the models is mostly for ease of visualization in the P stage as it will just be a linear composition.
Revised: The reason we choose to use the second last layer in the models is mostly for ease of visualization in the P stage as it will just be a linear composition.

Original: If we use a different layer as our representation, the
Revised: If we use a different layer as our representation, the

Original: P stage will contain multiple layers with different weights and more complex interactions, making it more difficult for a human user to visualize how each neuron in the representation related to the final prediction.
Revised: P stage will contain multiple layers with different weights and more complex interactions, making it more difficult for a human user to visualize how each neuron in the representation related to the final prediction.

Original: C D ATASETS AND M ODELS
Revised: C D ATASETS AND M ODELS

Original: All of our datasets build upon a diverse and standardized set of multimodal benchmarks in MultiBench (Liang et al., 2021a).
Revised: All of our datasets build upon a diverse and standardized set of multimodal benchmarks in MultiBench (Liang et al., 2021a).

Original: We briefly describe the datasets and preprocessing here:
Revised: We briefly describe the datasets and preprocessing here:

Original: C. D ATASETS IN M ULTI V IZ C.1. M ULTIMODAL FUSION
Revised: C. D ATASETS IN M ULTI V IZ C.1. M ULTIMODAL FUSION

Original: In multimodal fusion, the main challenge is to join information from two or more modalities to perform a prediction.
Revised: In multimodal fusion, the main challenge is to join information from two or more modalities to perform a prediction.

Original: Classic examples include audio-visual speech recognition, where visual lip motion is fused with speech signals to predict spoken words (Dupont and Luettin, 2000).
Revised: Classic examples include audio-visual speech recognition, where visual lip motion is fused with speech signals to predict spoken words (Dupont and Luettin, 2000).

Original: Information coming from different modalities have varying predictive power by themselves and also when complemented by each other (i.e., higher-order interactions).
Revised: Information coming from different modalities have varying predictive power by themselves and also when complemented by each other (i.e., higher-order interactions).

Original: In order to capture higher-order interactions, there is also a need to identify the relations between granular units from two or more different modalities (i.e., alignment).
Revised: In order to capture higher-order interactions, there is also a need to identify the relations between granular units from two or more different modalities (i.e., alignment).

Original: When dealing with temporal data, it also requires capturing possible long-range dependencies across time (i.e., temporal alignment).
Revised: When dealing with temporal data, it also requires capturing possible long-range dependencies across time (i.e., temporal alignment).

Original: M ULTI V IZ contains the following datasets for multimodal fusion spanning:
Revised: M ULTI V IZ contains the following datasets for multimodal fusion spanning:

Original: (1) CMU-MOSEI is the largest dataset of sentence-level sentiment analysis and emotion recognition in real-world online videos (Liang et al., 2018a;
Revised: (1) CMU-MOSEI is the largest dataset of sentence-level sentiment analysis and emotion recognition in real-world online videos (Liang et al., 2018a;

Original: Zadeh et al., 2018) with more than 65 hours of annotated video from more than 1 , 000 speakers and 250 topics.
Revised: Zadeh et al., 2018) with more than 65 hours of annotated video from more than 1 , 000 speakers and 250 topics.

Original: Each video is annotated for sentiment as well as the presence of 9 discrete emotions (angry, excited, fear, sad, surprised, frustrated, happy, disappointed, and neutral) as well as continuous emotions (valence, arousal, and dominance).
Revised: Each video is annotated for sentiment as well as the presence of 9 discrete emotions (angry, excited, fear, sad, surprised, frustrated, happy, disappointed, and neutral) as well as continuous emotions (valence, arousal, and dominance).

Original: The diversity of prediction tasks makes CMU-MOSEI a valuable dataset to test multimodal models across a range of real-world affective computing tasks.
Revised: The diversity of prediction tasks makes CMU-MOSEI a valuable dataset to test multimodal models across a range of real-world affective computing tasks.

Original: The dataset has been continuously used in workshops and competitions revolving around human multimodal language.
Revised: The dataset has been continuously used in workshops and competitions revolving around human multimodal language.

Original: Dataset
Revised: Dataset

Original: preprocessing
Revised: preprocessing

Original: : We follow current work (Liang et al., 2018b; Zadeh et al.,
Revised: : We follow current work (Liang et al., 2018b; Zadeh et al.,

Original: ) and apply standard preliminary feature extraction for the CMU-MOSEI dataset.
Revised: ) and apply standard preliminary feature extraction for the CMU-MOSEI dataset.

Original: Train, validation, and test splits : Each dataset contains several videos, and each video is further split into short segments (roughly 10 − 20 seconds) that are annotated.
Revised: Train, validation, and test splits : Each dataset contains several videos, and each video is further split into short segments (roughly 10 − 20 seconds) that are annotated.

Original: We split the data at the level of videos so that segments from the same video will not appear across train, valid, and test splits.
Revised: We split the data at the level of videos so that segments from the same video will not appear across train, valid, and test splits.

Original: This enables us to train user-independent models instead of having a model potentially memorizing the average affective state of a user.
Revised: This enables us to train user-independent models instead of having a model potentially memorizing the average affective state of a user.

Original: There are a total of 16 , 265 , 1 , 869 , and 4 , 643 segments in train, valid, and test datasets respectively for a total of 22 , 777 data points.
Revised: There are a total of 16 , 265 , 1 , 869 , and 4 , 643 segments in train, valid, and test datasets respectively for a total of 22 , 777 data points.

Original: (2) MM-IMD B is the largest publicly available multimodal dataset for genre prediction on movies (Arevalo et al., 2017).
Revised: (2) MM-IMD B is the largest publicly available multimodal dataset for genre prediction on movies (Arevalo et al., 2017).

Original: MM-IMD B starts from the movies of the MovieLens 20 M dataset and expands this dataset by collecting genre, poster, and plot information for each movie.
Revised: MM-IMD B starts from the movies of the MovieLens 20 M dataset and expands this dataset by collecting genre, poster, and plot information for each movie.

Original: The final dataset contains ratings for 25 , 959 movies.
Revised: The final dataset contains ratings for 25 , 959 movies.

Original: MM-IMD B is a realistic real-world multimodal dataset and is a popular benchmark for multimodal learning (Arevalo et al., 2017; Kiela et al., 2019; Pérez-Rúa et al., 2019).
Revised: MM-IMD B is a realistic real-world multimodal dataset and is a popular benchmark for multimodal learning (Arevalo et al., 2017; Kiela et al., 2019; Pérez-Rúa et al., 2019).

Original: Dataset preprocessing : We used the same method as (Arevalo et al., 2017) to extract features from texts and images.
Revised: Dataset preprocessing : We used the same method as (Arevalo et al., 2017) to extract features from texts and images.

Original: Train, validation, and test splits : The MM-IMDb dataset is split by genre into train, valid, and test datasets containing 15552 , 2608 , and 7799 .
Revised: Train, validation, and test splits : The MM-IMDb dataset is split by genre into train, valid, and test datasets containing 15552 , 2608 , and 7799 .

Original: The split was performed so that training, valid and test sets comprise 60% , 10% , 30% samples of each genre respectively.
Revised: The split was performed so that training, valid and test sets comprise 60% , 10% , 30% samples of each genre respectively.

Original: (3) MIMIC-III (Medical Information Mart for Intensive Care III) (Johnson et al., 2016) is a large, freely-available database comprising de-identified health-related data associated with over 40 ,patients who stayed in critical care units of the Beth Israel Deaconess Medical Center betweenand 2012 .
Revised: (3) MIMIC-III (Medical Information Mart for Intensive Care III) (Johnson et al., 2016) is a large, freely-available database comprising de-identified health-related data associated with over 40 ,patients who stayed in critical care units of the Beth Israel Deaconess Medical Center betweenand 2012 .

Original: Following (Purushotham et al., 2018), we organized numerous patient data into two major modalities (using the 17 features in feature set A in (Purushotham et al., 2018)): time series modality, which is a set of medical measurements of the patient taken every 1 hour in a period of 24 hours.
Revised: Following (Purushotham et al., 2018), we organized numerous patient data into two major modalities (using the 17 features in feature set A in (Purushotham et al., 2018)): time series modality, which is a set of medical measurements of the patient taken every 1 hour in a period of 24 hours.

Original: Each measurement is a vector of size 12 ( 12 different measured numerical values); static modality, which is a set of medical information about the patient, represented in a vector of size 5 .
Revised: Each measurement is a vector of size 12 ( 12 different measured numerical values); static modality, which is a set of medical information about the patient, represented in a vector of size 5 .

Original: We use these modalities for 3 tasks: mortality prediction ( 6 -class prediction on whether the patient dies in 1 day,day, 3 day, 1 week, 1 year, or longer than 1 year), and 2 ICD- 9 code predictions (binary classification on whether the patient fits any ICD- 9 code in group 1 ( 140 − 239 ) and binary classification on whether the patient fits any ICD- 9 code in group 7 460 − 519 ).
Revised: We use these modalities for 3 tasks: mortality prediction ( 6 -class prediction on whether the patient dies in 1 day,day, 3 day, 1 week, 1 year, or longer than 1 year), and 2 ICD- 9 code predictions (binary classification on whether the patient fits any ICD- 9 code in group 1 ( 140 − 239 ) and binary classification on whether the patient fits any ICD- 9 code in group 7 460 − 519 ).

Original: MIMIC poses unique challenges in integrating time-varying and static modalities, reinforcing the need of aligning multimodal information at correct granularity.
Revised: MIMIC poses unique challenges in integrating time-varying and static modalities, reinforcing the need of aligning multimodal information at correct granularity.

Original: Dataset preprocessing : We followed the instructions on https://mimic.physionet.
Revised: Dataset preprocessing : We followed the instructions on https://mimic.physionet.

Original: org/gettingstarted/access/ to download the dataset in the form of raw tables, then generated preprocessed data following the steps described in https://github.
Revised: org/gettingstarted/access/ to download the dataset in the form of raw tables, then generated preprocessed data following the steps described in https://github.

Original: com/USC-Melady/Benchmarking_DL_MIMICIII (which takes 1 − 2 weeks running time) to get the data used for experiments.
Revised: com/USC-Melady/Benchmarking_DL_MIMICIII (which takes 1 − 2 weeks running time) to get the data used for experiments.

Original: Specifically, we will use data in the file 24hrs/series/imputed-normed-ep_1_24-stdized.npz .
Revised: Specifically, we will use data in the file 24hrs/series/imputed-normed-ep_1_24-stdized.npz .

Original: When accessing this data from our code repo, set the imputed_path of the npz file above in the get_data.py and the script will generate the PyTorch data loader for the tasks (where we will normalize the data).
Revised: When accessing this data from our code repo, set the imputed_path of the npz file above in the get_data.py and the script will generate the PyTorch data loader for the tasks (where we will normalize the data).

Original: Train, validation, and test splits : We split the data into train/valid/test sets randomly (using a fixed random seed) in a 80 : 10 : 10 ratio (so 28 , 970 train, 3 , 621 valid, and 3 , 621 test data points) for a total of 36 , 212 data points.
Revised: Train, validation, and test splits : We split the data into train/valid/test sets randomly (using a fixed random seed) in a 80 : 10 : 10 ratio (so 28 , 970 train, 3 , 621 valid, and 3 , 621 test data points) for a total of 36 , 212 data points.

Original: C.1. M ULTIMODAL RETRIEVAL
Revised: C.1. M ULTIMODAL RETRIEVAL

Original: Another area of great interest lies in cross-modal retrieval (Liang et al., 2021b;
Revised: Another area of great interest lies in cross-modal retrieval (Liang et al., 2021b;

Original: Zhen et al., 2019), where the goal is to retrieve semantically similar data from a new modality using a modality as a query (e.g., given a phrase, retrieve the closest image describing that phrase).
Revised: Zhen et al., 2019), where the goal is to retrieve semantically similar data from a new modality using a modality as a query (e.g., given a phrase, retrieve the closest image describing that phrase).

Original: The core challenge is to perform alignment of representations across both modalities.
Revised: The core challenge is to perform alignment of representations across both modalities.

Original: M ULTI V IZ contains the following datasets for multimodal retrieval and grounding:
Revised: M ULTI V IZ contains the following datasets for multimodal retrieval and grounding:

Original: (1) F LICKR -30 K (Plummer et al., 2015) contains 32 , 000 images collected from Flickr, together with 5 reference sentences provided by human annotators enabling the tasks of text-to-image reference resolution, localizing textual entity mentions in an image, and bidirectional image-caption retrieval.
Revised: (1) F LICKR -30 K (Plummer et al., 2015) contains 32 , 000 images collected from Flickr, together with 5 reference sentences provided by human annotators enabling the tasks of text-to-image reference resolution, localizing textual entity mentions in an image, and bidirectional image-caption retrieval.

Original: Train, validation, and test splits : The training items are generated from the captions of 25 ,images, and the test items are generated from a disjoint set of 3 , 000 images.
Revised: Train, validation, and test splits : The training items are generated from the captions of 25 ,images, and the test items are generated from a disjoint set of 3 , 000 images.

Original: C.1. M ULTIMODAL QUESTION ANSWERING
Revised: C.1. M ULTIMODAL QUESTION ANSWERING

Original: Within the domain of language and vision, there has been growing interest in language-based question answering (i.e., “query” modality) of entities in the visual, video, or embodied domain (i.e., “queried” modality).
Revised: Within the domain of language and vision, there has been growing interest in language-based question answering (i.e., “query” modality) of entities in the visual, video, or embodied domain (i.e., “queried” modality).

Original: Datasets such as Visual Question Answering (Agrawal et al., 2017), Social IQ (Zadeh et al., 2019), and Embodied Question Answering (Das et al., 2018) have been proposed to benchmark the performance of multimodal models in these settings.
Revised: Datasets such as Visual Question Answering (Agrawal et al., 2017), Social IQ (Zadeh et al., 2019), and Embodied Question Answering (Das et al., 2018) have been proposed to benchmark the performance of multimodal models in these settings.

Original: A core challenge lies in aligning words asked in the question with entities in the queried modalities, which typically take the form of visual entities in images or videos (i.e., alignment).
Revised: A core challenge lies in aligning words asked in the question with entities in the queried modalities, which typically take the form of visual entities in images or videos (i.e., alignment).

Original: M ULTI V IZ contains the following datasets for multimodal question answering spanning several research areas:
Revised: M ULTI V IZ contains the following datasets for multimodal question answering spanning several research areas:

Original: (1) CLEVR (Johnson et al., 2017) is a diagnostic dataset for studying the ability of VQA systems to perform visual reasoning.
Revised: (1) CLEVR (Johnson et al., 2017) is a diagnostic dataset for studying the ability of VQA systems to perform visual reasoning.

Original: It contains 100 , 000 rendered images and about 853 , 000 unique automatically generated questions that test visual reasoning abilities such as counting, comparing, logical reasoning, and storing information in memory.
Revised: It contains 100 , 000 rendered images and about 853 , 000 unique automatically generated questions that test visual reasoning abilities such as counting, comparing, logical reasoning, and storing information in memory.

Original: Train, validation, and test splits :
Revised: Train, validation, and test splits :

Original: The complete dataset contains more than 608 K train, 140 K val and 140 K test (question, image) pairs.
Revised: The complete dataset contains more than 608 K train, 140 K val and 140 K test (question, image) pairs.

Original: (2) VQA 2.0 (Goyal et al., 2017) is a balanced version of the popular VQA (Agrawal et al., 2017) dataset by collecting complementary images such that every question is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question.
Revised: (2) VQA 2.0 (Goyal et al., 2017) is a balanced version of the popular VQA (Agrawal et al., 2017) dataset by collecting complementary images such that every question is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question.

Original: The reduces the occurrence of spurious correlations
Revised: The reduces the occurrence of spurious correlations

Original: in
Revised: in

Original: the dataset and
Revised: the dataset and

Original: enables training of
Revised: enables training of

Original: more robust
Revised: more robust

Original: models
Revised: models

Original: Train, validation
Revised: Train, validation

Original: and test splits :
Revised: and test splits :

Original: The complete balanced dataset contains more than 443 K train, 214 K val, and 453 K test (question, image) pairs.
Revised: The complete balanced dataset contains more than 443 K train, 214 K val, and 453 K test (question, image) pairs.

Original: [Figure]
Revised: [Figure]

Original: high
Revised: high

Original: [Figure]
Revised: [Figure]

Original: MultiViz: Interpreting Internal Mechanics
Revised: MultiViz: Interpreting Internal Mechanics

Original: [Figure]
Revised: [Figure]

Original: D A DDITIONAL E XPERIMENTS AND D ETAILS
Revised: D A DDITIONAL E XPERIMENTS AND D ETAILS

Original: In this section, we provide additional details on the experiments and additional results on several other multimodal datasets.
Revised: In this section, we provide additional details on the experiments and additional results on several other multimodal datasets.

Original: Computational resources : Preparations for all experiments (i.e. generating the necessary visualizations for the points for each dataset) are done on a private server with 2 GPUs.
Revised: Computational resources : Preparations for all experiments (i.e. generating the necessary visualizations for the points for each dataset) are done on a private server with 2 GPUs.

Original: The preparation time for model simulation experiment using 2 GPUs is about 12 hours for VQA,hour for MM-IMDb and 2 hours for CMU-MOSEI.
Revised: The preparation time for model simulation experiment using 2 GPUs is about 12 hours for VQA,hour for MM-IMDb and 2 hours for CMU-MOSEI.

Original: For the representation interpretation experiment, we generated all visualizations for the VQA data points in the experiment in about 3 hours on 1 GPU.
Revised: For the representation interpretation experiment, we generated all visualizations for the VQA data points in the experiment in about 3 hours on 1 GPU.

Original: For the error analysis, in addition to the visualizations already present on the MultiViz website, we also have 1 GPU available live during the human annotation (so human annotators can request second order gradient analysis on specific words, each second order gradient computation only takes 2seconds).
Revised: For the error analysis, in addition to the visualizations already present on the MultiViz website, we also have 1 GPU available live during the human annotation (so human annotators can request second order gradient analysis on specific words, each second order gradient computation only takes 2seconds).

Original: In all of the above analysis, we re-use the sparse linear model we had already trained for each dataset when building the main MultiViz webpage (the initial training can take some time - scaling the Sparse
Revised: In all of the above analysis, we re-use the sparse linear model we had already trained for each dataset when building the main MultiViz webpage (the initial training can take some time - scaling the Sparse

Original: Linear Model to the large VQA took over 72 hours with 1 GPU).
Revised: Linear Model to the large VQA took over 72 hours with 1 GPU).

Original: Note that VQA visualization generation is much slower than those for MM-IMDb and CMU-MOSEI.
Revised: Note that VQA visualization generation is much slower than those for MM-IMDb and CMU-MOSEI.

Original: This is because in VQA we used DIME for cross-modal interaction interpretation, but in MM-IMDb and CMU-MOSEI we use second-order gradient.
Revised: This is because in VQA we used DIME for cross-modal interaction interpretation, but in MM-IMDb and CMU-MOSEI we use second-order gradient.

Original: The newly proposed second order gradient is much faster compared to DIME since it only requires running the model once instead of
Revised: The newly proposed second order gradient is much faster compared to DIME since it only requires running the model once instead of

Original: up to 10,000 times in DIME.
Revised: up to 10,000 times in DIME.

Original: Overall, the proposed M ULTI V IZ interpretation stages are efficient and only add negligible time on top of existing trained models, especially for our newly-proposed second-order gradient method.
Revised: Overall, the proposed M ULTI V IZ interpretation stages are efficient and only add negligible time on top of existing trained models, especially for our newly-proposed second-order gradient method.

Original: Participant risks and compensation : Participation in these human studies were fully voluntary and without compensation.
Revised: Participant risks and compensation : Participation in these human studies were fully voluntary and without compensation.

Original: There are no participant risks involved.
Revised: There are no participant risks involved.

Original: We obtained consent from all participants prior to each short study.
Revised: We obtained consent from all participants prior to each short study.

Original: All annotations are fully anonymous and we do not store any information regarding the participants at all.
Revised: All annotations are fully anonymous and we do not store any information regarding the participants at all.

Original: D. M ODEL SIMULATION D.1. S ETUP
Revised: D. M ODEL SIMULATION D.1. S ETUP

Original: We design a large-scale use case of model simulation to determine if M ULTI V IZ helps users of multimodal models gain a deeper understanding of model behavior, as shown in Figure 17.
Revised: We design a large-scale use case of model simulation to determine if M ULTI V IZ helps users of multimodal models gain a deeper understanding of model behavior, as shown in Figure 17.

Original: We design a human study to see what humans predict given M ULTI V IZ explanations at each step (and across all steps).
Revised: We design a human study to see what humans predict given M ULTI V IZ explanations at each step (and across all steps).

Original: If M ULTI V IZ indeed generates human-understandable explanations, humans should be able to make a prediction on the task given these explanations only.
Revised: If M ULTI V IZ indeed generates human-understandable explanations, humans should be able to make a prediction on the task given these explanations only.

Original: Specifically, we compare the full version of M ULTI V IZ with a set of local ablations, each consisting of only 1 additional stage:
Revised: Specifically, we compare the full version of M ULTI V IZ with a set of local ablations, each consisting of only 1 additional stage:

Original: U :
Revised: U :

Original: Users are only shown the unimodal importance (U) of each modality towards the prediction.
Revised: Users are only shown the unimodal importance (U) of each modality towards the prediction.

Original: U + C : Users are shown both unimodal importance (U) and cross-modal interactions (C) highlighted towards the final prediction.
Revised: U + C : Users are shown both unimodal importance (U) and cross-modal interactions (C) highlighted towards the final prediction.

Original: U + C + R ℓ : Users are shown unimodal importance (U) and cross-modal interactions (C) of the given datapoint highlighted towards the final prediction, as well as local analysis (R ℓ ) of unimodal
Revised: U + C + R ℓ : Users are shown unimodal importance (U) and cross-modal interactions (C) of the given datapoint highlighted towards the final prediction, as well as local analysis (R ℓ ) of unimodal

Original: and cross-modal interactions of top ranked feature representations z top =
Revised: and cross-modal interactions of top ranked feature representations z top =

Original: { z (1) , z (2) , ... } with respect to that local datapoint.
Revised: { z (1) , z (2) , ... } with respect to that local datapoint.

Original: U + C + R ℓ + R g : Users are additionally shown global analysis (R g ) through similar datapoints that also maximally activate those same top ranked feature representations.
Revised: U + C + R ℓ + R g : Users are additionally shown global analysis (R g ) through similar datapoints that also maximally activate those same top ranked feature representations.

Original: M ULTI V IZ
Revised: M ULTI V IZ

Original: (U + C + R ℓ + R g + P) :
Revised: (U + C + R ℓ + R g + P) :

Original: This constitutes the entire M ULTI V IZ framework by including visualizations of the final prediction (P) stage: sorting all top ranked feature neurons z top =
Revised: This constitutes the entire M ULTI V IZ framework by including visualizations of the final prediction (P) stage: sorting all top ranked feature neurons z top =

Original: { z (1) , z (2) , ... } with respect to their coefficients β top = { β (1) , β (2) , ... } and showing these coefficients to the user.
Revised: { z (1) , z (2) , ... } with respect to their coefficients β top = { β (1) , β (2) , ... } and showing these coefficients to the user.

Original: We ask human annotators (who all have or are currently working towards a B.S. in a STEM field and have at least basic knowledge of machine learning models) to predict the output of a model analysis results and visualizations.
Revised: We ask human annotators (who all have or are currently working towards a B.S. in a STEM field and have at least basic knowledge of machine learning models) to predict the output of a model analysis results and visualizations.

Original: In each of the following datasets (VQA 2.0, MM-IMDb, CMU-MOSEI), we divide 15 total human annotators into 5 groups of 3, each group getting one of the five settings above, and then we compute average accuracy and inter-rater agreement within each group.
Revised: In each of the following datasets (VQA 2.0, MM-IMDb, CMU-MOSEI), we divide 15 total human annotators into 5 groups of 3, each group getting one of the five settings above, and then we compute average accuracy and inter-rater agreement within each group.

Original: The full results are shown in Table 2.
Revised: The full results are shown in Table 2.

Original: D.1. VQA 2. 
Revised: D.1. VQA 2.

Original: In this experiment, we will perform model simulation on VQA 2.0 dataset with pretrained LXMERT ( https://huggingface.co/unc-nlp/lxmert-vqa-uncased ).
Revised: In this experiment, we will perform model simulation on VQA 2.0 dataset with pretrained LXMERT ( https://huggingface.co/unc-nlp/lxmert-vqa-uncased ).

Original: We randomly selected 22 points from the validation split of the VQA dataset under the following criterion: (1) it is not a yes/no question and (2) the answer to the question is not infrequent (i.e. it occurs at least 220 times over 220K+ validation points).
Revised: We randomly selected 22 points from the validation split of the VQA dataset under the following criterion: (1) it is not a yes/no question and (2) the answer to the question is not infrequent (i.e. it occurs at least 220 times over 220K+ validation points).

Original: For each of the point, we run M ULTI V IZ analysis and visualization: for U stage we run LIME on each modality; for C stage we run DIME; for R ℓ we run LIME with respect to the representation feature on this data point; and for R g we run LIME on each modality with respect to the representation feature on 3 examples that maximally activates the feature; and for
Revised: For each of the point, we run M ULTI V IZ analysis and visualization: for U stage we run LIME on each modality; for C stage we run DIME; for R ℓ we run LIME with respect to the representation feature on this data point; and for R g we run LIME on each modality with respect to the representation feature on 3 examples that maximally activates the feature; and for

Original: P we show the top 5 representation features with the highest weights with respect to the predicted class in a Sparse Linear Model trained on the training set of VQA.
Revised: P we show the top 5 representation features with the highest weights with respect to the predicted class in a Sparse Linear Model trained on the training set of VQA.

Original: The webpage for each datapoint is organized into Overview page (containing U and C ) as well as five Features page ( R ℓ and R g for each of the top 5 representation features) as well as a "graph" on the right showing P .
Revised: The webpage for each datapoint is organized into Overview page (containing U and C ) as well as five Features page ( R ℓ and R g for each of the top 5 representation features) as well as a "graph" on the right showing P .

Original: An example
Revised: An example

Original: Overview page is shown in Figure 18 and an example Features page is shown in Figure 19.
Revised: Overview page is shown in Figure 18 and an example Features page is shown in Figure 19.

Original: In settings (1)-(4), we will use versions of the webpage with certain stages removed (for example,
Revised: In settings (1)-(4), we will use versions of the webpage with certain stages removed (for example,

Original: Figure 20 is the webpage for setting (2), only showing U and C ).
Revised: Figure 20 is the webpage for setting (2), only showing U and C ).

Original: Within each of the five groups, on each of the 22 points, human annotators are asked to predict what the model (LXMERT) predicts given a website containing some or all of the stages of analysis visualizations (depending on the group’s setting).
Revised: Within each of the five groups, on each of the 22 points, human annotators are asked to predict what the model (LXMERT) predicts given a website containing some or all of the stages of analysis visualizations (depending on the group’s setting).

Original: In addition, they are given an answer sheet (see
Revised: In addition, they are given an answer sheet (see

Original: Figure 21) where they are given 4 answer choices for each data point to predict with, and they have to select one of the choices they think LXMERT most likely predicted as the answer to each data point.
Revised: Figure 21) where they are given 4 answer choices for each data point to predict with, and they have to select one of the choices they think LXMERT most likely predicted as the answer to each data point.

Original: Before each annotator starts, they are taught how to interpret each analysis visualization, and then the instructor goes over 2 points together with the annotators as examples and the annotators need to finish the remaining 20 points on their own.
Revised: Before each annotator starts, they are taught how to interpret each analysis visualization, and then the instructor goes over 2 points together with the annotators as examples and the annotators need to finish the remaining 20 points on their own.

Original: Only the remaining 20 points counts towards the data collected in the experiment.
Revised: Only the remaining 20 points counts towards the data collected in the experiment.

Original: We then compute average accuracy and inter-rater agreement score (Krippendorff’s alpha) within each group.
Revised: We then compute average accuracy and inter-rater agreement score (Krippendorff’s alpha) within each group.

Original: In addition, groups under settings (3), (4) and (5) are asked whether they found the Overview or Features page more helpful.
Revised: In addition, groups under settings (3), (4) and (5) are asked whether they found the Overview or Features page more helpful.

Original: As shown in Table 2, in general, human annotators were able to better predict the model’s predictions when they were given more information, as the groups that got more information almost always end up with both higher average accuracy and higher inter-rater agreement.
Revised: As shown in Table 2, in general, human annotators were able to better predict the model’s predictions when they were given more information, as the groups that got more information almost always end up with both higher average accuracy and higher inter-rater agreement.

Original: Moreover, annotators in settings (3), (4), (5) reported that they found Features page most helpful compared to Overview page 31.7%, 61.7% and 80.0% of the time respectively, therefore showing that R g and P helps make representation analysis a lot more useful.
Revised: Moreover, annotators in settings (3), (4), (5) reported that they found Features page most helpful compared to Overview page 31.7%, 61.7% and 80.0% of the time respectively, therefore showing that R g and P helps make representation analysis a lot more useful.

Original: D.1. MM-IMD B
Revised: D.1. MM-IMD B

Original: In this experiment, we perform model simulation on MM-IMDb dataset with the LRTF model from
Revised: In this experiment, we perform model simulation on MM-IMDb dataset with the LRTF model from

Original: MultiBench (Liang et al., 2021a).
Revised: MultiBench (Liang et al., 2021a).

Original: We randomly selected 21 points from the test split of MM-IMDb dataset.
Revised: We randomly selected 21 points from the test split of MM-IMDb dataset.

Original: The original MM-IMDb dataset is designed for multi-label classification, but for simplicity, we only take the label with the highest prediction probability from LRTF as the predicted class, and effectively treat it as a single-label classification task during analysis, visualization and model simulation experiment.
Revised: The original MM-IMDb dataset is designed for multi-label classification, but for simplicity, we only take the label with the highest prediction probability from LRTF as the predicted class, and effectively treat it as a single-label classification task during analysis, visualization and model simulation experiment.

Original: For each of the points, we run M ULTI V IZ analysis and visualization: for
Revised: For each of the points, we run M ULTI V IZ analysis and visualization: for

Original: U stage we show first order gradient analysis on image and text; for C stage we perform second order gradient analysis on the top ten words with maximum first order gradient; for R ℓ we show first order gradient on image and text with respect to each representation feature; for R g , on each representation feature we present 3 data points that maximally activates the feature, and also show first order gradient visualization for each; for P stage we show the "graph" on the right that ranks the top 5 representation features from Sparse Linear Model analysis as well as their respective weights.
Revised: U stage we show first order gradient analysis on image and text; for C stage we perform second order gradient analysis on the top ten words with maximum first order gradient; for R ℓ we show first order gradient on image and text with respect to each representation feature; for R g , on each representation feature we present 3 data points that maximally activates the feature, and also show first order gradient visualization for each; for P stage we show the "graph" on the right that ranks the top 5 representation features from Sparse Linear Model analysis as well as their respective weights.

Original: The webpage organization is the same as the webpage for VQA with the Overview page (Figure 22) and Features pages (Figure 23).
Revised: The webpage organization is the same as the webpage for VQA with the Overview page (Figure 22) and Features pages (Figure 23).

Original: Within each of the five groups, on each of the 21 points, human annotators are asked to predict what the model (LRTF) predicts given a website containing some or all of the stages of analysis visualizations (depending on the group’s setting).
Revised: Within each of the five groups, on each of the 21 points, human annotators are asked to predict what the model (LRTF) predicts given a website containing some or all of the stages of analysis visualizations (depending on the group’s setting).

Original: In addition, we give human annotators 10 possible movie classes that the model could predict for these 21 points ("Drama/Romance", "Crime", "Sci-Fi", "Comedy", "Thriller", "Western", "Action", "War", "Documentary", "Horror").
Revised: In addition, we give human annotators 10 possible movie classes that the model could predict for these 21 points ("Drama/Romance", "Crime", "Sci-Fi", "Comedy", "Thriller", "Western", "Action", "War", "Documentary", "Horror").

Original: Note that in reality, some of these categories are not mutually exclusive, but we intentionally designed our experiment this way to see if human annotators were able to determine the model’s prediction by looking at what specific properties within the movie’s poster or description the model focused on during the prediction process.
Revised: Note that in reality, some of these categories are not mutually exclusive, but we intentionally designed our experiment this way to see if human annotators were able to determine the model’s prediction by looking at what specific properties within the movie’s poster or description the model focused on during the prediction process.

Original: Before each human annotator starts, they are taught how to interpret each analysis visualization, and then the instructor goes over the first point together with the annotator as example and the annotator need to finish the remaining 20 points on their own.
Revised: Before each human annotator starts, they are taught how to interpret each analysis visualization, and then the instructor goes over the first point together with the annotator as example and the annotator need to finish the remaining 20 points on their own.

Original: Only the remaining 20 points counts towards the data collected in the experiment.
Revised: Only the remaining 20 points counts towards the data collected in the experiment.

Original: We then compute average accuracy and inter-rater agreement score (Krippendorff’s alpha) within each group.
Revised: We then compute average accuracy and inter-rater agreement score (Krippendorff’s alpha) within each group.

Original: As shown in Table 2, in general, human annotators were able to better predict the model’s predictions when they were given more information, as the groups that got more information almost always end up with both higher average accuracy as well as higher inter-rater agreement.
Revised: As shown in Table 2, in general, human annotators were able to better predict the model’s predictions when they were given more information, as the groups that got more information almost always end up with both higher average accuracy as well as higher inter-rater agreement.

Original: We were especially surprised to find that including C stage actually helped, since MM-IMDb did not seem to be a task that relies much on cross-modal interaction.
Revised: We were especially surprised to find that including C stage actually helped, since MM-IMDb did not seem to be a task that relies much on cross-modal interaction.

Original: D.1. CMU-MOSEI
Revised: D.1. CMU-MOSEI

Original: In this experiment, we perform model simulation on CMU-MOSEI dataset with the MulT model from
Revised: In this experiment, we perform model simulation on CMU-MOSEI dataset with the MulT model from

Original: MultiBench (Liang et al., 2021a).
Revised: MultiBench (Liang et al., 2021a).

Original: We randomly selected 20 points from the test split of CMU-MOSEI dataset.
Revised: We randomly selected 20 points from the test split of CMU-MOSEI dataset.

Original: The original CMU-MOSEI dataset is designed for a 7-way sentiment classification (-3 to +3), but we follow the preprocessing in MultiBench and convert it into a binary classification problem
Revised: The original CMU-MOSEI dataset is designed for a 7-way sentiment classification (-3 to +3), but we follow the preprocessing in MultiBench and convert it into a binary classification problem

Original: [Table]
Revised: [Table]

Original: Within each of the five groups, on each of the 20 points, human annotators are asked to predict what the model (MulT) predicts given a website containing some or all of the stages of analysis visualizations (depending on the group’s setting).
Revised: Within each of the five groups, on each of the 20 points, human annotators are asked to predict what the model (MulT) predicts given a website containing some or all of the stages of analysis visualizations (depending on the group’s setting).

Original: Before each human annotator starts, they are taught how to interpret each analysis visualization, and the annotator needs to finish the 20 points on their own.
Revised: Before each human annotator starts, they are taught how to interpret each analysis visualization, and the annotator needs to finish the 20 points on their own.

Original: We then compute average accuracy and inter-rater agreement score (Krippendorff’s alpha) within each group.
Revised: We then compute average accuracy and inter-rater agreement score (Krippendorff’s alpha) within each group.

Original: As shown in Table 2, in general, human annotators were able to better predict the model’s predictions when they were given more information, as the groups that got more information almost always end up with both higher average accuracy and higher inter-rater agreement.
Revised: As shown in Table 2, in general, human annotators were able to better predict the model’s predictions when they were given more information, as the groups that got more information almost always end up with both higher average accuracy and higher inter-rater agreement.

Original: Moreover, human annotators were able to get perfect accuracy and agreement in settings (4) and (5), showing that including global analysis R g provides enough information to simulate model predictions.
Revised: Moreover, human annotators were able to get perfect accuracy and agreement in settings (4) and (5), showing that including global analysis R g provides enough information to simulate model predictions.

Original: D. C ROSS - MODAL INTERACTIONS
Revised: D. C ROSS - MODAL INTERACTIONS

Original: In order to verify local faithfulness of interpreting cross-modal interactions, we take a closer look at the qualitative and quantitative performance of our proposed second-order gradient method.
Revised: In order to verify local faithfulness of interpreting cross-modal interactions, we take a closer look at the qualitative and quantitative performance of our proposed second-order gradient method.

Original: D.2. CLEVR
Revised: D.2. CLEVR

Original: One gold standard for evaluating visualizations of cross-modal interactions involves using
Revised: One gold standard for evaluating visualizations of cross-modal interactions involves using

Original: CLEVR (Johnson et al., 2017) (for image question answering), because in this dataset we are given ground truth bounding boxes of each object and there are often cross-modal alignments that are obvious and without any controversy.
Revised: CLEVR (Johnson et al., 2017) (for image question answering), because in this dataset we are given ground truth bounding boxes of each object and there are often cross-modal alignments that are obvious and without any controversy.

Original: We picked two representative models: MDETR (Kamath et al., 2021), which is near-perfect (with 99.5% accuracy); and CNN+LSTM+SA (Johnson et al., 2017), which was the best model amongst the baselines included in the paper that introduced CLEVR dataset (Johnson et al., 2017).
Revised: We picked two representative models: MDETR (Kamath et al., 2021), which is near-perfect (with 99.5% accuracy); and CNN+LSTM+SA (Johnson et al., 2017), which was the best model amongst the baselines included in the paper that introduced CLEVR dataset (Johnson et al., 2017).

Original: We randomly selected 52 ground-truth alignment pairs, all of which aligns between a phrase in the question (1-4 words) and the one single object in the image.
Revised: We randomly selected 52 ground-truth alignment pairs, all of which aligns between a phrase in the question (1-4 words) and the one single object in the image.

Original: Then, for each pair, we compute the first-order gradient of each word with respect to the sum of all entries in the prediction logit vector, sum up the absolute gradients of the words in the phrase, before taking the gradient of each pixel with respect to the sum.
Revised: Then, for each pair, we compute the first-order gradient of each word with respect to the sum of all entries in the prediction logit vector, sum up the absolute gradients of the words in the phrase, before taking the gradient of each pixel with respect to the sum.

Original: We end up with a second-order gradient (SOG) on each pixel.
Revised: We end up with a second-order gradient (SOG) on each pixel.

Original: Then, we then compute the average absolute SOG per pixel within bounding boxes of each object (given by CLEVR).
Revised: Then, we then compute the average absolute SOG per pixel within bounding boxes of each object (given by CLEVR).

Original: We compute 2 metrics: alignment picked up by top 1 bounding box (how often does the aligned object match with the bounding box with the highest average SOG) and alignment picked up by top 2 bounding box (how often does the aligned object match with one of the bounding boxes with top 2 highest average SOG).
Revised: We compute 2 metrics: alignment picked up by top 1 bounding box (how often does the aligned object match with the bounding box with the highest average SOG) and alignment picked up by top 2 bounding box (how often does the aligned object match with one of the bounding boxes with top 2 highest average SOG).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We show the results in Table 5.
Revised: We show the results in Table 5.

Original: We found that under near-perfect setting (MDETR) where it is safe to assume that the model actually picks up all ground-truth alignments, our method was able to pick up over 80% of the alignments using top-2 bounding boxes, thus indicating that our method is quite faithful to the model’s actual prediction process.
Revised: We found that under near-perfect setting (MDETR) where it is safe to assume that the model actually picks up all ground-truth alignments, our method was able to pick up over 80% of the alignments using top-2 bounding boxes, thus indicating that our method is quite faithful to the model’s actual prediction process.

Original: Moreover, we found that CNN+LSTM+SA, which is a relatively simple late fusion model with relatively poor performance, was much less likely to pick up the correct alignments according to our method, which makes sense.
Revised: Moreover, we found that CNN+LSTM+SA, which is a relatively simple late fusion model with relatively poor performance, was much less likely to pick up the correct alignments according to our method, which makes sense.

Original: Below, we show examples of when the model picks up or is unable to pick up the ground-truth alignments in Figure 26.
Revised: Below, we show examples of when the model picks up or is unable to pick up the ground-truth alignments in Figure 26.

Original: D.2. F LICKR -30 K
Revised: D.2. F LICKR -30 K

Original: In addition, we perform a similar experiment for Flickr-30k image-text retrieval by modifying the above approach slightly.
Revised: In addition, we perform a similar experiment for Flickr-30k image-text retrieval by modifying the above approach slightly.

Original: We select 20 image-text pairs from the annotated dataset, and for each of them we take between 8-15 phrases to find the second-order-gradient (SOG) on each pixel.
Revised: We select 20 image-text pairs from the annotated dataset, and for each of them we take between 8-15 phrases to find the second-order-gradient (SOG) on each pixel.

Original: We take the ground-truth boxes from Flickr30k Entities (Plummer et al., 2015) and calculate the average SOG for a given object per pixel across all the available boxes for the object.
Revised: We take the ground-truth boxes from Flickr30k Entities (Plummer et al., 2015) and calculate the average SOG for a given object per pixel across all the available boxes for the object.

Original: Additionally, we match the phrase against ground-truth phrase annotations to find relevant boxes.
Revised: Additionally, we match the phrase against ground-truth phrase annotations to find relevant boxes.

Original: Finally, we calculate what percentage of the objects were recovered by double gradient from the ground-truth annotations, if any.
Revised: Finally, we calculate what percentage of the objects were recovered by double gradient from the ground-truth annotations, if any.

Original: For ViLT (Kim et al., 2021) model, we observe that second-order gradient is able to do so with 44% matching accuracy, as compared to 34% using random matching (see some examples of detected interactions in Figure 27).
Revised: For ViLT (Kim et al., 2021) model, we observe that second-order gradient is able to do so with 44% matching accuracy, as compared to 34% using random matching (see some examples of detected interactions in Figure 27).

Original: For CLIP, the matching performance is worse (35%) as the gradients are very scattered across examples, making it hard to localize one particular object (see some examples of detected interactions in Figure 28).
Revised: For CLIP, the matching performance is worse (35%) as the gradients are very scattered across examples, making it hard to localize one particular object (see some examples of detected interactions in Figure 28).

Original: Both these findings indicate potential future directions towards quantifying intermediate cross-modal interactions learned by a model beyond looking at final task performance.
Revised: Both these findings indicate potential future directions towards quantifying intermediate cross-modal interactions learned by a model beyond looking at final task performance.

Original: [Figure]
Revised: [Figure]

Original: with
Revised: with

Original: [Figure]
Revised: [Figure]

Original: and face covering uses a small knife to sculpt a piece of ice.
Revised: and face covering uses a small knife to sculpt a piece of ice.

Original: A man standing on a street with a suitcase in frontofhimwhile anotherman bendsdowntolookat whatisdisplayed on top of it.
Revised: A man standing on a street with a suitcase in frontofhimwhile anotherman bendsdowntolookat whatisdisplayed on top of it.

Original: D. R EPRESENTATION INTERPRETATION
Revised: D. R EPRESENTATION INTERPRETATION

Original: We now take a deeper look to check that M ULTI V IZ generates accurate explanations of multimodal representations.
Revised: We now take a deeper look to check that M ULTI V IZ generates accurate explanations of multimodal representations.

Original: Using local and global representation visualizations, can humans consistently assign interpretable concepts in natural language to previously uninterpretable features?
Revised: Using local and global representation visualizations, can humans consistently assign interpretable concepts in natural language to previously uninterpretable features?

Original: D.3. VQA 2. 
Revised: D.3. VQA 2.

Original: For VQA 2.0 dataset, we perform a representation interpretation experiment, where we give human annotators some visualizations on a particular representation feature and ask them to describe what concept they think that feature represents.
Revised: For VQA 2.0 dataset, we perform a representation interpretation experiment, where we give human annotators some visualizations on a particular representation feature and ask them to describe what concept they think that feature represents.

Original: We found 15 human annotators (with same qualifications as those in model simulation experiment), and divide them into 3 groups of 5.
Revised: We found 15 human annotators (with same qualifications as those in model simulation experiment), and divide them into 3 groups of 5.

Original: Each group is given a different setting (with different amounts of M ULTI V IZ visualizations available):
Revised: Each group is given a different setting (with different amounts of M ULTI V IZ visualizations available):

Original: R ℓ : R ℓ only, i
Revised: R ℓ : R ℓ only, i

Original: e. one random example and Unimodal LIME explanation on the example with respect to
Revised: e. one random example and Unimodal LIME explanation on the example with respect to

Original: this example
Revised: this example

Original: See Figure 29 for example
Revised: See Figure 29 for example

Original: R ℓ + R g (no viz) :
Revised: R ℓ + R g (no viz) :

Original: In addition to R ℓ with LIME, we also provide R g (top 3 examples that maximizes the feature’s value and top 3 examples that minimizes the feature’s value), but no LIME visualizations for R g .
Revised: In addition to R ℓ with LIME, we also provide R g (top 3 examples that maximizes the feature’s value and top 3 examples that minimizes the feature’s value), but no LIME visualizations for R g .

Original: See Figure 30 for example.
Revised: See Figure 30 for example.

Original: R ℓ + R g :
Revised: R ℓ + R g :

Original: Same as setting 2, but we also provide Unimodal LIME visualizations for all examples in R g .
Revised: Same as setting 2, but we also provide Unimodal LIME visualizations for all examples in R g .

Original: See Figure 31 for example.
Revised: See Figure 31 for example.

Original: We gave the same 13 representation features to all 15 human annotators, where the first feature serves as an example and the other 12 are the ones we actually record for the experiment.
Revised: We gave the same 13 representation features to all 15 human annotators, where the first feature serves as an example and the other 12 are the ones we actually record for the experiment.

Original: The instructor first explains to each annotator what each visualization means, and then goes over the first feature together.
Revised: The instructor first explains to each annotator what each visualization means, and then goes over the first feature together.

Original: Then, the annotator must write down a concept for the other 12 features on their own.
Revised: Then, the annotator must write down a concept for the other 12 features on their own.

Original: We also ask each annotator to rate a confidence of 1-5 on how confident they are that this feature indeed represents this concept.
Revised: We also ask each annotator to rate a confidence of 1-5 on how confident they are that this feature indeed represents this concept.

Original: Once we have collected all 180 annotations (15 annotators each on 12 features), we manually cluster these into 29 distinct concepts that we show in Figure 32.
Revised: Once we have collected all 180 annotations (15 annotators each on 12 features), we manually cluster these into 29 distinct concepts that we show in Figure 32.

Original: For example, annotations like "things to wear", "t-shirts" and "clothes" all belong to "clothes" concept; all color-related annotations belong to "colors" concept; "material question", "made-of question" and "material of object" all belongs to "material" concept.
Revised: For example, annotations like "things to wear", "t-shirts" and "clothes" all belong to "clothes" concept; all color-related annotations belong to "colors" concept; "material question", "made-of question" and "material of object" all belongs to "material" concept.

Original: We then compute inter-rater agreement score on each feature within each group of 5 annotators using Krippendorff’s alpha with 29 possible categories.
Revised: We then compute inter-rater agreement score on each feature within each group of 5 annotators using Krippendorff’s alpha with 29 possible categories.

Original: We report both inter-rater agreement and average confidence in Table 3.
Revised: We report both inter-rater agreement and average confidence in Table 3.

Original: As shown in Table 3, as we give annotators more information, they were able to assign concepts more consistently (higher inter-rater agreement) and more confidently (higher average confidence score).
Revised: As shown in Table 3, as we give annotators more information, they were able to assign concepts more consistently (higher inter-rater agreement) and more confidently (higher average confidence score).

Original: Under setting 3 with full M ULTI V IZ visualizations on feature representations, the 5 annotators completely agreed with each other on 7 out of 12 features, which is really impressive since there
Revised: Under setting 3 with full M ULTI V IZ visualizations on feature representations, the 5 annotators completely agreed with each other on 7 out of 12 features, which is really impressive since there

Original: [Figure]
Revised: [Figure]

Original: + ! "
Revised: + ! "

Original: (no viz) ! !
Revised: (no viz) ! !

Original: + ! "
Revised: + ! "

Original: [Figure]
Revised: [Figure]

Original: are so many possible concepts annotators could assign to each feature.
Revised: are so many possible concepts annotators could assign to each feature.

Original: Therefore, this shows that our visualizations, i.e. R ℓ and R g , really helps humans to better understand what concept (if any) that each feature in representation represents, and that R g examples and visualizations are especially helpful.
Revised: Therefore, this shows that our visualizations, i.e. R ℓ and R g , really helps humans to better understand what concept (if any) that each feature in representation represents, and that R g examples and visualizations are especially helpful.

Original: A concrete example : In Figure 33, we show a concrete example of human annotators using M ULTIV IZ to assign concepts to feature representations in multimodal models trained on VQA 2.0.
Revised: A concrete example : In Figure 33, we show a concrete example of human annotators using M ULTIV IZ to assign concepts to feature representations in multimodal models trained on VQA 2.0.

Original: We show the information provided to users in each of the 3 ablation cases as part of the experiment, along with the actual user annotations from the user study:
Revised: We show the information provided to users in each of the 3 ablation cases as part of the experiment, along with the actual user annotations from the user study:

Original: In R ℓ , we only provide the original seed datapoint and show visualizations of unimodal and cross-modal interactions with respect to a feature z for that datapoint.
Revised: In R ℓ , we only provide the original seed datapoint and show visualizations of unimodal and cross-modal interactions with respect to a feature z for that datapoint.

Original: Using just local information, annotators struggle to identify the concept captured by the feature z , with disagreement between ‘mirror’, ‘brushing teeth’, ‘bathroom’, ‘material’, and ‘none’, each with relatively lower confidence.
Revised: Using just local information, annotators struggle to identify the concept captured by the feature z , with disagreement between ‘mirror’, ‘brushing teeth’, ‘bathroom’, ‘material’, and ‘none’, each with relatively lower confidence.

Original: Indeed, any of the concepts are present in the image and question, which makes it hard to choose a precise one.
Revised: Indeed, any of the concepts are present in the image and question, which makes it hard to choose a precise one.

Original: In R ℓ + R g (no viz) , we provide both the original seed datapoint (local analysis), along withsimilar datapoints that also maximally activate the feature z (global analysis), for 3 datapoints in total.
Revised: In R ℓ + R g (no viz) , we provide both the original seed datapoint (local analysis), along withsimilar datapoints that also maximally activate the feature z (global analysis), for 3 datapoints in total.

Original: Using both local and global information, users are better able to identify the commonalities between all 3 datapoints which all active feature z , leading to 3 / 5 users identifying the concept as ‘asking about material’.
Revised: Using both local and global information, users are better able to identify the commonalities between all 3 datapoints which all active feature z , leading to 3 / 5 users identifying the concept as ‘asking about material’.

Original: However, the remaining 2 users answered ‘household objects/components’, which is another valid concept shared across those datapoints.
Revised: However, the remaining 2 users answered ‘household objects/components’, which is another valid concept shared across those datapoints.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: We find that the features separately capture image-only, language-only, and multimodal concepts.
Revised: We find that the features separately capture image-only, language-only, and multimodal concepts.

Original: In R ℓ + R g , we show both local and global analysis (so 3 datapoints in total), in addition to the visualizations of unimodal and cross-modal interactions with respect to a feature z for all datapoints.
Revised: In R ℓ + R g , we show both local and global analysis (so 3 datapoints in total), in addition to the visualizations of unimodal and cross-modal interactions with respect to a feature z for all datapoints.

Original: With all pieces of information, all 5 / 5 users identified the concept as ‘asking about material’.
Revised: With all pieces of information, all 5 / 5 users identified the concept as ‘asking about material’.

Original: Providing visualizations helps to resolve ambiguity in feature interpretation - the text importance identifies words like ‘counter’, ‘countertop’, and ‘wall’, along with the image crossmodal interactions highlighting these entities, which leads to high agreement and confidence among annotators in identifying the ‘material’ concept.
Revised: Providing visualizations helps to resolve ambiguity in feature interpretation - the text importance identifies words like ‘counter’, ‘countertop’, and ‘wall’, along with the image crossmodal interactions highlighting these entities, which leads to high agreement and confidence among annotators in identifying the ‘material’ concept.

Original: D.3. CLEVR
Revised: D.3. CLEVR

Original: A few examples of interpreted representations are shown in Figure 34, in addition to the examples in
Revised: A few examples of interpreted representations are shown in Figure 34, in addition to the examples in

Original: Figure 4 of the main paper.
Revised: Figure 4 of the main paper.

Original: D.3. MM-IMD B
Revised: D.3. MM-IMD B

Original: A few examples of interpreted representations are shown in Figure 35, in addition to the examples in
Revised: A few examples of interpreted representations are shown in Figure 35, in addition to the examples in

Original: Figure 4 of the main paper.
Revised: Figure 4 of the main paper.

Original: D. E RROR ANALYSIS
Revised: D. E RROR ANALYSIS

Original: In this section, we conduct an experiment to see if human annotators will be able to categorize the reasons why the model fails to predict the correct answer.
Revised: In this section, we conduct an experiment to see if human annotators will be able to categorize the reasons why the model fails to predict the correct answer.

Original: D.4. S ETUP
Revised: D.4. S ETUP

Original: We present three categories of errors:
Revised: We present three categories of errors:

Original: Unimodal perception error :
Revised: Unimodal perception error :

Original: The model fails to recognize certain unimodal features or aspects.
Revised: The model fails to recognize certain unimodal features or aspects.

Original: (For example, in Figure 5 top left example, the FRCNN object detector was unable to recognize the thin red streak as an object).
Revised: (For example, in Figure 5 top left example, the FRCNN object detector was unable to recognize the thin red streak as an object).

Original: Cross-modal interaction error :
Revised: Cross-modal interaction error :

Original: The model fails to capture important cross-modal interactions such as aligning words in question with relevant parts or detected objects in image.
Revised: The model fails to capture important cross-modal interactions such as aligning words in question with relevant parts or detected objects in image.

Original: (For example, in Figure 5 first one in middle column, the model is erroneously aligning "creamy" with the piece of carrot).
Revised: (For example, in Figure 5 first one in middle column, the model is erroneously aligning "creamy" with the piece of carrot).

Original: Prediction errors : The model is able to perceive correct unimodal features and their cross-modal interactions, but fails to reason through them to produce the correct prediction.
Revised: Prediction errors : The model is able to perceive correct unimodal features and their cross-modal interactions, but fails to reason through them to produce the correct prediction.

Original: (For example, in Figure 5 top right example, the model was able to both perfectly identify the chair with object
Revised: (For example, in Figure 5 top right example, the model was able to both perfectly identify the chair with object