Original: (For example, in Figure 5 first one in middle column, the model is erroneously aligning "creamy" with the piece of carrot).
Revised: (For example, in Figure 5 first one in middle column, the model is erroneously aligning "creamy" with the piece of carrot).

Original: Prediction errors : The model is able to perceive correct unimodal features and their crossmodal interactions, but fails to reason through them to produce the correct prediction.
Revised: Prediction errors : The model is able to perceive correct unimodal features and their cross-modal interactions, but fails to reason through them to produce the correct prediction.

Original: (For example, in Figure 5 top right example, the model was able to both perfectly identify the chair with object detector and associate it with the word "chair" in the question (as shown by second-order-gradient analysis), but the model was still unable to reason with the given information correctly to predict the correct answer).
Revised: (For example, in Figure 5 top right example, the model was able to both perfectly identify the chair with object detector and associate it with the word "chair" in the question (as shown by second-order-gradient analysis), but the model was still unable to reason with the given information correctly to predict the correct answer).

Original: For each of the 2 datasets we used in this experiment (VQA and CLEVR), we found 10 human annotators and divide them into 2 groups of 5, one group for each setting: (1) under M ULTI V IZ setting, for each data point, the human annotator is given access to full M ULTI V IZ webpage as well as live Second-Order Gradient (i.e. the human annotator may request to compute second order gradient for a specific subset of words in the question, and he will be presented with the resulting second order gradient result); (2) under No Viz setting, the human annotator is given nothing but the original data point, the correct answer and the predicted answer.
Revised: For each of the 2 datasets we used in this experiment (VQA and CLEVR), we found 10 human annotators and divide them into 2 groups of 5, one group for each setting: (1) under M ULTI V IZ setting, for each data point, the human annotator is given access to full M ULTI V IZ webpage as well as live Second-Order Gradient (i.e. the human annotator may request to compute second order gradient for a specific subset of words in the question, and he will be presented with the resulting second order gradient result); (2) under No Viz setting, the human annotator is given nothing but the original data point, the correct answer and the predicted answer.

Original: Each human annotator needs to classify each point into one of the three categories above, and they are also asked to rate their confidence in categorizing the error on a scale of 
Revised: Each human annotator needs to classify each point into one of the three categories above, and they are also asked to rate their confidence in categorizing the error on a scale of 1-5.

Original: D.4. VQA 2. 
Revised: D.4. VQA 2.

Original: In this experiment, we perform error analysis on VQA 2.0 with LXMERT.
Revised: In this experiment, we perform error analysis on VQA 2.0 with LXMERT.

Original: We first randomly selected 24 data points which the model got wrong, and then we ask 10 human annotators to categorize each point into one of the 3 categories above (5 annotators under M ULTI V IZ setting and 5 annotators under No Viz setting).
Revised: We first randomly selected 24 data points which the model got wrong, and then we ask 10 human annotators to categorize each point into one of the 3 categories above (5 annotators under M ULTI V IZ setting and 5 annotators under No Viz setting).

Original: The webpage that the human annotators under M ULTI V IZ setting sees is the same as the ones described in Appendix B.
Revised: The webpage that the human annotators under M ULTI V IZ setting sees is the same as the ones described in Appendix B.

Original: In addition, since the LXMERT prediction pipeline is differentiable with respect to the detected objects by FRCNN object detector but not with respect to each pixel in the original image, the human annotators under M ULTI V IZ setting will also be given all the bounding boxes of objects detected by FRCNN and also which ones have the highest second order gradient with respect to the specific words they picked.
Revised: In addition, since the LXMERT prediction pipeline is differentiable with respect to the detected objects by FRCNN object detector but not with respect to each pixel in the original image, the human annotators under M ULTI V IZ setting will also be given all the bounding boxes of objects detected by FRCNN and also which ones have the highest second order gradient with respect to the specific words they picked.

Original: See Figure 36 for an example of all bounding boxes detected by FRCNN as well as second-order gradient analysis results for LXMERT.
Revised: See Figure 36 for an example of all bounding boxes detected by FRCNN as well as second-order gradient analysis results for LXMERT.

Original: During the experiment, the instructor first informs the annotators what each of the 3 categories of errors mean, and then explains each part of the visualizations they are given (if under M ULTI V IZ setting).
Revised: During the experiment, the instructor first informs the annotators what each of the 3 categories of errors mean, and then explains each part of the visualizations they are given (if under M ULTI V IZ setting).

Original: Then the instructor goes over the first data point together with the human annotators, and the human annotators must categorize the remaining 23 points on their own, and only those 23 points’ annotations will count towards the final result.
Revised: Then the instructor goes over the first data point together with the human annotators, and the human annotators must categorize the remaining 23 points on their own, and only those 23 points’ annotations will count towards the final result.

Original: The result for VQA error analysis experiment is shown in Table 3.
Revised: The result for VQA error analysis experiment is shown in Table 3.

Original: As shown in the table, on average the human annotators are much more confident in categorizing each error, and also tend to agree with each other a lot more often when given M ULTI V IZ
Revised: As shown in the table, on average the human annotators are much more confident in categorizing each error, and also tend to agree with each other a lot more often when given M ULTI V IZ

Original: compared to No Viz .
Revised: compared to No Viz .

Original: This shows that M ULTI V IZ can indeed help humans identify types of errors within a multimodal model.
Revised: This shows that M ULTI V IZ can indeed help humans identify types of errors within a multimodal model.

Original: In addition, human annotators from the M ULTI V IZ setting report that they can tell whether a model is able to perceive unimodal information correctly via U stage analysis as well as the bounding boxes produced by
Revised: In addition, human annotators from the M ULTI V IZ setting report that they can tell whether a model is able to perceive unimodal information correctly via U stage analysis as well as the bounding boxes produced by

Original: FRCNN, and they found second order gradient requested on specific words most helpful among all
Revised: FRCNN, and they found second order gradient requested on specific words most helpful among all

Original: C stage visualizations (such as DIME) when determining if the model was able to find the correct cross-modal interactions.
Revised: C stage visualizations (such as DIME) when determining if the model was able to find the correct cross-modal interactions.

Original: The data point presented in Figure 36 is one good example of this.
Revised: The data point presented in Figure 36 is one good example of this.

Original: Error breakdown :
Revised: Error breakdown :

Original: Out of the 23 total errors, human annotators reported that on average 8.8 of them are category 1 (unimodal perception error), 6.8 of them are category 2 (cross modal interaction error) and 7.4 of them are category 3 (prediction error).
Revised: Out of the 23 total errors, human annotators reported that on average 8.8 of them are category 1 (unimodal perception error), 6.8 of them are category 2 (cross modal interaction error), and 7.4 of them are category 3 (prediction error).

Original: This suggests that the majority of errors present in LXMERT is still caused by misunderstanding the basic unimodal concepts and cross-modal alignments rather than high-level reasoning of the perceived information, and that one possible future direction for improving the model pipeline is to use better unimodal encoders (than FRCNN) and find out some way to force the model to learn to align visual and text concepts correctly.
Revised: This suggests that the majority of errors present in LXMERT is still caused by misunderstanding the basic unimodal concepts and cross-modal alignments rather than high-level reasoning of the perceived information, and that one possible future direction for improving the model pipeline is to use better unimodal encoders (than FRCNN) and find out some way to force the model to learn to align visual and text concepts correctly.

Original: [Figure] A concrete example : In Figure 37, we show a concrete example of human annotators using M ULTIV IZ to perform error analysis on incorrect predictions made by trained models, specifically into one of 3 stages: failures in (1) unimodal perception, (2) capturing cross-modal interaction, and (3) prediction with perceived unimodal and cross-modal information.
Revised: A concrete example : In Figure 37, we show a concrete example of human annotators using M ULTIV IZ to perform error analysis on incorrect predictions made by trained models, specifically into one of 3 stages: failures in (1) unimodal perception, (2) capturing cross-modal interaction, and (3) prediction with perceived unimodal and cross-modal information.

Original: We show the information provided to users in each of the 2 ablation cases, along with the actual user annotations from the user study: 1.
Revised: We show the information provided to users in each of the 2 ablation cases, along with the actual user annotations from the user study: 1.

Original: No Viz does not provide the user with any information.
Revised: No Viz does not provide the user with any information.

Original: Note that there are no intermediate stages we can ablate, since errors can occur at all stages, so removing any stage from MultiViz by definition cripples its ability to detect errors at that stage.
Revised: Note that there are no intermediate stages we can ablate, since errors can occur at all stages, so removing any stage from MultiViz by definition cripples its ability to detect errors at that stage.

Original: However, users still use their intuition to make a most educated guess on which stage the model is likely to make an error in.
Revised: However, users still use their intuition to make a most educated guess on which stage the model is likely to make an error in

Original: For example, if some odd object seems hard to detect, users tend to guess unimodal error, and if the prediction involves complex reasoning that is hard even for humans, users tend to guess prediction error.
Revised: For example, if some odd object seems hard to detect, users tend to guess unimodal error, and if the prediction involves complex reasoning that is hard even for humans, users tend to guess prediction error.

Original: M ULTI V IZ provides the user with the unimodal importance and cross-modal interactions visualized for that incorrectly predicted datapoint.
Revised: M ULTI V IZ provides the user with the unimodal importance and cross-modal interactions visualized for that incorrectly predicted datapoint.

Original: In the top example, users can tell that the unimodal importance on ‘cheese’ and ‘pizza’ are correct, along with the right imagetext interaction highlighting the bounding pizza around pizza.
Revised: In the top example, users can tell that the unimodal importance on ‘cheese’ and ‘pizza’ are correct, along with the right image-text interaction highlighting the bounding pizza around pizza.

Original: Hence, it is a prediction error, which all users agree on.
Revised: Hence, it is a prediction error, which all users agree on.

Original: In the bottom example, users can see that while ‘man’ and ‘jeans’ are unimodally highlighted correctly, none of the image-text interactions highlight the bounding box around the man’s jeans, so they agree on a cross-modal interaction error. 
Revised: In the bottom example, users can see that while ‘man’ and ‘jeans’ are unimodally highlighted correctly, none of the image-text interactions highlight the bounding box around the man’s jeans, so they agree on a cross-modal interaction error. [Figure]

Revised: : failures in (1) unimodal perception, (2) capturing cross-modal interaction, and (3) prediction with perceived unimodal and cross-modal information (Section 3.3 in the main paper), along with actual error categories annotated by the users.

Original: D.4. CLEVR
Revised: D.4. CLEVR

Original: In this experiment, we perform error analysis on CLEVR with CNN+LSTM+SA model.
Revised: In this experiment, we perform error analysis on CLEVR with CNN+LSTM+SA model.

Original: We first randomly selected 11 data points which the model got wrong, and then we ask 10 human annotators to categorize each point into one of the 3 categories above (5 annotators under M ULTI V IZ setting and 5 annotators under No Viz setting).
Revised: We first randomly selected 11 data points which the model got wrong, and then we ask 10 human annotators to categorize each point into one of the 3 categories above (5 annotators under M ULTI V IZ setting and 5 annotators under No Viz setting).

Original: The webpage that the human annotators under M ULTI V IZ setting sees is the same as the ones described in Appendix B.
Revised: The webpage that the human annotators under M ULTI V IZ setting sees is the same as the ones described in Appendix B.

Original: In addition, the human annotators under
Revised: In addition, the human annotators under

Original: M ULTI V IZ setting can request the second-order gradient analysis result on specific words or phrases they pick, both the pixel-wise heatmap and top 2 bounding boxes with the highest average absolute gradient per pixel (same procedure as described in Appendix D.2).
Revised: M ULTI V IZ setting can request the second-order gradient analysis result on specific words or phrases they pick, both the pixel-wise heatmap and top 2 bounding boxes with the highest average absolute gradient per pixel (same procedure as described in Appendix D.2).

Original: See the bottom half of Figurefor an example of second-order gradient analysis result of CNN+LSTM+SA.
Revised: See the bottom half of Figurefor an example of second-order gradient analysis result of CNN+LSTM+SA.

Original: During the experiment, the instructor first informs the annotators what each of the 3 categories of errors mean, and then explains each part of the visualizations they are given (if under M ULTI V IZ setting).
Revised: During the experiment, the instructor first informs the annotators what each of the 3 categories of errors mean, and then explains each part of the visualizations they are given (if under M ULTI V IZ setting).

Original: Then the instructor goes over the first data point together with the human annotators, and the  human annotators must categorize the remaining 10 points on their own, and only those 10 points’ annotations will count towards the final result.
Revised: Then the instructor goes over the first data point together with the human annotators, and the [Figure] [Figure] human annotators must categorize the remaining 10 points on their own, and only those 10 points’ annotations will count towards the final result.

Original: The result for CLEVR error analysis experiment is shown in Table 3.
Revised: The result for CLEVR error analysis experiment is shown in Table 3.

Original: As shown in the table, on average the human annotators are much more confident in categorizing each error, and also tend to agree with each other a lot more often when given M ULTI V IZ
Revised: As shown in the table, on average the human annotators are much more confident in categorizing each error, and also tend to agree with each other a lot more often when given M ULTI V IZ

Original: compared to No Viz .
Revised: compared to No Viz .

Original: This shows that
Revised: This shows that

Original: M ULTI V IZ can indeed help humans identify types of errors within a multimodal model.
Revised: M ULTI V IZ can indeed help humans identify types of errors within a multimodal model.

Original: Error breakdown :
Revised: Error breakdown :

Original: Out of the 10 total errors, human annotators on average reported 6 of them belonging to category 2 (cross modal interaction error).
Revised: Out of the 10 total errors, human annotators on average reported 6 of them belonging to category 2 (cross modal interaction error).

Original: This suggests that the major weakness of
Revised: This suggests that the major weakness of

Original: CNN+LSTM+SA is that it is not great at aligning phrases in text with the object the phrase refers to.
Revised: CNN+LSTM+SA is that it is not great at aligning phrases in text with the object the phrase refers to.

Original: This is expected because CNN+LSTM+SA is a late fusion model, which is known to be not great at capturing low-level cross-modal interactions.[Figure] identifying [Figure]
Revised: This is expected because CNN+LSTM+SA is a late fusion model, which is known to be not great at capturing low-level cross-modal interactions.

Original: [Figure] Model debugging Can humans find bugs in the model [Figure] improvement? 
Revised: D. M ODEL DEBUGGING D.5. VQA 2.

Original: [Figure]

Original: targeted

Original: [Figure] [Figure] [Figure] Following error analysis, we take a deeper investigation into one of the errors on a pretrained
Revised: Following error analysis, we take a deeper investigation into one of the errors on a pretrained

Original: LXMERT (Tan and Bansal, 2019) model fine-tuned on VQA 2.0 (Goyal et al., 2017).
Revised: LXMERT (Tan and Bansal, 2019) model fine-tuned on VQA 2.0 (Goyal et al., 2017).

Original: We compute the penultimate features (the input to the last linear layer in the classification head) of the V set, and train a linear model that best maps the absolute values of these penultimate features to a binary label where 0 means the original LXMERT model got this point right
Revised: We compute the penultimate features (the input to the last linear layer in the classification head) of the V set, and train a linear model that best maps the absolute values of these penultimate features to a binary label where 0 means the original LXMERT model got this point right

Original: and 1 means the original LXMERT model got this point wrong.
Revised: and 1 means the original LXMERT model got this point wrong.

Original: Then, we pick the top 5 dimensions in the penultimate feature with the highest positive weight in the linear model, and task human annotators to inspect these neurons carefully through M ULTI V IZ local and global representation analysis.
Revised: Then, we pick the top 5 dimensions in the penultimate feature with the highest positive weight in the linear model, and task human annotators to inspect these neurons carefully through M ULTI V IZ local and global representation analysis.

Original: Human annotators found that 2 of the 5 neurons were consistently related to questions asking about color, which highlighted the model’s failure to identify color correctly (especially blue).
Revised: Human annotators found that 2 of the 5 neurons were consistently related to questions asking about color, which highlighted the model’s failure to identify color correctly (especially blue).

Original: The model has an accuracy of only 5 . 5% amongst all blue-related points (i.e., either have blue as correct answer or predicted answer), and these failures account for 8 .
Revised: The model has an accuracy of only 5 . 5% amongst all blue-related points (i.e., either have blue as correct answer or predicted answer), and these failures account for 8 .

Original: 8% of all model errors.
Revised: 8% of all model errors.

Original: We show examples of such datapoints and their M ULTI V IZ visualizations in Figure 6.
Revised: We show examples of such datapoints and their M ULTI V IZ visualizations in Figure 6.

Original: Observe that the model is often able to capture unimodal and cross-modal interactions perfectly, but fails to identify color at the prediction stage.
Revised: Observe that the model is often able to capture unimodal and cross-modal interactions perfectly, but fails to identify color at the prediction stage.

Original: In this section, we describe our initial attempt at fixing this color-related bug by adding targeted data in an active learning scenario.
Revised: In this section, we describe our initial attempt at fixing this color-related bug by adding targeted data in an active learning scenario.

Original: If M ULTI V IZ indeed provides accurate insights for model debugging, we should be able to improve model performance using less data as compared to a control experiment that adds randomly sampled data (see Figure 38).
Revised: If M ULTI V IZ indeed provides accurate insights for model debugging, we should be able to improve model performance using less data as compared to a control experiment that adds randomly sampled data (see Figure 38).

Original: We first split the validation set (about 220K points) into 3 parts: the first 110K were called the V set (stands for "val"), the next 50K were called the U set (stands for "unlabeled"), and the last 60K were called the T test (stands for "test").
Revised: We first split the validation set (about 220K points) into 3 parts: the first 110K were called the V set (stands for "val"), the next 50K were called the U set (stands for "unlabeled"), and the last 60K were called the T test (stands for "test").

Original: We are simulating a situation where in addition to the 450K training set, we have a labeled 110K validation set (V set), another 50K unlabeled points (U set), and 60K held-out test set (T set).
Revised: We are simulating a situation where in addition to the 450K training set, we have a labeled 110K validation set (V set), another 50K unlabeled points (U set), and 60K held-out test set (T set).

Original: Our goal is to debug or improve the given model (LXMERT) by selecting N points from U set to label and finetune the model with these N points.
Revised: Our goal is to debug or improve the given model (LXMERT) by selecting N points from U set to label and finetune the model with these N points.

Original: We compare the following settings:
Revised: We compare the following settings:

Revised: [Table]

Revised: [Table]

Original: Random :
Revised: Random :

Original: We randomly sample N points from the U set.
Revised: We randomly sample N points from the U set.

Original: Uncertainty :
Revised: Uncertainty :

Original: A common active learning baseline which selects the top N datapoints from the U set that the model is uncertain about based on the entropy of its predicted label distribution (Lewis and Catlett, 1994; Lewis and Gale, 1994; Settles, 2009).
Revised: A common active learning baseline which selects the top N datapoints from the U set that the model is uncertain about based on the entropy of its predicted label distribution (Lewis and Catlett, 1994; Lewis and Gale, 1994; Settles, 2009).

Original: M ULTI V IZ 2 color :
Revised: M ULTI V IZ 2 color :

Original: For each of these 2 erroneous features, we picked N 2 points from the U set that has the highest absolute values on the feature, and together these points form the N points related to color that we select from the U set.
Revised: For each of these 2 erroneous features, we picked N 2 points from the U set that has the highest absolute values on the feature, and together these points form the N points related to color that we select from the U set.

Original: Note that we do not use label information about these additional datapoints.
Revised: Note that we do not use label information about these additional datapoints.

Original: M ULTI V IZ
Revised: M ULTI V IZ

Original: no color : Same as above, but we use 2 features that do not represent color.
Revised: no color : Same as above, but we use 2 features that do not represent color.

Original: [Table]
Revised: M ULTI V IZ 1 color :

Original: [Table] [Table]
Revised: Same as above, but we use 1 feature that represents color and 1 that does not represent color.

Original: Under each of these active learning settings [Table] we finetune the last layer of LXMERT with the N selected points from U set for one epoch (batch size 32, learning rate tuned to the best performance), and the result is evaluated on the T set.
Revised: Under each of these active learning settings, we finetune the last layer of LXMERT with the N selected points from U set for one epoch (batch size 32, learning rate tuned to the best performance), and the result is evaluated on the T set.

Original: In addition, since through M ULTI V IZ analysis we found out that LXMERT is particularly bad on data points that either have ground truth correct answer "blue" or the original LXMERT predicts as "blue", we define a subset of T set we call "bluelist" that contains all 1729 points in the T set that either have ground truth correct answer "blue" or the original
Revised: In addition, since through M ULTI V IZ analysis we found out that LXMERT is particularly bad on data points that either have ground truth correct answer "blue" or the original LXMERT predicts as "blue", we define a subset of T set we call "bluelist" that contains all 1729 points in the T set that either have ground truth correct answer "blue" or the original

Original: LXMERT predicts as "blue".
Revised: LXMERT predicts as "blue".

Original: The original LXMERT only has a 6% accuracy on bluelist.
Revised: The original LXMERT only has a 6% accuracy on bluelist.

Original: We try each setting 10 times (with different random seeds) and report average and standard deviation on improvement in accuracies on both the entire T set and bluelist over the original LXMERT.
Revised: We try each setting 10 times (with different random seeds) and report average and standard deviation on improvement in accuracies on both the entire T set and bluelist over the original LXMERT.

Original: We show these results in Table 6 and find that M ULTI V IZ significantly improves upon either random or uncertainty-based sampling as measured by performance on the overall VQA 2.0 test set.
Revised: We show these results in Table 6 and find that M ULTI V IZ significantly improves upon either random or uncertainty-based sampling as measured by performance on the overall VQA 2.0 test set.

Original: To obtain a deeper look at performance, we further evaluate performance on a targeted test set only containing questions asking about color (reflecting the main bug we found in the model).
Revised: To obtain a deeper look at performance, we further evaluate performance on a targeted test set only containing questions asking about color (reflecting the main bug we found in the model).

Original: On this targeted test set, M ULTI V IZ significantly improves performance by 30% as compared to only 1 . 4% for random sampling.
Revised: On this targeted test set, M ULTI V IZ significantly improves performance by 30% as compared to only 1 . 4% for random sampling.

Original: Using more features related to color also improved performance: 27% with 1 feature and 30% with both features.
Revised: Using more features related to color also improved performance: 27% with 1 feature and 30% with both features.

Original: Surprisingly, we find uncertainty sampling had no effect ( 0 . 0% ) since the model predicted these incorrect answers on color-related questions with high certainty , so none of these color-related questions were additionally introduced to the model.
Revised: Surprisingly, we find uncertainty sampling had no effect ( 0 . 0% ) since the model predicted these incorrect answers on color-related questions with high certainty , so none of these color-related questions were additionally introduced to the model.

Original: D. S UMMARY OF TAKEAWAY MESSAGES
Revised: D. S UMMARY OF TAKEAWAY MESSAGES

Original: From these results we emphasize the main take-away messages:
Revised: From these results, we emphasize the main take-away messages:

Original: From the model simulation experiment, we found that on all 3 settings of datasets and models, human annotators were able to get higher accuracy and better agreement when given strictly more stages of visualization from M ULTI V IZ .
Revised: From the model simulation experiment, we found that on all 3 settings of datasets and models, human annotators were able to get higher accuracy and better agreement when given strictly more stages of visualization from M ULTI V IZ .

Original: This suggests that each stage in M ULTI V IZ is complementary to helping humans better understand the models’ decision-making process.
Revised: This suggests that each stage in M ULTI V IZ is complementary to helping humans better understand the models’ decision-making process.

Original: Through a deeper inspection of cross-modal interaction visualization, we showed that second-order gradient is faithful to what the model internally aligns most of the time (over 80% using top 2 alignment accuracy on MDETR).
Revised: Through a deeper inspection of cross-modal interaction visualization, we showed that second-order gradient is faithful to what the model internally aligns most of the time (over 80% using topalignment accuracy on MDETR).

Original: From the representation interpretation experiment, we found that having both local and global representation visualizations helps human annotators assign interpretable concepts in natural language to deep features with higher confidence and agreement.
Revised: From the representation interpretation experiment, we found that having both local and global representation visualizations helps human annotators assign interpretable concepts in natural language to deep features with higher confidence and agreement.

Original: From the error analysis experiment, we showed that M ULTI V IZ can help users locate the stage of model that caused the error when the model makes a mistake, which provides insights for model error analysis and debugging.
Revised: From the error analysis experiment, we showed that M ULTI V IZ can help users locate the stage of model that caused the error when the model makes a mistake, which provides insights for model error analysis and debugging.

Original: Finally, we showcase a real-world model debugging case study: using each stage of M ULTIV IZ to localize the error, we were able to locate a real bug in the HuggingFace Transformers LXMERT library.
Revised: Finally, we showcase a real-world model debugging case study: using each stage of M ULTI V IZ to localize the error, we were able to locate a real bug in the HuggingFace Transformers LXMERT library.

Original: E L IMITATIONS AND F UTURE D IRECTIONS E. L IMITATIONS
Revised: E L IMITATIONS AND F UTURE D IRECTIONS E. L IMITATIONS

Original: We are aware of some directions in which M ULTI V IZ can still be improved and outline these for future work:
Revised: We are aware of some directions in which M ULTI V IZ can still be improved and outline these for future work:

Original: Large number of prediction classes: for complex tasks like VQA 2.0 where there are over three thousand prediction classes, a lot of rarely used answer choices will get "sparsed out" in the Sparse Linear Model analysis (since setting their weights to 0 barely affects overall accuracy), which makes it difficult to find related datapoints for local and global representation analysis.
Revised: Large number of prediction classes: for complex tasks like VQA 2.0 where there are over three thousand prediction classes, a lot of rarely used answer choices will get "sparsed out" in the Sparse Linear Model analysis (since setting their weights to 0 barely affects overall accuracy), which makes it difficult to find related datapoints for local and global representation analysis.

Original: For example, Sparse Linear Model Analysis on LXMERT have zero weight from all representation neurons to the rare answer choice "abstract", so the five most important feature neurons are completely randomly selected.
Revised: For example, Sparse Linear Model Analysis on LXMERT have zero weight from all representation neurons to the rare answer choice "abstract", so the five most important feature neurons are completely randomly selected.

Original: Too few prediction classes: for VQA 2.0 subsets with ‘yes/no’ answer choices, we found that the final-layer activated features contain too much overlap to reliably visualize, and we have to extend MultiViz to rely on more intermediate-layer features.
Revised: Too few prediction classes: for VQA 2.0 subsets with ‘yes/no’ answer choices, we found that the final-layer activated features contain too much overlap to reliably visualize, and we have to extend MultiViz to rely on more intermediate-layer features.

Original: We added this experiment in Appendix B.3.
Revised: We added this experiment in Appendix B.3.

Original: Overall, M ULTI V IZ (like general ML models), work best with a reasonable number of prediction classes, such as those in multimodal emotion recognition, standard multiple-choice multimodal question answering, and others.
Revised: Overall, M ULTI V IZ (like general ML models), work best with a reasonable number of prediction classes, such as those in multimodal emotion recognition, standard multiple-choice multimodal question answering, and others.

Original: Model requirements: Currently the two requirement of models is that they have categorical outputs (classification) and we can easily compute gradients via AutoGrad.
Revised: Model requirements: Currently the two requirement of models is that they have categorical outputs (classification) and we can easily compute gradients via AutoGrad.

Original: The classification requirement is so that we can visualize given specific model outputs (e.g., word answers, emotion categories, video categories).
Revised: The classification requirement is so that we can visualize given specific model outputs (e.g., word answers, emotion categories, video categories).

Original: For regression, we can extend MultiViz via discretizing the output space into categorical outputs.
Revised: For regression, we can extend MultiViz via discretizing the output space into categorical outputs.

Original: The second requirement enables us to perform first and higher-order gradient analysis, which means that we cannot currently support some neuro-symbolic multimodal architectures that have discrete steps (e.g., parsing and executing the question as a program (Mao et al., 2019)) in the middle of the model that prevents gradient flow.
Revised: The second requirement enables us to perform first and higher-order gradient analysis, which means that we cannot currently support some neuro-symbolic multimodal architectures that have discrete steps (e.g., parsing and executing the question as a program (Mao et al., 2019)) in the middle of the model that prevents gradient flow.

Original: We plan to extend MultiViz via approximate gradients such as perturbation or policy gradientsin the future to handle these cases.
Revised: We plan to extend MultiViz via approximate gradients such as perturbation or policy gradients to handle these cases.

Original: Visualization testing: We spent a lot of time into finding and training users.
Revised: Visualization testing: We spent a lot of time into finding and training users.

Original: We carefully found users (who are not the authors and are not part of the same research groups) that have or are working towards a graduate degree in a STEM field and have knowledge of ML models.
Revised: We carefully found users (who are not the authors and are not part of the same research groups) that have or are working towards a graduate degree in a STEM field and have knowledge of ML models.

Original: We showed them a training video describing how MultiViz can be used before each study session (see Appendix D for all experiment and user study details).
Revised: We showed them a training video describing how MultiViz can be used before each study session (see Appendix D for all experiment and user study details).

Original: Consequently, our user studies span over 60 hours of human testing on close to 100 total datapoints, which has enabled us to draw preliminary conclusions regarding the efficacy of multiple proposed stages towards model understanding and debugging.
Revised: Consequently, our user studies span overhours of human testing on close to 100 total datapoints, which has enabled us to draw preliminary conclusions regarding the efficacy of multiple proposed stages towards model understanding and debugging.

Original: Future work can explore more standardized ways of human-in-the-loop interpretation and debugging of multimodal models, and we hope that M ULTI V IZ can provide the initial data, models, tools, and evaluation as a step in this direction.
Revised: Future work can explore more standardized ways of human-in-the-loop interpretation and debugging of multimodal models, and we hope that M ULTI V IZ can provide the initial data, models, tools, and evaluation as a step in this direction.

Original: We plan to ensure the continual availability, maintenance, and expansion of M ULTI V IZ .
Revised: We plan to ensure the continual availability, maintenance, and expansion of M ULTI V IZ .

Original: Several immediatefuture directions include expansions in the datasets and models used for interpretation with M ULTI V IZ , interpretation algorithmsimplemented in M ULTI V IZ , and holistic evaluation of interpretation methods.E. N EW DATASETS AND MODELS There are a suite of new datasets in MultiBench (Liang et al., 2021a) that we plan to include in M ULTI V IZ , including those in the multimodal reinforcement learning (Luketina et al., 2019) (e.g., RTFM (Zhong et al., 2020) and SILG benchmark (Zhong et al., 2021)) and multimodal co-learning (Zadeh et al., 2020) research fields (e.g., transferring information from CMU-MOSI to
Revised: Several immediate directions include new interpretation algorithms and holistic evaluation of interpretation methods.

Original: SST, CMU-MOSEI to SST (Zadeh et al., 2020), G LO V E word embeddings for CIFAR10 image classification (Socher et al., 2013), and knowledge graphs to image classification on Visual Genome dataset (Krishna et al., 2017; Marino et al., 2017)).

Original: We attempted to include several models in the neuro-symbolic category (Amizadeh et al., 2020;

Original: Mao et al., 2019), but found them incompatible with gradient-based interpretation methods as well as LIME-based methods.

Original: We plan to investigate ways to analyze these models as well, perhaps in architecture-specific ways.

Original: There have also been recent attempts at building multimodal models that are interpretable by design, such as through soft (Park et al., 2018) and hard attention mechanisms (Chen et al., 2017), graphs (Liang et al., 2018b; Zadeh et al., 2018) or routing networks (Tsai et al., 2020).

Original: We plan to investigate the tradeoffs between black-box post-hoc interpretable models versus those interpretable by design, with respect to model simulation, error analysis, and debugging with stakeholders, as well as performance.

Original: E. N EW CLASSES OF INTERPRETATION TOOLS
Revised: E. N EW CLASSES OF INTERPRETATION TOOLS

Original: M ULTI V IZ is designed to be modular and support interpretation tools at each stage.
Revised: M ULTI V IZ is designed to be modular and support interpretation tools at each stage.

Original: While we have explored some directions, we plan to include the followinginterpretation methods in future work:E
Revised: While we have explored some directions, we plan to include the following methods in future work:

Original: [Equation]
Revised: [Equation]

Original: U NIMODAL IMPORTANCE
Revised: U NIMODAL IMPORTANCE

Original: We also plan to include methods for segmenting data into atoms: a basic unit of real-world data in that modality which cannot (or rather, the user chooses to not) be broken down into further units.
Revised: We also plan to include methods for segmenting data into atoms: a basic unit of real-world data in that modality which cannot (or rather, the user chooses to not) be broken down into further units.

Original: For example, when working with text, a user may choose the level of words or characters as the most basic unit.
Revised: For example, when working with text, a user may choose the level of words or characters as the most basic unit.

Original: We plan for M ULTI V IZ to provide a standard set of segmentations for each modality’s raw data, such as words for sentences or object bounding boxes for images.
Revised: We plan for M ULTI V IZ to provide a standard set of segmentations for each modality’s raw data, such as words for sentences or object bounding boxes for images.

Original: At the same time, it also allows the user to flexibly segment their own modalities of choice.
Revised: At the same time, it also allows the user to flexibly segment their own modalities of choice.

Original: Given segmented atoms, representations are learned for each atom to capture unimodal meaning contextualized in the presence of other atoms in the same modality.
Revised: Given segmented atoms, representations are learned for each atom to capture unimodal meaning contextualized in the presence of other atoms in the same modality.

Original: While the included datasets provide relatively clear segmentation, this task becomes nontrivial in the face of new datasets we plan to include, such as videos and graphs.
Revised: While the included datasets provide relatively clear segmentation, this task becomes nontrivial in the face of new datasets we plan to include, such as videos and graphs.

Original: E.3. C ROSS - MODAL INTERACTIONS
Revised: E.2. C ROSS - MODAL INTERACTIONS

Original: There have been several attempts at building multimodal models that are interpretable by design, with a particular focus on cross-modal interactions.
Revised: There have been several attempts at building multimodal models that are interpretable by design, with a particular focus on cross-modal interactions.

Original: Many of these involve parameterizing cross-modal interactions through attention models (Hu and Singh, 2021; Lu et al., 2019) or graph-based models (Liang et al., 2018b; Zadeh et al., 2018).
Revised: Many of these involve parameterizing cross-modal interactions through attention models (Hu and Singh, 2021; Lu et al., 2019) or graph-based models (Liang et al., 2018b; Zadeh et al., 2018).

Original: As a result, there have been several approaches to study these specific types of cross-modal interactions, such as M2Lens (Wang et al., 2021), an interactive visual analytics system to visualize multimodal models for sentiment analysis through both unimodal and cross-modal contributions, and VL-InterpreT
Revised: As a result, there have been several approaches to study these specific types of cross-modal interactions, such as M2Lens (Wang et al., 2021), an interactive visual analytics system to visualize multimodal models for sentiment analysis through both unimodal and cross-modal contributions, and VL-InterpreT

Original: (Aflalo et al., 2022), an interactive visualization tool for interpreting vision-language transformers.
Revised: (Aflalo et al., 2022), an interactive visualization tool for interpreting vision-language transformers.

Original: We plan to include these in M ULTI V IZ to compare black-box post-hoc interpretationof cross-modal interactions versus models interpretable by design.
Revised: We plan to include these in M ULTI V IZ to compare black-box post-hoc interpretation versus models interpretable by design.

Original: E.3. M ULTIMODAL PREDICTION
Revised: E.2. M ULTIMODAL PREDICTION

Original: For future work beyond linear prediction, we also plan to investigate 2 other parameterizations to interpret the multimodal prediction process:
Revised: For future work beyond linear prediction, we also plan to investigate 2 other parameterizations to interpret the multimodal prediction process:

Original: Tree: Insights from integrating neural networks with decision trees (Wan et al., 2020) can enable us to generalize linear reasoning into one based on compositionality defined by a decision tree.
Revised: Tree: Insights from integrating neural networks with decision trees (Wan et al., 2020) can enable us to generalize linear reasoning into one based on compositionality defined by a decision tree.

Original: Specifically, we will investigate replacing the final layer of a black-box multimodal model with a learned decision tree to generalize linear prediction to hierarchical tree-structured prediction.
Revised: Specifically, we will investigate replacing the final layer of a black-box multimodal model with a learned decision tree to generalize linear prediction to hierarchical tree-structured prediction.

Original: Hierarchical: How can we extend these ideas towards visualizing general compositionality of multimodal evidences, either predefined by the user or learned by the model?
Revised: Hierarchical: How can we extend these ideas towards visualizing general compositionality of multimodal evidences, either predefined by the user or learned by the model?

Original: We propose knowledge distillation as a general paradigm for understanding the reasoning process in multimodal models.
Revised: We propose knowledge distillation as a general paradigm for understanding the reasoning process in multimodal models.

Original: For example, we may attempt to distill the knowledge in a black-box ViLBERT (Lu et al., 2019) model into a neuro-symbolic model with predefined multimodal composition (e.g., Neural Module
Revised: For example, we may attempt to distill the knowledge in a black-box ViLBERT (Lu et al., 2019) model into a neuro-symbolic model with predefined multimodal composition (e.g., Neural Module

Original: Networks (Andreas et al., 2016) or Neuro-Symbolic VQA (Yi et al., 2018)).
Revised: Networks (Andreas et al., 2016) or Neuro-Symbolic VQA (Yi et al., 2018)).

Original: This results in a model that preserves the interpretable benefits of predefined reasoning while at the same time enjoying the benefits of large-scale pretraining.
Revised: This results in a model that preserves the interpretable benefits of predefined reasoning while at the same time enjoying the benefits of large-scale pretraining.

Original: By comparing the decisions made bythe ViLBERTmodel before and after distillation, we can also obtain a measure of the reasoning capabilities of theseblack-box models.
Revised: By comparing the decisions made by ViLBERT before and after distillation, we can also obtain a measure of the reasoning capabilities of these models.

Original: E. E VALUATING INTERPRETABILITY
Revised: E. E VALUATING INTERPRETABILITY

Original: Progress towards interpretability is challenging to evaluate (Chan et al., 2022; Dasgupta et al., 2022;
Revised: Progress towards interpretability is challenging to evaluate (Chan et al., 2022; Dasgupta et al., 2022;

Original: Jacovi and Goldberg, 2020; Shah et al., 2021; Srinivas and Fleuret, 2020).
Revised: Jacovi and Goldberg, 2020; Shah et al., 2021; Srinivas and Fleuret, 2020).

Original: Model interpretability (1) is highly subjective across different population subgroups (Arora et al., 2021; Krishna et al., 2022), (2) requires high-dimensional model outputs as opposed to low-dimensional prediction objectives (Park et al., 2018), and (3) has desiderata that change across research fields, populations, and time (Murdoch et al., 2019).
Revised: Model interpretability (1) is highly subjective across different population subgroups (Arora et al., 2021; Krishna et al., 2022), (2) requires high-dimensional model outputs as opposed to low-dimensional prediction objectives (Park et al., 2018), and (3) has desiderata that change across research fields, populations, and time (Murdoch et al., 2019).

Original: We plan to continuously expand M ULTI V IZ through community inputs for new metrics to evaluate interpretability methods.
Revised: We plan to continuously expand M ULTI V IZ through community inputs for new metrics to evaluate interpretability methods.

Original: Some metrics we have in mind include those for measuring faithfulness, as proposed in recent work (Chan et al., 2022; Dasgupta et al., 2022; Jacovi and Goldberg, 2020; Madsen et al., 2021; Shah et al., 2021; Srinivas and Fleuret, 2020; Wu and Mooney, 2019).
Revised: Some metrics we have in mind include those for measuring faithfulness, as proposed in recent work (Chan et al., 2022; Dasgupta et al., 2022; Jacovi and Goldberg, 2020; Madsen et al., 2021; Shah et al., 2021; Srinivas and Fleuret, 2020).

Original: E. E NGAGEMENT WITH REAL - WORLD STAKEHOLDERS
Revised: E. E NGAGEMENT WITH REAL - WORLD STAKEHOLDERS

Original: Finally, we have plans for engagement with real-world stakeholders to evaluate the usefulness of these multimodal interpretation tools.
Revised: Finally, we have plans for engagement with real-world stakeholders to evaluate the usefulness of these multimodal interpretation tools.

Original: We plan to engage these stakeholders in the healthcare domain to evaluate interpretability on the MIMIC dataset and those in the affective computing domain to evaluate interpretability on the CMU-MOSEI dataset.
Revised: We plan to engage these stakeholders in the healthcare domain to evaluate interpretability on the MIMIC dataset and those in the affective computing domain to evaluate interpretability on the CMU-MOSEI dataset.

Original: We also refer the reader to recent work examining the issues surrounding real-world deployment of interpretable machine learning (Bhatt et al., 2020; Chen et al., 2022; Krishna et al., 2022).
Revised: We also refer the reader to recent work examining the issues surrounding real-world deployment of interpretable machine learning (Bhatt et al., 2020; Chen et al., 2022; Krishna et al., 2022).