Original: O BJECTIVE E VALUATION OF D EEP V ISUAL I NTERPRETATIONS ON T IME S ERIES D ATA Anonymous authors
Revised: O BJECTIVE E VALUATION OF D EEP V ISUAL I NTERPRETATIONS ON T IME S ERIES D ATA Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The correct interpretation and understanding of deep learning models is essential in many applications.
Revised: The correct interpretation and understanding of deep learning models is essential in many applications.

Original: (Explanatory) visual interpretation approaches for image and natural language processing allow domain experts to validate and understand almost any deep learning model.
Revised: (Explanatory) visual interpretation approaches for image and natural language processing allow domain experts to validate and understand almost any deep learning model.

Original: However, they fall short when generalizing to arbitrary time series data that is less intuitive and more diverse.
Revised: However, they fall short when generalizing to arbitrary time series data that is less intuitive and more diverse.

Original: Whether a visualization explains the true reasoning or captures the real features is more difﬁcult to judge.
Revised: Whether a visualization explains the true reasoning or captures the real features is more difﬁcult to judge.

Original: Hence, instead of blind trust we need an objective evaluation to obtain reliable quality metrics.
Revised: Hence, instead of blind trust we need an objective evaluation to obtain reliable quality metrics.

Original: This paper proposes a framework of six orthogonal quality metrics for gradient- or perturbation-based post-hoc visual interpretation methods designed for time series classiﬁcation and segmentation tasks.
Revised: This paper proposes a framework of six orthogonal quality metrics for gradient- or perturbation-based post-hoc visual interpretation methods designed for time series classiﬁcation and segmentation tasks.

Original: This comprehensive set is either based on ”human perception” or on ”functional properties”.
Revised: This comprehensive set is either based on ”human perception” or on ”functional properties”.

Original: An extensive experimental study includes commonly used neural network architectures for time series and nine visual interpretation methods.
Revised: An extensive experimental study includes commonly used neural network architectures for time series and nine visual interpretation methods.

Original: We evaluate the visual interpretation methods with diverse datasets from the UCR repository and another complex real-world dataset.
Revised: We evaluate the visual interpretation methods with diverse datasets from the UCR repository and another complex real-world dataset.

Original: We show that none of the methods consistently outperforms any of the others on all metrics while some of them are ahead in either functional or human-based metrics.
Revised: We show that none of the methods consistently outperforms any of the others on all metrics while some of them are ahead in either functional or human-based metrics.

Original: Our results allow experts to make an informed choice of suitable visualization techniques for the model and task at hand
Revised: Our results allow experts to make an informed choice of suitable visualization techniques for the model and task at hand

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Due to its high performance on complex multi-modal data, deep learning (DL) becomes increasingly popular in many real-world applications that process time series data (Fawaz et al., 2019b).
Revised: Due to its high performance on complex multi-modal data, deep learning (DL) becomes increasingly popular in many real-world applications that process time series data (Fawaz et al., 2019b).

Original: While we fundamentally rely on their classiﬁcation accuracy in many safety-relevant applications (Berkenkamp et al., 2017) they remain difﬁcult to interpret.
Revised: While we fundamentally rely on their classiﬁcation accuracy in many safety-relevant applications (Berkenkamp et al., 2017) they remain difﬁcult to interpret.

Original: Typical applications are the monitoring of industrial processes (L ¨ ofﬂer et al., 2021), the support of health care and sports (Dorschky et al., 2020), or safety in autonomous driving (Schmidt et al., 2021).
Revised: Typical applications are the monitoring of industrial processes (L ¨ ofﬂer et al., 2021), the support of health care and sports (Dorschky et al., 2020), or safety in autonomous driving (Schmidt et al., 2021).

Original: The need for improved model understanding (Carvalho et al., 2019), along with regulatory guidelines (Goodman & Flaxman, 2017), led to a myriad of new approaches to the visual interpretation problem (Zhang & Zhu, 2018).
Revised: The need for improved model understanding (Carvalho et al., 2019), along with regulatory guidelines (Goodman & Flaxman, 2017), led to a myriad of new approaches to the visual interpretation problem (Zhang & Zhu, 2018).

Original: Post-hoc visual interpretation allows a domain expert to validate and understand how (almost) arbitrary deep learning models operate.
Revised: Post-hoc visual interpretation allows a domain expert to validate and understand how (almost) arbitrary deep learning models operate.

Original: Their central idea lies in highlighting features on the input that are ”relevant” for the prediction of a learned model (Adebayo et al., 2018).
Revised: Their central idea lies in highlighting features on the input that are ”relevant” for the prediction of a learned model (Adebayo et al., 2018).

Original: Many of these techniques do not require a modiﬁcation of the original model (Simonyan et al., 2014; Ribeiro et al., 2016) and are compatible with different architectures, which makes them useful as a general-purpose validation tool for neural networks across different tasks (Arrieta et al., 2020).
Revised: Many of these techniques do not require a modiﬁcation of the original model (Simonyan et al., 2014; Ribeiro et al., 2016) and are compatible with different architectures, which makes them useful as a general-purpose validation tool for neural networks across different tasks (Arrieta et al., 2020).

Original: However, while visual interpretation yields intuitive and correct explanations for images (Samek et al., 2021), the application of these methods on time series data is still an unsolved problem (Rojat et al., 2021).
Revised: However, while visual interpretation yields intuitive and correct explanations for images (Samek et al., 2021), the application of these methods on time series data is still an unsolved problem (Rojat et al., 2021).

Original: Time series inherently are more diverse (Rojat et al., 2021)
Revised: Time series inherently are more diverse (Rojat et al., 2021)

Original: (because they may originate from a variety of different sensors and processes), and often do not allow an obvious patch- or texture-based localization of critical features for human observers.
Revised: (because they may originate from a variety of different sensors and processes), and often do not allow an obvious patch- or texture-based localization of critical features for human observers.

Original: This makes the application and the evaluation of visual interpretability methods difﬁcult.
Revised: This makes the application and the evaluation of visual interpretability methods difﬁcult.

Original: A domain expert cannot easily judge if explanations are correct in (i) explaining the reason for the decision process in the DL model and in (ii) capturing the real features in the dataset that lead to a correct classiﬁcation.
Revised: A domain expert cannot easily judge if explanations are correct in (i) explaining the reason for the decision process in the DL model and in (ii) capturing the real features in the dataset that lead to a correct classiﬁcation.

Original: Hence, it is important not to blindly apply different visualization methods.
Revised: Hence, it is important not to blindly apply different visualization methods.

Original: This requires quality metrics that evaluate visual interpretations and that enable an expert to select a suitable visualization for a given model and task at hand.
Revised: This requires quality metrics that evaluate visual interpretations and that enable an expert to select a suitable visualization for a given model and task at hand.

Original: However, both state-of-the-art visualization techniques and metrics that evaluate visual interpretations (e.g. Pixel Flipping (Samek et al., 2017), Sanity Check (Adebayo et al., 2018), and sensitivity checks (Rebufﬁ et al., 2020)) have so far only been examined on images (Rojat et al., 2021) or on NLP tasks (Arras et al., 2017).
Revised: However, both state-of-the-art visualization techniques and metrics that evaluate visual interpretations (e.g. Pixel Flipping (Samek et al., 2017), Sanity Check (Adebayo et al., 2018), and sensitivity checks (Rebufﬁ et al., 2020)) have so far only been examined on images (Rojat et al., 2021) or on NLP tasks (Arras et al., 2017).

Original: This lack of objective and subjective evaluation seriously limits the application and utility of them for time series.
Revised: This lack of objective and subjective evaluation seriously limits the application and utility of them for time series.

Original: In this paper, we propose to evaluate visual interpretations for time series combining six orthogonal metrics: ”sanity” (Adebayo et al., 2018), ”faithfulness” (Alvarez Melis & Jaakkola, 2018), ”sensitivity” (Rebufﬁ et al., 2020), ”robustness” (Yeh et al., 2019), ”stability” (Fel & Vigouroux, 2020; Li et al., 2021), and a novel metric based on human preferences: ”localization”.
Revised: In this paper, we propose to evaluate visual interpretations for time series combining six orthogonal metrics: ”sanity” (Adebayo et al., 2018), ”faithfulness” (Alvarez Melis & Jaakkola, 2018), ”sensitivity” (Rebufﬁ et al., 2020), ”robustness” (Yeh et al., 2019), ”stability” (Fel & Vigouroux, 2020; Li et al., 2021), and a novel metric based on human preferences: ”localization”.

Original: These metrics rate and validate distinct qualities of saliency.
Revised: These metrics rate and validate distinct qualities of saliency.

Original: Our metrics are both based on the functional perspective, i.e., based on the model-speciﬁc operation, and on how well they represent annotators’ semantics.
Revised: Our metrics are both based on the functional perspective, i.e., based on the model-speciﬁc operation, and on how well they represent annotators’ semantics.

Original: In an extensive evaluation, we train four different architectures on two different types of tasks:
Revised: In an extensive evaluation, we train four different architectures on two different types of tasks:

Original: UTime model (Perslev et al., 2019) and bidirectional Long Short-Term Memory (bi-LSTM) (Schuster & Paliwal, 1997) on segmentation tasks, and Fully Convolutional Network (FCN) (Long et al., 2015) and Temporal Convolutional Network (TCN) (Bai et al., 2018) on classiﬁcation tasks.
Revised: UTime model (Perslev et al., 2019) and bidirectional Long Short-Term Memory (bi-LSTM) (Schuster & Paliwal, 1997) on segmentation tasks, and Fully Convolutional Network (FCN) (Long et al., 2015) and Temporal Convolutional Network (TCN) (Bai et al., 2018) on classiﬁcation tasks.

Original: We use diverse datasets from the UCR repository (Dau et al., 2018)
Revised: We use diverse datasets from the UCR repository (Dau et al., 2018)

Original: (GunPointAgeSpan, FordA, FordB, ElectricDevices, MelbournePedestrian, NATOPS) and for segmentation the more complex real-world tool tracking dataset (L ¨ ofﬂer et al., 2021).
Revised: (GunPointAgeSpan, FordA, FordB, ElectricDevices, MelbournePedestrian, NATOPS) and for segmentation the more complex real-world tool tracking dataset (L ¨ ofﬂer et al., 2021).

Original: The experiments show the necessity of all categories to create an objective rating for methods, models and tasks, and allow domain experts to understand, rate, and validate saliency for time series in safety-critical applications.
Revised: The experiments show the necessity of all categories to create an objective rating for methods, models and tasks, and allow domain experts to understand, rate, and validate saliency for time series in safety-critical applications.

Original: The rest of the paper is structured as follows.
Revised: The rest of the paper is structured as follows.

Original: Section 2 discusses background and related work.
Revised: Section 2 discusses background and related work.

Original: Section 3 introduces extended and novel metrics for both the classiﬁcation and segmentation task.
Revised: Section 3 introduces extended and novel metrics for both the classiﬁcation and segmentation task.

Original: Section 4 discusses the experimental results and Section 5 proposes recommendations.
Revised: Section 4 discusses the experimental results and Section 5 proposes recommendations.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: Interpretation methods for DL models can be divided into ante-hoc methods, i.e., methods that are inherently part of the model, and post-hoc methods, i.e., methods that provide the interpretation after training (Rojat et al., 2021).
Revised: Interpretation methods for DL models can be divided into ante-hoc methods, i.e., methods that are inherently part of the model, and post-hoc methods, i.e., methods that provide the interpretation after training (Rojat et al., 2021).

Original: We focus on post-hoc methods and divide them into gradient-based and perturbation-based methods (Li et al., 2021; Warnecke et al., 2020; Ismail et al., 2020).
Revised: We focus on post-hoc methods and divide them into gradient-based and perturbation-based methods (Li et al., 2021; Warnecke et al., 2020; Ismail et al., 2020).

Original: Gradient-based methods compute the relevance for all input features by passing gradients backwards through the neural network (Ancona et al., 2018).
Revised: Gradient-based methods compute the relevance for all input features by passing gradients backwards through the neural network (Ancona et al., 2018).

Original: Gradient (Simonyan et al., 2014) computes a class c ’s saliency map M c using the derivative of the class score P c of the model with respect to the input sample x , as M c ( x ) =
Revised: Gradient (Simonyan et al., 2014) computes a class c ’s saliency map M c using the derivative of the class score P c of the model with respect to the input sample x , as M c ( x ) =

Original: ∂P
Revised: ∂P

Original: c ∂x .
Revised: c ∂x .

Original: The gradient indicates the importance of points in the input sequence for predictions.
Revised: The gradient indicates the importance of points in the input sequence for predictions.

Original: The advantage of gradient-based methods lies in their computational efﬁciency, as they use only a small number of backward passes to compute M c ( x ) .
Revised: The advantage of gradient-based methods lies in their computational efﬁciency, as they use only a small number of backward passes to compute M c ( x ) .

Original: Perturbation-based methods perturb known input samples and measure the effects of speciﬁc perturbations on the predicted class via a forward pass through the network.
Revised: Perturbation-based methods perturb known input samples and measure the effects of speciﬁc perturbations on the predicted class via a forward pass through the network.

Original: For instance, Local
Revised: For instance, Local

Original: Interpretable Model-Agnostic Explanations (LIME)
Revised: Interpretable Model-Agnostic Explanations (LIME)

Original: (Ribeiro et al., 2016) ﬁts a local surrogate model (e.g., a linear regression) as an explanation, and calculates relevance based on this surrogate.
Revised: (Ribeiro et al., 2016) ﬁts a local surrogate model (e.g., a linear regression) as an explanation, and calculates relevance based on this surrogate.

Original: Perturbation-based methods are computationally expensive as they require multiple forward passes per sample.
Revised: Perturbation-based methods are computationally expensive as they require multiple forward passes per sample.

Original: However, they do not need gradient information and work with black-box models.
Revised: However, they do not need gradient information and work with black-box models.

Original: 2. M ETRICS FOR S ALIENCY ON T IME S ERIES
Revised: 2. M ETRICS FOR S ALIENCY ON T IME S ERIES

Original: Most interpretation methods were originally designed for image or text data.
Revised: Most interpretation methods were originally designed for image or text data.

Original: Understanding and comparing visual interpretation is intuitive on image data, compared to more abstract time series.
Revised: Understanding and comparing visual interpretation is intuitive on image data, compared to more abstract time series.

Original: Furthermore, the diversity of interpretation methods complicates an objective choice for the model and task (Rojat et al., 2021).
Revised: Furthermore, the diversity of interpretation methods complicates an objective choice for the model and task (Rojat et al., 2021).

Original: For example, when Wang et al.
Revised: For example, when Wang et al.

Original: (2017) and Fawaz et al.
Revised: (2017) and Fawaz et al.

Original: (2019b) apply
Revised: (2019b) apply

Original: Class Activation Maps (CAM) (Zhou et al., 2016) on well-known UCR datasets (Dau et al., 2018), they notice a qualitative difference of CAM interpretations between network architectures.
Revised: Class Activation Maps (CAM) (Zhou et al., 2016) on well-known UCR datasets (Dau et al., 2018), they notice a qualitative difference of CAM interpretations between network architectures.

Original: Similarly, other work relies on domain experts that perform a (costly) qualitative evaluation (Strodthoff &
Revised: Similarly, other work relies on domain experts that perform a (costly) qualitative evaluation (Strodthoff &

Original: Strodthoff, 2019;
Revised: Strodthoff, 2019;

Original: Fawaz et al., 2019a; Oviedo et al., 2019).
Revised: Fawaz et al., 2019a; Oviedo et al., 2019).

Original: Hence, there is an increasing need for objective evaluation metrics to make the interpretations measurable and comparable.
Revised: Hence, there is an increasing need for objective evaluation metrics to make the interpretations measurable and comparable.

Original: [Figure]
Revised: [Figure]

Original: a y
Revised: a y

Original: [Figure]
Revised: [Figure]

Original: l a y
Revised: l a y

Original: [Figure]
Revised: [Figure]

Original: l a y
Revised: l a y

Original: [Figure]
Revised: [Figure]

Original: Metrics.
Revised: Metrics.

Original: There exists a large variety of different metrics.
Revised: There exists a large variety of different metrics.

Original: First, the stability of saliency maps can be evaluated with respect to the sensitivity of the majority class (Yeh et al., 2019).
Revised: First, the stability of saliency maps can be evaluated with respect to the sensitivity of the majority class (Yeh et al., 2019).

Original: This metric evaluates the change of interpretation when the input samples are attacked by adversarial noise.
Revised: This metric evaluates the change of interpretation when the input samples are attacked by adversarial noise.

Original: It is a special form of perturbation, that was so far only applied to image data.
Revised: It is a special form of perturbation, that was so far only applied to image data.

Original: Second, Rebufﬁ et al.
Revised: Second, Rebufﬁ et al.

Original: (2020) measure class sensitivity by comparing saliency maps of a min- and max-class.
Revised: (2020) measure class sensitivity by comparing saliency maps of a min- and max-class.

Original: Third, Cho et al.
Revised: Third, Cho et al.

Original: (2020) perturb the unimportant input samples and preserve only the important inputs.
Revised: (2020) perturb the unimportant input samples and preserve only the important inputs.

Original: If the predictions are stable compared to the unperturbed input, the interpretation is robust.
Revised: If the predictions are stable compared to the unperturbed input, the interpretation is robust.

Original: Similarly, Ates et al.
Revised: Similarly, Ates et al.

Original: (2021) evaluate the local robustness (i.e., similar samples should lead to similar interpretations) of the interpretation by measuring the ratio of change of the interpretation compared to the amount of input perturbation.
Revised: (2021) evaluate the local robustness (i.e., similar samples should lead to similar interpretations) of the interpretation by measuring the ratio of change of the interpretation compared to the amount of input perturbation.

Original: Finally, faithfulness is a perturbation-based metric that aims to measure the relationship between saliency and predictive features (Alvarez Melis & Jaakkola, 2018).
Revised: Finally, faithfulness is a perturbation-based metric that aims to measure the relationship between saliency and predictive features (Alvarez Melis & Jaakkola, 2018).

Original: Schlegel et al.
Revised: Schlegel et al.

Original: (2019) focus on continuous univariate time series.
Revised: (2019) focus on continuous univariate time series.

Original: Sub-sequences with high relevance values are perturbed either by swapping their samples or replacing them with a mean value.
Revised: Sub-sequences with high relevance values are perturbed either by swapping their samples or replacing them with a mean value.

Original: Similarly, Ismail et al.
Revised: Similarly, Ismail et al.

Original: propose input perturbations to evaluate models on synthetic time series.
Revised: propose input perturbations to evaluate models on synthetic time series.

Original: As they perturb inputs point-wise instead of sub-sequences their metric does not consider time dependency between consecutive points.
Revised: As they perturb inputs point-wise instead of sub-sequences their metric does not consider time dependency between consecutive points.

Original: Our faithfulness metric perturbs sub-sequences of high saliency (Schlegel et al., 2019).
Revised: Our faithfulness metric perturbs sub-sequences of high saliency (Schlegel et al., 2019).

Original: We adapt each type of metric into our set to provide a visual interpretation of time series.
Revised: We adapt each type of metric into our set to provide a visual interpretation of time series.

Original: Categories of metrics.
Revised: Categories of metrics.

Original: Metrics can be divided into different categories, depending on the question they answer.
Revised: Metrics can be divided into different categories, depending on the question they answer.

Original: Doshi-Velez & Kim (2017) propose a distinction between human-grounded and functional metrics.
Revised: Doshi-Velez & Kim (2017) propose a distinction between human-grounded and functional metrics.

Original: The former involve human perception and intuition with the goal of generating qualitative, intuitive visualizations, e.g., bounding boxes for image object detection for testing the localization of saliency maps (Jianming et al., 2016), or questionnaires to indicate testers’ opinions on the quality of explanations (Li et al., 2021).
Revised: The former involve human perception and intuition with the goal of generating qualitative, intuitive visualizations, e.g., bounding boxes for image object detection for testing the localization of saliency maps (Jianming et al., 2016), or questionnaires to indicate testers’ opinions on the quality of explanations (Li et al., 2021).

Original: Functional metrics provide statistical measures, e.g., to aggregate performance metrics automatically (Rojat et al., 2021), and make use of proxy tasks to generate a quantitative evaluation.
Revised: Functional metrics provide statistical measures, e.g., to aggregate performance metrics automatically (Rojat et al., 2021), and make use of proxy tasks to generate a quantitative evaluation.

Original: Another categorization (Li et al., 2021) identiﬁes multiple broad categories for image data, that we may transfer to time series, i.e., the faithfulness of salient features, the class sensitivity of an explanation, and the stability of explanations given noisy inputs.
Revised: Another categorization (Li et al., 2021) identiﬁes multiple broad categories for image data, that we may transfer to time series, i.e., the faithfulness of salient features, the class sensitivity of an explanation, and the stability of explanations given noisy inputs.

Original: This paper proposes a framework of metrics with orthogonal categories, speciﬁcally to time series.
Revised: This paper proposes a framework of metrics with orthogonal categories, speciﬁcally to time series.

Original: We adapt and extend metrics to (multivariate) time series and propose an intra-class stability metric and a concept of relevance localization: we build on the pointing game (Jianming et al., 2016) and combine it with the precision and recall for time series framework (Tatbul et al., 2018).
Revised: We adapt and extend metrics to (multivariate) time series and propose an intra-class stability metric and a concept of relevance localization: we build on the pointing game (Jianming et al., 2016) and combine it with the precision and recall for time series framework (Tatbul et al., 2018).

Original: To our knowledge we are the ﬁrst to apply and evaluate sanity check (Adebayo et al., 2018) on time series.
Revised: To our knowledge we are the ﬁrst to apply and evaluate sanity check (Adebayo et al., 2018) on time series.

Original:  S CORING C ATEGORIES
Revised:  S CORING C ATEGORIES

Original: We propose a set of six distinct categories (sanity, faithfulness, sensitivity, robustness, stability, and localization) to assess visual interpretation methods and to determine their performance and trustworthiness in classiﬁcation or segmentation tasks on time series.
Revised: We propose a set of six distinct categories (sanity, faithfulness, sensitivity, robustness, stability, and localization) to assess visual interpretation methods and to determine their performance and trustworthiness in classiﬁcation or segmentation tasks on time series.

Original: For each of them we propose a metric that enables a comparative evaluation of diverse types of visual interpretation methods.
Revised: For each of them we propose a metric that enables a comparative evaluation of diverse types of visual interpretation methods.

Original: F a i t
Revised: F a i t

Original: h
Revised: h

Original: f u l n e ss
Revised: f u l n e ss

Original: I n t r a - C l a ss
Revised: I n t r a - C l a ss

Original: S t a b ili t y
Revised: S t a b ili t y

Original: R o b u s t n e ss
Revised: R o b u s t n e ss

Original: S a n i t y
Revised: S a n i t y

Original: S e n s i t i v i
Revised: S e n s i t i v i

Original: t y
Revised: t y

Original: Faithfulness
Revised: Faithfulness

Original: Intra-Class Stability
Revised: Intra-Class Stability

Original: Robustness
Revised: Robustness

Original: Sanity
Revised: Sanity

Original: Sensitivity 0.2 -0.00029 -0.17 -0.
Revised: Sensitivity 0.2 -0.00029 -0.17 -0.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Why do we need six scoring categories?
Revised: Why do we need six scoring categories?

Original: It seems tempting to rely on a single metric or on a single aggregated score across multiple metrics to assess the quality of a visual interpretation.
Revised: It seems tempting to rely on a single metric or on a single aggregated score across multiple metrics to assess the quality of a visual interpretation.

Original: However, we show that the six presented categories are inherently orthogonal (see Fig. 2) and capture distinct qualities.
Revised: However, we show that the six presented categories are inherently orthogonal (see Fig. 2) and capture distinct qualities.

Original: Interpretations depend on model parameters (sanity check (Adebayo et al., 2018)), predictive features (faithfulness (Alvarez Melis &
Revised: Interpretations depend on model parameters (sanity check (Adebayo et al., 2018)), predictive features (faithfulness (Alvarez Melis &

Original: Jaakkola, 2018)), coherence of class predictions (intra-class stability (Fel & Vigouroux, 2020)), robustness against noise (max sensitivity with adversarial noise (Yeh et al., 2019)), speciﬁc sequences (or even points) in a time series like by Tatbul et al.
Revised: Jaakkola, 2018)), coherence of class predictions (intra-class stability (Fel & Vigouroux, 2020)), robustness against noise (max sensitivity with adversarial noise (Yeh et al., 2019)), speciﬁc sequences (or even points) in a time series like by Tatbul et al.

Original: (2018) (novel localization metric), and the relevance map’s speciﬁcity (interclass sensitivity (Rebufﬁ et al., 2020)).
Revised: (2018) (novel localization metric), and the relevance map’s speciﬁcity (interclass sensitivity (Rebufﬁ et al., 2020)).

Original: It is important to assess if a given interpretation accurately captures these dependencies.
Revised: It is important to assess if a given interpretation accurately captures these dependencies.

Original: We give a brief overview of our frameworks’ functional (Fig.
Revised: We give a brief overview of our frameworks’ functional (Fig.

Original: 1a-e) and human-grounded metrics (Fig. 1f) that we evaluate with time series data.
Revised: 1a-e) and human-grounded metrics (Fig. 1f) that we evaluate with time series data.

Original: (a)
Revised: (a)

Original: Notation.
Revised: Notation.

Original: We follow the notation adapted from Fawaz et al.
Revised: We follow the notation adapted from Fawaz et al.

Original: (2019b):
Revised: (2019b):

Original: A multivariate time series is deﬁned by X =
Revised: A multivariate time series is deﬁned by X =

Original: [ X 1 , ..., X H ] , where H is the number of input channels, X i =
Revised: [ X 1 , ..., X H ] , where H is the number of input channels, X i =

Original: ( x i 1 , ..., x iT ) ∈ R T is an ordered set of real values, and T denotes the number of time steps.
Revised: ( x i 1 , ..., x iT ) ∈ R T is an ordered set of real values, and T denotes the number of time steps.

Original: For H equal to 1, we consider a univariate time series, otherwise we consider a multivariate time series.
Revised: For H equal to 1, we consider a univariate time series, otherwise we consider a multivariate time series.

Original: Time series often include complex temporal dependencies, i e., distinct points are not independent.
Revised: Time series often include complex temporal dependencies, i e., distinct points are not independent.

Original: Time series classiﬁcation deﬁnes a mapping X
Revised: Time series classiﬁcation deﬁnes a mapping X

Original: → y that minimizes the error on a dataset D
Revised: → y that minimizes the error on a dataset D

Original: = { ( X 1 , y 1 ) , ..., ( X N , y N ) } , where N is the number of data samples, X ∈ D is a time series, y i ∈ R C denotes the one-hot vector of a class label the input belongs to, and C is the number of classes.
Revised: = { ( X 1 , y 1 ) , ..., ( X N , y N ) } , where N is the number of data samples, X ∈ D is a time series, y i ∈ R C denotes the one-hot vector of a class label the input belongs to, and C is the number of classes.

Original: In time series segmentation, we search X
Revised: In time series segmentation, we search X

Original: → Y that maps an input sample to a dense classiﬁcation Y =
Revised: → Y that maps an input sample to a dense classiﬁcation Y =

Original: [ y 1 , ..., y T ] ∈ R C × T (Perslev et al., 2019), i.e., a class label is predicted for each time step.
Revised: [ y 1 , ..., y T ] ∈ R C × T (Perslev et al., 2019), i.e., a class label is predicted for each time step.

Original: Post-hoc visual interpretation methods compute a relevance map M cm ∈ R H × T , M cm ( X ) =
Revised: Post-hoc visual interpretation methods compute a relevance map M cm ∈ R H × T , M cm ( X ) =

Original: [ R 1 , ..., R H ] , where
Revised: [ R 1 , ..., R H ] , where

Original: R i =
Revised: R i =

Original: ( r i 1 , ..., r iT ) , representing the importance of each input feature with respect to the class c and a model m , for each time step.
Revised: ( r i 1 , ..., r iT ) , representing the importance of each input feature with respect to the class c and a model m , for each time step.

Original: We use M as a function to produce the saliency map.
Revised: We use M as a function to produce the saliency map.

Original: For clarity, we will omit the dependency on m , i.e., M cm ≡ M c , if it is not explicitly required.
Revised: For clarity, we will omit the dependency on m , i.e., M cm ≡ M c , if it is not explicitly required.

Original: An evaluation metric for visual interpretation methods deﬁnes a score S metric ( · ) that rates the quality of the relevance map
Revised: An evaluation metric for visual interpretation methods deﬁnes a score S metric ( · ) that rates the quality of the relevance map

Original: M at a sample X given a model m and optional parameters.
Revised: M at a sample X given a model m and optional parameters.

Original: We provide a uniﬁed view, so that for all scores, a higher score corresponds to a better visualization according to the perspective.
Revised: We provide a uniﬁed view, so that for all scores, a higher score corresponds to a better visualization according to the perspective.

Original: 3. S ANITY
Revised: 3. S ANITY

Original: Intuitively, if the weights and biases of trained networks models were re-initialized with random values, the networks predictions and generated saliency maps should also be different from the original maps.
Revised: Intuitively, if the weights and biases of trained networks models were re-initialized with random values, the networks predictions and generated saliency maps should also be different from the original maps.

Original: However, this is not always the case.
Revised: However, this is not always the case.

Original: Despite a drop in model accuracy, saliency may remain stable.
Revised: Despite a drop in model accuracy, saliency may remain stable.

Original: Hence, we test sanity using a variant of sanity check (Adebayo et al., 2018), that performs a layer-wise cascading randomization of the network’s weights and biases, starting from the output to the input.
Revised: Hence, we test sanity using a variant of sanity check (Adebayo et al., 2018), that performs a layer-wise cascading randomization of the network’s weights and biases, starting from the output to the input.

Original: In contrast to independent randomization, the cascading approach results in a mostly continuous performance degradation of predictions, see Fig.
Revised: In contrast to independent randomization, the cascading approach results in a mostly continuous performance degradation of predictions, see Fig.

Original: 1a for an illustration.
Revised: 1a for an illustration.

Original: Network accuracy should increasingly resemble random guessing.
Revised: Network accuracy should increasingly resemble random guessing.

Original: Following Adebayo et al.
Revised: Following Adebayo et al.

Original: (2018), we compare saliency using the structural similarity index measure (SSIM) (Wang et al., 2004), that compares the distribution of sub-sequences of M c .
Revised: (2018), we compare saliency using the structural similarity index measure (SSIM) (Wang et al., 2004), that compares the distribution of sub-sequences of M c .

Original: We deﬁne the sanity score as
Revised: We deﬁne the sanity score as

Original: [Equation]
Revised: [Equation]

Original: where | D | =
Revised: where | D | =

Original: N , i enumerates the L layers of m , whose parameters are randomized.
Revised: N , i enumerates the L layers of m , whose parameters are randomized.

Original: M cm i ( x ) is the saliency map after randomizing layer i of m .
Revised: M cm i ( x ) is the saliency map after randomizing layer i of m .

Original: We average the SSIM over L layers and compute the average over all samples in D .
Revised: We average the SSIM over L layers and compute the average over all samples in D .

Original: 3. F AITHFULNESS
Revised: 3. F AITHFULNESS

Original: A relevance measure is faithful, if input features with a high relevance (w.r.t.
Revised: A relevance measure is faithful, if input features with a high relevance (w.r.t.

Original: M c ) have a high inﬂuence on the model prediction, see Fig.
Revised: M c ) have a high inﬂuence on the model prediction, see Fig.

Original: 1b for an example.
Revised: 1b for an example.

Original: Alvarez Melis & Jaakkola (2018) propose a perturbation-based metric that evaluates the faithfulness of predictions.
Revised: Alvarez Melis & Jaakkola (2018) propose a perturbation-based metric that evaluates the faithfulness of predictions.

Original: The metric measures the correlation between input (features) with high saliency on the one hand and predictive accuracy on the other hand.
Revised: The metric measures the correlation between input (features) with high saliency on the one hand and predictive accuracy on the other hand.

Original: We choose a sub-sequence of the input to perturb by ranking each time step of
Revised: We choose a sub-sequence of the input to perturb by ranking each time step of

Original: X T ∈ R T according to the relevance of its saliency map.
Revised: X T ∈ R T according to the relevance of its saliency map.

Original: Next, we perturb connected sub-sequences similar to Schlegel et al.
Revised: Next, we perturb connected sub-sequences similar to Schlegel et al.

Original: We extend the metric to multivariate data X HT :
Revised: We extend the metric to multivariate data X HT :

Original: Based on the relevance r , we select time points t ∈ { 1 , ..., T } and features i ∈ { 1 , ..., H } .
Revised: Based on the relevance r , we select time points t ∈ { 1 , ..., T } and features i ∈ { 1 , ..., H } .

Original: We select t according to the maximum relevance r it and set t as the middle point of the sub sequence.
Revised: We select t according to the maximum relevance r it and set t as the middle point of the sub sequence.

Original: Therefore, we select and reverse the sequence as x rev =
Revised: Therefore, we select and reverse the sequence as x rev =

Original: ( x it + L 2 , ..., x it , ..., x it − L 2 ) to break the temporal correlation.
Revised: ( x it + L 2 , ..., x it , ..., x it − L 2 ) to break the temporal correlation.

Original: Next, we insert it back into a copy of X that we denote as X (cid:48) .
Revised: Next, we insert it back into a copy of X that we denote as X (cid:48) .

Original: With this perturbation we compute the mean faithfulness score over the whole dataset D as
Revised: With this perturbation we compute the mean faithfulness score over the whole dataset D as

Original: [Equation]
Revised: [Equation]

Original: where m c : R c → R is the softmax prediction of the target class c .
Revised: where m c : R c → R is the softmax prediction of the target class c .

Original: m c ( X ) − m c
Revised: m c ( X ) − m c

Original: ( X (cid:48) ) is the gap score between the softmax prediction of the original and the perturbed sample.
Revised: ( X (cid:48) ) is the gap score between the softmax prediction of the original and the perturbed sample.

Original: 3. I NTER -C LASS S ENSITIVITY
Revised: 3. I NTER -C LASS S ENSITIVITY

Original: In multi-class prediction tasks the classiﬁer needs to identify relevant features for each of the classes to make a correct prediction.
Revised: In multi-class prediction tasks the classiﬁer needs to identify relevant features for each of the classes to make a correct prediction.

Original: Hence, the relevance map M c should identify different salient features for those different classes (Li et al., 2021), see Fig. 1c.
Revised: Hence, the relevance map M c should identify different salient features for those different classes (Li et al., 2021), see Fig. 1c.

Original: If a method is not sensitive to the class, the saliency could be misleading, especially if the classiﬁer fails to learn correct features for some classes.
Revised: If a method is not sensitive to the class, the saliency could be misleading, especially if the classiﬁer fails to learn correct features for some classes.

Original: Inter-class sensitivity (Rebufﬁ et al., 2020) measures class speciﬁc sensitivity of the generated relevance map with respect to the most ( c max ) and least ( c min ) likely class according to the model.
Revised: Inter-class sensitivity (Rebufﬁ et al., 2020) measures class speciﬁc sensitivity of the generated relevance map with respect to the most ( c max ) and least ( c min ) likely class according to the model.

Original: We compute the mean inter-class sensitivity score as
Revised: We compute the mean inter-class sensitivity score as

Original: [Equation]
Revised: [Equation]

Original: We compute similarity of two saliency maps as sim ( M c max ( x ) , M c min ( x )) where sim ( · , · ) is a similarity function (e.g. a cosine similarity) that is easy to interpret via its geometric interpretation.
Revised: We compute similarity of two saliency maps as sim ( M c max ( x ) , M c min ( x )) where sim ( · , · ) is a similarity function (e.g. a cosine similarity) that is easy to interpret via its geometric interpretation.

Original: It is deﬁned as the angle between two non-zero vectors that measures the similarity between their inner product space (Han et al., 2012).
Revised: It is deﬁned as the angle between two non-zero vectors that measures the similarity between their inner product space (Han et al., 2012).

Original: Similarity of M in binary classiﬁcation would result in a negative cosine similarity, meaning nearly inverted saliency maps for max- and min-classes.
Revised: Similarity of M in binary classiﬁcation would result in a negative cosine similarity, meaning nearly inverted saliency maps for max- and min-classes.

Original: 3. R OBUSTNESS
Revised: 3. R OBUSTNESS

Original: We evaluate a method’s robustness with respect to adversarial noise (Yeh et al., 2019) via its sensitivity of the most likely class ( c max ) see Fig.
Revised: We evaluate a method’s robustness with respect to adversarial noise (Yeh et al., 2019) via its sensitivity of the most likely class ( c max ) see Fig.

Original: 1d.
Revised: 1d.

Original: Intuitively, even with noisy inputs the saliency map of a model should not change signiﬁcantly (Alvarez-Melis & Jaakkola, 2018) and saliency maps with high sensitivity are less reliable.
Revised: Intuitively, even with noisy inputs the saliency map of a model should not change signiﬁcantly (Alvarez-Melis & Jaakkola, 2018) and saliency maps with high sensitivity are less reliable.

Original: Yeh et al.
Revised: Yeh et al.

Original: (2019) deﬁne the sensitivity of the saliency map derived from the gradient as
Revised: (2019) deﬁne the sensitivity of the saliency map derived from the gradient as

Original: [Equation]
Revised: [Equation]

Original: for any j ∈ { 1 , ..., | H · T |} , where e j ∈ R H
Revised: for any j ∈ { 1 , ..., | H · T |} , where e j ∈ R H

Original: × T is the j -th coordinate vector and the j -th entry is one while the others are zero.
Revised: × T is the j -th coordinate vector and the j -th entry is one while the others are zero.

Original: We use Monte Carlo sampling of (cid:15)e j , where | (cid:15)e j | <a ( a is a user-speciﬁed radius), to generate different ˆ X =
Revised: We use Monte Carlo sampling of (cid:15)e j , where | (cid:15)e j | <a ( a is a user-speciﬁed radius), to generate different ˆ X =

Original: X
Revised: X

Original: + (cid:15)e j .
Revised: + (cid:15)e j .

Original: We compare ˆ X with the original X to compute
Revised: We compare ˆ X with the original X to compute

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 3. I NTRA -C LASS S TABILITY
Revised: 3. I NTRA -C LASS S TABILITY

Original: A method is stable if similar input samples from the same class produce similar interpretations (Fel & Vigouroux, 2020).
Revised: A method is stable if similar input samples from the same class produce similar interpretations (Fel & Vigouroux, 2020).

Original: We implement this concept to test the intra-class stability of saliency maps
Revised: We implement this concept to test the intra-class stability of saliency maps

Original: M c for a given dataset D using distance statistics, see Fig.
Revised: M c for a given dataset D using distance statistics, see Fig.

Original: 1e for an example.
Revised: 1e for an example.

Original: To this end, we compute pairwise distances between saliency maps M c for different samples
Revised: To this end, we compute pairwise distances between saliency maps M c for different samples

Original: X i , X j ∈ D .
Revised: X i , X j ∈ D .

Original: Then, we aggregate these distances for all samples from one class:
Revised: Then, we aggregate these distances for all samples from one class:

Original: [Equation]
Revised: [Equation]

Original: We use Dynamic Time Warping (Vintsyuk, 1968) as the distance function d DTW .
Revised: We use Dynamic Time Warping (Vintsyuk, 1968) as the distance function d DTW .

Original: The score compares each classes’ sample’s M c with all other samples’ saliency maps in the dataset, using the distance d DTW .
Revised: The score compares each classes’ sample’s M c with all other samples’ saliency maps in the dataset, using the distance d DTW .

Original: In summary, the mean distance for a class should be small and the variance low.
Revised: In summary, the mean distance for a class should be small and the variance low.

Original: 3. L OCALIZATION
Revised: 3. L OCALIZATION

Original: The temporal location of class-speciﬁc features with high relevance in a time series segmentation task should be situated within (or close-by) its labeled segment (temporal sub-sequence), see Fig. 1f.
Revised: The temporal location of class-speciﬁc features with high relevance in a time series segmentation task should be situated within (or close-by) its labeled segment (temporal sub-sequence), see Fig. 1f.

Original: Due to its connection to the annotation task we call it a human-grounded metric.
Revised: Due to its connection to the annotation task we call it a human-grounded metric.

Original: Speciﬁcally, we propose a novel metric that adapts Pointing Game (Jianming et al., 2016) used for object detection to time series using range-based metrics (Tatbul et al., 2018).
Revised: Speciﬁcally, we propose a novel metric that adapts Pointing Game (Jianming et al., 2016) used for object detection to time series using range-based metrics (Tatbul et al., 2018).

Original: We argue that Pointing Game’s original hit-and-miss accuracy is inadequate for time-series methods, as Segmentation is point-wise exact.
Revised: We argue that Pointing Game’s original hit-and-miss accuracy is inadequate for time-series methods, as Segmentation is point-wise exact.

Original: Thus, we replace it with range-based testing, yielding Localization.
Revised: Thus, we replace it with range-based testing, yielding Localization.

Original: Furthermore, we observed that some saliency methods can show strong temporal biases towards the beginning, middle or end of a segmented class.
Revised: Furthermore, we observed that some saliency methods can show strong temporal biases towards the beginning, middle or end of a segmented class.

Original: We would expect high saliency for features within the predicted segment, and low saliency otherwise, to accurately capture the model’s predictive behavior.
Revised: We would expect high saliency for features within the predicted segment, and low saliency otherwise, to accurately capture the model’s predictive behavior.

Original: Hence, Localization measures agreement between annotators, models’ predictions and saliency, and can discover issues like temporal biases and imprecise margins.
Revised: Hence, Localization measures agreement between annotators, models’ predictions and saliency, and can discover issues like temporal biases and imprecise margins.

Original: First, we ﬁlter out non-relevant predictions (according to the relevance map) as follows.
Revised: First, we ﬁlter out non-relevant predictions (according to the relevance map) as follows.

Original: A prediction at time t , denoted with ˆ Y ( t ) , is relevant if there exists an i ∈ { 1 , ..., M } such that r it > max( | r | ) · θ .
Revised: A prediction at time t , denoted with ˆ Y ( t ) , is relevant if there exists an i ∈ { 1 , ..., M } such that r it > max( | r | ) · θ .

Original: We select the model’s predicted class at time t if the prediction is relevant, otherwise we set the class to none .
Revised: We select the model’s predicted class at time t if the prediction is relevant, otherwise we set the class to none .

Original: The resulting relevancy-ﬁltered prediction is denoted as ˆ Y (cid:48) .
Revised: The resulting relevancy-ﬁltered prediction is denoted as ˆ Y (cid:48) .

Original: Finally, we can compare ˆ Y (cid:48) with the ground truth Y and evaluate how well high saliency features lie within the annotated sub-sequences (Tatbul et al., 2018).
Revised: Finally, we can compare ˆ Y (cid:48) with the ground truth Y and evaluate how well high saliency features lie within the annotated sub-sequences (Tatbul et al., 2018).

Original: We compute the mean localization score for the whole dataset D .
Revised: We compute the mean localization score for the whole dataset D .

Original: We compare any of the N sub existing labeled sub-sequence Y sub i ∈ R T from D each with its temporally co-located prediction ˆ Y (cid:48)
Revised: We compare any of the N sub existing labeled sub-sequence Y sub i ∈ R T from D each with its temporally co-located prediction ˆ Y (cid:48)

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For each pair ( Y sub i , ˆ Y (cid:48)
Revised: For each pair ( Y sub i , ˆ Y (cid:48)

Original: i )
Revised: i )

Original: we calculate a range-based recall score based on the point-wise comparison proposed by Tatbul et al.
Revised: we calculate a range-based recall score based on the point-wise comparison proposed by Tatbul et al.

Original: (2018) as
Revised: (2018) as

Original: [Equation]
Revised: [Equation]

Original: where α weighs the two reward terms for ”existence” and ”overlap”.
Revised: where α weighs the two reward terms for ”existence” and ”overlap”.

Original: Existence is 1 if any time point was correctly predicted within the labeled region, and
Revised: Existence is 1 if any time point was correctly predicted within the labeled region, and

Original: 0 otherwise.
Revised: 0 otherwise.

Original: The parameterized ”overlap” function determines the ﬁner properties of cardinality, size and position.
Revised: The parameterized ”overlap” function determines the ﬁner properties of cardinality, size and position.

Original: The cardinality parameter discounts the score if the prediction is an interrupted, fragment range instead of being continuous.
Revised: The cardinality parameter discounts the score if the prediction is an interrupted, fragment range instead of being continuous.

Original: The overall size of the overlap of predicted and label ranges depends on a positional bias.
Revised: The overall size of the overlap of predicted and label ranges depends on a positional bias.

Original: It may favor ”front”, ”middle”, or ”back” overlap.
Revised: It may favor ”front”, ”middle”, or ”back” overlap.

Original: Practically, for some applications an early detection is preferable over a late detection.
Revised: Practically, for some applications an early detection is preferable over a late detection.

Original: For further details see Appendix A or Tatbul et al.
Revised: For further details see Appendix A or Tatbul et al.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our evaluation compares visual interpretation methods on a set of network architectures and diverse classiﬁcation datasets from the UCR repository, and on a more complex segmentation dataset.
Revised: Our evaluation compares visual interpretation methods on a set of network architectures and diverse classiﬁcation datasets from the UCR repository, and on a more complex segmentation dataset.

Original: In total, we aggregate the results from 540 experiments.
Revised: In total, we aggregate the results from 540 experiments.

Original: This section is divided into two parts.
Revised: This section is divided into two parts.

Original: We ﬁrst discuss the new localization metric for the segmentation task.
Revised: We ﬁrst discuss the new localization metric for the segmentation task.

Original: Next, we discuss the metrics for the classiﬁcation task (faithfulness, sensitivity, stability, robustness, sanity).
Revised: Next, we discuss the metrics for the classiﬁcation task (faithfulness, sensitivity, stability, robustness, sanity).

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: We evaluate nine visual interpretation methods: Gradient (Simonyan et al., 2014), Integrated Gradient (Sundararajan et al., 2017), SmoothGrad (Smilkov et al., 2017), Guided Backpropagation (Springenberg et al., 2015), GradCAM (Selvaraju et al., 2017), Guided-GradCAM (Selvaraju et al., 2017),
Revised: We evaluate nine visual interpretation methods: Gradient (Simonyan et al., 2014), Integrated Gradient (Sundararajan et al., 2017), SmoothGrad (Smilkov et al., 2017), Guided Backpropagation (Springenberg et al., 2015), GradCAM (Selvaraju et al., 2017), Guided-GradCAM (Selvaraju et al., 2017),

Original: Layer-Wise Relevance Propagation (LRP) (Bach et al., 2015), LIME (Ribeiro et al., 2016) and Kernel SHAP (Lundberg & Lee, 2017).
Revised: Layer-Wise Relevance Propagation (LRP) (Bach et al., 2015), LIME (Ribeiro et al., 2016) and Kernel SHAP (Lundberg & Lee, 2017).

Original: See Appendix B.2 for method-speciﬁc hyper parameters.
Revised: See Appendix B.2 for method-speciﬁc hyper parameters.

Original: For the segmentation task, we consider two models: U-time (Perslev et al., 2019) (derived for time series from U-Net (Ronneberger et al., 2015)), and a bi-LSTM (Schuster & Paliwal, 1997).
Revised: For the segmentation task, we consider two models: U-time (Perslev et al., 2019) (derived for time series from U-Net (Ronneberger et al., 2015)), and a bi-LSTM (Schuster & Paliwal, 1997).

Original: We provide details on each architecture in Appendix B.3.
Revised: We provide details on each architecture in Appendix B.3.

Original: We evaluate them on the tool tracking (L¨ofﬂer et al., 2021) dataset, a complex multivariate, multi-class time series from a 9-D magneto-inertial sensor, and use the electric screwdriver’s data.
Revised: We evaluate them on the tool tracking (L¨ofﬂer et al., 2021) dataset, a complex multivariate, multi-class time series from a 9-D magneto-inertial sensor, and use the electric screwdriver’s data.

Original: For the classiﬁcation task, we select two commonly used model architectures, i.e, a Fully Convolutional Network (FCN)
Revised: For the classiﬁcation task, we select two commonly used model architectures, i.e, a Fully Convolutional Network (FCN)

Original: (Long et al., 2015) and a Temporal Convolutional Network (TCN) (Bai et al., 2018).
Revised: (Long et al., 2015) and a Temporal Convolutional Network (TCN) (Bai et al., 2018).

Original: We do not focus on simpler architectures such as Multilayer Perceptrons and LSTMs (Hochreiter & Schmidhuber, 1997) due to noisy or vanishing saliency in preliminary studies.
Revised: We do not focus on simpler architectures such as Multilayer Perceptrons and LSTMs (Hochreiter & Schmidhuber, 1997) due to noisy or vanishing saliency in preliminary studies.

Original: Our dataset selection follows related work (Fawaz et al., 2019b;
Revised: Our dataset selection follows related work (Fawaz et al., 2019b;

Original: Wang et al., 2017; Schlegel et al., 2019; Ates et al., 2021) that uses large, multivariate, multi-class datasets from diverse domains (GunPointAgeSpan, FordA, FordB, ElectricDevices,
Revised: Wang et al., 2017; Schlegel et al., 2019; Ates et al., 2021) that uses large, multivariate, multi-class datasets from diverse domains (GunPointAgeSpan, FordA, FordB, ElectricDevices,

Original: MelbournePedestrian and NATOPS from the UCR repository (Dau et al., 2018)).
Revised: MelbournePedestrian and NATOPS from the UCR repository (Dau et al., 2018)).

Original: 4. E VALUATION ON S EGMENTATION T ASK
Revised: 4. E VALUATION ON S EGMENTATION T ASK

Original: This section presents the key ﬁndings from our evaluation when applying the localization metric on saliency maps generated from U-time (results for bi-LSTM are in Appendix C.1), trained for a segmentation task on the tool tracking dataset.
Revised: This section presents the key ﬁndings from our evaluation when applying the localization metric on saliency maps generated from U-time (results for bi-LSTM are in Appendix C.1), trained for a segmentation task on the tool tracking dataset.

Original: Fig.
Revised: Fig.

Original: 3a shows the localization metrics’ positional biases ”classic”, ”front”, ”middle” and ”back” for each visualization method.
Revised: 3a shows the localization metrics’ positional biases ”classic”, ”front”, ”middle” and ”back” for each visualization method.

Original: Fig.
Revised: Fig.

Original: 3b shows an exemplary saliency map for each method.
Revised: 3b shows an exemplary saliency map for each method.

Original: [Figure]
Revised: [Figure]

Original: GradCAM clearly outperforms the other methods on the localization metric.
Revised: GradCAM clearly outperforms the other methods on the localization metric.

Original: It’s saliency maps are smoother, with high relevance located on the annotated segments.
Revised: It’s saliency maps are smoother, with high relevance located on the annotated segments.

Original: This may result from its coarser relevance, as its gradients do not ﬂow to the input but stop at the last convolution layer.
Revised: This may result from its coarser relevance, as its gradients do not ﬂow to the input but stop at the last convolution layer.

Original: In contrast, LRP and Integrated
Revised: In contrast, LRP and Integrated

Original: Gradient perform worst.
Revised: Gradient perform worst.

Original: As Fig.
Revised: As Fig.

Original: 3b shows, the saliency maps highlight fewer relevant input features for the segmentation task, but instead features within the labeled sub-sequence that may be important for a classiﬁcation.
Revised: 3b shows, the saliency maps highlight fewer relevant input features for the segmentation task, but instead features within the labeled sub-sequence that may be important for a classiﬁcation.

Original: Saliency maps are located better in the ”middle” of the annotated segments, compared to ”front” and ”back”, see Fig. 3a.
Revised: Saliency maps are located better in the ”middle” of the annotated segments, compared to ”front” and ”back”, see Fig. 3a.

Original: This is also visible in the example in Fig.
Revised: This is also visible in the example in Fig.

Original: 3b where the relevance values start decreasing below the threshold of 0 .
Revised: 3b where the relevance values start decreasing below the threshold of 0 .

Original: 5 already before the end of the labeled segment at 100 ms .
Revised: 5 already before the end of the labeled segment at 100 ms .

Original: This points towards a biased labeling process with longer sub-sequences assigned to the classes than necessary, and can help identifying faulty labels.
Revised: This points towards a biased labeling process with longer sub-sequences assigned to the classes than necessary, and can help identifying faulty labels.

Original: Furthermore, the high variance of most results (except for GradCAM) reveals that saliency maps for time series segmentation are noisy.
Revised: Furthermore, the high variance of most results (except for GradCAM) reveals that saliency maps for time series segmentation are noisy.

Original: This is especially the case for Kernel SHAP, see Fig.
Revised: This is especially the case for Kernel SHAP, see Fig.

Original: 3b, where relevance is assigned seemingly at random.
Revised: 3b, where relevance is assigned seemingly at random.

Original: In summary, GradCAM performs best with respect to the human-grounded metric of localization, compared to all other methods in this study.
Revised: In summary, GradCAM performs best with respect to the human-grounded metric of localization, compared to all other methods in this study.

Original: It exhibits low variance in the results and focus on the ”middle” segment, while other methods hardly align with annotated relevant segments in the input features or their saliency maps are noisy.
Revised: It exhibits low variance in the results and focus on the ”middle” segment, while other methods hardly align with annotated relevant segments in the input features or their saliency maps are noisy.

Original: Conversely, no method works reliably on the bi-LSTM.
Revised: Conversely, no method works reliably on the bi-LSTM.

Original: We identify a lack of useful visualization methods for LSTMs, as CAM variants (GradCAM, Guided GradCAM, and Guided Backprop) are not applicable, see Fig.
Revised: We identify a lack of useful visualization methods for LSTMs, as CAM variants (GradCAM, Guided GradCAM, and Guided Backprop) are not applicable, see Fig.

Original: 6a and 6b in Appendix C.1 for detailed results.
Revised: 6a and 6b in Appendix C.1 for detailed results.

Original: 4. E VALUATION ON C LASSIFICATION T ASK
Revised: 4. E VALUATION ON C LASSIFICATION T ASK

Original: In this section, we discuss the ﬁve visual interpretation quality metrics for the classiﬁcation task.
Revised: In this section, we discuss the ﬁve visual interpretation quality metrics for the classiﬁcation task.

Original: Fig.
Revised: Fig.

Original: 4a summarizes the scores for each metric over model architectures (FCN, TCN) and datasets.
Revised: 4a summarizes the scores for each metric over model architectures (FCN, TCN) and datasets.

Original: Sanity.
Revised: Sanity.

Original: Across all datasets, we ﬁnd that Gradient, Integrated Gradients, LIME, and Kernel-SHAP consistently achieve high sanity scores.
Revised: Across all datasets, we ﬁnd that Gradient, Integrated Gradients, LIME, and Kernel-SHAP consistently achieve high sanity scores.

Original: In contrast, LRP, Guided GradCAM and Guided Backpropagation often assign similar saliency even after randomizing network parameters.
Revised: In contrast, LRP, Guided GradCAM and Guided Backpropagation often assign similar saliency even after randomizing network parameters.

Original: This replicates sanity results by Adebayo et al.
Revised: This replicates sanity results by Adebayo et al.

Original: Low sanity methods highlight features in the input that may also be randomly extracted but do not necessarily predict classes.
Revised: Low sanity methods highlight features in the input that may also be randomly extracted but do not necessarily predict classes.

Original: Faithfulness.
Revised: Faithfulness.

Original: The scores differ widely across datasets.
Revised: The scores differ widely across datasets.

Original: While Gradient, Integrated Gradients, LRP and Guided Backprop have an above-average faithfulness, Guided Gradcam and GradCAM are unable to identify critical features for a prediction.
Revised: While Gradient, Integrated Gradients, LRP and Guided Backprop have an above-average faithfulness, Guided Gradcam and GradCAM are unable to identify critical features for a prediction.

Original: Especially scores for FordA, FordB and NATOPS are generally very low across all visualization methods.
Revised: Especially scores for FordA, FordB and NATOPS are generally very low across all visualization methods.

Original: Perturbating connected sub-sequences instead of random samples may be less suitable for these datasets.
Revised: Perturbating connected sub-sequences instead of random samples may be less suitable for these datasets.

Original: Sensitivity.
Revised: Sensitivity.

Original: GradCAM shows the highest sensitivity for classes - the saliency maps for the most and least likely classes differ largely.
Revised: GradCAM shows the highest sensitivity for classes - the saliency maps for the most and least likely classes differ largely.

Original: On the other hand, Integrated Gradient and Guided Backpropagation produce similar saliency maps, independent of a sample’s class.
Revised: On the other hand, Integrated Gradient and Guided Backpropagation produce similar saliency maps, independent of a sample’s class.

Original: This holds across all datasets.
Revised: This holds across all datasets.

Original: Robustness.
Revised: Robustness.

Original: As found by Smilkov et al.
Revised: As found by Smilkov et al.

Original: (2017) and us, Gradient is less robust to (adversarial) noisy.
Revised: (2017) and us, Gradient is less robust to (adversarial) noisy.

Original: LIME and, interestingly, SmoothGrads also have a low average robustness.
Revised: LIME and, interestingly, SmoothGrads also have a low average robustness.

Original: The additive noise of SmoothGrads may compound and lead to diverging saliency maps.
Revised: The additive noise of SmoothGrads may compound and lead to diverging saliency maps.

Original: The metric’s weakness is computational cost – { TCN,ElectricDevices } timed out after 12 days.
Revised: The metric’s weakness is computational cost – { TCN,ElectricDevices } timed out after 12 days.

Original: Intra-class Stability.
Revised: Intra-class Stability.

Original: This metric is highly dependent on the dataset, e.g., centered samples as in
Revised: This metric is highly dependent on the dataset, e.g., centered samples as in

Original: GunPointAgeSpan.
Revised: GunPointAgeSpan.

Original: However, when accounting for variance introduced by the dataset (see Fig.
Revised: However, when accounting for variance introduced by the dataset (see Fig.

Original: 7b in
Revised: 7b in

Original: Appendix C.2), SmoothGrads, LIME, and Kernel-SHAP show unstable saliency maps for the same classes.
Revised: Appendix C.2), SmoothGrads, LIME, and Kernel-SHAP show unstable saliency maps for the same classes.

Original: Guided GradCAM is able to produce highly stable saliency maps across all datasets.
Revised: Guided GradCAM is able to produce highly stable saliency maps across all datasets.

Original: Dataset Inﬂuence.
Revised: Dataset Inﬂuence.

Original: First, datasets heavily impact scores, see the outliers in Fig. 4a.
Revised: First, datasets heavily impact scores, see the outliers in Fig. 4a.

Original: Plotting the categories over each dataset for all { model architectures, methods } (Fig. 4b) emphasizes this ﬁnding.
Revised: Plotting the categories over each dataset for all { model architectures, methods } (Fig. 4b) emphasizes this ﬁnding.

Original: The choice of visual interpretation method depends, for the most part, on the task.
Revised: The choice of visual interpretation method depends, for the most part, on the task.

Original: Still, after normalizing for dataset bias, the relative scores do not diverge signiﬁcantly but conﬁrm the initial scores.
Revised: Still, after normalizing for dataset bias, the relative scores do not diverge signiﬁcantly but conﬁrm the initial scores.

Original: We report the relative scores in Fig.
Revised: We report the relative scores in Fig.

Original: 7b in Appendix C.2.
Revised: 7b in Appendix C.2.

Original: Second, the model architecture (FCN or TCN) matters little to the scores, see Fig.
Revised: Second, the model architecture (FCN or TCN) matters little to the scores, see Fig.

Original: 7c in Appendix C.2.
Revised: 7c in Appendix C.2.

Original: The scores for FCN and
Revised: The scores for FCN and

Original: TCN diverge slightly more for faithfulness.
Revised: TCN diverge slightly more for faithfulness.

Original: This may be due to the higher capacity of the TCN, which learns more features of lower relevance, so that accuracy is not as affected by perturbations as for the smaller FCN.
Revised: This may be due to the higher capacity of the TCN, which learns more features of lower relevance, so that accuracy is not as affected by perturbations as for the smaller FCN.

Original: A larger perturbation percentage may resolve this issue.
Revised: A larger perturbation percentage may resolve this issue.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  R ECOMMENDATIONS
Revised:  R ECOMMENDATIONS

Original: The quality of visualizations differs wildly between methods.
Revised: The quality of visualizations differs wildly between methods.

Original: No method passes the tests for all categories on all six datasets.
Revised: No method passes the tests for all categories on all six datasets.

Original: This emphasizes the need to evaluate all metrics for every visual interpretation.
Revised: This emphasizes the need to evaluate all metrics for every visual interpretation.

Original: We recommend to use a summary as in Fig.
Revised: We recommend to use a summary as in Fig.

Original: 5 to judge visualizations on every category, and propose the following guidelines for relative ranking.
Revised: 5 to judge visualizations on every category, and propose the following guidelines for relative ranking.

Original: The absolute scores may be understood in comparison with a random baseline, similar to shufﬂed AUC (Borji et al., 2013).
Revised: The absolute scores may be understood in comparison with a random baseline, similar to shufﬂed AUC (Borji et al., 2013).

Original: G r a d C A M G r a d i e n t
Revised: G r a d C A M G r a d i e n t

Original: G u i d e d B a c k p r o p .
Revised: G u i d . B a c k p r o p .

Original: G u i d e d G r a d C A M
Revised: G u i d . G r a d C A M

Original: I n t .
Revised: I n t .

Original: G r a d
Revised: G r a d

Original: i e n t s
Revised: i e n t s

Original: K [Figure] r n e l 
Revised: K e r n e l - S H A

Original: [Figure]

Original: S H A P L I M E L R P 
Revised: P L I M E L R P R a n d o m

Original: [Figure]
Revised: [Figure]

Original: G r a
Revised: r a

Original: [Figure]
Revised: [Figure]

Original: First: Ensure Faithfulness and Sanity.
Revised: First: Ensure Faithfulness and Sanity.

Original: The general purpose of interpretability methods is to provide insights into model behavior.
Revised: The general purpose of interpretability methods is to provide insights into model behavior.

Original: We propose to use the Faithfulness and Sanity scores to ascertain that a saliency map represents the model behavior.
Revised: We propose to use the Faithfulness and Sanity scores to ascertain that a saliency map represents the model behavior.

Original: Faithfulness ensures that the saliency matches the  predictive features that the model uses.
Revised: Faithfulness ensures that the saliency matches the model’s predictive features.

Original: Sanity checks conﬁrm that saliency maps are sensitive to model parameters.
Revised: Sanity checks conﬁrm that saliency maps are sensitive to model parameters.

Original: This is important to avoid cases where visualizations capture highly salient features, like edges in images (Adebayo et al., 2018), while being insensitive to model parameters.
Revised: This is important to avoid ﬁnding highly salient features, like edges in images (Adebayo et al., 2018), while being insensitive to model parameters.

Original: Saliency maps with low scores in either Faithfulness and Sanity should be disregarded, as they do not reﬂect the model behavior.
Revised: We avoid saliency maps with low scores in either metric.

Original: For an example, focus on the two top rows in Fig.
Revised: See Fig.

Original: 5: Gradient, IG and LIME achieve the three highest scores.
Revised: 5 for an example: Gradient, IG and LIME achieve the three highest scores.

Revised: Random saliency maps are dissimilar, hence the high Sanity score.

Revised: However, its Faithfulness hints at low reliability of the other methods.

Original: Guided Backprop, Guided
Revised: Guided Backprop, Guided

Original: GradCAM and LRP fail the Sanity Check, while Kernel-SHAP performs poorly in Faithfulness.
Revised: GradCAM and LRP fail the Sanity Check, while Kernel-SHAP performs poorly in Faithfulness.

Original: Second: Check Sensitivity and Robustness.
Revised: Second: Check Sensitivity and Robustness.

Original: Once Faithfulness and Sanity are established, we propose to look at Sensitivity and Robustness of the generated saliency maps.
Revised: Once Faithfulness and Sanity are established, we propose to look at Sensitivity and Robustness of the generated saliency maps.

Original: A low Inter-Class
Revised: A low Inter-Class

Original: Sensitivity can indicate that the saliency maps only focus on the predicted class and underestimate the importance of features that do not belong to this class.
Revised: Sensitivity can indicate that the saliency maps only focus on the predicted class and underestimate the importance of features that do not belong to this class.

Original: A low Robustness score suggests that the visualization method is susceptible to adversarial examples and small perturbations in the input.
Revised: A low Robustness score suggests that the visualization method is susceptible to adversarial examples and small perturbations in the input.

Original: Given that the model predictions are robust (Zhang et al., 2019), and the saliency is faithful, a low
Revised: Given that the model predictions are robust (Zhang et al., 2019), and the saliency is faithful, a low

Original: Robustness implies that the saliency method cannot be trusted, and may even be manipulated (Dombrowski et al., 2019).
Revised: Robustness implies that the saliency method cannot be trusted, and may even be manipulated (Dombrowski et al., 2019).

Original: For non-robust models we recommend Faithfulness.
Revised: For non-robust models we recommend Faithfulness.

Original: According to Fig.
Revised: According to Fig.

Original: 5, we keep Gradient and LIME, but disregard IG, due to its lower Sensitivity.
Revised: 5, we keep Gradient and LIME, but disregard IG, due to its lower Sensitivity.

Original: Detailed Analysis: Analyze Intra-class Stability and Localization. 
Revised: Detailed Analysis: Analyze Intra-class Stability and Localization. Intra-class stability measures

Original: Intra-class stability measures how much saliency maps for one class agree between different samples.
Revised: how much saliency maps for one class agree between different samples.

Original: The localization metric, like other Pointing Game-like metrics, measures semantic precision based on human annotations.
Revised: The localization metric, like other Pointing Game-like metrics, measures semantic precision based on human annotations.

Original: These human-grounded metrics allow an expert to ﬁlter visualizations that are more intuitive and understandable.
Revised: These human-grounded metrics allow an expert to ﬁlter visualizations that are more intuitive and understandable.

Original: Note that it is crucial to ensure the Faithfulness of a method before relying on this metric.
Revised: Note that it is crucial to ensure the Faithfulness of a method before relying on this metric.

Original: If the method is not faithful, stable and localized visualizations are visually pleasing, but do not reﬂect model behavior.
Revised: If the method is not faithful, stable and localized visualizations are visually pleasing, but do not reﬂect model behavior.

Original: This can hide issues like spurious correlations (Arjovsky et al., 2019) or the shortcut learning problem (Geirhos et al., 2020) behind a good score.
Revised: This can hide issues like spurious correlations (Arjovsky et al., 2019) or the shortcut learning problem (Geirhos et al., 2020) behind a good score.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposes a framework of six orthogonal metrics for the objective evaluation of visual interpretation methods for time series classiﬁcation or segmentation.
Revised: This paper proposes a framework of six orthogonal metrics for the objective evaluation of visual interpretation methods for time series classiﬁcation or segmentation.

Original: All these metrics should be evaluated for every visual interpretation to prevent a reliance on interesting, but spurious results.
Revised: All these metrics should be evaluated for every visual interpretation to prevent a reliance on interesting, but spurious results.

Original: We also show that each perspective emphasizes different strengths and weaknesses of visualization methods.
Revised: We also show that each perspective emphasizes different strengths and weaknesses of visualization methods.

Original: We show empirically for six datasets and different model architectures that none of nine state-of-the-art visual interpretability methods passes all tests.
Revised: We show empirically for six datasets and different model architectures that none of nine state-of-the-art visual interpretability methods passes all tests.

Original: We propose to use our framework to guide the selection of visual interpretations and to understand their weaknesses.
Revised: We propose to use our framework to guide the selection of visual interpretations and to understand their weaknesses.

Original: Further, we found that datasets highly inﬂuence the quality of visualizations.
Revised: Further, we found that datasets highly inﬂuence the quality of visualizations.

Original: Our extended and novel metrics provide important information, like the localization metric for time series.
Revised: Our extended and novel metrics provide important information, like the localization metric for time series.

Original: Interestingly, on the segmentation task only one visual interpretation method passes our novel evaluation metric.
Revised: Interestingly, on the segmentation task only one visual interpretation method passes our novel evaluation metric.

Original: None of the saliency maps for LSTMs produce satisfactory scores.
Revised: None of the saliency maps for LSTMs produce satisfactory scores.

Original: Future work combines range-based and functional metrics, or optimizes models speciﬁcally to achieve higher interpretability scores.
Revised: Future work combines range-based and functional metrics, or optimizes models speciﬁcally to achieve higher interpretability scores.

Original: R EPRODUCIBILITY
Revised: R EPRODUCIBILITY

Original: This paper includes supplemental materials to improve reproducibility.
Revised: This paper includes supplemental materials to improve reproducibility.

Original: Most importantly, we publish the code for model training and dataset loaders, for generating visual interpretations and of all evaluation metrics.
Revised: Most importantly, we publish the code for model training and dataset loaders, for generating visual interpretations and of all evaluation metrics.

Original: Furthermore, all datasets are publicly available
Revised: Furthermore, all datasets are publicly available

Original: A L OCALIZATION M ETRIC P OSITIONAL B IASES
Revised: A L OCALIZATION M ETRIC P OSITIONAL B IASES

Original: This section explains the ”existence” and ”overlap” terms of our localization metric.
Revised: This section explains the ”existence” and ”overlap” terms of our localization metric.

Original: Note again the recall score as
Revised: Note again the recall score as

Original: [Equation]
Revised: [Equation]

Original: The ”existence” and ”overlap” terms are deﬁned by Tatbul et al.
Revised: The ”existence” and ”overlap” terms are deﬁned by Tatbul et al.

Original: (2018) as follows.
Revised: (2018) as follows.

Original: First, existence is deﬁned as a correct prediction of one sample point at index j of the correct class within the labeled region:
Revised: First, existence is deﬁned as a correct prediction of one sample point at index j of the correct class within the labeled region:

Original: [Equation]
Revised: [Equation]

Original: The overlap determines the ﬁner properties cardinality γ () , size ω () and position δ () .
Revised: The overlap determines the ﬁner properties cardinality γ () , size ω () and position δ () .

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The term α ( 0 ≤ α ≤ 1 ) weights existence and the following qualitative measures of overlap.
Revised: The term α ( 0 ≤ α ≤ 1 ) weights existence and the following qualitative measures of overlap.

Original: The three user-deﬁned functions return values of 0 ≤ γ () ≤ 1 , 0 ≤ ω () ≤ 1 and δ () ≥ 1 .
Revised: The three user-deﬁned functions return values of 0 ≤ γ () ≤ 1 , 0 ≤ ω () ≤ 1 and δ () ≥ 1 .

Original: The cardinality γ () function weighs the prediction of a continuous range Y (cid:48) covering the whole labeled range Y sub versus interrupted ranges in a fragment manner.
Revised: The cardinality γ () function weighs the prediction of a continuous range Y (cid:48) covering the whole labeled range Y sub versus interrupted ranges in a fragment manner.

Original: The overall size ω () of the agreement of predicted and label ranges depends on the positional bias δ () and can favor ”front”, ”middle”, or ”back”.
Revised: The overall size ω () of the agreement of predicted and label ranges depends on the positional bias δ () and can favor ”front”, ”middle”, or ”back”.

Original: Practically, for some applications an early detection is preferable to a late detection.
Revised: Practically, for some applications an early detection is preferable to a late detection.

Original: For the four functions, we selected α is equal to 0, γ is equal to 1.
Revised: For the four functions, we selected α is equal to 0, γ is equal to 1.

Original: ω depends on the position bias δ and for δ we have three variants, ”front”, ”middle” or ”back”, proposed by Tatbul et al.
Revised: ω depends on the position bias δ and for δ we have three variants, ”front”, ”middle” or ”back”, proposed by Tatbul et al.

Original: B E XPERIMENTAL S ETUP
Revised: B E XPERIMENTAL S ETUP

Original: We describe each visual interpretation method shortly in Sec.
Revised: We describe each visual interpretation method shortly in Sec.

Original: B.1, then the hyper parameters for each visual interpretation method in Sec.
Revised: B.1, then the hyper parameters for each visual interpretation method in Sec.

Original: B.2 and for the optimizer and networks in Sec.
Revised: B.2 and for the optimizer and networks in Sec.

Original: B.3.
Revised: B.3.

Original: B. C ATEGORIZATION OF M ETHODS
Revised: B. C ATEGORIZATION OF M ETHODS

Original: This Section provides a short introduction to the methods that we use in our experiments.
Revised: This Section provides a short introduction to the methods that we use in our experiments.

Original: Gradient-based methods.
Revised: Gradient-based methods.

Original: Gradient (Simonyan et al., 2014) computes class c ’s saliency map
Revised: Gradient (Simonyan et al., 2014) computes class c ’s saliency map

Original: M c using the derivative of the class score P c of model with respect to the input sample x , as
Revised: M c using the derivative of the class score P c of model with respect to the input sample x , as

Original: M c ( x ) = ∂P
Revised: M c ( x ) = ∂P

Original: c ∂x .
Revised: c ∂x .

Original: However, Gradient suffers from the saturation problem (a feature may have global importance, but its local derivative is small (Smilkov et al., 2017)), local sensitivity, and noisy saliency maps (due to (sharp) local variations in the gradients (Smilkov et al., 2017)).
Revised: However, Gradient suffers from the saturation problem (a feature may have global importance, but its local derivative is small (Smilkov et al., 2017)), local sensitivity, and noisy saliency maps (due to (sharp) local variations in the gradients (Smilkov et al., 2017)).

Original: Follow-up work smooths the gradients to reduce noise (Smilkov et al., 2017), applies special propagation rules instead of propagating a gradient (Arras et al., 2017), or propagates only up to a speciﬁc intermediate layer (Selvaraju et al., 2017).
Revised: Follow-up work smooths the gradients to reduce noise (Smilkov et al., 2017), applies special propagation rules instead of propagating a gradient (Arras et al., 2017), or propagates only up to a speciﬁc intermediate layer (Selvaraju et al., 2017).

Original: Perturbation-based methods. Local Interpretable Model-Agnostic Explanations
Revised: Perturbation-based methods. Local Interpretable Model-Agnostic Explanations

Original: (LIME)
Revised: (LIME)

Original: (Ribeiro et al., 2016) ﬁts a local surrogate model (e.g., a linear regression) as an explanation and uses this surrogate to calculate relevance.
Revised: (Ribeiro et al., 2016) ﬁts a local surrogate model (e.g., a linear regression) as an explanation and uses this surrogate to calculate relevance.

Original: Kernel SHAP (Lundberg & Lee, 2017) builds on LIME, but calculates Shapley values that measure the contribution of individual features to the input more accurately.
Revised: Kernel SHAP (Lundberg & Lee, 2017) builds on LIME, but calculates Shapley values that measure the contribution of individual features to the input more accurately.

Original: B. V ISUALIZATION M ETHODS H YPER P ARAMETERS
Revised: B. V ISUALIZATION M ETHODS H YPER P ARAMETERS

Original: We set the hyper parameters for each visual interpretation method according to their recommendations from literature.
Revised: We set the hyper parameters for each visual interpretation method according to their recommendations from literature.

Original: We provide the reasoning behind the selected parameters in the following paragraphs.
Revised: We provide the reasoning behind the selected parameters in the following paragraphs.

Original: Gradient computes the derivative of the target class score with respect to the input sample and returns the saliency map of the input sample at the end.
Revised: Gradient computes the derivative of the target class score with respect to the input sample and returns the saliency map of the input sample at the end.

Original: Integrated Gradient uses the linear path method to compute gradient along the path from a baseline x (cid:48) .
Revised: Integrated Gradient uses the linear path method to compute gradient along the path from a baseline x (cid:48) .

Original: We use a zero vector for the baseline.
Revised: We use a zero vector for the baseline.

Original: As suggested by Smilkov et al.
Revised: As suggested by Smilkov et al.

Original: (2017), the number of steps for the path should be selected betweenand 300.
Revised: (2017), the number of steps for the path should be selected betweenand 300.

Original: Hence, we use steps N = 60 , meaning that it takes 60 steps from baseline x (cid:48) to the original input sample x , according to x = x (cid:48)
Revised: Hence, we use steps N = 60 , meaning that it takes 60 steps from baseline x (cid:48) to the original input sample x , according to x = x (cid:48)

Original: + ( x − x (cid:48) )
Revised: + ( x − x (cid:48) )

Original: N · n , n is the current step.
Revised: N · n , n is the current step.

Original: SmoothGrad also computes the derivative of the target class score with respect to the input sample.
Revised: SmoothGrad also computes the derivative of the target class score with respect to the input sample.

Original: However, it adds Gaussian noise N (0 , σ 2 ) to the input sample multiple times and computes the gradients from the perturbed samples ( x + N (0 , σ 2 )) .
Revised: However, it adds Gaussian noise N (0 , σ 2 ) to the input sample multiple times and computes the gradients from the perturbed samples ( x + N (0 , σ 2 )) .

Original: The number of iteration of adding noise is chosen N = 60 and the standard deviation of Gaussian noise is chosen σ = 0 .
Revised: The number of iteration of adding noise is chosen N = 60 and the standard deviation of Gaussian noise is chosen σ = 0 .

Original: For LRP, we select the (cid:15) -propagation rule for every DL model with (cid:15) = 1 e − 9 .
Revised: For LRP, we select the (cid:15) -propagation rule for every DL model with (cid:15) = 1 e − 9 .

Original: Due to the residual block in TCN model the propagated relevances should therefore be added together ( R = R 1 + R 2 ).
Revised: Due to the residual block in TCN model the propagated relevances should therefore be added together ( R = R 1 + R 2 ).

Original: GradCAM and Guided-GradCAM are designed for models with convolutional layers, i.e. FCN,
Revised: GradCAM and Guided-GradCAM are designed for models with convolutional layers, i.e. FCN,

Original: TCN and U-time.
Revised: TCN and U-time.

Original: GradCAM focuses on the last convolutional layer, which produces feature maps, whose shapes usually are smaller than the shape of input samples.
Revised: GradCAM focuses on the last convolutional layer, which produces feature maps, whose shapes usually are smaller than the shape of input samples.

Original: Therefore, we use interpolation to up-sample the saliency maps from the last convolution layer to match their shape to the shape of the input samples, which allows us to visualize them in the input space.
Revised: Therefore, we use interpolation to up-sample the saliency maps from the last convolution layer to match their shape to the shape of the input samples, which allows us to visualize them in the input space.

Original: Because of ReLU functions,
Revised: Because of ReLU functions,

Original: GradCAM returns only positive relevances.
Revised: GradCAM returns only positive relevances.

Original: For LIME, we use the cosine distance function as the kernel function with width w = 5 . 0 to weight the perturbed samples, and perform 1000 iterations.
Revised: For LIME, we use the cosine distance function as the kernel function with width w = 5 . 0 to weight the perturbed samples, and perform 1000 iterations.

Original: For the perturbation, we consider that neighbors along time dimension should have similar relevance to reduce the computational time, so we set the number of features along time dimension 50 for GunPointAgeSpan.
Revised: For the perturbation, we consider that neighbors along time dimension should have similar relevance to reduce the computational time, so we set the number of features along time dimension 50 for GunPointAgeSpan.

Original: This means that saliency maps for samples with length 150 in GunPointAgeSpan have same relevance for every 3 neighbors and saliency maps for dense labeling samples in tool tracking have same relevance for every 4 neighbors.
Revised: This means that saliency maps for samples with length 150 in GunPointAgeSpan have same relevance for every 3 neighbors and saliency maps for dense labeling samples in tool tracking have same relevance for every 4 neighbors.

Original: For Kernel SHAP, we use 1000 iterations and set the number of features along time dimension to 50 .
Revised: For Kernel SHAP, we use 1000 iterations and set the number of features along time dimension to 50 .

Original: Furthermore, the sampling of feature perturbation in Kernel SHAP is based on the distribution p ( f ) =
Revised: Furthermore, the sampling of feature perturbation in Kernel SHAP is based on the distribution p ( f ) =

Original: ( F − 1) ( f · ( F − f )) , where f is the number of selected features and F is the total number of features in interpretation space.
Revised: ( F − 1) ( f · ( F − f )) , where f is the number of selected features and F is the total number of features in interpretation space.

Original: B. N ETWORK A RCHITECTURES
Revised: B. N ETWORK A RCHITECTURES

Original: We use the Adam optimizer with a learning rate of 0 . 002 .
Revised: We use the Adam optimizer with a learning rate of 0 . 002 .

Original: We train every dataset for 600 epochs (with early stopping after 80 epochs).
Revised: We train every dataset for 600 epochs (with early stopping after 80 epochs).

Original: We use a Cross Entropy loss for time series classiﬁcation and a Generalized Dice loss with Cross Entropy function for time series segmentation.
Revised: We use a Cross Entropy loss for time series classiﬁcation and a Generalized Dice loss with Cross Entropy function for time series segmentation.

Original: For the time series segmentation task on the tool tracking dataset, we report an accuracy of 83% for
Revised: For the time series segmentation task on the tool tracking dataset, we report an accuracy of 83% for

Original: U-time and 85% for the bi-LSTM.
Revised: U-time and 85% for the bi-LSTM.

Original: Table 1 shows the classiﬁcation results for FCN and TCN with all classiﬁcation datasets.[Table]
Revised: Table 1 shows the classiﬁcation results for FCN and TCN with all classiﬁcation datasets.

Original: [Table]
Revised: [Table]

Original:  C ONVOLUTION N ETWORK
Revised: B.3. F ULLY C ONVOLUTION N ETWORK

Original: We use a slightly modiﬁed FCN, similar to Wang et al.
Revised: We use a slightly modiﬁed FCN, similar to Wang et al.

Original: Ours contains four convolution blocks with a convolutional layer, a batch normalization layer (Ioffe & Szegedy, 2015) and a ReLU layer in each block.
Revised: Ours contains four convolution blocks with a convolutional layer, a batch normalization layer (Ioffe & Szegedy, 2015) and a ReLU layer in each block.

Original: The kernel shapes and numbers of ﬁlter for convolution layers are { 7 , 5 , 3 , 3 } and { 16 , 32 , 32 , 16 } .
Revised: The kernel shapes and numbers of ﬁlter for convolution layers are { 7 , 5 , 3 , 3 } and { 16 , 32 , 32 , 16 } .

Original: Therefore, there are four convolutional layers.
Revised: Therefore, there are four convolutional layers.

Original: Each convolutional layer has unit stride and no padding, which means the time sequence will be reduced continuously by the blocks.
Revised: Each convolutional layer has unit stride and no padding, which means the time sequence will be reduced continuously by the blocks.

Original: The ﬁnal convolution block, which is behind the four convolution blocks, does not have a ReLU layer and contains a 1x1 convolutional layer.
Revised: The ﬁnal convolution block, which is behind the four convolution blocks, does not have a ReLU layer and contains a 1x1 convolutional layer.

Original: The 1x1 convolutional layer serves as a projection layer.
Revised: The 1x1 convolutional layer serves as a projection layer.

Original: It can not only reduce the channel size of feature maps but also keep their salient features.
Revised: It can not only reduce the channel size of feature maps but also keep their salient features.

Original: Finally, we apply Global Max Pooling on the features maps, before the softmax operation.
Revised: Finally, we apply Global Max Pooling on the features maps, before the softmax operation.

Original: B.3. T EMPORAL C ONVOLUTION N ETWORK
Revised: B.3. T EMPORAL C ONVOLUTION N ETWORK

Original: For the TCN, that was ﬁrst proposed by (Bai et al., 2018), we use a global pooling layer for the prediction.
Revised: For the TCN, that was ﬁrst proposed by (Bai et al., 2018), we use a global pooling layer for the prediction.

Original: In our architecture, TCN has the convolution ﬁlters { 16 , 32 , 32 , 32 } and kernel shapes for convolution layers { 7 , 5 , 5 , 5 } in four residual blocks.
Revised: In our architecture, TCN has the convolution ﬁlters { 16 , 32 , 32 , 32 } and kernel shapes for convolution layers { 7 , 5 , 5 , 5 } in four residual blocks.

Original: Therefore, the total number of layers is 8 .
Revised: Therefore, the total number of layers is 8 .

Original: B.3. BIDIRECTIONAL L ONG S HORT -T ERM M EMORY
Revised: B.3. BIDIRECTIONAL L ONG S HORT -T ERM M EMORY

Original: We use a standard, single-layer bi-LSTM to predict dense labels for the segmentation task, as implemented in PyTorch, with 512 hidden units.
Revised: We use a standard, single-layer bi-LSTM to predict dense labels for the segmentation task, as implemented in PyTorch, with 512 hidden units.

Original: There is a dense layer behind the LSTM model to ﬁt the hidden units 512 to the number of classes.
Revised: There is a dense layer behind the LSTM model to ﬁt the hidden units 512 to the number of classes.

Original: Also, the dropout rate is set to 0 .
Revised: Also, the dropout rate is set to 0 .

Original: 2 to prevent overﬁtting.
Revised: 2 to prevent overﬁtting.

Original: B.3. U- TIME
Revised: B.3. U- TIME

Original: We use the U-time (Perslev et al., 2019) architecture with the following conﬁguration.
Revised: We use the U-time (Perslev et al., 2019) architecture with the following conﬁguration.

Original: In each convolution block, there are two dilated convolution layers with dilation 3 , followed by a ReLU layer, a batch normalization layer, and a Max-Pooling layer at the end.
Revised: In each convolution block, there are two dilated convolution layers with dilation 3 , followed by a ReLU layer, a batch normalization layer, and a Max-Pooling layer at the end.

Original: The number of ﬁlters for convolution layers in four convolution blocks are { 16 , 32 , 64 , 128 } and the pooling window sizes are all 2 .
Revised: The number of ﬁlters for convolution layers in four convolution blocks are { 16 , 32 , 64 , 128 } and the pooling window sizes are all 2 .

Original: Two additional convolutions with ﬁlter numbers { 256 , 256 } follow after four convolution blocks.
Revised: Two additional convolutions with ﬁlter numbers { 256 , 256 } follow after four convolution blocks.

Original: In each transposed convolution block, a nearest-neighbor up-sampling (Odena et al., 2016) of its input is implemented, followed by a dilated convolution layer with dilation 3 , a ReLU layer and a batch normalization layer.
Revised: In each transposed convolution block, a nearest-neighbor up-sampling (Odena et al., 2016) of its input is implemented, followed by a dilated convolution layer with dilation 3 , a ReLU layer and a batch normalization layer.

Original: The number of ﬁlters for convolution layers in four transposed convolution blocks are in the reverse order of the encoder { 128 , 64 , 32 , 16 } .
Revised: The number of ﬁlters for convolution layers in four transposed convolution blocks are in the reverse order of the encoder { 128 , 64 , 32 , 16 } .

Original: The kernel size of convolution layers in both encoder and decoder is 7 .
Revised: The kernel size of convolution layers in both encoder and decoder is 7 .

Original: C A DDITIONAL RESULTS
Revised: C A DDITIONAL RESULTS

Original: We report additional results to supplement our discussion for a bi-LSTM architecture on the segmentation task in Sec.
Revised: We report additional results to supplement our discussion for a bi-LSTM architecture on the segmentation task in Sec.

Original: C.1, and for the classiﬁcation task in Sec.
Revised: C.1, and for the classiﬁcation task in Sec.

Original: C.2.
Revised: C.2.

Original: C. E VALUATION ON S EGMENTATION T ASK : BI -LSTM
Revised: C. E VALUATION ON S EGMENTATION T ASK : BI -LSTM

Original: [Figure]
Revised: [Figure]

Original: The results for bi-LSTM on the segmentation task for tool tracking in Fig.
Revised: The results for bi-LSTM on the segmentation task for tool tracking in Fig.

Original: 6 show that none of the visual interpretation methods, that are compatible with the model’s architecture, produce satisfactory saliency maps.
Revised: 6 show that none of the visual interpretation methods, that are compatible with the model’s architecture, produce satisfactory saliency maps.

Original: C. C LASSIFICATION T ASK
Revised: C. C LASSIFICATION T ASK

Original: We show the variability of each visualization method with respect to each dataset and metric category for the TCN in Fig.
Revised: We show the variability of each visualization method with respect to each dataset and metric category for the TCN in Fig.

Original: 7a.
Revised: 7a.

Original: It is noteworthy that some datasets are more problematic for the methods, especially for faithfulness and intra-class stability, than others.
Revised: It is noteworthy that some datasets are more problematic for the methods, especially for faithfulness and intra-class stability, than others.

Original: We also show that the model has only a small inﬂuence.
Revised: We also show that the model has only a small inﬂuence.

Original: For this result, we aggregate all datasets and visualization methods separately for FCN and TCN in Fig.
Revised: For this result, we aggregate all datasets and visualization methods separately for FCN and TCN in Fig.

Original: 7c.
Revised: 7c.

Original: In order to compare methods independent of datasets, we control for their bias by normalizing the scores of all methods for each { category, dataset } so that their mean is 0 and variance is 1 .
Revised: In order to compare methods independent of datasets, we control for their bias by normalizing the scores of all methods for each { category, dataset } so that their mean is 0 and variance is 1 .

Original: With this, we can correctly assess the ”relative” performance of the visual interpretation methods across different datasets.
Revised: With this, we can correctly assess the ”relative” performance of the visual interpretation methods across different datasets.

Original: The plot in Fig.
Revised: The plot in Fig.

Original: 7b shows these relative (or marginal) scores that a method can achieve, compared to other methods on the same datasets.
Revised: 7b shows these relative (or marginal) scores that a method can achieve, compared to other methods on the same datasets.

Original: We prove the orthogonality of metrics in Figure 8.
Revised: We prove the orthogonality of metrics in Figure 8.

Original: No combination of our metric scores shows a high correlation.
Revised: No combination of our metric scores shows a high correlation.

Original: This shows that each metric measures a quality independent of all other metrics.
Revised: This shows that each metric measures a quality independent of all other metrics.

Revised: We show examples for saliency of the TCN architecture on the FordB dataset for the visual interpretation methods for class 0 in Figure 9 and class 1 in Figure 10.

Revised: We argue that the choice of a suitable method to generate saliency maps should be guided by our framework

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: F a i t
Revised: F a i t

Original: h
Revised: h

Original: f u l n e ss
Revised: f u l n e ss

Original: I n t r a - C l a ss S t a b ili t y
Revised: I n t r a - C l a ss S t a b ili t y

Original: R o b u s t n e ss
Revised: R o b u s t n e ss

Original: S a n i t y
Revised: S a n i t y

Original: [Figure]
Revised: [Figure]

Original: e n s i t i v i
Revised: e n s i t i v i

Original: t y
Revised: t y

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]