Original: T OWARDS D IVERSE P ERSPECTIVE L EARNING WITH S WITCH OVER M ULTIPLE T EMPORAL P OOLING
Revised: T OWARDS D IVERSE P ERSPECTIVE L EARNING WITH S WITCH OVER M ULTIPLE T EMPORAL P OOLING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Pooling is a widely used method for classification problems.
Revised: Pooling is a widely used method for classification problems.

Original: In the time series classification (TSC) domain, pooling considering temporal information has been proposed.
Revised: In the time series classification (TSC) domain, pooling considering temporal information has been proposed.

Original: However, we found that each temporal pooling has a distinct and fixed perspective, which causes data dependency.
Revised: However, we found that each temporal pooling has a distinct and fixed perspective, which causes data dependency.

Original: In this paper, we propose a novel pooling architecture for diverse perspective learning switch over multiple pooling (SoM-TP).
Revised: In this paper, we propose a novel pooling architecture for diverse perspective learning switch over multiple pooling (SoM-TP).

Original: SoM-TP dynamically selects the optimal pooling method suitable for each data.
Revised: SoM-TP dynamically selects the optimal pooling method suitable for each data.

Original: The massive case study using layer-wise relevance propagation (LRP) reveals the distinct perspective each pooling has and ultimately demonstrates the need for diverse perspective learning.
Revised: The massive case study using layer-wise relevance propagation (LRP) reveals the distinct perspective each pooling has and ultimately demonstrates the need for diverse perspective learning.

Original: The ablation study on SoM-TP shows how diverse perspective learning is achieved and performed.
Revised: The ablation study on SoM-TP shows how diverse perspective learning is achieved and performed.

Original: Furthermore, the pooling classification experiment also supports the need for diverse perspective learning by showing that more suitable pooling exists depending on the data.
Revised: Furthermore, the pooling classification experiment also supports the need for diverse perspective learning by showing that more suitable pooling exists depending on the data.

Original: Extensive experiments are done with the UCR/UEA repository.
Revised: Extensive experiments are done with the UCR/UEA repository.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Time series classification (TSC) has been one of the most valuable subjects in data mining (L ¨ angkvist et al., 2014).
Revised: Time series classification (TSC) has been one of the most valuable subjects in data mining (L ¨ angkvist et al., 2014).

Original: Also, the revolutionary success of deep neural networks (DNN) has led to their application on TSC (Wang et al., 2017; Le Guennec et al., 2016;
Revised: Also, the revolutionary success of deep neural networks (DNN) has led to their application on TSC (Wang et al., 2017; Le Guennec et al., 2016;

Original: Zhao et al., 2017; Tanisaro & Heidemann, 2016; Serr`a et al., 2018).
Revised: Zhao et al., 2017; Tanisaro & Heidemann, 2016; Serr`a et al., 2018).

Original: Especially the convolutional neural networks (CNN) based model architecture, fully convolutional networks (FCNs) (Long et al., 2015) and residual networks (ResNet) (He et al., 2016), achieved the current state-of-the-art (SOTA) in an end-to-end manner (Ismail Fawaz et al., 2019; LeCun et al., 2015).
Revised: Especially the convolutional neural networks (CNN) based model architecture, fully convolutional networks (FCNs) (Long et al., 2015) and residual networks (ResNet) (He et al., 2016), achieved the current state-of-the-art (SOTA) in an end-to-end manner (Ismail Fawaz et al., 2019; LeCun et al., 2015).

Original: In CNN, pooling is the key component with two primary purposes: 1) Decreasing the number of parameters for less computational cost and preventing overfitting, and 2) Position invariance learning.
Revised: In CNN, pooling is the key component with two primary purposes: 1) Decreasing the number of parameters for less computational cost and preventing overfitting, and 2) Position invariance learning.

Original: To this end, pooling combines the high-dimensional values of feature outputs into low-dimensional (Gholamalinezhad & Khosravi, 2020).
Revised: To this end, pooling combines the high-dimensional values of feature outputs into low-dimensional (Gholamalinezhad & Khosravi, 2020).

Original: Thus, global pooling is widely used as the most suitable pooling method for the above purposes.
Revised: Thus, global pooling is widely used as the most suitable pooling method for the above purposes.

Original: However, a simple structure of global pooling has a drawback in the time series domain by losing the temporal information of hidden vectors.
Revised: However, a simple structure of global pooling has a drawback in the time series domain by losing the temporal information of hidden vectors.

Original: To solve this limitation, temporal poolings have been proposed to conserve temporal position information from convolutional hidden vectors(Lee et al., 2021).
Revised: To solve this limitation, temporal poolings have been proposed to conserve temporal position information from convolutional hidden vectors(Lee et al., 2021).

Original: Global temporal pooling (GTP) outputs only one position-invariant feature, which is a simplified version of common global pooling.
Revised: Global temporal pooling (GTP) outputs only one position-invariant feature, which is a simplified version of common global pooling.

Original: However, time-segmented pooling methods, statictemporal-pooling (STP) and dynamic-temporal-pooling (DTP) use the given number n ∈ Z + to segment the time-axis by temporal order.
Revised: However, time-segmented pooling methods, statictemporal-pooling (STP) and dynamic-temporal-pooling (DTP) use the given number n ∈ Z + to segment the time-axis by temporal order.

Original: STP divides equally, while DTP does dynamically with different segmentation lengths via soft dynamic time warping (soft-DTW)
Revised: STP divides equally, while DTP does dynamically with different segmentation lengths via soft dynamic time warping (soft-DTW)

Original: (Lee et al., 2021).
Revised: (Lee et al., 2021).

Original: Due to the segmentation, STP and DTP represent multiple local features, whereas GTP represents only one.
Revised: Due to the segmentation, STP and DTP represent multiple local features, whereas GTP represents only one.

Original: With multiple local features, the temporal order information in each segment is conserved by each pooled vector.
Revised: With multiple local features, the temporal order information in each segment is conserved by each pooled vector.

Original: How to aggregate convolution features in pooling is a significant matter.
Revised: How to aggregate convolution features in pooling is a significant matter.

Original: Each temporal pooling has a distinct mechanism for aggregation, and we term the different mechanisms of temporal pooling as a ‘perspective’.
Revised: Each temporal pooling has a distinct mechanism for aggregation, and we term the different mechanisms of temporal pooling as a ‘perspective’.

Original: Depending on the use of segmentation in pooling, the perspective is divided into ‘global’ and ‘local’, and according to the segmentation method, the local is divided into ‘rigid’ and ‘dynamic’.
Revised: Depending on the use of segmentation in pooling, the perspective is divided into ‘global’ and ‘local’, and according to the segmentation method, the local is divided into ‘rigid’ and ‘dynamic’.

Original: However, each temporal pooling only deals with a single perspective on hidden features as defined.
Revised: However, each temporal pooling only deals with a single perspective on hidden features as defined.

Original: And this fixed-perspective learning has the following constraints.
Revised: And this fixed-perspective learning has the following constraints.

Original: A global view cannot
Revised: A global view cannot

Original: (1, T)
Revised: (1, T)

Original: Input
Revised: Input

Original: Convolutional Stack Φ 1 : 𝑙𝑙
Revised: Convolutional Stack Φ 1 : 𝑙𝑙

Original: T ch𝑓𝑓 1𝑙𝑙 1 +1 : 𝑙𝑙𝑓𝑓 2𝑙𝑙 1 +1 : 𝑙𝑙selected
Revised: T ch𝑓𝑓 1𝑙𝑙 1 +1 : 𝑙𝑙𝑓𝑓 2𝑙𝑙 1 +1 : 𝑙𝑙selected

Original: Pooling Block
Revised: Pooling Block

Original: Attention
Revised: Attention

Original: Masking ⨂
Revised: Masking ⨂

Original: Switching Block
Revised: Switching Block

Original: AlignMatrix
Revised: AlignMatrix

Original: S o f t D T W
Revised: S o f t D T W

Original: GTP
Revised: GTP

Original: STP
Revised: STP

Original: DTP
Revised: DTP

Original: Convolutionallayer forencoding attention weight 𝑦𝑦 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝
Revised: Convolutionallayer forencoding attention weight 𝑦𝑦 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝

Original: Diverse Perspective Learning Network
Revised: Diverse Perspective Learning Network

Original: Hidden feature with temporal position information 𝑝𝑝𝑝𝑝𝑝𝑝�𝑃𝑃 𝐴𝐴 0 ⨂ =
Revised: Hidden feature with temporal position information 𝑝𝑝𝑝𝑝𝑝𝑝�𝑃𝑃 𝐴𝐴 0 ⨂ =

Original: 𝑀𝑀 𝐴𝐴 𝑝𝑝�𝑃𝑃 𝐴𝐴 𝑦𝑦 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 = 𝐸𝐸 𝐻𝐻
Revised: 𝑀𝑀 𝐴𝐴 𝑝𝑝�𝑃𝑃 𝐴𝐴 𝑦𝑦 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 = 𝐸𝐸 𝐻𝐻

Original: Classification Network capture multiple classification points, while a local view cannot concentrate on one dominant classification point.
Revised: Classification Network capture multiple classification points, while a local view cannot concentrate on one dominant classification point.

Original: Therefore, the performance loss is inevitable if the task requires diverse viewpoints to simultaneously capture a dominant feature and hidden local features.
Revised: Therefore, the performance loss is inevitable if the task requires diverse viewpoints to simultaneously capture a dominant feature and hidden local features.

Original: Motivated by the limitations, the novel approach of pooling to fully utilize diverse perspectives is suggested.
Revised: Motivated by the limitations, the novel approach of pooling to fully utilize diverse perspectives is suggested.

Original: In this paper, we propose “Switch over Multiple Temporal Pooling (SoM-TP)”, a pooling architecture with diverse perspective learning.
Revised: In this paper, we propose “Switch over Multiple Temporal Pooling (SoM-TP)”, a pooling architecture with diverse perspective learning.

Original: Diverse perspective learning is the opposite concept of fixedperspective learning, which can overcome the limitation of existing temporal poolings.
Revised: Diverse perspective learning is the opposite concept of fixedperspective learning, which can overcome the limitation of existing temporal poolings.

Original: SoM-TP can reflect various views by dynamically selecting proper pooling based on attention.
Revised: SoM-TP can reflect various views by dynamically selecting proper pooling based on attention.

Original: The challenge here is to train attention weight to dynamically select the optimal pooling for each specific data sample.
Revised: The challenge here is to train attention weight to dynamically select the optimal pooling for each specific data sample.

Original: For this, we propose three methods: 1) convolutional encoded attention weight; 2) diverse perspective learning network which is a sub-network to deal with various poolings; 3) perspective loss which is the combination of KL divergence and cross-entropy loss function.
Revised: For this, we propose three methods: 1) convolutional encoded attention weight; 2) diverse perspective learning network which is a sub-network to deal with various poolings; 3) perspective loss which is the combination of KL divergence and cross-entropy loss function.

Original: The attention score selects proper pooling by highlighting important pooled vector’s index.
Revised: The attention score selects proper pooling by highlighting important pooled vector’s index.

Original: To get an attention score, the convolutional layer encodes the vectors of multiplying pooling features and attention weight to compress the information from all pooling features.
Revised: To get an attention score, the convolutional layer encodes the vectors of multiplying pooling features and attention weight to compress the information from all pooling features.

Original: Also, the sub-network and perspective loss is for effective optimization to SoM-TP to learn diverse perspectives.
Revised: Also, the sub-network and perspective loss is for effective optimization to SoM-TP to learn diverse perspectives.

Original: Next, by minimizing perspective loss from the KL divergence between each output from the main network and a diverse perspective learning network, all learnable weights in the SoM-TP learns diverse perspectives.
Revised: Next, by minimizing perspective loss from the KL divergence between each output from the main network and a diverse perspective learning network, all learnable weights in the SoM-TP learns diverse perspectives.

Original: By making the two output distributions similar, the model learns various views, which is considered as using the concept of parameter tying.
Revised: By making the two output distributions similar, the model learns various views, which is considered as using the concept of parameter tying.

Original: This process makes a model robust by learning any pooled observations from different views.
Revised: This process makes a model robust by learning any pooled observations from different views.

Original: The research scope is limited to temporal pooling methods with CNN.
Revised: The research scope is limited to temporal pooling methods with CNN.

Original: A model architecture with
Revised: A model architecture with

Original: FCN and ResNet, customized with three primary temporal pooling schemes, is specifically used.
Revised: FCN and ResNet, customized with three primary temporal pooling schemes, is specifically used.

Original:  B ACKGROUND 2. D EFINE THE PERSPECTIVE OF TEMPORAL POOLINGS
Revised:  B ACKGROUND 2. D EFINE THE PERSPECTIVE OF TEMPORAL POOLINGS

Original: Convolutional Neural Network in Time Series Classification CNN outperformed conventional methods such as nearest neighbor classifiers (Yuan et al., 2019a) or COTE (Bagnall et al., 2015;
Revised: Convolutional Neural Network in Time Series Classification CNN outperformed conventional methods such as nearest neighbor classifiers (Yuan et al., 2019a) or COTE (Bagnall et al., 2015;

Original: Lines et al., 2016) by capturing local patterns in TSC problems (Ismail Fawaz et al., 2019; Wang et al., 2017).
Revised: Lines et al., 2016) by capturing local patterns in TSC problems (Ismail Fawaz et al., 2019; Wang et al., 2017).

Original: As for TSC, the CNN model can be generally formulated as follows: a time series data T
Revised: As for TSC, the CNN model can be generally formulated as follows: a time series data T

Original: = { ( X 1 , y 1 ) , ..., ( X t , y t ) } , where X ∈ R d × t of length t with d variables and y ∈ { 1 , ..., C } from
Revised: = { ( X 1 , y 1 ) , ..., ( X t , y t ) } , where X ∈ R d × t of length t with d variables and y ∈ { 1 , ..., C } from

Original: C classes.
Revised: C classes.

Original: Then, convolution stack Φ of out channel dimension k encodes features as hidden representations with temporal position information H =
Revised: Then, convolution stack Φ of out channel dimension k encodes features as hidden representations with temporal position information H =

Original: { h 0 , ..., h t } ∈ R k × t (Lee et al., 2021).
Revised: { h 0 , ..., h t } ∈ R k × t (Lee et al., 2021).

Original: [Equation]
Revised: [Equation]

Original: Global Temporal Pooling.
Revised: Global Temporal Pooling.

Original: GTP is a global pooling by using H as input.
Revised: GTP is a global pooling by using H as input.

Original: GTP pools just one representation p 1 =
Revised: GTP pools just one representation p 1 =

Original: [ p 1 ] ∈ R k × 1 in whole time range with ignoring temporal information.
Revised: [ p 1 ] ∈ R k × 1 in whole time range with ignoring temporal information.

Original: GTP entirely aggregates the hidden convolution output features by the time axis t .
Revised: GTP entirely aggregates the hidden convolution output features by the time axis t .

Original: As a result, GTP has a global view since the whole time axis information is represented as one pooled vector.
Revised: As a result, GTP has a global view since the whole time axis information is represented as one pooled vector.

Original: [Equation]
Revised: [Equation]

Original: However, GTP has the crucial limitation that it cannot contain sequential information which can be an important feature.
Revised: However, GTP has the crucial limitation that it cannot contain sequential information which can be an important feature.

Original: To utilize temporal position from H , other temporal pooling methods have been proposed (Figure 1 with a pooling block) (Lee et al., 2021).
Revised: To utilize temporal position from H , other temporal pooling methods have been proposed (Figure 1 with a pooling block) (Lee et al., 2021).

Original: Both STP and DTP are multiple local pooling methods with time-axis segmentation.
Revised: Both STP and DTP are multiple local pooling methods with time-axis segmentation.

Original: The main difference between the two methods is whether to consider a temporal relationship which means the relationship between each time steps in a time series data.
Revised: The main difference between the two methods is whether to consider a temporal relationship which means the relationship between each time steps in a time series data.

Original: Considering that time series data generally consist of a combination of specific local patterns, the reflection of temporal relationship can be effective in terms of loss of meaningful sequential information.
Revised: Considering that time series data generally consist of a combination of specific local patterns, the reflection of temporal relationship can be effective in terms of loss of meaningful sequential information.

Original: Static Temporal Pooling.
Revised: Static Temporal Pooling.

Original: STP divides the time axis equally into n segments with a length ℓ = tn .
Revised: STP divides the time axis equally into n segments with a length ℓ = tn .

Original: In other words, STP ϕ 2 pools each of the segments h ℓ , where ¯H =
Revised: In other words, STP ϕ 2 pools each of the segments h ℓ , where ¯H =

Original: { h 0 : ℓ , h ℓ : 2 ℓ , ..., h ( n − 1 ) ℓ : n ℓ } (Lee et al., 2021).
Revised: { h 0 : ℓ , h ℓ : 2 ℓ , ..., h ( n − 1 ) ℓ : n ℓ } (Lee et al., 2021).

Original: Note that h ℓ keeps local temporal information within each time segment, but there is no consideration of temporal relationship in the segmentation process.
Revised: Note that h ℓ keeps local temporal information within each time segment, but there is no consideration of temporal relationship in the segmentation process.

Original: The pooling representations increase as p 2 =
Revised: The pooling representations increase as p 2 =

Original: [ p 1 , ..., p n ] ∈ R k × n .
Revised: [ p 1 , ..., p n ] ∈ R k × n .

Original: In terms of the pooling perspective, STP has a rigid local view since the whole time axis is represented by n equally segmented pooled vectors.
Revised: In terms of the pooling perspective, STP has a rigid local view since the whole time axis is represented by n equally segmented pooled vectors.

Original: [Equation]
Revised: [Equation]

Original: Dynamic Temporal Pooling.
Revised: Dynamic Temporal Pooling.

Original: DTP is the layer that is optimized by soft-DTW to segment the time axis while considering temporal relationship (Algorithm 1 in Appendix B).
Revised: DTP is the layer that is optimized by soft-DTW to segment the time axis while considering temporal relationship (Algorithm 1 in Appendix B).

Original: The DTW distance is calculated by point-to-point matching with temporal consistency,
Revised: The DTW distance is calculated by point-to-point matching with temporal consistency,

Original: [Equation]
Revised: [Equation]

Original: where X and Y are time series with lengths t 1 and t 2 , and the cost matrix ∆(
Revised: where X and Y are time series with lengths t 1 and t 2 , and the cost matrix ∆(

Original: X , Y ) ∈ R t 1 × trepresents the distance between X t 1 and Y t 2 .
Revised: X , Y ) ∈ R t 1 × trepresents the distance between X t 1 and Y t 2 .

Original: DTW is defined as the minimum inner product of the cost matrix with any binary alignment matrix A ∈ { 0 , 1 } t 1 × t 2 (Lee et al., 2021; Cuturi & Blondel, 2017).
Revised: DTW is defined as the minimum inner product of the cost matrix with any binary alignment matrix A ∈ { 0 , 1 } t 1 × t 2 (Lee et al., 2021; Cuturi & Blondel, 2017).

Original: With the soft-DTW algorithm, a similar time sequence is grouped with different lengths.
Revised: With the soft-DTW algorithm, a similar time sequence is grouped with different lengths.

Original: In this way, H is segmented in diverse time length ¯ ℓ =
Revised: In this way, H is segmented in diverse time length ¯ ℓ =

Original: [ ℓ 1 , ℓ 2 , ..., ℓ n ] , where t = (cid:80) ¯ ℓ .
Revised: [ ℓ 1 , ℓ 2 , ..., ℓ n ] , where t = (cid:80) ¯ ℓ .

Original: Finally, DTP ϕeffectively optimizes all the parameters, and the optimal pooled vectors p 3 =
Revised: Finally, DTP ϕeffectively optimizes all the parameters, and the optimal pooled vectors p 3 =

Original: [ p ℓ 1 , ..., p ℓ n ] is pooled from each of h ¯ ℓ , where p (¯ ℓ ) =
Revised: [ p ℓ 1 , ..., p ℓ n ] is pooled from each of h ¯ ℓ , where p (¯ ℓ ) =

Original: ϕ 3 ( h ¯ ℓ ) .
Revised: ϕ 3 ( h ¯ ℓ ) .

Original: Note that DTP has a dynamic local view with n segmentation with different lengths, which is formed by considering the temporal relationship between time steps.
Revised: Note that DTP has a dynamic local view with n segmentation with different lengths, which is formed by considering the temporal relationship between time steps.

Original: [Figure]
Revised: [Figure]

Original: circle is where pooling is highly focused and the green circle is
Revised: circle is where pooling is highly focused and the green circle is

Original: SoM-TP is highly focused on by new perspective.
Revised: SoM-TP is highly focused on by new perspective.

Original: 2. C ONSTRAINTS OF TEMPORAL POOLING METHODS
Revised: 2. C ONSTRAINTS OF TEMPORAL POOLING METHODS

Original: Each temporal pooling has different perspectives based on different mechanisms.
Revised: Each temporal pooling has different perspectives based on different mechanisms.

Original: And only one viewpoint exists, we define it as fixed-perspective learning.
Revised: And only one viewpoint exists, we define it as fixed-perspective learning.

Original: Through qualitative analysis with layerwise relevance propagation (LRP) and loss landscape, we examine how the mechanisms of each pooling operate in the model to capture time series patterns.
Revised: Through qualitative analysis with layerwise relevance propagation (LRP) and loss landscape, we examine how the mechanisms of each pooling operate in the model to capture time series patterns.

Original: LRP is a gradient-based method of computing input attribution for input features to explain network predictions (Binder et al., 2016b;a; Bach et al., 2015).
Revised: LRP is a gradient-based method of computing input attribution for input features to explain network predictions (Binder et al., 2016b;a; Bach et al., 2015).

Original: The input attribution from LRP shows how each temporal poolings focus on different points: LRP z + rule for Φ 1: l 1 and ϵ rule for f l 1 +1: l 2 .
Revised: The input attribution from LRP shows how each temporal poolings focus on different points: LRP z + rule for Φ 1: l 1 and ϵ rule for f l 1 +1: l 2 .

Original: GTP perspective.
Revised: GTP perspective.

Original: GTP focuses on dominant features with a global view.
Revised: GTP focuses on dominant features with a global view.

Original: However, GTP cannot capture multiple classification points dispersed on a time axis.
Revised: However, GTP cannot capture multiple classification points dispersed on a time axis.

Original: Thus, the global view has constraints on complex time series data.
Revised: Thus, the global view has constraints on complex time series data.

Original: STP Perspective.
Revised: STP Perspective.

Original: STP captures multiple local features.
Revised: STP captures multiple local features.

Original: However, STP segments important consecutive patterns by compulsory segmentation of equal length.
Revised: However, STP segments important consecutive patterns by compulsory segmentation of equal length.

Original: Also, STP pools the unimportant segmentations.
Revised: Also, STP pools the unimportant segmentations.

Original: This inefficiency causes representation power to be distributed loosely, leading to a performance loss.
Revised: This inefficiency causes representation power to be distributed loosely, leading to a performance loss.

Original: DTP Perspective.
Revised: DTP Perspective.

Original: DTP has the most complexity by optimizing segmentation length dynamically.
Revised: DTP has the most complexity by optimizing segmentation length dynamically.

Original: This flexibility enables the model to fully utilize segmentation power.
Revised: This flexibility enables the model to fully utilize segmentation power.

Original: However, since DTP is based on distance similarity, segmentation occurs at the change point.
Revised: However, since DTP is based on distance similarity, segmentation occurs at the change point.

Original: For this reason, DTP is hard to capture the consecutive pattern divided at the change point, such as an inflection point.
Revised: For this reason, DTP is hard to capture the consecutive pattern divided at the change point, such as an inflection point.

Original:  S O M-TP: T OWARDS D IVERSE P ERSPECTIVE L EARNING
Revised:  S O M-TP: T OWARDS D IVERSE P ERSPECTIVE L EARNING

Original: Diverse perspective learning is a new concept, fully utilizing all the views from each temporal pooling.
Revised: Diverse perspective learning is a new concept, fully utilizing all the views from each temporal pooling.

Original: Note that one fixed view, which is fixed-perspective learning done by individual temporal pooling, causes the model to have data dependency and performance degradation.
Revised: Note that one fixed view, which is fixed-perspective learning done by individual temporal pooling, causes the model to have data dependency and performance degradation.

Original: However, SoM-TP can overcome these limitations with diverse perspective learning.
Revised: However, SoM-TP can overcome these limitations with diverse perspective learning.

Original: We first introduce how SoM-TP obtains valuable information from multiple poolings.
Revised: We first introduce how SoM-TP obtains valuable information from multiple poolings.

Original: Next, we analyze the performance of SoM-TP in two directions: 1) What are the mechanisms that make the model learn diverse perspectives in SoM-TP? and 2)
Revised: Next, we analyze the performance of SoM-TP in two directions: 1) What are the mechanisms that make the model learn diverse perspectives in SoM-TP? and 2)

Original: Is SoM-TP more robust than other pooling methods?
Revised: Is SoM-TP more robust than other pooling methods?

Original: 3. S WITCH OVER M ULTIPLE P OOLING
Revised: 3. S WITCH OVER M ULTIPLE P OOLING

Original: SoM-TP is a framework to induce diverse perspective learning to achieve robustness through the pooling method.
Revised: SoM-TP is a framework to induce diverse perspective learning to achieve robustness through the pooling method.

Original: To leverage existing temporal poolings, SoM-TP needs three components; attention score, ensemble network, and perspective loss.
Revised: To leverage existing temporal poolings, SoM-TP needs three components; attention score, ensemble network, and perspective loss.

Original: Through these components, SoM-TP selects proper pooling by specific time series samples in one dataset.
Revised: Through these components, SoM-TP selects proper pooling by specific time series samples in one dataset.

Original: Learnable attention weight
Revised: Learnable attention weight

Original: The critical component of SoM-TP is learnable attention weight vectors for scoring the importance of each temporal pooling, which allows a single model to learn diverse perspectives.
Revised: The critical component of SoM-TP is learnable attention weight vectors for scoring the importance of each temporal pooling, which allows a single model to learn diverse perspectives.

Original: The purpose of the attention score is to properly select a specific pooling for data samples in each mini-batch unit.
Revised: The purpose of the attention score is to properly select a specific pooling for data samples in each mini-batch unit.

Original: For data-specific learning, learnable attention weights and a convolutional layer are used to calculate the attention score.
Revised: For data-specific learning, learnable attention weights and a convolutional layer are used to calculate the attention score.

Original: The attention score is derived from pooled vectors and one convolutional layer.
Revised: The attention score is derived from pooled vectors and one convolutional layer.

Original: The concatenated pooled vector ¯ P =
Revised: The concatenated pooled vector ¯ P =

Original: [ p 1 , p 2 , p 3 ] ∈ R k × 3 · n , where p ∈ R k × n from each temporal pooling, and attention weight vector A 0 ∈ R 1 × 3 · n are multiplied as the input of convolutional layer.
Revised: [ p 1 , p 2 , p 3 ] ∈ R k × 3 · n , where p ∈ R k × n from each temporal pooling, and attention weight vector A 0 ∈ R 1 × 3 · n are multiplied as the input of convolutional layer.

Original: For the attention weight vector, A 0 is the parameter to weight pooled vectors to highlight important pooling.
Revised: For the attention weight vector, A 0 is the parameter to weight pooled vectors to highlight important pooling.

Original: The multiplied input M ∈ R k × 3 · n are encoded to attention score A ∈ R 1 × 3 · n through the convolutional layer.
Revised: The multiplied input M ∈ R k × 3 · n are encoded to attention score A ∈ R 1 × 3 · n through the convolutional layer.

Original: The final attention sore is normalized with softmax, y j = exp ( x j ) (cid:80) i exp ( x i ) ,
Revised: The final attention sore is normalized with softmax, y j = exp ( x j ) (cid:80) i exp ( x i ) ,

Original: where x ∈ A , to compare the contribution of each feature from temporal poolings.
Revised: where x ∈ A , to compare the contribution of each feature from temporal poolings.

Original: For selecting proper pooling based on attention, there are two options: 1)
Revised: For selecting proper pooling based on attention, there are two options: 1)

Original: One pooling output p ∈ ¯ P with a maximum y j is selected, or 2) average y n = (cid:80) n y j n is proceeded in each pooling output, in the unit of length n which is pooled number, then the pooling output with the highest average y n is selected.
Revised: One pooling output p ∈ ¯ P with a maximum y j is selected, or 2) average y n = (cid:80) n y j n is proceeded in each pooling output, in the unit of length n which is pooled number, then the pooling output with the highest average y n is selected.

Original: Diverse perspective learning network and KL Divergence.
Revised: Diverse perspective learning network and KL Divergence.

Original: The SoM-TP framework has two fully-connected networks; the main decision network which is termed a ‘classification network’ and a sub-network which is termed a ‘diverse perspective learning network (DPLN)’.
Revised: The SoM-TP framework has two fully-connected networks; the main decision network which is termed a ‘classification network’ and a sub-network which is termed a ‘diverse perspective learning network (DPLN)’.

Original: To optimize attention weight and the whole model to learn diverse perspectives, a new loss function is introduced.
Revised: To optimize attention weight and the whole model to learn diverse perspectives, a new loss function is introduced.

Original: We define “perspective loss” which is the addition of DPLN loss and KL-divergence.
Revised: We define “perspective loss” which is the addition of DPLN loss and KL-divergence.

Original: While the classification network only focuses on minimizing the whole model of cross-entropy loss with selected pooling for each mini-batch, DPLN minimizes the classification loss with all pooled vectors.
Revised: While the classification network only focuses on minimizing the whole model of cross-entropy loss with selected pooling for each mini-batch, DPLN minimizes the classification loss with all pooled vectors.

Original: In detail, the ensemble output E =
Revised: In detail, the ensemble output E =

Original: A × ¯ P ∈ R k × 3 · n , from concatenated pooling vectors ¯ P multiplied by attention A , is given to DPLN.
Revised: A × ¯ P ∈ R k × 3 · n , from concatenated pooling vectors ¯ P multiplied by attention A , is given to DPLN.

Original: L DPLN is included in L perspective to optimize attention A with considering the classification result of DPLN.
Revised: L DPLN is included in L perspective to optimize attention A with considering the classification result of DPLN.

Original: The Kullback-Leibler divergence (KL-divergence) is specially used as a regularizer in the perspective loss.
Revised: The Kullback-Leibler divergence (KL-divergence) is specially used as a regularizer in the perspective loss.

Original: Through the optimization process, KL-divergence pulls y pred 1 to y pred 2 to reflect all perspectives, which prevents a single perspective pooling to be dominant.
Revised: Through the optimization process, KL-divergence pulls y pred 1 to y pred 2 to reflect all perspectives, which prevents a single perspective pooling to be dominant.

Original: The perspective loss driven by DPLN is defined as follows,
Revised: The perspective loss driven by DPLN is defined as follows,

Original: [Equation]
Revised: [Equation]

Original: where input time series { ( X 1 , y 1 ) , ..., ( X t , y t ) } , Φ with learnable parameter W 0 of CNN, y pred 1 ∈
Revised: where input time series { ( X 1 , y 1 ) , ..., ( X t , y t ) } , Φ with learnable parameter W 0 of CNN, y pred 1 ∈

Original: R 1 × c from the classification network W ( c ) , and y pred 2 ∈ R 1 × c from the DPLN W ( p ) .
Revised: R 1 × c from the classification network W ( c ) , and y pred 2 ∈ R 1 × c from the DPLN W ( p ) .

Original: We define pooling weight matrix W ( p ) =
Revised: We define pooling weight matrix W ( p ) =

Original: [ w ( p 1 ), ..., w ( p 3 ) 3 n ] ∈ R K × 3 · n , where w ( p ) ∈
Revised: [ w ( p 1 ), ..., w ( p 3 ) 3 n ] ∈ R K × 3 · n , where w ( p ) ∈

Original: R k that weighted importance of each latent dimension for pooling p by fully-connected layer, whereas
Revised: R k that weighted importance of each latent dimension for pooling p by fully-connected layer, whereas

Original: W ( c ) =
Revised: W ( c ) =

Original: [ w ( c ) 1 , ..., w ( c ) n ] ∈ R k × n is the class weight matrix (Lee et al., 2021).
Revised: [ w ( c ) 1 , ..., w ( c ) n ] ∈ R k × n is the class weight matrix (Lee et al., 2021).

Original: Therefore, the final loss function of the SoM-TP framework is defined as follows,
Revised: Therefore, the final loss function of the SoM-TP framework is defined as follows,

Original: [Equation]
Revised: [Equation]

Original: where W ∈ { W ( c ) , W ( p ) , A } of learnable parameters.
Revised: where W ∈ { W ( c ) , W ( p ) , A } of learnable parameters.

Original: With classification accuracy as a priority, the loss of the classification network L classification from y pred 2 is calculated through cross-entropy, and L perspective is added as λ decay.
Revised: With classification accuracy as a priority, the loss of the classification network L classification from y pred 2 is calculated through cross-entropy, and L perspective is added as λ decay.

Original: Therefore, CNN with learnable parameter W 0 adopts two fully-connected layers relatively while minimizing the similarity between each network output, as the concept of parameter tying.
Revised: Therefore, CNN with learnable parameter W 0 adopts two fully-connected layers relatively while minimizing the similarity between each network output, as the concept of parameter tying.

Original: Furthermore, SoM-TP proceeds with additional optimization for A .
Revised: Furthermore, SoM-TP proceeds with additional optimization for A .

Original: The dot product similarity is considered to regulate A .
Revised: The dot product similarity is considered to regulate A .

Original: L attn is learned in the direction of lowering the dot product similarity between the outputs of the classification network and the DPLN.
Revised: L attn is learned in the direction of lowering the dot product similarity between the outputs of the classification network and the DPLN.

Original: [Equation]
Revised: [Equation]

Original: L attn plays a role similar to KL-divergence in perspective loss but directly affects only attention weight.
Revised: L attn plays a role similar to KL-divergence in perspective loss but directly affects only attention weight.

Original: Shortly, SoM-TP learns various perspectives through attention, DPLN, and perspective loss.
Revised: Shortly, SoM-TP learns various perspectives through attention, DPLN, and perspective loss.

Original: Different pooled features affect convolutional and fully-connected networks to learn weight in distinct ways with individual pooling.
Revised: Different pooled features affect convolutional and fully-connected networks to learn weight in distinct ways with individual pooling.

Original: This is the most significant distinction between SoM-TP and fixedperspective learning.
Revised: This is the most significant distinction between SoM-TP and fixedperspective learning.

Original: 3. A BLATION STUDY ON S O M-TP
Revised: 3. A BLATION STUDY ON S O M-TP

Original: The novel point of SoM-TP is to make diversification of pooling selection that fits specific data even in one dataset.
Revised: The novel point of SoM-TP is to make diversification of pooling selection that fits specific data even in one dataset.

Original: Therefore, the ablation study is done to show how diverse perspective learning occurs.
Revised: Therefore, the ablation study is done to show how diverse perspective learning occurs.

Original: 3.2. E XPERIMENTAL S ETTING
Revised: 3.2. E XPERIMENTAL S ETTING

Original: For extensive evaluation, 112 univariate and 21 multivariate time series datasets from the UCR/UEA repositories are used (Bagnall et al., 2018; Dau et al., 2019); collected from a wide range of domains and publicly available.
Revised: For extensive evaluation, 112 univariate and 21 multivariate time series datasets from the UCR/UEA repositories are used (Bagnall et al., 2018; Dau et al., 2019); collected from a wide range of domains and publicly available.

Original: CNN classifiers with pooling layers are built on a common ground of model architecture.
Revised: CNN classifiers with pooling layers are built on a common ground of model architecture.

Original: FCN and
Revised: FCN and

Original: ResNet are specifically designed for TSC (Wang et al., 2017), and three different temporal poolings are customized with settings: normalization with BatchNorm (Ioffe & Szegedy, 2015), activation
Revised: ResNet are specifically designed for TSC (Wang et al., 2017), and three different temporal poolings are customized with settings: normalization with BatchNorm (Ioffe & Szegedy, 2015), activation

Original: [Table] [Table]
Revised: [Table] [Table]

Original: function with ReLU, optimizer with Adam (Kingma & Ba, 2014).
Revised: function with ReLU, optimizer with Adam (Kingma & Ba, 2014).

Original: The validation dataset is made from 20% of the training set for a more accurate evaluation.
Revised: The validation dataset is made from 20% of the training set for a more accurate evaluation.

Original: For an imbalanced class, the weighted loss is used to train.
Revised: For an imbalanced class, the weighted loss is used to train.

Original: The prototype number n is distributed greedily while accounting for each dataset’s unique class number.
Revised: The prototype number n is distributed greedily while accounting for each dataset’s unique class number.

Revised: A more detailed experimental setting is in Appendix A.

Revised: [Table]

Original: 3.2. L EARNING PROCESS ANALYSIS
Revised: 3.2. L EARNING PROCESS ANALYSIS

Original: How specific pooling is chosen by each data?
Revised: How specific pooling is chosen by each data?

Original: SoM-TP identifies data for each mini-batch unit and dynamically selects appropriate pooling.
Revised: SoM-TP identifies data for each mini-batch unit and dynamically selects appropriate pooling.

Original: In Figure 3, the learning process is introduced: How the attention finds optimal pooling.
Revised: In Figure 3, the learning process is introduced: How the attention finds optimal pooling.

Original: SoM-TP reflects every data without converging to one dominant pooling.
Revised: SoM-TP reflects every data without converging to one dominant pooling.

Original: Dynamic selection can be done during the inference process without the use of a subnetwork DPLN.
Revised: Dynamic selection can be done during the inference process without the use of a subnetwork DPLN.

Original: SoM-TP extended ablation study on the individual datasets is in Appendix A.1.
Revised: SoM-TP extended ablation study on the individual datasets is in Appendix A.3.

Original: The difference in perspective when compared to individual poolings.
Revised: The difference in perspective when compared to individual poolings.

Original: A qualitative analysis using LRP is performed to examine how the perspectives of SoM-TP are different from those of other temporal pooling methods.
Revised: A qualitative analysis using LRP is performed to examine how the perspectives of SoM-TP are different from those of other temporal pooling methods.

Original: In Figure 2, SoM-TP overcomes the limitation of independent pooling and learns better by taking mixed views of global and local.
Revised: In Figure 2, SoM-TP overcomes the limitation of independent pooling and learns better by taking mixed views of global and local.

Original: In detail, SoM-TP outperforms by focusing on hidden important features that others are not concentrating on.
Revised: In detail, SoM-TP outperforms by focusing on hidden important features that others are not concentrating on.

Original: What is the role of DPLN?
Revised: What is the role of DPLN?

Original: DPLN is an ensemble network in that the network uses the weighted pooling outputs.
Revised: DPLN is an ensemble network in that the network uses the weighted pooling outputs.

Original: SoM-TP employs DPLN’s result for the perspective loss.
Revised: SoM-TP employs DPLN’s result for the perspective loss.

Original: In other words, SoM-TP uses its ensemble result for optimization.
Revised: In other words, SoM-TP uses its ensemble result for optimization.

Revised: Furthermore, the λ ablation study on perspective loss is in

Revised: Appendix A.2.

Original: 3.2. P ERFORMANCE ANALYSIS
Revised: 3.2. P ERFORMANCE ANALYSIS

Original: SoM-TP shows robust performance for the various domain TSC datasets.
Revised: SoM-TP shows robust performance for the various domain TSC datasets.

Original: To investigate the performance of SoM-TP in detail, we analyzed both ways; quantitative results and robustness.
Revised: To investigate the performance of SoM-TP in detail, we analyzed both ways; quantitative results and robustness.

Original: We calculated the average performance of the entire dataset (Table 2).
Revised: We calculated the average performance of the entire dataset (Table 2).

Original: We considered accuracy and f1 macro score to deal with the imbalanced class in datasets.
Revised: We considered accuracy and f1 macro score to deal with the imbalanced class in datasets.

Original: SoM-TP outperforms the existing temporal pooling methods both in univariate and multivariate time series datasets.
Revised: SoM-TP outperforms the existing temporal pooling methods both in univariate and multivariate time series datasets.

Original: Furthermore,
Revised: Furthermore,

Original: Figure 4 histograms show the superior performance of SoM-TP than DTP which is the SOTA of temporal pooling.
Revised: Figure 4 histograms show the superior performance of SoM-TP than DTP which is the SOTA of temporal pooling.

Original: SoM-TP has long tail than DTP, which means that highly beats the DTP in accuracy.
Revised: SoM-TP has long tail than DTP, which means that highly beats the DTP in accuracy.

Original: The most highest beaten in one dataset achieved a 5% increase.
Revised: The most highest beaten in one dataset achieved a 5% increase.

Original: We take additionalevaluation metrics as a histogram under area. As shown in (a), (b)-histogram, the area of SoM-TP is
Revised: We take additional

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We choose histograms and bar charts to show the superiority of SoM-TP in a different way.
Revised: We choose histograms and bar charts to show the superiority of SoM-TP in a different way.

Original: The performance in the graph is based on accuracy.
Revised: The performance in the graph is based on accuracy.

Original: For the histogram, the x-axis means the value obtained by subtracting DTP from SoM-TP and the y-axis is the number of datasets.
Revised: For the histogram, the x-axis means the value obtained by subtracting DTP from SoM-TP and the y-axis is the number of datasets.

Original: Here, we can see that there are both cases in which each pooling performs better than the other.
Revised: Here, we can see that there are both cases in which each pooling performs better than the other.

Original: However, there are more cases in that SoM-TP outperforms with a big gap in both UCR and UEA repositories.
Revised: However, there are more cases in that SoM-TP outperforms with a big gap in both UCR and UEA repositories.

Original: Also, as shown in the bar charts with performance ranks between four temporal poolings, SoM-TP has the most robust performance than any other pooling with rank 1 highest and the following ranks becoming lower, clearly shown in UEA.
Revised: Also, as shown in the bar charts with performance ranks between four temporal poolings, SoM-TP has the most robust performance than any other pooling with rank 1 highest and the following ranks becoming lower, clearly shown in UEA.

Original: However, for DTP, which is the SOTA of temporal pooling, it is not clear that overall performance is better than STP.
Revised: However, for DTP, which is the SOTA of temporal pooling, it is not clear that overall performance is better than STP.

Original: Through these results, we can check the robustness and performance of SoM-TP compared to other temporal pooling.
Revised: Through these results, we can check the robustness and performance of SoM-TP compared to other temporal pooling.

Original:  bigger than DTP’s as much as +1 .
Revised: evaluation metrics as a histogram under area. As shown in (a), (b)-histogram, the area of SoM-TP is bigger than DTP’s as much as +1 .

Original: 1083 and +0 .
Revised: 1083 and +0 .

Original: 1763 respectively.
Revised: 1763 respectively.

Original: Through (a), (b)-bar charts, we also check that SoM-TP has the most rank 1 and least rank 4.
Revised: Through (a), (b)-bar charts, we also check that SoM-TP has the most rank 1 and least rank 4.

Original: However, there are a few cases where fixed-perspective learning is dominant due to its data nature.
Revised: However, there are a few cases where fixed-perspective learning is dominant due to its data nature.

Original: The high degree of freedom of the SoM-TP mechanism can cause performance degradation for data with simple classification points.
Revised: The high degree of freedom of the SoM-TP mechanism can cause performance degradation for data with simple classification points.

Original: This problem appears as an outlier, especially in the feature extraction with FCN.
Revised: This problem appears as an outlier, especially in the feature extraction with FCN.

Original: 3. P OOLING CLASSIFICATION
Revised: 3. P OOLING CLASSIFICATION

Original: SoM-TP selects the most appropriate pooling method based on specific data.
Revised: SoM-TP selects the most appropriate pooling method based on specific data.

Original: However, if there is no relationship between data and pooling, selecting suitable pooling according to each characteristic of the data samples cannot be generalized.
Revised: However, if there is no relationship between data and pooling, selecting suitable pooling according to each characteristic of the data samples cannot be generalized.

Original: Without this proposition, SoM-TP is not different from the random selection of poolings.
Revised: Without this proposition, SoM-TP is not different from the random selection of poolings.

Original: Through an empirical study of pooling classification, we prove the existence of a relationship between distinct data and best-fitted poolings.
Revised: Through an empirical study of pooling classification, we prove the existence of a relationship between distinct data and best-fitted poolings.

Original: The challenge here is to prove not only the relationship between data and pooling but also its perspective.

Original: Therefore, we use relevance score as a multivariate input along with raw time series.

Original: The

Original: [Figure]
Revised: [Figure]

Original: GTP
Revised: GTP

Original: STP
Revised: STP

Original: DTP

Original: [Figure]
Revised: [Figure]

Original: Then multivariate input with raw time series and input attribution goes through the model to classify the best pooling methods for each dataset.
Revised: Then multivariate input with raw time series and input attribution goes through the model to classify the best pooling methods for each dataset.

Original: In this network, the simple global pooling layer is used.
Revised: In this network, the simple global pooling layer is used.

Revised: The challenge here is to prove not only the relationship between data and pooling but also its perspective.

Revised: Therefore, we use relevance score as a multivariate input along with raw time series.

Original:  relevance score, which is input attribution, is the LRP result from fixed-perspective learning.
Revised: The relevance score, which is input attribution, is the LRP result from fixed-perspective learning.

Original: By adding LRP values as input, the model can recognize important features that contribute to classification, which is captured by distinct perspective of each polings.
Revised: By adding LRP values as input, the model can recognize important features that contribute to classification, which is captured by distinct perspective of each polings.

Original: This setting is also consistent with the fact that TSC is mainly done by capturing a particular pattern in time series.
Revised: This setting is also consistent with the fact that TSC is mainly done by capturing a particular pattern in time series.

Original: Experimental settings.
Revised: Experimental settings.

Original: Given a training set of N samples of time series and an LRP value, T = { ( X 1 , L 1 , y ) , ..., ( X N , L N , y ) } , from class c of three temporal pooling { GTP: 0, STP: 1, DTP: 2 } , we aim to learn the CNN classifier.
Revised: Given a training set of N samples of time series and an LRP value, T = { ( X 1 , L 1 , y ) , ..., ( X N , L N , y ) } , from class c of three temporal pooling { GTP: 0, STP: 1, DTP: 2 } , we aim to learn the CNN classifier.

Original: Note that we set the best pooling method as a target class.
Revised: Note that we set the best pooling method as a target class.

Original: Performance analysis.
Revised: Performance analysis.

Original: Considering the imbalance of classification target labels, we use F1-score as an evaluation metric.
Revised: Considering the imbalance of classification target labels, we use F1-score as an evaluation metric.

Original: When only time series input is used as univariate classification, the performance is 74% of the f1-score.
Revised: When only time series input is used as univariate classification, the performance is 74% of the f1-score.

Original: However, with a relevance score, the performance increases to 78%.
Revised: However, with a relevance score, the performance increases to 78%.

Original: This result shows that the data categorization is possible with the relationship proved; each time series has appropriate pooling with distinct views.
Revised: This result shows that the data categorization is possible with the relationship proved; each time series has appropriate pooling with distinct views.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposes SoM-TP, switch over multiple pooling that learns diverse perspectives to fully utilize temporal pooling.
Revised: This paper proposes SoM-TP, switch over multiple pooling that learns diverse perspectives to fully utilize temporal pooling.

Original: The pooling perspective that we defined is from the distinct mechanism of either segmentation on temporal order, or dynamic segmentation length by considering temporal relationships.
Revised: The pooling perspective that we defined is from the distinct mechanism of either segmentation on temporal order, or dynamic segmentation length by considering temporal relationships.

Original: SoM-TP is not one pooling layer, but an architecture, which has a learning framework with perspective loss.
Revised: SoM-TP is not one pooling layer, but an architecture, which has a learning framework with perspective loss.

Original: With the attention weight vector in SoM-TP, pooling is selected dynamically based on the each data.
Revised: With the attention weight vector in SoM-TP, pooling is selected dynamically based on the each data.

Original: The extensive experiment with the UCR/UEA repository validated the performance of our new method, and with LRP, we saw SoM-TP view every data sample with a new perspective that is not seen in existing temporal pooling methods.
Revised: The extensive experiment with the UCR/UEA repository validated the performance of our new method, and with LRP, we saw SoM-TP view every data sample with a new perspective that is not seen in existing temporal pooling methods.

Original: A E XTENDED ANALYSIS ON S O M-TP A. E XTENDED S O M-TP ABLATION STUDY WITH AN INDIVIDUAL DATASET
Revised: A E XTENDED ANALYSIS ON S O M-TP A. D ETAILED EXPERIMENTAL SETTING

Revised: I npu t I np u t

Original: [Figure]
Revised: [Figure]

Revised: ResNet

Original: -TP works robustly and dynamically selects appropriate pooling both at training and inference procedure.
Revised: B N + R e L U B N + R e L U B N + R e L U B N + R e L U

Revised: B N + R e L U

Revised: B N + R e L U

Revised: B N + R e L U

Revised: B N + R e L U

Revised: B N + R e L U

Revised: B N + R e L U

Revised: B N + R e L U

Revised: B N + R e L U p oo li n g p oo li n g

Revised: F C + S o f t M a x

Revised: F C + S o f t M a x +

Revised: + +

Revised: A. P ERSPECTIVE LOSS : λ ABLATION STUDY

Revised: [Figure]

Revised: A. E XTENDED S O M-TP ABLATION STUDY WITH AN INDIVIDUAL DATASET

Revised: [Figure]

Revised: ate pooling both at training and inference procedure.

Original: With the first column, the x-axis, we can see the different time lengths of each dataset.
Revised: With the first column, the x-axis, we can see the different time lengths of each dataset.

Original: Also, with the same 20 epoch and 8 batch size, the batch number is different, which means the difference in data size of each individual dataset.
Revised: Also, with the same 20 epoch and 8 batch size, the batch number is different, which means the difference in data size of each individual dataset.

Original: However,
Revised: However,

Original: SoM-TP learns diverse perspectives from the training procedure and peaks different but appropriate pooling by specific data samples.
Revised: SoM-TP learns diverse perspectives from the training procedure and peaks different but appropriate pooling by specific data samples.

Original: [Figure]
Revised: [Figure]

Original: robustly performs with proper selection.
Revised: robustly performs with proper selection.

Original: B DTP ALGORITHM
Revised: B DTP ALGORITHM

Original: Function DTP( P , H ) : δ ( p l , h t ) =
Revised: Function DTP( P , H ) : δ ( p l , h t ) =

Original: 1 − p l · h t ||
Revised: 1 − p l · h t ||

Original: p l
Revised: p l

Original: || 2 ||
Revised: || 2 ||

Original: h t
Revised: h t

Original: ||
Revised: ||

Original: Function forward( P , H ) :
Revised: Function forward( P , H ) :

Original: ▷ fill the alignment cost matrix R ∈
Revised: ▷ fill the alignment cost matrix R ∈

Original: R L × T
Revised: R L × T

Original: R 0 , 0 = 0 , R : , 0 = R 0 , : = ∞ for l = 1 , ..., L do for t = 1 , ...,
Revised: R 0 , 0 = 0 , R : , 0 = R 0 , : = ∞ for l = 1 , ..., L do for t = 1 , ...,

Original: T do
Revised: T do

Original: R l,t = δ ( p l , h t ) + min γ { R l − 1 ,t − 1 , R l,t − 1 } return DTW γ ( P , H )
Revised: R l,t = δ ( p l , h t ) + min γ { R l − 1 ,t − 1 , R l,t − 1 } return DTW γ ( P , H )

Original: =
Revised: =

Original: R L,T
Revised: R L,T

Original: Function backward( P , H ) :
Revised: Function backward( P , H ) :

Original: ▷ fill the soft alignment matrix E ∈ R L × T
Revised: ▷ fill the soft alignment matrix E ∈ R L × T

Original: E l,t = ∂R
Revised: E l,t = ∂R

Original: L,T /∂R l,t
Revised: L,T /∂R l,t

Original: E : , T + 1 = E L +1 , : =
Revised: E : , T + 1 = E L +1 , : =

Original: R : ,T +1 =
Revised: R : ,T +1 =

Original: R L +1 , : =
Revised: R L +1 , : =

Original: −∞ for l = L, ..., 1 do for t =
Revised: −∞ for l = L, ..., 1 do for t =

Original: T, ..., 1 do a = exp 1 γ
Revised: T, ..., 1 do a = exp 1 γ

Original: ( R l,t +1 − R l,t − δ ( p l , h t +1 ))
Revised: ( R l,t +1 − R l,t − δ ( p l , h t +1 ))

Original: b = exp 1 γ
Revised: b = exp 1 γ

Original: ( R l +1 ,t +1 − R l,t − δ ( p l +1 , h t +1 ))
Revised: ( R l +1 ,t +1 − R l,t − δ ( p l +1 , h t +1 ))

Original: E l,t = a · E l,t +1 + b · E l +1 ,t + return ∇ P DTW
Revised: E l,t = a · E l,t +1 + b · E l +1 ,t + return ∇ P DTW

Original: γ
Revised: γ

Original: ( P , H )
Revised: ( P , H )

Original: = ( ∂ ∆( P , H )
Revised: = ( ∂ ∆( P , H )

Original: ∂ P ) T E
Revised: ∂ P ) T E

Original: Function optimization( ( X , y ) , P , Φ ) :
Revised: Function optimization( ( X , y ) , P , Φ ) :

Original: ▷ W : network Φ parameter ▷w ( c ) ∈ R K of class weight vector ▷ W c
Revised: ▷ W : network Φ parameter ▷w ( c ) ∈ R K of class weight vector ▷ W c

Original: =
Revised: =

Original: [ w ( c ) 1 , ..., w ( c ) L ] ∈ R k
Revised: [ w ( c ) 1 , ..., w ( c ) L ] ∈ R k

Original: × L of class weight matrix ▷P
Revised: × L of class weight matrix ▷P

Original: ( y = c | X ) =
Revised: ( y = c | X ) =

Original: [Equation]
Revised: [Equation]

Original: Lc ′
Revised: Lc ′

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: P [Equation] P
Revised: P ← P

Original: [Equation]
Revised: [Equation]

Original: proto 
Revised: proto /

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: class
Revised: class

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: W
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: c 
Revised: c )

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: c )

Original: C E XTENDED RELATED WORK
Revised: C E XTENDED RELATED WORK

Original: Conventional approach for TSC.
Revised: Conventional approach for TSC.

Original: In early TSC, there were various approaches to classifying time series data based on the similarity of distance or feature pattern (Geurts, 2001; Alcock et al., 1999;
Revised: In early TSC, there were various approaches to classifying time series data based on the similarity of distance or feature pattern (Geurts, 2001; Alcock et al., 1999;

Original: Fulcher & Jones, 2014; Abanda et al., 2019; Bagnall et al., 2017; Giusti & Batista, 2013).
Revised: Fulcher & Jones, 2014; Abanda et al., 2019; Bagnall et al., 2017; Giusti & Batista, 2013).

Original: The distance-based metric developed variously, and especially DTW, which is also used in DTP, is applied dynamically to TSC (Batista et al., 2014; Lines & Bagnall, 2015; Cuturi & Blondel, 2017;
Revised: The distance-based metric developed variously, and especially DTW, which is also used in DTP, is applied dynamically to TSC (Batista et al., 2014; Lines & Bagnall, 2015; Cuturi & Blondel, 2017;

Original: Jeong et al., 2011; Marteau, 2008; Stefan et al., 2012; Yuan et al., 2019b; Zhao & Itti, 2018).
Revised: Jeong et al., 2011; Marteau, 2008; Stefan et al., 2012; Yuan et al., 2019b; Zhao & Itti, 2018).

Original: There are also various algorithms that serve as a noble foundation (Bagnall et al., 2015; Lines et al., 2016;
Revised: There are also various algorithms that serve as a noble foundation (Bagnall et al., 2015; Lines et al., 2016;

Original: Baydogan et al., 2013; Lines et al., 2012; Ma et al., 2020; Sch¨afer, 2015; Yuan et al., 2019a).
Revised: Baydogan et al., 2013; Lines et al., 2012; Ma et al., 2020; Sch¨afer, 2015; Yuan et al., 2019a).

Original: However, these baselines are defeated by DNN architecture (Lee et al., 2021; Ismail Fawaz et al., 2019).
Revised: However, these baselines are defeated by DNN architecture (Lee et al., 2021; Ismail Fawaz et al., 2019).

Original: Deep Neural Network for TSC.
Revised: Deep Neural Network for TSC.

Original: Especially, CNN has diverse architecture for TSC, which tried to consider time series characteristics (Cui et al., 2016; Karim et al., 2017; Ismail Fawaz et al., 2020;
Revised: Especially, CNN has diverse architecture for TSC, which tried to consider time series characteristics (Cui et al., 2016; Karim et al., 2017; Ismail Fawaz et al., 2020;

Original: Zheng et al., 2016; Kashiparekh et al., 2019;
Revised: Zheng et al., 2016; Kashiparekh et al., 2019;

Original: He et al., 2022; Duan et al., 2022).
Revised: He et al., 2022; Duan et al., 2022).

Original: Also, there were novel trials to interpret time series data and its classification (Senin & Malinchik, 2013), specifically, prototype and attention-based approaches were usually driven (Zhang et al., 2020;
Revised: Also, there were novel trials to interpret time series data and its classification (Senin & Malinchik, 2013), specifically, prototype and attention-based approaches were usually driven (Zhang et al., 2020;

Original: Gee et al., 2019;
Revised: Gee et al., 2019;

Original: Ghods & Cook, 2022; Cho et al., 2021).
Revised: Ghods & Cook, 2022; Cho et al., 2021).

Original: However, for TSC with CNN, there were few studies that used LRP for explaining and interpreting the model and its time series input attribution (Rojat et al., 2021).
Revised: However, for TSC with CNN, there were few studies that used LRP for explaining and interpreting the model and its time series input attribution (Rojat et al., 2021).

Original: Pooling for TSC.
Revised: Pooling for TSC.

Original: In other fields, especially computer vision and natural language processing, there were many approaches to novel pooling methods (Mao et al., 2021; Sun et al., 2017; Ibrahim et al., 2021;
Revised: In other fields, especially computer vision and natural language processing, there were many approaches to novel pooling methods (Mao et al., 2021; Sun et al., 2017; Ibrahim et al., 2021;

Original: Christlein et al., 2019; Dong et al., 2020; Zhang et al., 2018; Williams & Li, 2018; Chan et al., 2021;
Revised: Christlein et al., 2019; Dong et al., 2020; Zhang et al., 2018; Williams & Li, 2018; Chan et al., 2021;

Original: Zhai et al., 2017; Passalis & Tefas, 2017; Gao et al., 2019).
Revised: Zhai et al., 2017; Passalis & Tefas, 2017; Gao et al., 2019).

Original: There are, however, few approaches for the time series domain (Dempster et al., 2020; Tan et al., 2021; Wang et al., 2016).
Revised: There are, however, few approaches for the time series domain (Dempster et al., 2020; Tan et al., 2021; Wang et al., 2016).