Original: T EMPORAL A LIGNMENT P REDICTION FOR S UPERVISED R EPRESENTATION L EARNING AND F EW -S HOT S EQUENCE C LASSIFICATION
Revised: T EMPORAL A LIGNMENT P REDICTION FOR S UPERVISED R EPRESENTATION L EARNING AND F EW -S HOT S EQUENCE C LASSIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Explainable distances for sequence data depend on temporal alignment to tackle sequences with different lengths and local variances.
Revised: Explainable distances for sequence data depend on temporal alignment to tackle sequences with different lengths and local variances.

Original: Most sequence alignment methods infer the optimal alignment by solving an optimization problem under pre-deﬁned feasible alignment constraints, which not only is time
Revised: Most sequence alignment methods infer the optimal alignment by solving an optimization problem under pre-deﬁned feasible alignment constraints, which not only is time

Original: -consuming, but also makes end-to-end sequence learning intractable
Revised: -consuming, but also makes end-to-end sequence learning intractable

Original: In this paper, we propose a learnable sequence distance called Temporal Alignment Prediction (
Revised: In this paper, we propose a learnable sequence distance called Temporal Alignment Prediction (

Original: TAP
Revised: TAP

Original: TAP employs a lightweight convolutional neural network to directly predict the opti
Revised: TAP employs a lightweight convolutional neural network to directly predict the opti

Original: - mal alignment between two sequences
Revised: - mal alignment between two sequences

Original: , so that only straightforward calculations are required and no optimization is involved in inference.
Revised: , so that only straightforward calculations are required and no optimization is involved in inference.

Original: TAP can be applied in different distance-based machine learning tasks.
Revised: TAP can be applied in different distance-based machine learning tasks.

Original: For supervised sequence representation learning, we show that TAP trained with various metric learning losses achieves completive performances with much faster inference speed.
Revised: For supervised sequence representation learning, we show that TAP trained with various metric learning losses achieves completive performances with much faster inference speed.

Original: For fewshot action classiﬁcation, we apply TAP as the distance measure in the metric learning-based episode-training paradigm.
Revised: For fewshot action classiﬁcation, we apply TAP as the distance measure in the metric learning-based episode-training paradigm.

Original: This simple strategy achieves comparable results with state-of-the-art few-shot action recognition methods.
Revised: This simple strategy achieves comparable results with state-of-the-art few-shot action recognition methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Distance between sequences plays a crucial role in sequence classiﬁcation (Sakoe & Chiba, 1978), retrieval (Su et al., 2019), clustering (Garc ´ ıa-Garc ´ ıa et al., 2008), etc.
Revised: Distance between sequences plays a crucial role in sequence classiﬁcation (Sakoe & Chiba, 1978), retrieval (Su et al., 2019), clustering (Garc ´ ıa-Garc ´ ıa et al., 2008), etc.

Original: Measuring distance between sequences is difﬁcult, since different sequences may have different sampling rates, execution speeds, local distortions, initial states, and elastic warpage.
Revised: Measuring distance between sequences is difﬁcult, since different sequences may have different sampling rates, execution speeds, local distortions, initial states, and elastic warpage.

Original: To tackle such temporal variances, existing sequence distances either encode each sequence into a feature vector invariant to temporal variances (Abid & Zou, 2018; Lohit et al., 2019) or employ alignment for temporal correspondence calibration (Sakoe & Chiba, 1978; Su & Hua, 2019).
Revised: To tackle such temporal variances, existing sequence distances either encode each sequence into a feature vector invariant to temporal variances (Abid & Zou, 2018; Lohit et al., 2019) or employ alignment for temporal correspondence calibration (Sakoe & Chiba, 1978; Su & Hua, 2019).

Original: Typical feature-based methods use recurrent neural networks (RNNs) (Ramachandran et al., 2017) to encode sequences and measure the Euclidean distance between corresponding features.
Revised: Typical feature-based methods use recurrent neural networks (RNNs) (Ramachandran et al., 2017) to encode sequences and measure the Euclidean distance between corresponding features.

Original: The feature of a sequence is ﬁxed when compared with any other sequence.
Revised: The feature of a sequence is ﬁxed when compared with any other sequence.

Original: Although such methods are naturally learnable and only perform forward calculations in inference, they require large amounts of sequences to train complex RNNs and how the learned features handle temporal variances and what kinds of variances can be handled are not clear.
Revised: Although such methods are naturally learnable and only perform forward calculations in inference, they require large amounts of sequences to train complex RNNs. Moreover, how the learned features handle temporal variances and what kinds of variances can be handled are not clear.

Original: Alignment-based methods determine different optimal alignments for different sequence pairs.
Revised: Alignment-based methods determine different optimal alignments for different sequence pairs.

Original: This is more intuitive and ﬂexible because temporal variances may be different when comparing different sequences.
Revised: This is more intuitive and ﬂexible because temporal variances may be different when comparing different sequences.

Original: The inferred alignments clearly indicate how and where the two sequences differ in temporal steps.
Revised: The inferred alignments clearly indicate how and where the two sequences differ in temporal steps.

Original: Most alignment methods solve an optimization problem under pre-deﬁned feasible constraints to infer the optimal alignment.
Revised: Most alignment methods solve an optimization problem under pre-deﬁned feasible constraints to infer the optimal alignment.

Original: E.g., DTW (Sakoe & Chiba, 1978) requires dynamic programming and OPW (Su & Hua, 2019) employs ﬁxed-point iterations.
Revised: E.g., DTW (Sakoe & Chiba, 1978) requires dynamic programming and OPW (Su & Hua, 2019) employs ﬁxed-point iterations.

Original: Such optimizations are often time-consuming and cannot fully utilize GPU.
Revised: Such optimizations are often time-consuming and cannot fully utilize GPU.

Original: Moreover, since inferring the alignment is itself an optimization problem and has its own objective, sequence distance-based end-to-end learning using other objectives becomes intractable.
Revised: Moreover, since inferring the alignment is itself an optimization problem and has its own objective, sequence distance-based end-to-end learning using other objectives becomes intractable.

Original: For instance, learning discriminative temporal representations for elements in sequences often adopt the objective that sequences of different classes are better separated w.r.t.
Revised: For instance, learning discriminative temporal representations for elements in sequences often adopt the objective that sequences of different classes are better separated w.r.t.

Original: a sequence distance (Mei et al.,
Revised: a sequence distance (Mei et al.,

Original: 2014; Su & Wu, 2020b).
Revised: 2014; Su & Wu, 2020b).

Original: Gradients of this overall objective are difﬁcult to pass through alignments since they are latent variables determined by another optimization problem.
Revised: Gradients of this overall objective are difﬁcult to pass through alignments since they are latent variables determined by another optimization problem.

Original: In this paper, we propose a learnable alignment-based sequence distance, namely Temporal Alignment Prediction (TAP) .
Revised: In this paper, we propose a learnable alignment-based sequence distance, namely Temporal Alignment Prediction (TAP) .

Original: TAP simulates the optimization-based inference from the input sequence pair to the optimal alignment as a function.
Revised: TAP simulates the optimization-based inference from the input sequence pair to the optimal alignment as a function.

Original: The function is modeled by a lightweight convolutional neural network to directly predict the optimal alignment.
Revised: The function is modeled by a lightweight convolutional neural network to directly predict the optimal alignment.

Original: TAP can be applied in different distance-based machine learning tasks under different settings.
Revised: TAP can be applied in different distance-based machine learning tasks under different settings.

Original: As instances, we show the applications of TAP in supervised representation learning and few-shot learning for sequence data.
Revised: As instances, we show the applications of TAP in supervised representation learning and few-shot learning for sequence data.

Original: For supervised learning, we employ metric learning losses by using TAP as the distance measure to learn the frame-wise temporal representations.
Revised: For supervised learning, we employ metric learning losses by using TAP as the distance measure to learn the frame-wise temporal representations.

Original: For few-shot learning, we adopt TAP as the distance measure in the metric learning based paradigm to compare the query and support sequence samples.
Revised: For few-shot learning, we adopt TAP as the distance measure in the metric learning based paradigm to compare the query and support sequence samples.

Original: In both cases, owing to the straightforward structure of TAP, the alignment prediction network and the feature extraction or transformation network can be jointly trained in an end-to-end manner, resulting in principled yet straightforward supervised learning and few-shot learning solutions for sequences.
Revised: In both cases, owing to the straightforward structure of TAP, the alignment prediction network and the feature extraction or transformation network can be jointly trained in an end-to-end manner, resulting in principled yet straightforward supervised learning and few-shot learning solutions for sequences.

Original: We further show the application of TAP in self-supervised alignment learning in the appendix.
Revised: We further show the application of TAP in self-supervised alignment learning in the appendix.

Original: The main contributions of this paper are three-fold. 1.
Revised: The main contributions of this paper are three-fold. 1.

Original: We propose TAP, a learnable and fast sequence distance that only requires straightforward calculations in inference.
Revised: We propose TAP, a learnable and fast sequence distance that only requires straightforward calculations in inference.

Original: We show that TAP conveniently enables end-to-end supervised sequence representation learning with different losses.
Revised: We show that TAP conveniently enables end-to-end supervised sequence representation learning with different losses.

Original: We adopt TAP to dynamically align the support and query sequences for few-shot action recognition, so that alignments and temporal representations can be jointly learned with the episode-training strategy.
Revised: We adopt TAP to dynamically align the support and query sequences for few-shot action recognition, so that alignments and temporal representations can be jointly learned with the episode-training strategy.

Original: Experiments on six datasets demonstrate the effectiveness of TAP for both supervised learning and few-shot learning.
Revised: Experiments on seven datasets demonstrate the effectiveness of TAP for both supervised learning and few-shot learning.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Alignment-based sequence distance.
Revised: Alignment-based sequence distance.

Original: Various variations of DTW have been proposed to speed up the inference (Salvador & Chan, 2007; Al-Naymat et al., 2009), or adapt to additional or modiﬁed constraints (Ratanamahatana & Keogh, 2004), or tackle sequences from different modalities (Zhou & Torre, 2009;
Revised: Various variations of DTW have been proposed to speed up the inference (Salvador & Chan, 2007; Al-Naymat et al., 2009), or adapt to additional or modiﬁed constraints (Ratanamahatana & Keogh, 2004), or tackle sequences from different modalities (Zhou & Torre, 2009;

Original: Zhou & De la Torre, 2016; Trigeorgis et al., 2016; 2017; Cohen et al., 2021).
Revised: Zhou & De la Torre, 2016; Trigeorgis et al., 2016; 2017; Cohen et al., 2021).

Original: They rely on non-differential dynamic programming for inference.
Revised: They rely on non-differential dynamic programming for inference.

Original: Soft-DTW (Cuturi & Blondel, 2017) optimizes a differentiable loss by using the soft-minimum cost of all feasible alignments as the objective.
Revised: Soft-DTW (Cuturi & Blondel, 2017) optimizes a differentiable loss by using the soft-minimum cost of all feasible alignments as the objective.

Original: TCOT (Su et al., 2019) and OPW (Su & Hua, 2019) view the alignment problem as the optimal transport problem with temporal regularizations.
Revised: TCOT (Su et al., 2019) and OPW (Su & Hua, 2019) view the alignment problem as the optimal transport problem with temporal regularizations.

Original: These methods solve an optimization problem to infer the optimal alignment.
Revised: These methods solve an optimization problem to infer the optimal alignment.

Original: In contrast, our method directly predicts the alignment using a neural network, does not have its own objective, and only involves differentiable calculations.
Revised: In contrast, our method directly predicts the alignment using a neural network, does not have its own objective, and only involves differentiable calculations.

Original: Data-driven alignment methods.
Revised: Data-driven alignment methods.

Original: In Abid & Zou (2018), Autowarp ﬁnds the warping distance that best mimics the distance between the features encoded by LSTM, which is difﬁcult to be integrated into an end-to-end framework because the autoencoder needs to be pre-trained and a betaCV-based optimization is required.
Revised: In Abid & Zou (2018), Autowarp ﬁnds the warping distance that best mimics the distance between the features encoded by LSTM, which is difﬁcult to be integrated into an end-to-end framework because the autoencoder needs to be pre-trained and a betaCV-based optimization is required.

Original: It is shown in Tallec & Ollivier (2018) that RNN can learn to warp sequences.
Revised: It is shown in Tallec & Ollivier (2018) that RNN can learn to warp sequences.

Original: In Lohit et al.
Revised: In Lohit et al.

Original: Weber et al.
Revised: Weber et al.

Original: (2019), a ﬁxed warping function is predicted for a given sequence when comparing with different sequences.
Revised: (2019), a ﬁxed warping function is predicted for a given sequence when comparing with different sequences.

Original: This may be viewed as warping all sequences w.r.t.
Revised: This may be viewed as warping all sequences w.r.t.

Original: a ﬁxed timing scale, which may bias to the training sequences.
Revised: a ﬁxed timing scale, which may bias to the training sequences.

Original: In contrast, since the relative local variations and correspondences between different sequences are different, our method predicts different alignments for different sequence pairs.
Revised: In contrast, since the relative local variations and correspondences between different sequences are different, our method predicts different alignments for different sequence pairs.

Original: Supervised representation learning for sequences.
Revised: Supervised representation learning for sequences.

Original: Generally, since alignment-based distances need to solve their own optimization problems to infer alignments in the latent space, the gradients of the supervised metric learning loss cannot be directly propagated.
Revised: Generally, since alignment-based distances need to solve their own optimization problems to infer alignments in the latent space, the gradients of the supervised metric learning loss cannot be directly propagated.

Original: Existing methods exploit either an approximate proxy loss (Su et al., 2018; Lajugie et al., 2014), or a time-consuming optimization method to iteratively infer alignments (Mei et al., 2014), or both (Su & Wu, 2020a;b).
Revised: Existing methods exploit either an approximate proxy loss (Su et al., 2018; Lajugie et al., 2014), or a time-consuming optimization method to iteratively infer alignments (Mei et al., 2014), or both (Su & Wu, 2020a;b).

Original: In this paper, we show that existing deep metric learning methods can be readily applied to the proposed TAP distance in an end-to-end manner.
Revised: In this paper, we show that existing deep metric learning methods can be readily applied to the proposed TAP distance in an end-to-end manner.

Original: Few-shot action recognition.
Revised: Few-shot action recognition.

Original: Most existing few-shot action recognition methods follow the metric learning based paradigm.
Revised: Most existing few-shot action recognition methods follow the metric learning based paradigm.

Original: In Ben-Ari et al.
Revised: In Ben-Ari et al.

Original: and Tan & Yang (2019), TAEN and FAN encode each action into a representation with a ﬁxed dimension and apply vector-wise metrics.
Revised: and Tan & Yang (2019), TAEN and FAN encode each action into a representation with a ﬁxed dimension and apply vector-wise metrics.

Original: Recent works identify the importance of temporal alignment for tackling non-linear temporal variations.
Revised: Recent works identify the importance of temporal alignment for tackling non-linear temporal variations.

Original: [Figure]
Revised: [Figure]

Original: Various alignment-based distances are proposed to compare sequence pairs, such as the temporal attentive relation network (TARN) (Bishay et al., 2019), a variant of DTW used in OTAM (Cao et al., 2020), the permutation-invariant spatial and temporal attention reweighted distance in ARN (Zhang et al., 2020), the temporal CrossTransformer for comparing all possible subsequences in TRX (Perrett et al., 2021), and the two-stage temporal alignment network (TTAN) (Li et al., 2021).
Revised: Various alignment-based distances are proposed to compare sequence pairs, such as the temporal attentive relation network (TARN) (Bishay et al., 2019), a variant of DTW used in OTAM (Cao et al., 2020), the permutation-invariant spatial and temporal attention reweighted distance in ARN (Zhang et al., 2020), the temporal CrossTransformer for comparing all possible subsequences in TRX (Perrett et al., 2021), and the two-stage temporal alignment network (TTAN) (Li et al., 2021).

Original: Following this spirit, we apply the proposed TAP to perform temporal alignment and measure the distance between support and query sequences without any complicated attention mechanism, yielding a simple yet effective few-shot sequence learning solution.
Revised: Following this spirit, we apply the proposed TAP to perform temporal alignment and measure the distance between support and query sequences without any complicated attention mechanism, yielding a simple yet effective few-shot sequence learning solution.

Original:  M ETHODOLOGY 3. T EMPORAL ALIGNMENT PREDICTION
Revised:  M ETHODOLOGY 3. T EMPORAL ALIGNMENT PREDICTION

Original: We ﬁrst revisit the uniﬁed formulation of sequence distance in Su & Wu (2019).
Revised: We ﬁrst revisit the uniﬁed formulation of sequence distance in Su & Wu (2019).

Original: Let X =
Revised: Let X =

Original: [ x 1 , · · · , x L X ] ∈ R d × L X and Y =
Revised: [ x 1 , · · · , x L X ] ∈ R d × L X and Y =

Original: [ y 1 , · · · , y L Y ] ∈ R d × L Y be two sequences with lengths of
Revised: [ y 1 , · · · , y L Y ] ∈ R d × L Y be two sequences with lengths of

Original: L X and L Y , respectively.
Revised: L X and L Y , respectively.

Original: The elements x i , i = 1 , · · · , L X of X and y j , j = 1 , · · · , L Y of Y lie in a d -dimensional feature space R d .
Revised: The elements x i , i = 1 , · · · , L X of X and y j , j = 1 , · · · , L Y of Y lie in a d -dimensional feature space R d .

Original: Many alignment-based distances between X and Y can be uniﬁed as follows:
Revised: Many alignment-based distances between X and Y can be uniﬁed as follows:

Original: [Equation]
Revised: [Equation]

Original: where (cid:104) T , D (cid:105) =
Revised: where (cid:104) T , D (cid:105) =

Original: tr ( T T D ) is the Frobenius dot product.
Revised: tr ( T T D ) is the Frobenius dot product.

Original: D :=
Revised: D :=

Original: [ e ( x i , y j )] ij ∈ R L X × L Y is the matrix of pairwise distances between elements in X and Y , e ( x i , y j ) is a vector-wise distance between two elements x i and y j , which we use Euclidean distance in this paper.
Revised: [ e ( x i , y j )] ij ∈ R L X × L Y is the matrix of pairwise distances between elements in X and Y , e ( x i , y j ) is a vector-wise distance between two elements x i and y j , which we use Euclidean distance in this paper.

Original: T is an alignment matrix whose element T ij indicates whether or how likely x i and y j are aligned.
Revised: T is an alignment matrix whose element T ij indicates whether or how likely x i and y j are aligned.

Original: Φ is the set of all feasible T with some constraints, which is a subset of R L X × L Y .
Revised: Φ is the set of all feasible T with some constraints, which is a subset of R L X × L Y .

Original: R ( T ) is a regularization term on
Revised: R ( T ) is a regularization term on

Original: T .
Revised: T .

Original: T ∗ is the optimal alignment which is the solution of the optimization problem in Eq.
Revised: T ∗ is the optimal alignment which is the solution of the optimization problem in Eq.

Original: Different sequence distances impose different constraints on the feasible set, have different regularization terms, and use different optimization methods for inference.
Revised: Different sequence distances impose different constraints on the feasible set, have different regularization terms, and use different optimization methods for inference.

Original: For instance, DTW optimizes
Revised: For instance, DTW optimizes

Original: Eq.
Revised: Eq.

Original: (2) with R ( T ) = 

Revised: (2) with R ( T ) =

Original: and the boundary, continuity, and monotonicity constraints via dynamic programming, while OPW optimizes Eq.
Revised: and the boundary, continuity, and monotonicity constraints via dynamic programming, while OPW optimizes Eq.

Original: (2) with two temporal regularization terms and the coupling constraints via the Sinkhorn’s matrix scaling algorithm.
Revised: (2) with two temporal regularization terms and the coupling constraints via the Sinkhorn’s matrix scaling algorithm.

Original: Solving T ∗ by the optimization Eq.
Revised: Solving T ∗ by the optimization Eq.

Original: (2) not only requires a long inference time, but also makes it difﬁcult to apply a loss on the sequence distance Eq.
Revised: (2) not only requires a long inference time, but also makes it difﬁcult to apply a loss on the sequence distance Eq.

Original: (1) for learning element representations, because T ∗ is a latent variable that needs to be inferred and its gradient cannot be calculated.
Revised: (1) for learning element representations, because T ∗ is a latent variable that needs to be inferred and its gradient cannot be calculated.

Original: To avoid solving the optimization problem, we propose a feedforward framework for measuring the distances between sequences, namely Temporal Alignment Prediction (TAP) .
Revised: To avoid solving the optimization problem, we propose a feedforward framework for measuring the distances between sequences, namely Temporal Alignment Prediction (TAP) .

Original: Fig.
Revised: Fig.

Original: 1(a) illustrates the TAP framework.
Revised: 1(a) illustrates the TAP framework.

Original: For two sequences X = [ x 1 , · · · , x L X ] and Y =
Revised: For two sequences X = [ x 1 , · · · , x L X ] and Y =

Original: [ y 1 , · · · , y L Y ] , their TAP distance also has the form of Eq.
Revised: [ y 1 , · · · , y L Y ] , their TAP distance also has the form of Eq.

Original: (1), i.e., the Frobenius dot product of D and T ∗ .
Revised: (1), i.e., the Frobenius dot product of D and T ∗ .

Original: Different from other sequence distances which infer the alignment with predeﬁned objectives and constraints, TAP uses an alignment prediction neural network f to directly predict the optimal T ∗ =
Revised: Different from other sequence distances which infer the alignment with predeﬁned objectives and constraints, TAP uses an alignment prediction neural network f to directly predict the optimal T ∗ =

Original: f ( X , Y ) by taking the two sequences as inputs and learns f from data.
Revised: f ( X , Y ) by taking the two sequences as inputs and learns f from data.

Original: The alignment prediction network f can be instantiated by different architectures.
Revised: The alignment prediction network f can be instantiated by different architectures.

Original: In this paper, we propose a lightweight convolutional neural network architecture for its simplicity and fast inference speed.
Revised: In this paper, we propose a lightweight convolutional neural network architecture for its simplicity and fast inference speed.

Original: Speciﬁcally, we reuse the spatial ground matrix D to measure all pairwise distances between elements in X and Y .
Revised: Speciﬁcally, we reuse the spatial ground matrix D to measure all pairwise distances between elements in X and Y .

Original: The relative positions of ˆ x i and ˆ y j are i/L X and j/L Y , respectively.
Revised: The relative positions of ˆ x i and ˆ y j are i/L X and j/L Y , respectively.

Original: To incorporate the temporal dissimilarities, TAP further calculates all pairwise Euclidean distances between the relative positions between x i and y j into a matrix D t :=
Revised: To incorporate the temporal dissimilarities, TAP further calculates all pairwise Euclidean distances between the relative positions between x i and y j into a matrix D t :=

Original: [ e ( i/L X , j/L Y )] ij ∈ R L X × L Y .
Revised: [ e ( i/L X , j/L Y )] ij ∈ R L X × L Y .

Original: Then, we concatenate D and D t along the channel dimension to form D s with a size of L x × L y × 2 .
Revised: Then, we concatenate D and D t along the channel dimension to form D s with a size of L x × L y × 2 .

Original: We use a CNN g to predict the ﬁnal alignment matrix from D s .
Revised: We use a CNN g to predict the ﬁnal alignment matrix from D s .

Original: The CNN uses three convolutional layers each followed by ReLu.
Revised: The CNN uses three convolutional layers each followed by ReLu.

Original: The kernel size in the three layers is 5, 5, and 3, respectively, the stride is ﬁxed to 1, and the padding is set to 2, 2, and 1, respectively, to keep the spatial size.
Revised: The kernel size in the three layers is 5, 5, and 3, respectively, the stride is ﬁxed to 1, and the padding is set to 2, 2, and 1, respectively, to keep the spatial size.

Original: The numbers of kernels in the three layers are set to 30, 30, and 1, respectively.
Revised: The numbers of kernels in the three layers are set to 30, 30, and 1, respectively.

Original: The learned kernels are expected to capture the local alignment patterns.
Revised: The learned kernels are expected to capture the local alignment patterns.

Original: The output of f is augmented by a residual connection with D , resulting in the similarity matrix S :
Revised: The output of f is augmented by a residual connection with D , resulting in the similarity matrix S :

Original: = − ( D + g ( D s ))
Revised: = − ( D + g ( D s ))

Original: ∈
Revised: ∈

Original: R L X × L Y .
Revised: R L X × L Y .

Original: The attentions of elements in Y on x i are obtained by performing Softmax on the i -th row of
Revised: The attentions of elements in Y on x i are obtained by performing Softmax on the i -th row of

Original: S .
Revised: S .

Original: The attentions on all elements in X form an attention matrix A , which can be obtained by performing Softmax along the 2nd dimension of S .
Revised: The attentions on all elements in X form an attention matrix A , which can be obtained by performing Softmax along the 2nd dimension of S .

Original: To generate the predicted alignment T ∗ , TAP ﬁnally performs a global L 1 normalization on A :
Revised: To generate the predicted alignment T ∗ , TAP ﬁnally performs a global L 1 normalization on A :

Original: [Equation]
Revised: [Equation]

Original: T ∗ ij indicates the probability of aligning x i and y j .
Revised: T ∗ ij indicates the probability of aligning x i and y j .

Original: The TAP distance between X and Y is calculated as in Eq.
Revised: The TAP distance between X and Y is calculated as in Eq.

Original: The prediction network f of TAP is lightweight since it only contains a few convolutional kernels of g as parameters.
Revised: The prediction network f of TAP is lightweight since it only contains a few convolutional kernels of g as parameters.

Original: Limitations.
Revised: Limitations.

Original: The family of alignments for TAP is Φ :=
Revised: The family of alignments for TAP is Φ :=

Original: { T ∈ R
Revised: { T ∈ R

Original: L x × L y + | T 1 L y =
Revised: L x × L y + | T 1 L y =

Original: 1 L x /L x } , where 1 L is a L -dimensional vector with all 1 elements.
Revised: 1 L x /L x } , where 1 L is a L -dimensional vector with all 1 elements.

Original: Since strict order-preserving is not guaranteed in Φ , the performance of TAP may be limited when data are strictly ordered.
Revised: Since strict order-preserving is not guaranteed in Φ , the performance of TAP may be limited when data are strictly ordered.

Original: T T 1 L x = 1 L y /L
Revised: T T 1 L x = 1 L y /L

Original: y does not necessarily hold, hence TAP is asymmetric and is not a real metric.
Revised: y does not necessarily hold, hence TAP is asymmetric and is not a real metric.

Original: These limitations make TAP more ﬂexible in turn.
Revised: These limitations make TAP more ﬂexible in turn.

Original: Without strict order-preserving constraint,
Revised: Without strict order-preserving constraint,

Original: TAP can tackle local temporal reorders and generalize to non-sequential (e.g. spatial, cross-modal, etc) correspondences.
Revised: TAP can tackle local temporal reorders and generalize to non-sequential (e.g. spatial, cross-modal, etc) correspondences.

Original: Asymmetric alignment distinguishes the source sequence X and the target sequence Y , where all elements in X must be transported with the same mass when aligning to different target sequences.
Revised: Asymmetric alignment distinguishes the source sequence X and the target sequence Y , where all elements in X must be transported with the same mass when aligning to different target sequences.

Original: To perform classiﬁcation or retrieval, we can always set the test or query sequence as the source, which serves as a standard template to be aligned.
Revised: To perform classiﬁcation or retrieval, we can always set the test or query sequence as the source, which serves as a standard template to be aligned.

Original: The symmetric distance can be obtained by averaging d ( X , Y ) and d ( Y , X ) .
Revised: The symmetric distance can be obtained by averaging d ( X , Y ) and d ( Y , X ) .

Original: 3. S UPERVISED REPRESENTATION LEARNING WITH TAP
Revised: 3. S UPERVISED REPRESENTATION LEARNING WITH TAP

Original: The performance of conventional alignment-based distances and TAP highly depends on D , where
Revised: The performance of conventional alignment-based distances and TAP highly depends on D , where

Original: D depends on the representation space or the ground metric for elements in sequences.
Revised: D depends on the representation space or the ground metric for elements in sequences.

Original: Therefore, learning more discriminative representations or ground metrics is of great interest.
Revised: Therefore, learning more discriminative representations or ground metrics is of great interest.

Original: We transform element-wise features x i of X into a new space by a transformation network h .
Revised: We transform element-wise features x i of X into a new space by a transformation network h .

Original: We denote the sequence of transformed features h ( x i ) by h ( X ) .
Revised: We denote the sequence of transformed features h ( x i ) by h ( X ) .

Original: When labeled training sequences are available, the objective is often to make the distance better separate sequences from different classes.
Revised: When labeled training sequences are available, the objective is often to make the distance better separate sequences from different classes.

Original: However, such objectives are difﬁcult to optimize for conventional alignment-based distances because they solve another optimization problem to obtain the alignments, as presented in Appendix A.2.
Revised: However, such objectives are difﬁcult to optimize for conventional alignment-based distances because they solve another optimization problem to obtain the alignments, as presented in Appendix A.2.

Original: In contrast, TAP enables supervised learning of temporal representations without seeking surrogates objectives.
Revised: In contrast, TAP enables supervised learning of temporal representations without seeking surrogates objectives.

Original: Given a batch of B training sequences X n , n = 1 , · · · , B with corresponding labels y n , n = 1 , · · · , B , for each sequence X n , we generate its augmentation ˜ X n , whose class label is the same as X n , and calculate its TAP distances to all other B sequences including ˜ X n after transformation by h .
Revised: Given a batch of B training sequences X n , n = 1 , · · · , B with corresponding labels y n , n = 1 , · · · , B , for each sequence X n , we generate its augmentation ˜ X n , whose class label is the same as X n , and calculate its TAP distances to all other B sequences including ˜ X n after transformation by h .

Original: We obtain its largest TAP distance to h ( X n (cid:48) ) of the same class as d nl = max n (cid:48) :
Revised: We obtain its largest TAP distance to h ( X n (cid:48) ) of the same class as d nl = max n (cid:48) :

Original: y n (cid:48)
Revised: y n (cid:48)

Original: =
Revised: =

Original: y n d ( h ( X n ) , h ( X n (cid:48) ); f ) and its smallest TAP distance to h ( X n (cid:48) ) of different classes as d ns = min n (cid:48) :
Revised: y n d ( h ( X n ) , h ( X n (cid:48) ); f ) and its smallest TAP distance to h ( X n (cid:48) ) of different classes as d ns = min n (cid:48) :

Original: y n (cid:48) (cid:54) =
Revised: y n (cid:48) (cid:54) =

Original: y n d ( h ( X n ) , h ( X n (cid:48) ); f ) .
Revised: y n d ( h ( X n ) , h ( X n (cid:48) ); f ) .

Original: Any metric learning losses can be used to train TAP.
Revised: Any metric learning losses can be used to train TAP.

Original: We employ three metric learning losses to learn h and f jointly, including the triplet loss with hard-mining (HM)
Revised: We employ three metric learning losses to learn h and f jointly, including the triplet loss with hard-mining (HM)

Original: (Hermans et al., 2017), the lifted structure (Lifted) loss (Song et al., 2016), and binomial deviance loss (Binomial) (Yi et al., 2014).
Revised: (Hermans et al., 2017), the lifted structure (Lifted) loss (Song et al., 2016), and binomial deviance loss (Binomial) (Yi et al., 2014).

Original: For the triplet loss, we minimize the TAP distances to sequences of the same class that are larger than d ns − m , and maximize those to different classes that are smaller than d nl + m , where m is a margin and set to 0 .
Revised: For the triplet loss, we minimize the TAP distances to sequences of the same class that are larger than d ns − m , and maximize those to different classes that are smaller than d nl + m , where m is a margin and set to 0 .

Original: The objective to be minimized over f and h is formulated as follows:
Revised: The objective to be minimized over f and h is formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: For the lifted structure loss
Revised: For the lifted structure loss

Original: [Equation]
Revised: [Equation]

Original: we minimize the following objective to utilize the TAP distances among all the positive and negative pairs:
Revised: we minimize the following objective to utilize the TAP distances among all the positive and negative pairs:

Original: [Equation]
Revised: [Equation]

Original: For the binomial deviance loss
Revised: For the binomial deviance loss

Original: [Equation]
Revised: [Equation]

Original: for the n -th sequence
Revised: for the n -th sequence

Original: [Equation]
Revised: [Equation]

Original: we ﬁrst perform hard mining to select its hard positive and negative samples within the batch:
Revised: we ﬁrst perform hard mining to select its hard positive and negative samples within the batch:

Original: [Equation]
Revised: [Equation]

Original: We then minimize the following objective:
Revised: We then minimize the following objective:

Original: [Equation]
Revised: [Equation]

Original: where | P n | and | N n | denote the number of mined positive and negative sequences, respectively.
Revised: where | P n | and | N n | denote the number of mined positive and negative sequences, respectively.

Original: In all losses, α and β are hyper-parameters controlling the weights of positive and negative pairs, respectively.
Revised: In all losses, α and β are hyper-parameters controlling the weights of positive and negative pairs, respectively.

Original: For all the losses, we also incorporate the self-supervised loss of minimizing the TAP distance between each sequence and its augmentation, as presented in the appendix, where the augmentation is obtained by random blurring and random merge.
Revised: For all the losses, we also incorporate the self-supervised loss of minimizing the TAP distance between each sequence and its augmentation, as presented in the appendix, where the augmentation is obtained by random blurring and random merge.

Original: For any supervised loss, the feature transformation network h and the alignment prediction network f in TAP are learned jointly by back-propagation in an end-to-end manner.
Revised: For any supervised loss, the feature transformation network h and the alignment prediction network f in TAP are learned jointly by back-propagation in an end-to-end manner.

Original: 3. F EW - SHOT ACTION RECOGNITION WITH TAP
Revised: 3. F EW - SHOT ACTION RECOGNITION WITH TAP

Original: In this subsection, we demonstrate the application of TAP in few-shot action recognition.
Revised: In this subsection, we demonstrate the application of TAP in few-shot action recognition.

Original: For a Nway K -shot classiﬁcation task, there are N action classes and each class has K support videos.
Revised: For a Nway K -shot classiﬁcation task, there are N action classes and each class has K support videos.

Original: The task aims to classify each query video into one of the N classes.
Revised: The task aims to classify each query video into one of the N classes.

Original: We follow the episode training and evaluation.
Revised: We follow the episode training and evaluation.

Original: The video dataset is divided into a training set, a validation set, and a test set.
Revised: The video dataset is divided into a training set, a validation set, and a test set.

Original: Classes in the three sets are disjoint.
Revised: Classes in the three sets are disjoint.

Original: In the training phase, we randomly construct one N -way K -shot task in each episode to train the model.
Revised: In the training phase, we randomly construct one N -way K -shot task in each episode to train the model.

Original: The best model is selected according to the performance on tasks constructed from the validation set and evaluated on tasks constructed from the test set.
Revised: The best model is selected according to the performance on tasks constructed from the validation set and evaluated on tasks constructed from the test set.

Original: We adopt the metric learning strategy with the TAP distance.
Revised: We adopt the metric learning strategy with the TAP distance.

Original: For each N -way K -shot classiﬁcation task, the support videos are denoted by V nk , n = 1 , · · · , N ; k = 1 , · · · , K and the query videos are denoted by V q , q = 1 , · · · , Q , where Q is the number of query videos.
Revised: For each N -way K -shot classiﬁcation task, the support videos are denoted by V nk , n = 1 , · · · , N ; k = 1 , · · · , K and the query videos are denoted by V q , q = 1 , · · · , Q , where Q is the number of query videos.

Original: We follow Cao et al.
Revised: We follow Cao et al.

Original: Li et al.
Revised: Li et al.

Original: (2021) to sample a ﬁxed number L of frames per video.
Revised: (2021) to sample a ﬁxed number L of frames per video.

Original: We use a feature embedding backbone h to represent each frame v i by a d -dimensional feature vector x i
Revised: We use a feature embedding backbone h to represent each frame v i by a d -dimensional feature vector x i

Original: = h ( v i ) , thus each video V is represented by a feature sequence X
Revised: = h ( v i ) , thus each video V is represented by a feature sequence X

Original: = h ( V ) =
Revised: = h ( V ) =

Original: [ x 1 , · · · , x L ] ∈ R d × L .
Revised: [ x 1 , · · · , x L ] ∈ R d × L .

Original: A self-attention (SA) module can be performed to enhance X =
Revised: A self-attention (SA) module can be performed to enhance X =

Original: SA ( X ) as presented in Appendix A.3.
Revised: SA ( X ) as presented in Appendix A.3.

Original: To classify a query video V q , we calculate the TAP distances between X q and all the support sequences.
Revised: To classify a query video V q , we calculate the TAP distances between X q and all the support sequences.

Original: Since TAP is non-symmetric, we add d ( X q , X nk ; f ) and d ( X nk , X q ; f ) as the dissimilarity between X q and the k -th support sequence of the n -th class X nk .
Revised: Since TAP is non-symmetric, we add d ( X q , X nk ; f ) and d ( X nk , X q ; f ) as the dissimilarity between X q and the k -th support sequence of the n -th class X nk .

Original: The similarity between V q and the n -th class is then obtained as:
Revised: The similarity between V q and the n -th class is then obtained as:

Original: [Equation]
Revised: [Equation]

Original: X q is classiﬁed to the class with the largest similarity.
Revised: X q is classiﬁed to the class with the largest similarity.

Original: In the training phase, we jointly learn the backbone h and the prediction network f of TAP with the cross-entropy loss.
Revised: In the training phase, we jointly learn the backbone h and the prediction network f of TAP with the cross-entropy loss.

Original: The loss over a single episode is formulated as follows:
Revised: The loss over a single episode is formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: where y q is the ground-truth class label of the query video V q .
Revised: where y q is the ground-truth class label of the query video V q .

Original: If SA is applied to query and support sequences before performing alignment by TAP, the method is denoted by SA-TAP.
Revised: If SA is applied to query and support sequences before performing alignment by TAP, the method is denoted by SA-TAP.

Original:  E XPERIMENTS 4. D ATASETS
Revised:  E XPERIMENTS 4. D ATASETS

Original: For supervised representation learning, we evaluate TAP on four sequence datasets.
Revised: For supervised representation learning, we evaluate TAP on four sequence datasets.

Original: MSR Action3D (Li et al., 2010; Wang et al., 2012) contains 557 skeleton sequences from 20 action classes.
Revised: MSR Action3D (Li et al., 2010; Wang et al., 2012) contains 557 skeleton sequences from 20 action classes.

Original: We follow the splitting of the dataset in Wang et al.
Revised: We follow the splitting of the dataset in Wang et al.

Original: ; Wang & Wu (2013).
Revised: ; Wang & Wu (2013).

Original: We use the 192 -dimensional element-wise features in Wang et al.
Revised: We use the 192 -dimensional element-wise features in Wang et al.

Original: ; Wang & Wu (2013).
Revised: ; Wang & Wu (2013).

Original: MSR Daily
Revised: MSR Daily

Original: Activity3D (Wang et al., 2012) contains 320 skeleton sequences from 16 activity classes.
Revised: Activity3D (Wang et al., 2012) contains 320 skeleton sequences from 16 activity classes.

Original: We also follow the splitting of the dataset in Wang et al.
Revised: We also follow the splitting of the dataset in Wang et al.

Original: ; Wang & Wu (2013) and use the 390dimensional element-wise features in Wang et al.
Revised: ; Wang & Wu (2013) and use the 390dimensional element-wise features in Wang et al.

Original: ; Wang & Wu (2013).
Revised: ; Wang & Wu (2013).

Original: The “Spoken Arabic Digits (SAD)” dataset from the UCI Machine Learning Repository (Bache & Lichman, 2013) contains 8,800 sequences of 13-dimensional mel-frequency cepstrum coefﬁcients from 10 classes.
Revised: The “Spoken Arabic Digits (SAD)” dataset from the UCI Machine Learning Repository (Bache & Lichman, 2013) contains 8,800 sequences of 13-dimensional mel-frequency cepstrum coefﬁcients from 10 classes.

Original: The dataset has 6,600 training sequences and 2,200 testing sequences.
Revised: The dataset has 6,600 training sequences and 2,200 testing sequences.

Original: ChaLearn (Escalera et al., 2013b;a) contains 955 Italian gesture sequences from 20 classes.
Revised: ChaLearn (Escalera et al., 2013b;a) contains 955 Italian gesture sequences from 20 classes.

Original: The dataset is split into training, validation, and test sets.
Revised: The dataset is split into training, validation, and test sets.

Original: Following Fernando et al.
Revised: Following Fernando et al.

Original: , we conduct experiments on the segmented sequences where each sequence contains one gesture.
Revised: , we conduct experiments on the segmented sequences where each sequence contains one gesture.

Original: We use the 100 -dimensional element-wise features in Fernando et al.
Revised: We use the 100 -dimensional element-wise features in Fernando et al.

Original: On all datasets, the lengths of different sequences are different.
Revised: On all datasets, the lengths of different sequences are different.

Original: For few-shot action recognition, we evaluate TAP on three video datasets.
Revised: For few-shot action recognition, we evaluate TAP on three video datasets.

Original: UCF101 (Soomro et al., 2012) contains 13,320 videos of 101 classes.
Revised: UCF101 (Soomro et al., 2012) contains 13,320 videos of 101 classes.

Original: We follow the protocol used in Zhang et al.
Revised: We follow the protocol used in Zhang et al.

Original: (2020) to split the dataset, where the training, validation, and test sets contain 70, 10, and 21 classes with 9154, 1421, and 2745 videos, respectively.
Revised: (2020) to split the dataset, where the training, validation, and test sets contain 70, 10, and 21 classes with 9154, 1421, and 2745 videos, respectively.

Original: HMDB51 (Kuehne et al., 2011) consists of 6,840 videos from 51 classes.
Revised: HMDB51 (Kuehne et al., 2011) consists of 6,840 videos from 51 classes.

Original: We also follow the protocol used in Zhang et al.
Revised: We also follow the protocol used in Zhang et al.

Original: (2020) to split the dataest, where the training, validation, and test sets contain 31, 10, and 10 classes with 4280, 1194, and 1292 videos, respectively.
Revised: (2020) to split the dataest, where the training, validation, and test sets contain 31, 10, and 10 classes with 4280, 1194, and 1292 videos, respectively.

Original: Something-Something-v2 (SSv2) (Goyal et al., 2017) contains 220,847 videos ofclasses.
Revised: Something-Something-v2 (SSv2) (Goyal et al., 2017) contains 220,847 videos ofclasses.

Original: We follow the protocol in Cao et al.
Revised: We follow the protocol in Cao et al.

Original: (2020) to split the dataset, where the training, validation, and test sets contain 64, 12, and 24 classes with 77,500, 1925, and 2854 videos, respectively.
Revised: (2020) to split the dataset, where the training, validation, and test sets contain 64, 12, and 24 classes with 77,500, 1925, and 2854 videos, respectively.

Original: 4. I MPLEMENTATION DETAILS
Revised: 4. I MPLEMENTATION DETAILS

Original: The structure and hyper-parameters of the alignment prediction network are ﬁxed in TAP, as introduced in Section 3.1.
Revised: The structure and hyper-parameters of the alignment prediction network are ﬁxed in TAP, as introduced in Section 3.1.

Original: For supervised representation learning, we perform two tasks on each dataset: sequence classiﬁcation and sequence retrieval.
Revised: For supervised representation learning, we perform two tasks on each dataset: sequence classiﬁcation and sequence retrieval.

Original: For classiﬁcation, we use the nearest neighbor (1NN) classiﬁer.
Revised: For classiﬁcation, we use the nearest neighbor (1NN) classiﬁer.

Original: For each test sequence, we calculate the distances to all training sequences w.r.t.
Revised: For each test sequence, we calculate the distances to all training sequences w.r.t.

Original: the evaluated distance and its label is determined by the training sequence with the smallest distance.
Revised: the evaluated distance and its label is determined by the training sequence with the smallest distance.

Original: We use accuracy as the performance measure.
Revised: We use accuracy as the performance measure.

Original: For retrieval, we use each test sequence as a query and rank all training sequences according to their distances to the query in ascending order.
Revised: For retrieval, we use each test sequence as a query and rank all training sequences according to their distances to the query in ascending order.

Original: We use mean average precision (MAP) as the performance measure.
Revised: We use mean average precision (MAP) as the performance measure.

Original: Following Su & Wu (2020b), we use a three-layer perceptron as the transformation network h , where the two hidden layers have 1024 nodes followed by ReLU activations and the output has the same dimension with the input element-wise features.
Revised: Following Su & Wu (2020b), we use a three-layer perceptron as the transformation network h , where the two hidden layers have 1024 nodes followed by ReLU activations and the output has the same dimension with the input element-wise features.

Original: For learning the TAP distance, we use the Adam optimizer with a momentum of 0.9 and a weight decay of 1 e − 4 and train a maximum of 100 epochs.
Revised: For learning the TAP distance, we use the Adam optimizer with a momentum of 0.9 and a weight decay of 1 e − 4 and train a maximum of 100 epochs.

Original: The learning rate is ﬁxed to 0.0001 on all datasets.
Revised: The learning rate is ﬁxed to 0.0001 on all datasets.

Original: The batch size is set to 64 except for the MSR
Revised: The batch size is set to 64 except for the MSR

Original: Activity3D dataset, where it is reduced to 16 due to memory limitation.
Revised: Activity3D dataset, where it is reduced to 16 due to memory limitation.

Original: We ﬁx the hyper-parameters α and β in the lifted structure loss and the binomial deviance loss to 40 and 2, respectively.
Revised: We ﬁx the hyper-parameters α and β in the lifted structure loss and the binomial deviance loss to 40 and 2, respectively.

Original: To evaluate TAP on few-shot action recognition, we perform 5-way 1-shot and 5-way 5-shot classiﬁcation tasks.
Revised: To evaluate TAP on few-shot action recognition, we perform 5-way 1-shot and 5-way 5-shot classiﬁcation tasks.

Original: Following Cao et al.
Revised: Following Cao et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: , we uniformly sample L = 8 frames
Revised: , we uniformly sample L = 8 frames

Original: [Table]
Revised: [Table]

Original: per video, resize each frame to 256 × 256 , augment the frames with random horizontal ﬂipping, perform 224 × 224 random crops for training and center crops for testing, and use the pretrained
Revised: per video, resize each frame to 256 × 256 , augment the frames with random horizontal ﬂipping, perform 224 × 224 random crops for training and center crops for testing, and use the pretrained

Original: ResNet-50 as the backbone.
Revised: ResNet-50 as the backbone.

Original: We jointly tune the backbone and train TAP by the SGD optimizer with a learning rate of 0 .
Revised: We jointly tune the backbone and train TAP by the SGD optimizer with a learning rate of 0 .

Original: Each episode contains 1 task due to the memory constraint.
Revised: Each episode contains 1 task due to the memory constraint.

Original: We adapt the toolbox 1 and all other hyper-parameters remain the same.
Revised: We adapt the toolbox 1 and all other hyper-parameters remain the same.

Original: We train a maximum of 100000 iterations on SSv2 and 35000 iterations on other datasets.
Revised: We train a maximum of 100000 iterations on SSv2 and 35000 iterations on other datasets.

Original: We select the best iteration every 5000 iterations according to the performance on the validation set.
Revised: We select the best iteration every 5000 iterations according to the performance on the validation set.

Original: For SA-TAP, we increase the numbers of kernels in the middle layers of f from 30 to 64.
Revised: For SA-TAP, we increase the numbers of kernels in the middle layers of f from 30 to 64.

Original: For 1-shot task with SA-TAP on HMDB51, the validation accuracy starts to ﬂuctuate after 5000 iterations, so we train 5000 iterations and select everyiterations.
Revised: For 1-shot task with SA-TAP on HMDB51, the validation accuracy starts to ﬂuctuate after 5000 iterations, so we train 5000 iterations and select everyiterations.

Original: We randomly sample 10,000 tasks from the test set and report the average accuracy.
Revised: We randomly sample 10,000 tasks from the test set and report the average accuracy.

Original: 4. C OMPARISON WITH SUPERVISED REPRESENTATION LEARNING METHODS
Revised: 4. C OMPARISON WITH SUPERVISED REPRESENTATION LEARNING METHODS

Original: TAP can be used to learn temporal representations in a supervised manner as presented in Section 3.2.
Revised: TAP can be used to learn temporal representations in a supervised manner as presented in Section 3.2.

Original: Comparisons with other supervised sequence representation learning methods on the four datasets are shown in Table 1, where we follow the setting and compare with the results reported in Su & Wu (2020b).
Revised: Comparisons with other supervised sequence representation learning methods on the four datasets are shown in Table 1, where we follow the setting and compare with the results reported in Su & Wu (2020b).

Original: LDMLT (Mei et al., 2014), SWMD (Huang et al., 2016), and RVSML (Su &
Revised: LDMLT (Mei et al., 2014), SWMD (Huang et al., 2016), and RVSML (Su &

Original: Wu, 2019) learn linear transformations for elements in sequences, while deep metric learning methods with binomial deviance loss (Binomial)
Revised: Wu, 2019) learn linear transformations for elements in sequences, while deep metric learning methods with binomial deviance loss (Binomial)

Original: (Yi et al., 2014), lifted structured loss (Lifted) (Song et al., 2016), triplet loss with hard-mining (HM)
Revised: (Yi et al., 2014), lifted structured loss (Lifted) (Song et al., 2016), triplet loss with hard-mining (HM)

Original: (Hermans et al., 2017), and multi-similarity loss (MS) (Wang et al., 2019) learn non-linear temporal representations by treating elements in each sequence as independent samples of the same class.
Revised: (Hermans et al., 2017), and multi-similarity loss (MS) (Wang et al., 2019) learn non-linear temporal representations by treating elements in each sequence as independent samples of the same class.

Original: DeepRVSML (D-R) (Su & Wu, 2020b) is the deep extension of RVSML.
Revised: DeepRVSML (D-R) (Su & Wu, 2020b) is the deep extension of RVSML.

Original: All methods use the same transformation network.
Revised: All methods use the same transformation network.

Original: TAP generally outperforms these methods on all datasets.
Revised: TAP generally outperforms these methods on all datasets.

Original: Especially, TAP achieves much higher MAP, which reﬂects that the TAP distances between sequences from the same class are generally smaller than those from different classes.
Revised: Especially, TAP achieves much higher MAP, which reﬂects that the TAP distances between sequences from the same class are generally smaller than those from different classes.

Original: This shows the effectiveness of the end-to-end joint training of the feature transformation network and the alignment prediction network of TAP.
Revised: This shows the effectiveness of the end-to-end joint training of the feature transformation network and the alignment prediction network of TAP.

Original: SoftDTW is differentiable and can also be combined with different losses to train h .
Revised: SoftDTW is differentiable and can also be combined with different losses to train h .

Original: This can be achieved by replacing the TAP distance d ( h ( X n ) , h ( X n (cid:48) ); f ) with the SoftDTW distance in Eq.
Revised: This can be achieved by replacing the TAP distance d ( h ( X n ) , h ( X n (cid:48) ); f ) with the SoftDTW distance in Eq.

Original: [Table]
Revised: [Table]

Original: Eq.
Revised: Eq.

Original: (5), and Eq.
Revised: (5), and Eq.

Original: (7), respectively.
Revised: (7), respectively.

Original: Comparisons between TAP and SoftDTW for supervised representation learning are also presented in Tab. 1.
Revised: Comparisons between TAP and SoftDTW for supervised representation learning are also presented in Tab. 1.

Original: We employ the PyTorch implementation 2 of SoftDTW.
Revised: We employ the PyTorch implementation 2 of SoftDTW.

Original: The triplet loss is relatively straightforward, since the sequence distance does not undergo a nonlinear function.
Revised: The triplet loss is relatively straightforward, since the sequence distance does not undergo a nonlinear function.

Original: SoftDTW outperforms TAP when the HM loss is used on some datasets.
Revised: SoftDTW outperforms TAP when the HM loss is used on some datasets.

Original: In the other two losses, exponential and logarithmic transformations are applied to the sequence distance, so the backpropagation through the alignment variable is more complicated.
Revised: In the other two losses, exponential and logarithmic transformations are applied to the sequence distance, so the backpropagation through the alignment variable is more complicated.

Original: For SoftDTW, using a large α will lead to inﬁnite initial losses, so we reduce α and β to 2 and 1, respectively.
Revised: For SoftDTW, using a large α will lead to inﬁnite initial losses, so we reduce α and β to 2 and 1, respectively.

Original: When using the lifted structure loss and binomial deviance loss to optimize the transformation network h , the results of SoftDTW are inferior to those of TAP on most datasets.
Revised: When using the lifted structure loss and binomial deviance loss to optimize the transformation network h , the results of SoftDTW are inferior to those of TAP on most datasets.

Original: In addition, it is easy to fall into a trivial solution when incorporating SoftDTW into complex losses on some datasets.
Revised: In addition, it is easy to fall into a trivial solution when incorporating SoftDTW into complex losses on some datasets.

Original: This may be because that gradients backpropagated through SoftDTW cannot be automatically calculated and such manually derived backward processes with respect to complex losses may be more unstable.
Revised: This may be because that gradients backpropagated through SoftDTW cannot be automatically calculated and such manually derived backward processes with respect to complex losses may be more unstable.

Original: On all datasets, TAP outperforms all other methods by a signiﬁcant margin on MAP.
Revised: On all datasets, TAP outperforms all other methods by a signiﬁcant margin on MAP.

Original: In cases where the accuracy of TAP is not the best, it is also comparable with the best result.
Revised: In cases where the accuracy of TAP is not the best, it is also comparable with the best result.

Original: These results show that
Revised: These results show that

Original: TAP can be better learned with different losses, the training of TAP is easier and more consistent, and TAP has better potential to be applied to more complex tasks.
Revised: TAP can be better learned with different losses, the training of TAP is easier and more consistent, and TAP has better potential to be applied to more complex tasks.

Original: In addition, TAP can be generalized to tackle non-sequential data, e.g., determining the spatial or set-to-set correspondences; while
Revised: In addition, TAP can be generalized to tackle non-sequential data, e.g., determining the spatial or set-to-set correspondences; while

Original: SoftDTW is more restricted to strictly order-preserving sequence data.
Revised: SoftDTW is more restricted to strictly order-preserving sequence data.

Original: 4. E VALUATION ON COMPUTATIONAL TIMES AND ALIGNMENTS
Revised: 4. E VALUATION ON COMPUTATIONAL TIMES AND ALIGNMENTS

Original: We compare the average training time per epoch of SoftDTW and TAP on a single TITAN V GPU by using the triplet loss in the middle part of Table 2.
Revised: We compare the average training time per epoch of SoftDTW and TAP on a single TITAN V GPU by using the triplet loss in the middle part of Table 2.

Original: Since the asymmetric TAP distance between any two sequences in a batch is calculated twice, the training time of TAP is longer than SoftDTW.
Revised: Since the asymmetric TAP distance between any two sequences in a batch is calculated twice, the training time of TAP is longer than SoftDTW.

Original: Table 2 (bottom) compares the average running times for computing the distances between one test sequence and all training sequences with different distances including DTW, optimal transport (OT),
Revised: Table 2 (bottom) compares the average running times for computing the distances between one test sequence and all training sequences with different distances including DTW, optimal transport (OT),

Original: OPW, SoftDTW, and TAP.
Revised: OPW, SoftDTW, and TAP.

Original: l in the top row indicates the average length of sequences in the dataset.
Revised: l in the top row indicates the average length of sequences in the dataset.

Original: TAP runs faster than OT, DTW, and OPW by at least an order of magnitude.
Revised: TAP runs faster than OT, DTW, and OPW by at least an order of magnitude.

Original: The larger the dataset, the more signiﬁcant the advantage.
Revised: The larger the dataset, the more signiﬁcant the advantage.

Original: This is because TAP avoids complex optimizations required by other distances and can be accelerated conveniently by GPU.
Revised: This is because TAP avoids complex optimizations required by other distances and can be accelerated conveniently by GPU.

Original: When both running on GPU, TAP is faster than SoftDTW.
Revised: When both running on GPU, TAP is faster than SoftDTW.

Original: As shown in Appendix A.5, the advantage increases as l increases.
Revised: As shown in Appendix A.5, the advantage increases as l increases.

Original: Fig.2 shows visualizations on the MSR Action3D dataset.
Revised: Fig.2 shows visualizations on the MSR Action3D dataset.

Original: For the test sequence of “draw tick”, the nearest training sequence by TAP comes from the same class, but the one by Soft-DTW comes from another class “horizontal arm wave”.
Revised: For the test sequence of “draw tick”, the nearest training sequence by TAP comes from the same class, but the one by Soft-DTW comes from another class “horizontal arm wave”.

Original: Actions contain different movement cycles and similar poses.
Revised: Actions contain different movement cycles and similar poses.

Original: Each alignment path of Soft-DTW is strictly order-preserving, causing some meaningless alignments.
Revised: Each alignment path of Soft-DTW is strictly order-preserving, causing some meaningless alignments.

Original: The alignments of TAP show periodicity while preserving the general orders.
Revised: The alignments of TAP show periodicity while preserving the general orders.

Original: In the alignment prediction network f , we implement g by CNN to model the temporal constraints since multiple convolution kernels in different layers are suitable for capturing different local alignment patterns and variances of local motions.
Revised: In the alignment prediction network f , we implement g by CNN to model the temporal constraints since multiple convolution kernels in different layers are suitable for capturing different local alignment patterns and variances of local motions.

Original: The local receptive ﬁeld of CNN can be viewed as imposing a constraint on the local temporal warping.
Revised: The local receptive ﬁeld of CNN can be viewed as imposing a constraint on the local temporal warping.

Original: Ablation studies on the architecture and hyperparameters of g are presented in Appendix A.7.
Revised: Ablation studies on the architecture and hyperparameters of g are presented in Appendix A.7.

Original: To evaluate the effect of g ( D s ) , we apply the trained
Revised: To evaluate the effect of g ( D s ) , we apply the trained

Original: [Table]
Revised: [Table]

Original: TAP model on each test sequence and calculate the Frobenius norm ratio:
Revised: TAP model on each test sequence and calculate the Frobenius norm ratio:

Original: [Table]
Revised: [Table]

Original: D
Revised: D

Original: [Table]
Revised: [Table]

Original: F .
Revised: F .

Original: The average ratios of all test sequences with different losses are shown in Tab. 3.
Revised: The average ratios of all test sequences with different losses are shown in Tab. 3.

Original: The ratio ranges from 2 to 21, showing that g ( D s ) contributes more than D .
Revised: The ratio ranges from 2 to 21, showing that g ( D s ) contributes more than D .

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: To quantitatively evaluate the alignments, we generate augmentations and corresponding groundtruth alignments for each sequence in the test set.
Revised: To quantitatively evaluate the alignments, we generate augmentations and corresponding groundtruth alignments for each sequence in the test set.

Original: Comparison on MSE between ground-truths and predicted alignments by the supervised learned TAP (TAP-s) and SoftDTW (SoftDTW-s) is shown in the left part of Tab. 4.
Revised: Comparison on MSE between ground-truths and predicted alignments by the supervised learned TAP (TAP-s) and SoftDTW (SoftDTW-s) is shown in the left part of Tab. 4.

Original: f in TAP can also be directly learned by minimizing MSE on the training set via augmentations.
Revised: f in TAP can also be directly learned by minimizing MSE on the training set via augmentations.

Original: We directly apply TAP learned in this way and SoftDTW to align the original test sequences and their augmentations.
Revised: We directly apply TAP learned in this way and SoftDTW to align the original test sequences and their augmentations.

Original: Results are shown in the right part of Tab. 4.
Revised: Results are shown in the right part of Tab. 4.

Original: Details of the evaluation methods are presented in Appendix A.8.
Revised: Details of the evaluation methods are presented in Appendix A.8.

Original: In both cases, TAP achieves lower MSEs.
Revised: In both cases, TAP achieves lower MSEs.

Original: 4. E VALUATION ON FEW - SHOT ACTION RECOGNITION
Revised: 4. E VALUATION ON FEW - SHOT ACTION RECOGNITION

Original: We compare TAP and SA-TAP with state-of-the-art few shot action recognition methods including
Revised: We compare TAP and SA-TAP with state-of-the-art few shot action recognition methods including

Original: ARN Zhang et al.
Revised: ARN Zhang et al.

Original: (2020), CMN-J (Zhu & Yang, 2018; 2020), OTAM (Cao et al., 2020), TRX (Perrett et al., 2021), and TTAN (Li et al., 2021).
Revised: (2020), CMN-J (Zhu & Yang, 2018; 2020), OTAM (Cao et al., 2020), TRX (Perrett et al., 2021), and TTAN (Li et al., 2021).

Original: Results on 5-way 1-shot and 5-way 5-shot tasks are shown in Table 5, where the results of OTAM and TRX on UCF101 and HMDB51 are reported in (Li et al., 2021).
Revised: Results on 5-way 1-shot and 5-way 5-shot tasks are shown in Table 5, where the results of OTAM and TRX on UCF101 and HMDB51 are reported in (Li et al., 2021).

Original: OTAM employs a modiﬁed DTW as the distance.
Revised: OTAM employs a modiﬁed DTW as the distance.

Original: TAP generally outperforms OTAM and SoftDTW signiﬁcantly.
Revised: TAP generally outperforms OTAM and SoftDTW signiﬁcantly.

Original: When only 1 support sample per class is available, TAP also achieves better results than TRX.
Revised: When only 1 support sample per class is available, TAP also achieves better results than TRX.

Original: TAP with much fewer parameters outperforms TTAN consisting of two stages on UCF101.
Revised: TAP with much fewer parameters outperforms TTAN consisting of two stages on UCF101.

Original: TAP has tighter 95% conﬁdence intervals and is more stable than SoftDTW
Revised: TAP has tighter 95% conﬁdence intervals and is more stable than SoftDTW

Original: and
Revised: and

Original: ARN.
Revised: ARN.

Original: SA-TAP further improves TAP.
Revised: SA-TAP further improves TAP.

Original: It outperforms all other methods in 1-shot tasks on UCFand HMDB-51.
Revised: It outperforms all other methods in 1-shot tasks on UCFand HMDB-51.

Original: On SSv2, SA-TAP achieves comparable results to TTAN in 1-shot learning and comparable results to TRXs in 5-shot learning, while signiﬁcantly outperforming other methods.
Revised: On SSv2, SA-TAP achieves comparable results to TTAN in 1-shot learning and comparable results to TRXs in 5-shot learning, while signiﬁcantly outperforming other methods.

Original: These results validate the effectiveness of TAP and SA-TAP on few-shot action recognition.
Revised: These results validate the effectiveness of TAP and SA-TAP on few-shot action recognition.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present TAP, which is a learnable distance for sequences.
Revised: In this paper, we present TAP, which is a learnable distance for sequences.

Original: TAP calculates the optimal alignment from the input sequences using a prediction network.
Revised: TAP calculates the optimal alignment from the input sequences using a prediction network.

Original: We demonstrate the application of TAP in supervised representation learning for sequence data and few-shot action recognition.
Revised: We demonstrate the application of TAP in supervised representation learning for sequence data and few-shot action recognition.

Original: For supervised learning, TAP can be learned with various metric learning-based losses.
Revised: For supervised learning, TAP can be learned with various metric learning-based losses.

Original: Experimental results on four real-world datasets show that TAP achieves competitive performances with much faster inference speed.
Revised: Experimental results on four real-world datasets show that TAP achieves competitive performances with much faster inference speed.

Original: For few-shot learning, simply applying TAP in the metric-based strategy achieves comparable results with some state-of-the-art few-shot action recognition methods.
Revised: For few-shot learning, simply applying TAP in the metric-based strategy achieves comparable results with some state-of-the-art few-shot action recognition methods.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: This paper proposes a learnable sequence distance that employs a lightweight neural network to predict the optimal alignment between two sequences and its applications in supervised learning, few-shot learning and self-supervised learning (in the appendix) for sequence data.
Revised: This paper proposes a learnable sequence distance that employs a lightweight neural network to predict the optimal alignment between two sequences and its applications in supervised learning, few-shot learning and self-supervised learning (in the appendix) for sequence data.

Original: The proposed distance can be applied to any sequence data to speed up the inference of optimal alignment and improve the discrimination ability.
Revised: The proposed distance can be applied to any sequence data to speed up the inference of optimal alignment and improve the discrimination ability.

Original: Because this work presents such a general distance measure and its learning methods under different conditions, we did not see any particular foreseeable negative ethics impacts.
Revised: Because this work presents such a general distance measure and its learning methods under different conditions, we did not see any particular foreseeable negative ethics impacts.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The architecture and hyper-parameters of the alignment prediction network in TAP are ﬁxed and speciﬁed in Section 3.1.
Revised: The architecture and hyper-parameters of the alignment prediction network in TAP are ﬁxed and speciﬁed in Section 3.1.

Original: We specify all the training details on hyper-parameters and experimental settings in Section 4.2.
Revised: We specify all the training details on hyper-parameters and experimental settings in Section 4.2.

Original: Code will be made publicly available
Revised: Code will be made publicly available

Original: A A PPENDIX A. S ELF - SUPERVISED LOSS FOR TAP
Revised: A A PPENDIX A. S ELF - SUPERVISED LOSS FOR TAP

Original: We show that the proposed TAP distance can also be learned in a self-supervised manner with unlabeled sequence samples.
Revised: We show that the proposed TAP distance can also be learned in a self-supervised manner with unlabeled sequence samples.

Original: We learn the prediction network of TAP by minimizing the TAP distances between sequences and their corresponding augmentations.
Revised: We learn the prediction network of TAP by minimizing the TAP distances between sequences and their corresponding augmentations.

Original: Data of speciﬁc modalities such as images and videos can be augmented through low-level transformations, where self-supervised learning methods learn the feature embedding network by encouraging consistent features extracted from original data and their augmentations.
Revised: Data of speciﬁc modalities such as images and videos can be augmented through low-level transformations, where self-supervised learning methods learn the feature embedding network by encouraging consistent features extracted from original data and their augmentations.

Original: Differently, we focus on learning the distance between sequences, where the element-wise features are already given no matter where and how the features are extracted, so low-level transformations are not applicable.
Revised: Differently, we focus on learning the distance between sequences, where the element-wise features are already given no matter where and how the features are extracted, so low-level transformations are not applicable.

Original: To this end, we propose two augmentation methods for sequences: random blur and random merge.
Revised: To this end, we propose two augmentation methods for sequences: random blur and random merge.

Original: Random blur.
Revised: Random blur.

Original: Random blur constructs a blur kernel K with a randomly selected size.
Revised: Random blur constructs a blur kernel K with a randomly selected size.

Original: The values in the kernel are also randomly selected from the range of [0 , 1] so that values in the kernel decrease from middle to both sides.
Revised: The values in the kernel are also randomly selected from the range of [0 , 1] so that values in the kernel decrease from middle to both sides.

Original: Given a sequence X =
Revised: Given a sequence X =

Original: [ x 1 , · · · , x L X ] ∈ R d × L X , random blur generates an augmented sequence ˜ X by convolving X with K .
Revised: [ x 1 , · · · , x L X ] ∈ R d × L X , random blur generates an augmented sequence ˜ X by convolving X with K .

Original: The stride along the 1st dimension is set to 1.
Revised: The stride along the 1st dimension is set to 1.

Original: The stride along the 2nd dimension is randomly selected from 1 , 2 , 3 .
Revised: The stride along the 2nd dimension is randomly selected from 1 , 2 , 3 .

Original: No padding is applied.
Revised: No padding is applied.

Original: Thus, each column element ˜ x i of ˜ X is the weighted average of several successive elements of X .
Revised: Thus, each column element ˜ x i of ˜ X is the weighted average of several successive elements of X .

Original: The ground-truth alignment values between ˜ x
Revised: The ground-truth alignment values between ˜ x

Original: i and the N k elements are set as the values in K , while those between ˜ x i and all other elements of X are set to 0.
Revised: i and the N k elements are set as the values in K , while those between ˜ x i and all other elements of X are set to 0.

Original: In this way, we construct the ground-truth alignment matrix ˜ T between X and ˜ X .
Revised: In this way, we construct the ground-truth alignment matrix ˜ T between X and ˜ X .

Original: Random merge.
Revised: Random merge.

Original: For X with a length of L X , random merge ﬁrst randomly selects an integer ˜ L from [0 .
Revised: For X with a length of L X , random merge ﬁrst randomly selects an integer ˜ L from [0 .

Original: 5 L X , 0 .
Revised: 5 L X , 0 .

Original: 8 L X ] .
Revised: 8 L X ] .

Original: X is then randomly segmented into ˜ L parts, where each part has at least one element.
Revised: X is then randomly segmented into ˜ L parts, where each part has at least one element.

Original: For the k -th part with ˜ L k elements, ˜ L k values are randomly sampled from a standard
Revised: For the k -th part with ˜ L k elements, ˜ L k values are randomly sampled from a standard

Original: Gaussian distribution and then normalized by softmax to form ˜
Revised: Gaussian distribution and then normalized by softmax to form ˜

Original: L k weights.
Revised: L k weights.

Original: The ˜ L k elements are averaged with the weights to form a new element ˜ x k .
Revised: The ˜ L k elements are averaged with the weights to form a new element ˜ x k .

Original: The augmented sequence consists of all the generated elements for all parts sequentially: ˜ X =
Revised: The augmented sequence consists of all the generated elements for all parts sequentially: ˜ X =

Original: [ ˜ x 1 , · · · , ˜ x ˜ L k ] .
Revised: [ ˜ x 1 , · · · , ˜ x ˜ L k ] .

Original: The ground-truth alignment values between ˜ x k and the ˜ L k elements in the k -th part of X are set as the ˜ L k corresponding weights, while those between ˜ x k and elements in other parts are set to 0.
Revised: The ground-truth alignment values between ˜ x k and the ˜ L k elements in the k -th part of X are set as the ˜ L k corresponding weights, while those between ˜ x k and elements in other parts are set to 0.

Original: In this way, the groundtruth alignment matrix ˜ T between X and ˜ X is constructed.
Revised: In this way, the groundtruth alignment matrix ˜ T between X and ˜ X is constructed.

Original: Self-supervised learning of TAP.
Revised: Self-supervised learning of TAP.

Original: Given N unlabeled training sequences X n , n = 1 , · · · , N , the goal is to learn the TAP distance by learning the prediction network.
Revised: Given N unlabeled training sequences X n , n = 1 , · · · , N , the goal is to learn the TAP distance by learning the prediction network.

Original: For each X n , we ﬁrst randomly select one of the two methods to generate its augmented sequence ˜ X n and the corresponding ground-truth alignment ˜ T n .
Revised: For each X n , we ﬁrst randomly select one of the two methods to generate its augmented sequence ˜ X n and the corresponding ground-truth alignment ˜ T n .

Original: We then calculate the TAP distance d ( X n , ˜ X n ; f ) and the corresponding non-normalized alignment A n in Eq.
Revised: We then calculate the TAP distance d ( X n , ˜ X n ; f ) and the corresponding non-normalized alignment A n in Eq.

Original: (3) between X n and ˜ X n .
Revised: (3) between X n and ˜ X n .

Original: Since TAP is asymmetric, we also calculate the dual TAP distance d ( ˜ X n , X n ; f ) and dual alignment A nd between ˜ X n and X n .
Revised: Since TAP is asymmetric, we also calculate the dual TAP distance d ( ˜ X n , X n ; f ) and dual alignment A nd between ˜ X n and X n .

Original: We minimize the TAP and its dual distances.
Revised: We minimize the TAP and its dual distances.

Original: We also minimize the Mean Squared Errors (MSEs) between the predicted dual alignments and the ground-truth alignments, which indirectly forces the alignment and the dual alignment to be consistent.
Revised: We also minimize the Mean Squared Errors (MSEs) between the predicted dual alignments and the ground-truth alignments, which indirectly forces the alignment and the dual alignment to be consistent.

Original: The objective for a batch of B sequences is:
Revised: The objective for a batch of B sequences is:

Original: [Equation]
Revised: [Equation]

Original: Parameters of the alignment prediction network f in TAP are learned by back-propagation.
Revised: Parameters of the alignment prediction network f in TAP are learned by back-propagation.

Original: Since
Revised: Since

Original: D is ﬁxed with given sequences and T is obtained after softmax and L 1 normalization, the TAP dis- tance between two sequences is bounded.
Revised: D is ﬁxed with given sequences and T is obtained after softmax and L 1 normalization, the TAP dis- tance between two sequences is bounded.

Original: Therefore, f has no trivial collapse solution, so negative sequence samples are naturally not required.
Revised: Therefore, f has no trivial collapse solution, so negative sequence samples are naturally not required.

Original: Once f is learned, TAP can be applied to measure the difference between two sequences by feedforward calculations.
Revised: Once f is learned, TAP can be applied to measure the difference between two sequences by feedforward calculations.

Original: In Section 3.2, the self-supervised loss 10 is added to all supervised losses to improve the stability of the learned TAP.
Revised: In Section 3.2, the self-supervised loss 10 is added to all supervised losses to improve the stability of the learned TAP.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Comparison with alignment-based distances.
Revised: Comparison with alignment-based distances.

Original: We compare the self-supervised learned TAP with 8 alignment-based sequence distances in the retrieval and classiﬁcation tasks, including DTW, length-normalized DTW (lDTW), matching step-normalized DTW (nDTW), optimal transport (OT) (Kolouri et al., 2017), Sinkhorn (Cuturi, 2013), TCOT (Su & Hua, 2019), OPW (Su & Hua, 2019), and SoftDTW (Cuturi & Blondel, 2017).
Revised: We compare the self-supervised learned TAP with 8 alignment-based sequence distances in the retrieval and classiﬁcation tasks, including DTW, length-normalized DTW (lDTW), matching step-normalized DTW (nDTW), optimal transport (OT) (Kolouri et al., 2017), Sinkhorn (Cuturi, 2013), TCOT (Su & Hua, 2019), OPW (Su & Hua, 2019), and SoftDTW (Cuturi & Blondel, 2017).

Original: We follow the setting in (Su & Hua, 2019) and directly compare with the results reported in Su & Hua (2019).
Revised: We follow the setting in (Su & Hua, 2019) and directly compare with the results reported in Su & Hua (2019).

Original: The comparisons on the MSR Action3D and MSR Activity3D datasets are shown in Table 6 and Table 7, respectively.
Revised: The comparisons on the MSR Action3D and MSR Activity3D datasets are shown in Table 6 and Table 7, respectively.

Original: We observe that TAP generally outperforms other distances on the MSR Activity3D dataset, but performs worse on the MSR Action3D dataset.
Revised: We observe that TAP generally outperforms other distances on the MSR Activity3D dataset, but performs worse on the MSR Action3D dataset.

Original: On the small-scale MSR Action3D dataset, the training sequences may not exhibit enough variances and TAP may not be fully trained.
Revised: On the small-scale MSR Action3D dataset, the training sequences may not exhibit enough variances and TAP may not be fully trained.

Original: SoftDTW can be used as a differentiable loss on top of a single-layer embedding.
Revised: SoftDTW can be used as a differentiable loss on top of a single-layer embedding.

Original: SoftDTW ∗ trains the additional linear layer in a self-supervised manner by minimizing the SoftDTW distances between the original training sequences and their augmented ones, but the effect is equivalent to random classiﬁcation.
Revised: SoftDTW ∗ trains the additional linear layer in a self-supervised manner by minimizing the SoftDTW distances between the original training sequences and their augmented ones, but the effect is equivalent to random classiﬁcation.

Original: Without negative sample sampling, SoftDTW ∗ falls into the trivial solution of all zeros.
Revised: Without negative sample sampling, SoftDTW ∗ falls into the trivial solution of all zeros.

Original: In self-supervised learning, TAP naturally avoids the trivial solution since it calculates D from the original sequences and performs Softmax on T .
Revised: In self-supervised learning, TAP naturally avoids the trivial solution since it calculates D from the original sequences and performs Softmax on T .

Original: A. P ROBLEM OF SUPERVISED REPRESENTATION LEARNING WITH ALIGNMENT - BASED DISTANCES
Revised: A. P ROBLEM OF SUPERVISED REPRESENTATION LEARNING WITH ALIGNMENT - BASED DISTANCES

Original: For supervised learning, the objective is often to make the sequence distance better separate sequences from different classes.
Revised: For supervised learning, the objective is often to make the sequence distance better separate sequences from different classes.

Original: However, such objectives are difﬁcult to optimize for conventional alignment-based distances because they solve another optimization problem to obtain the alignments.
Revised: However, such objectives are difﬁcult to optimize for conventional alignment-based distances because they solve another optimization problem to obtain the alignments.

Original: For example, if we apply the triplet loss to the OPW distance, the objective becomes:
Revised: For example, if we apply the triplet loss to the OPW distance, the objective becomes:

Original: [Equation]
Revised: [Equation]

Original: The gradients of this objective with respect to the parameters of h rely on T ∗ nn (cid:48) , which is determined by another optimization problem with additional constraints on T and this optimization problem depends on h in turn.
Revised: The gradients of this objective with respect to the parameters of h rely on T ∗ nn (cid:48) , which is determined by another optimization problem with additional constraints on T and this optimization problem depends on h in turn.

Original: Therefore, the gradients cannot be explicitly formulated and directly obtained.
Revised: Therefore, the gradients cannot be explicitly formulated and directly obtained.

Original: The overall optimization problem thus cannot be optimized via back propagation in an end-to-end manner.
Revised: The overall optimization problem thus cannot be optimized via back propagation in an end-to-end manner.

Original: An alternative solution is to alternatively update T ∗ and h by ﬁxing the other in an iterative way, which is time-consuming and there is no theoretical guarantee of convergence.
Revised: An alternative solution is to alternatively update T ∗ and h by ﬁxing the other in an iterative way, which is time-consuming and there is no theoretical guarantee of convergence.

Original: Existing meth- ods (Mei et al., 2014; Su & Wu, 2020b) optimize a proxy or approximate objective through complex iterative solutions.
Revised: Existing meth- ods (Mei et al., 2014; Su & Wu, 2020b) optimize a proxy or approximate objective through complex iterative solutions.

Original: A. S ELF - ATTENTION MODULE IN SA-TAP
Revised: A. S ELF - ATTENTION MODULE IN SA-TAP

Original: In few-shot action recognition, TAP can be combined with the self-attention mechanism.
Revised: In few-shot action recognition, TAP can be combined with the self-attention mechanism.

Original: Especially, the proposed TAP focuses on aligning different sequences, but cannot fully utilize the temporal dependencies among frames in the same sequence.
Revised: Especially, the proposed TAP focuses on aligning different sequences, but cannot fully utilize the temporal dependencies among frames in the same sequence.

Original: This can be alleviated by applying a single-head self-attention (SA) module to each sequence before performing alignment by TAP.
Revised: This can be alleviated by applying a single-head self-attention (SA) module to each sequence before performing alignment by TAP.

Original: Speciﬁcally, after obtaining the feature sequence X for a video, we ﬁrst obtain X p =
Revised: Speciﬁcally, after obtaining the feature sequence X for a video, we ﬁrst obtain X p =

Original: X + P e
Revised: X + P e

Original: ( X ) , where P e ( X ) is the sequence of position embeddings for elements in X .
Revised: ( X ) , where P e ( X ) is the sequence of position embeddings for elements in X .

Original: We then employ the self-attention module to transform X p into the following sequence:
Revised: We then employ the self-attention module to transform X p into the following sequence:

Original: [Equation]
Revised: [Equation]

Original: where W q , W k , and W v are three linear projections with the size of d × d .
Revised: where W q , W k , and W v are three linear projections with the size of d × d .

Original: The sequence X is ﬁnally transformed into:
Revised: The sequence X is ﬁnally transformed into:

Original: [Equation]
Revised: [Equation]

Original: Where LN is the layer normalization.
Revised: Where LN is the layer normalization.

Original: Both the support and query sequences are enhanced before calculating their TAP distance in Eq.
Revised: Both the support and query sequences are enhanced before calculating their TAP distance in Eq.

Original: We denote this method by SA-TAP.
Revised: We denote this method by SA-TAP.

Original: A. C OMPLEXITY AND MODEL SIZE
Revised: A. C OMPLEXITY AND MODEL SIZE

Original: The complexities for calculating D , performing convolutions with a few layers and small kernels, calculating S , and performing Softmax and L 1 normalization are O
Revised: The complexities for calculating D , performing convolutions with a few layers and small kernels, calculating S , and performing Softmax and L 1 normalization are O

Original: ( L X L Y d ) , O ( L X L Y d ) ,
Revised: ( L X L Y d ) , O ( L X L Y d ) ,

Original: O ( L X L Y ) , and O ( L X L Y ) , respectively, where the number of layers, the size of kernels, and the number of kernels are omitted since they are small.
Revised: O ( L X L Y ) , and O ( L X L Y ) , respectively, where the number of layers, the size of kernels, and the number of kernels are omitted since they are small.

Original: The overall complexity of TAP is O
Revised: The overall complexity of TAP is O

Original: ( L X L Y d ) .
Revised: ( L X L Y d ) .

Original: TAP only contains three convolution layers and hence the model size is quite small.
Revised: TAP only contains three convolution layers and hence the model size is quite small.

Original: For supervised learning, the feature transformation network is the same for all methods.
Revised: For supervised learning, the feature transformation network is the same for all methods.

Original: DTW, OPW, and Soft-DTW are parameter-free but require solving an optimization problem for inferring the optimal alignment, so they introduce no additional parameters.
Revised: DTW, OPW, and Soft-DTW are parameter-free but require solving an optimization problem for inferring the optimal alignment, so they introduce no additional parameters.

Original: In the three layers of TAP, we ﬁx the kernel sizes to 5, 5, and 3, respectively, and we ﬁx the number of ﬁlers to 30, 30, and 1, respectively.
Revised: In the three layers of TAP, we ﬁx the kernel sizes to 5, 5, and 3, respectively, and we ﬁx the number of ﬁlers to 30, 30, and 1, respectively.

Original: Therefore, TAP only introduces 30 × 5 × 5 × 2 + 30 × 5 × 5 × 30 + 1 × 3 × 3 × 30 = 0 .
Revised: Therefore, TAP only introduces 30 × 5 × 5 × 2 + 30 × 5 × 5 × 30 + 1 × 3 × 3 × 30 = 0 .

Original: M parameters.
Revised: M parameters.

Original: For few-shot action recognition, the feature extraction backbone is ﬁxed to ResNet-50 for all methods.
Revised: For few-shot action recognition, the feature extraction backbone is ﬁxed to ResNet-50 for all methods.

Original: OTAM employs a variant of DTW as the distance measure and hence does not introduce additional parameters.
Revised: OTAM employs a variant of DTW as the distance measure and hence does not introduce additional parameters.

Original: TRX contains two Ω × d × d k projections and one Ω × d × d v projection, where Ω is the length of a tuple, d = 2048 is the dimension of frame-wide features and d k = d v is set to 1152.
Revised: TRX contains two Ω × d × d k projections and one Ω × d × d v projection, where Ω is the length of a tuple, d = 2048 is the dimension of frame-wide features and d k = d v is set to 1152.

Original: Therefore, TRX introduces 3 × 2 × 2048 × 1152 × Ω parameters, Ω is often set to 2 for using frame pairs and 3 for frame tuples, so the number of parameters ranges from 27M to 40.5M.
Revised: Therefore, TRX introduces 3 × 2 × 2048 × 1152 × Ω parameters, Ω is often set to 2 for using frame pairs and 3 for frame tuples, so the number of parameters ranges from 27M to 40.5M.

Original: Moreover, TRX requires an exhaustive traversal of pairs and triplets of frames with high complexity.
Revised: Moreover, TRX requires an exhaustive traversal of pairs and triplets of frames with high complexity.

Original: In TTAN, the location network in the temporal transform module takes a feature sequence as input and outputs two indexes with several learnable layers, but the detailed network structure is not speciﬁed; the temporal attentive module contains 3 d × d projections and has 12M parameters; the late fusion module introduces another 3 d × d projections with 12M parameters.
Revised: In TTAN, the location network in the temporal transform module takes a feature sequence as input and outputs two indexes with several learnable layers, but the detailed network structure is not speciﬁed; the temporal attentive module contains 3 d × d projections and has 12M parameters; the late fusion module introduces another 3 d × d projections with 12M parameters.

Original: In contrast, TAP also only introduces 0.0231M parameters.
Revised: In contrast, TAP also only introduces 0.0231M parameters.

Original: For SA-TAP, the self-attention layer contains 3 d × d projections, the kernel sizes are enlarged to 64, 64, and 1, respectively.
Revised: For SA-TAP, the self-attention layer contains 3 d × d projections, the kernel sizes are enlarged to 64, 64, and 1, respectively.

Original: Therefore, SA-TAP introduces 3 d 2 + 64 × 5 × 5 × 2 + 64 × 5 × 5 × 64 + 1 × 3 × 3 × 64 = 12 .
Revised: Therefore, SA-TAP introduces 3 d 2 + 64 × 5 × 5 × 2 + 64 × 5 × 5 × 64 + 1 × 3 × 3 × 64 = 12 .

Original: 1013 M parameters.
Revised: 1013 M parameters.

Original: The model sizes of both TAP and SA-TAP are smaller than TRX and TTAN.
Revised: The model sizes of both TAP and SA-TAP are smaller than TRX and TTAN.

Original: A. C OMPARISON OF TAP AND S OFT DTW ON RUNNING TIMES
Revised: A. C OMPARISON OF TAP AND S OFT DTW ON RUNNING TIMES

Original: We study the scalability of the proposed TAP in comparison with SoftDTW as the length and dimensionality increase.
Revised: We study the scalability of the proposed TAP in comparison with SoftDTW as the length and dimensionality increase.

Original: We randomly generate two sequences of length L , both consists of d -dimensional features.
Revised: We randomly generate two sequences of length L , both consists of d -dimensional features.

Original: For different L and d , we compute the SoftDTW distance and the TAP distance with random parameters between the two sequences, respectively, and compare the corresponding running
Revised: For different L and d , we compute the SoftDTW distance and the TAP distance with random parameters between the two sequences, respectively, and compare the corresponding running

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: times.
Revised: times.

Original: We ﬁrst ﬁx d to 500, and increase L from 100 to 1000 with an interval of 100.
Revised: We ﬁrst ﬁx d to 500, and increase L from 100 to 1000 with an interval of 100.

Original: The running times of SoftDTW and TAP as functions of L are compared in Fig. 3(a).
Revised: The running times of SoftDTW and TAP as functions of L are compared in Fig. 3(a).

Original: The running time of
Revised: The running time of

Original: SoftDTW grows rapidly as the length increases due to the dynamic programming and backtracking processes.
Revised: SoftDTW grows rapidly as the length increases due to the dynamic programming and backtracking processes.

Original: In contrast, since only the forward matrix operations are required, the running time of
Revised: In contrast, since only the forward matrix operations are required, the running time of

Original: TAP does not change much as the length increases.
Revised: TAP does not change much as the length increases.

Original: The longer the sequences, the more obvious the speed advantage of the proposed TAP.
Revised: The longer the sequences, the more obvious the speed advantage of the proposed TAP.

Original: We then ﬁx L to
Revised: We then ﬁx L to

Original: [Table]
Revised: [Table]

Original: and increase d from 100 to 1000 with an interval of 100 .
Revised: and increase d from 100 to 1000 with an interval of 100 .

Original: The running times of
Revised: The running times of

Original: SoftDTW and TAP as functions of d are compared in Fig. 3(b).
Revised: SoftDTW and TAP as functions of d are compared in Fig. 3(b).

Original: We observe that the proposed TAP is consistently about 10 times faster than SoftDTW.
Revised: We observe that the proposed TAP is consistently about 10 times faster than SoftDTW.

Original: These results demonstrate that TAP can better scale to long and high-dimensional sequences.
Revised: These results demonstrate that TAP can better scale to long and high-dimensional sequences.

Original: A. E FFECT OF DATA AUGMENTATION ON SUPERVISED LEARNING
Revised: A. E FFECT OF DATA AUGMENTATION ON SUPERVISED LEARNING

Original: We apply the data augmentations, i.e., random blur and random merge, in supervised learning only for obtaining more stable results.
Revised: We apply the data augmentations, i.e., random blur and random merge, in supervised learning only for obtaining more stable results.

Original: For example, in some cases, all sequences in a small batch satisfy the constraints in Eq.
Revised: For example, in some cases, all sequences in a small batch satisfy the constraints in Eq.

Original: (4) and Eq.
Revised: (4) and Eq.

Original: (6), where data augmentation can enlarge the within-class distance.
Revised: (6), where data augmentation can enlarge the within-class distance.

Original: Comparison results of TAP with and without data augmentations on the MSR Action3D and MSR
Revised: Comparison results of TAP with and without data augmentations on the MSR Action3D and MSR

Original: Activity3D datasets are shown in Tab. 8.
Revised: Activity3D datasets are shown in Tab. 8.

Original: In some cases, the data augmentation only brings minor improvements, while in other cases, the data augmentation even causes performance degradation, which may be because augmented sequences obtained by such temporal blur and merge do not reﬂect the real intra-class variances.
Revised: In some cases, the data augmentation only brings minor improvements, while in other cases, the data augmentation even causes performance degradation, which may be because augmented sequences obtained by such temporal blur and merge do not reﬂect the real intra-class variances.

Original: Generally, in most cases, since the augmented sequence often has a small distance from the original sequence, it will not change the selection of hard negatives and hard positives in Eq.
Revised: Generally, in most cases, since the augmented sequence often has a small distance from the original sequence, it will not change the selection of hard negatives and hard positives in Eq.

Original: Without the proposed data augmentations, TAP still outperforms all other methods where it performs best in Tab. 1, and all conclusions still hold.
Revised: Without the proposed data augmentations, TAP still outperforms all other methods where it performs best in Tab. 1, and all conclusions still hold.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. A BLATION ON THE ARCHITECTURE OF THE ALIGNMENT PREDICTION NETWORK
Revised: A. A BLATION ON THE ARCHITECTURE OF THE ALIGNMENT PREDICTION NETWORK

Original: The intuition behind using CNN as g in the alignment prediction network f is that CNN can implicitly model the temporal constraints and capture local alignment patterns.
Revised: The intuition behind using CNN as g in the alignment prediction network f is that CNN can implicitly model the temporal constraints and capture local alignment patterns.

Original: Temporal constraints for performing temporal alignment are often reﬂected as follows: the probability of two frames being aligned is not only related to the distance between the two elements and the difference between their relative positions, but also depends on the distances and alignment probabilities between their adjacent frames.
Revised: Temporal constraints for performing temporal alignment are often reﬂected as follows: the probability of two frames being aligned is not only related to the distance between the two elements and the difference between their relative positions, but also depends on the distances and alignment probabilities between their adjacent frames.

Original: Convolution kernels aim at capturing such local information.
Revised: Convolution kernels aim at capturing such local information.

Original: There may exist different local alignment patterns at different levels.
Revised: There may exist different local alignment patterns at different levels.

Original: Multiple convolution kernels in different layers are suitable for capturing these different local patterns and variances of local motions.
Revised: Multiple convolution kernels in different layers are suitable for capturing these different local patterns and variances of local motions.

Original: A transformer-like cross-attention-based architecture can also be used to predict temporal alignment.
Revised: A transformer-like cross-attention-based architecture can also be used to predict temporal alignment.

Original: Speciﬁcally, for aligning Y to X , we use a linear projection W q ∈ R d (cid:48)
Revised: Speciﬁcally, for aligning Y to X , we use a linear projection W q ∈ R d (cid:48)

Original: × d to transform all elements x i of X to ˆ x i =
Revised: × d to transform all elements x i of X to ˆ x i =

Original: W q x i , resulting in a transformed sequence
Revised: W q x i , resulting in a transformed sequence

Original: ˆ X
Revised: ˆ X

Original: =
Revised: =

Original: [ ˆ x i , i = 1 , · · · , L X ] =
Revised: [ ˆ x i , i = 1 , · · · , L X ] =

Original: W q X , where d (cid:48)
Revised: W q X , where d (cid:48)

Original: = 30 is the transformed dimensionality.
Revised: = 30 is the transformed dimensionality.

Original: Similarly, Y is transformed into ˆ Y =
Revised: Similarly, Y is transformed into ˆ Y =

Original: [ ˆ y j , j = 1 , · · · , L Y ] =
Revised: [ ˆ y j , j = 1 , · · · , L Y ] =

Original: W k Y by another projection W k .
Revised: W k Y by another projection W k .

Original: We use each ˆ x i as the query, use ˆ y j , j = 1 , · · · , L Y as keys, and calculate all pairwise Euclidean distances between ˆ x i , i = 1 , · · · , L X and ˆ y j , j = 1 , · · · , L Y into a matrix D s :=
Revised: We use each ˆ x i as the query, use ˆ y j , j = 1 , · · · , L Y as keys, and calculate all pairwise Euclidean distances between ˆ x i , i = 1 , · · · , L X and ˆ y j , j = 1 , · · · , L Y into a matrix D s :=

Original: [ e (ˆ x i , ˆ y j ) / √ d (cid:48) ] ij ∈ R L X × L Y .
Revised: [ e (ˆ x i , ˆ y j ) / √ d (cid:48) ] ij ∈ R L X × L Y .

Original: The similarity matrix S is calculated as: S = − ( D s + λ D t ) , where the hyper-parameter λ is set to 50.
Revised: The similarity matrix S is calculated as: S = − ( D s + λ D t ) , where the hyper-parameter λ is set to 50.

Original: Similarly with TAP, we perform
Revised: Similarly with TAP, we perform

Original: Softmax and L 1 normalization in Eq.(3) to S to obtain the ﬁnal alignment T .
Revised: Softmax and L 1 normalization in Eq.(3) to S to obtain the ﬁnal alignment T .

Original: We denote this method by TAP-trans.
Revised: We denote this method by TAP-trans.

Original: Comparisons of TAP and TAP-trans are shown in Tab. 9.
Revised: Comparisons of TAP and TAP-trans are shown in Tab. 9.

Original: We observe that CNN outperforms the Transformer-like attention-based network.
Revised: We observe that CNN outperforms the Transformer-like attention-based network.

Original: This may indicate that CNN indeed can capture the local alignment patterns by integrating the local contexts.
Revised: This may indicate that CNN indeed can capture the local alignment patterns by integrating the local contexts.

Original: The local receptive ﬁeld of CNN can be viewed as imposing a constraint on the temporal deviation: only local warping within the receptive ﬁeld is considered since extremely unbalanced warping is not preferred.
Revised: The local receptive ﬁeld of CNN can be viewed as imposing a constraint on the temporal deviation: only local warping within the receptive ﬁeld is considered since extremely unbalanced warping is not preferred.

Original: Increasing the kernel sizes or using more convolution layers gains larger receptive ﬁelds, hence larger warping patterns are allowed and long-range dependencies can be utilized.
Revised: Increasing the kernel sizes or using more convolution layers gains larger receptive ﬁelds, hence larger warping patterns are allowed and long-range dependencies can be utilized.

Original: We have added comparisons with two variants, namely TAP (l) and TAP (d), to evaluate the effects of using larger kernels and using more convolution layers, respectively.
Revised: We have added comparisons with two variants, namely TAP (l) and TAP (d), to evaluate the effects of using larger kernels and using more convolution layers, respectively.

Original: In TAP (l), the kernel size in the three convolution layers is increased to 7, 7, and 5 with paddings 3, 3, and 2 to keep the spatial
Revised: In TAP (l), the kernel size in the three convolution layers is increased to 7, 7, and 5 with paddings 3, 3, and 2 to keep the spatial

Original: [Table]
Revised: [Table]

Original: size, respectively.
Revised: size, respectively.

Original: In TAP (d), another convolution layer is inserted after the second layer, where the kernel size, padding, and the number of kernels are set to 5, 2, and 30, respectively.
Revised: In TAP (d), another convolution layer is inserted after the second layer, where the kernel size, padding, and the number of kernels are set to 5, 2, and 30, respectively.

Original: Comparison results are shown in Tab. 10.
Revised: Comparison results are shown in Tab. 10.

Original: Generally, using larger convolution kernels or adding more convolution layers to relax the restriction on warping and model more levels of alignment patterns can further increase the performance of TAP.
Revised: Generally, using larger convolution kernels or adding more convolution layers to relax the restriction on warping and model more levels of alignment patterns can further increase the performance of TAP.

Original: We use the CNN architecture consisting of 3 convolution layers with small kernels to impose strong inductive bias and keep the alignment prediction network lightweight.
Revised: We use the CNN architecture consisting of 3 convolution layers with small kernels to impose strong inductive bias and keep the alignment prediction network lightweight.

Original: A. E VALUATION METHODS ON MSE BETWEEN PREDICTED AND GROUND - TRUTH ALIGNMENTS .
Revised: A. E VALUATION METHODS ON MSE BETWEEN PREDICTED AND GROUND - TRUTH ALIGNMENTS .

Original: In Tab. 4, we quantitatively evaluate the predicted alignments.
Revised: In Tab. 4, we quantitatively evaluate the predicted alignments.

Original: The detailed evaluation method is as follows.
Revised: The detailed evaluation method is as follows.

Original: For each test sequence, we randomly generate 5 augmentations with the proposed augmentation methods (random blur and random merge) and approximately construct the groundtruth alignment between the original sequence and its augmentations, respectively, as described in appendix A.1.
Revised: For each test sequence, we randomly generate 5 augmentations with the proposed augmentation methods (random blur and random merge) and approximately construct the groundtruth alignment between the original sequence and its augmentations, respectively, as described in appendix A.1.

Original: To evaluate the predicted alignments by supervised trained TAP (TAP-s), we ﬁrst apply the trained transformation network h to transform the sequence and each of its augmentations, and then predict their un-normalized alignments by the trained alignment prediction network f .
Revised: To evaluate the predicted alignments by supervised trained TAP (TAP-s), we ﬁrst apply the trained transformation network h to transform the sequence and each of its augmentations, and then predict their un-normalized alignments by the trained alignment prediction network f .

Original: For supervised learning with SoftDTW (SoftDTW-s), after transforming the sequences with the trained h , the alignments are directly inferred by SoftDTW.
Revised: For supervised learning with SoftDTW (SoftDTW-s), after transforming the sequences with the trained h , the alignments are directly inferred by SoftDTW.

Original: The MSE between the predicted alignments and the ground-truth alignments of all test-augmentation sequence pairs is calculated as the performance measure.
Revised: The MSE between the predicted alignments and the ground-truth alignments of all test-augmentation sequence pairs is calculated as the performance measure.

Original: Comparisons of TAP-s and SoftDTW-s by using the triplet loss are shown in the left part of Tab. 4.
Revised: Comparisons of TAP-s and SoftDTW-s by using the triplet loss are shown in the left part of Tab. 4.

Original: We can also directly train TAP without jointly learning the transformation network h by constraining the predicted alignments.
Revised: We can also directly train TAP without jointly learning the transformation network h by constraining the predicted alignments.

Original: We generate augmentations for each training sequence and construct the corresponding ground-truth alignment.
Revised: We generate augmentations for each training sequence and construct the corresponding ground-truth alignment.

Original: The prediction network f is trained by minimizing the
Revised: The prediction network f is trained by minimizing the

Original: MSE loss between the predicted un-normalized alignments by TAP and the constructed groundtruth alignments.
Revised: MSE loss between the predicted un-normalized alignments by TAP and the constructed groundtruth alignments.

Original: In the testing stage, we directly apply SoftDTW or the trained TAP to predict the alignments between test sequences and their corresponding augmentations.
Revised: In the testing stage, we directly apply SoftDTW or the trained TAP to predict the alignments between test sequences and their corresponding augmentations.

Original: Comparisons on the
Revised: Comparisons on the

Original: MSE between the ground-truth alignments and the alignments predicted by SoftDTW and TAP on the test set are shown in the right part of Tab. 4.
Revised: MSE between the ground-truth alignments and the alignments predicted by SoftDTW and TAP on the test set are shown in the right part of Tab. 4.

Original: We observe that in both cases, TAP achieves lower
Revised: We observe that in both cases, TAP achieves lower

Original: MSEs, which may indicate that TAP can indeed learn alignment patterns from the training sequences to better approximate the ground-truth alignments.
Revised: MSEs, which may indicate that TAP can indeed learn alignment patterns from the training sequences to better approximate the ground-truth alignments.

Original: A. E VALUATING THE LEARNED REPRESENTATIONS WITH A LINEAR CLASSIFIER
Revised: A. E VALUATING THE LEARNED REPRESENTATIONS WITH A LINEAR CLASSIFIER

Original: In supervised representation learning, we jointly learn the frame-wise feature extraction network h and the alignment prediction network f .
Revised: In supervised representation learning, we jointly learn the frame-wise feature extraction network h and the alignment prediction network f .

Original: We further use a linear classiﬁer to evaluate the learned representations.
Revised: We further use a linear classiﬁer to evaluate the learned representations.

Original: For each sequence, we freeze the backbone to extract frame-wise features and then encode them into a vector representation by mean pooling.
Revised: For each sequence, we freeze the backbone to extract frame-wise features and then encode them into a vector representation by mean pooling.

Original: We then train a linear classiﬁer on these vector representations.
Revised: We then train a linear classiﬁer on these vector representations.

Original: We use the Adam optimizer, where the hyper-parameters of the optimizer remain the same as in supervised representation learning.
Revised: We use the Adam optimizer, where the hyper-parameters of the optimizer remain the same as in supervised representation learning.

Original: The number of epochs is ﬁxed to 3000.
Revised: The number of epochs is ﬁxed to 3000.

Original: Comparisons of TAP and SoftDTW with the linear classiﬁer are shown in Tab. 11.
Revised: Comparisons of TAP and SoftDTW with the linear classiﬁer are shown in Tab. 11.

Original: We observe that TAP always outperforms SoftDTW with the linear classiﬁer no matter what supervised representation learning loss is adopted and whether SoftDTW falls into the trivial solution.
Revised: We observe that TAP always outperforms SoftDTW with the linear classiﬁer no matter what supervised representation learning loss is adopted and whether SoftDTW falls into the trivial solution.

Original: This shows that the representations learned by TAP are more discriminative.
Revised: This shows that the representations learned by TAP are more discriminative.

Original: The results obtained by the linear classiﬁer are worse than the 1-NN classiﬁer, which indicates that the mean pooling loses useful temporal information and TAP can take advantage of such information.
Revised: The results obtained by the linear classiﬁer are worse than the 1-NN classiﬁer, which indicates that the mean pooling loses useful temporal information and TAP can take advantage of such information.