Original: D
Revised: D

Original: EEP
Revised: EEP

Original: T EMPORAL C LUSTERING : F ULLY UNSUPERVISED LEARNING OF TIME - DOMAIN FEATURES
Revised: T EMPORAL C LUSTERING : F ULLY UNSUPERVISED LEARNING OF TIME - DOMAIN FEATURES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Unsupervised learning of time series data, also known as temporal clustering, is a challenging problem in machine learning.
Revised: Unsupervised learning of time series data, also known as temporal clustering, is a challenging problem in machine learning.

Original: Here we propose a novel algorithm,
Revised: Here we propose a novel algorithm,

Original: Deep Temporal Clustering (DTC), to naturally integrate dimensionality reduction and temporal clustering into a single end-to-end learning framework, fully unsupervised.
Revised: Deep Temporal Clustering (DTC), to naturally integrate dimensionality reduction and temporal clustering into a single end-to-end learning framework, fully unsupervised.

Original: The algorithm utilizes an autoencoder for temporal dimensionality reduction and a novel temporal clustering layer for cluster assignment.
Revised: The algorithm utilizes an autoencoder for temporal dimensionality reduction and a novel temporal clustering layer for cluster assignment.

Original: Then it jointly optimizes the clustering objective and the dimensionality reduction objective.
Revised: Then it jointly optimizes the clustering objective and the dimensionality reduction objective.

Original: Based on requirement and application, the temporal clustering layer can be customized with any temporal similarity metric.
Revised: Based on requirement and application, the temporal clustering layer can be customized with any temporal similarity metric.

Original: Several similarity metrics and state-of-the-art algorithms are considered and compared.
Revised: Several similarity metrics and state-of-the-art algorithms are considered and compared.

Original: To gain insight into temporal features that the network has learned for its clustering, we apply a visualization method that generates a region of interest heat map for the time series.
Revised: To gain insight into temporal features that the network has learned for its clustering, we apply a visualization method that generates a region of interest heatmap for the time series.

Original: The viability of the algorithm is demonstrated using time series data from diverse domains, ranging from earthquakes to spacecraft sensor data.
Revised: The viability of the algorithm is demonstrated using time series data from diverse domains, ranging from earthquakes to spacecraft sensor data.

Original: In each case, we show that the proposed algorithm outperforms traditional methods.
Revised: In each case, we show that the proposed algorithm outperforms traditional methods.

Original: The superior performance is attributed to the fully integrated temporal dimensionality reduction and clustering criterion.
Revised: The superior performance is attributed to the fully integrated temporal dimensionality reduction and clustering criterion.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning has become the dominant approach to supervised learning of labeled data (LeCun et al., 2015; Schmidhuber, 2015).
Revised: Deep learning has become the dominant approach to supervised learning of labeled data (LeCun et al., 2015; Schmidhuber, 2015).

Original: However, in many applications the data labels may not be available or be reliable.
Revised: However, in many applications the data labels may not be available or be reliable.

Original: A variety of techniques have been developed for unsupervised learning where the algorithms draw inferences from unlabeled data.
Revised: A variety of techniques have been developed for unsupervised learning where the algorithms draw inferences from unlabeled data.

Original: However, the progress in learning of complex structure in the data has so far been largely restricted to labeled datasets, while relatively little attention was paid to learning of complex, high-level structure and features of unlabeled data.
Revised: However, the progress in learning of complex structure in the data has so far been largely restricted to labeled datasets, while relatively little attention was paid to learning of complex, high-level structure and features of unlabeled data.

Original: The standard unsupervised techniques include clustering approaches which organize similar objects into clusters.
Revised: The standard unsupervised techniques include clustering approaches which organize similar objects into clusters.

Original: Such techniques differ in the method for organizing the data as well as the metrics to measure similarity.
Revised: Such techniques differ in the method for organizing the data as well as the metrics to measure similarity.

Original: While clustering techniques have been successfully applied to static data, their extension to time series data remains an open problem.
Revised: While clustering techniques have been successfully applied to static data, their extension to time series data remains an open problem.

Original: This has left a gap in technology for accurate unsupervised learning of time series data which encompass many areas of science and engineering such as ﬁnancial trading, medical monitoring, and event detection (Aghabozorgi et al., 2015).
Revised: This has left a gap in technology for accurate unsupervised learning of time series data which encompass many areas of science and engineering such as ﬁnancial trading, medical monitoring, and event detection (Aghabozorgi et al., 2015).

Original: The problem of unsupervised time series clustering is particularly challenging.
Revised: The problem of unsupervised time series clustering is particularly challenging.

Original: Time series data from different domains exhibit considerable variations in important properties and features, temporal scales, and dimensionality.
Revised: Time series data from different domains exhibit considerable variations in important properties and features, temporal scales, and dimensionality.

Original: Further, time series data from real world applications often have temporal gaps as well as high frequency noise due to the data acquisition method and/or the inherent nature of the data (Antunes & Oliveira, 2001).
Revised: Further, time series data from real world applications often have temporal gaps as well as high frequency noise due to the data acquisition method and/or the inherent nature of the data (Antunes & Oliveira, 2001).

Original: To address the issues and limitations of using standard clustering techniques on time series data, we present a novel algorithm called deep temporal clustering (DTC).
Revised: To address the issues and limitations of using standard clustering techniques on time series data, we present a novel algorithm called deep temporal clustering (DTC).

Original: A key element of DTC is the transformation of the time series data into a low dimensional latent space using a trainable network, which here we choose to be a deep autoencoder network that is fully integrated with a novel temporal clustering layer.
Revised: A key element of DTC is the transformation of the time series data into a low dimensional latent space using a trainable network, which here we choose to be a deep autoencoder network that is fully integrated with a novel temporal clustering layer.

Original: The overview of our method is illustrated in Figure 1.
Revised: The overview of our method is illustrated in Figure 1.

Original: The latent representation is compatible with any temporal similarity metric.
Revised: The latent representation is compatible with any temporal similarity metric.

Original: The proposed DTC algorithm was designed based on the observation that time series data have informative features on all time scales.
Revised: The proposed DTC algorithm was designed based on the observation that time series data have informative features on all time scales.

Original: To disentangle the data manifolds, i.e., to uncover the latent dimension(s) along which the temporal or spatio-temporal unlabeled data split into two or more classes, we propose the following three-level approach.
Revised: To disentangle the data manifolds, i.e., to uncover the latent dimension(s) along which the temporal or spatio-temporal unlabeled data split into two or more classes, we propose the following three-level approach.

Original: The ﬁrst level, implemented as a CNN, reduces the data dimensionality and learns the dominant short-time-scale waveforms.
Revised: The ﬁrst level, implemented as a CNN, reduces the data dimensionality and learns the dominant short-time-scale waveforms.

Original: The second level, implemented as a BI-LSTM, reduces the data dimensionality further and learns the temporal connections between waveforms across all time scales.
Revised: The second level, implemented as a BI-LSTM, reduces the data dimensionality further and learns the temporal connections between waveforms across all time scales.

Original: The third level performs non-parametric clustering of BI-LSTM latent representations, ﬁnding one or more spatio-temporal dimensions along which the data split into two or more classes.
Revised: The third level performs non-parametric clustering of BI-LSTM latent representations, ﬁnding one or more spatio-temporal dimensions along which the data split into two or more classes.

Original: The unique ability of this approach to untangle the data manifolds without discarding the information provided by the time course of the data (e.g., in contrast to PCA-based methods) allows our approach to achieve high performance on a variety of real-life and benchmark datasets without any parameter adjustment.
Revised: The unique ability of this approach to untangle the data manifolds without discarding the information provided by the time course of the data (e.g., in contrast to PCA-based methods) allows our approach to achieve high performance on a variety of real-life and benchmark datasets without any parameter adjustment.

Original: DTC also includes an algorithm to visualize the cluster-assignment activations across time, a feature not available in traditional clustering algorithms.
Revised: DTC also includes an algorithm to visualize the cluster-assignment activations across time, a feature not available in traditional clustering algorithms.

Original: This allows the localization of events in unlabeled time series data, and provides explanation (as opposed to black-box approaches) regarding the most informative data features for class assignment.
Revised: This allows the localization of events in unlabeled time series data, and provides explanation (as opposed to black-box approaches) regarding the most informative data features for class assignment.

Original: To the best of our knowledge, this is the ﬁrst work on the application of deep learning in temporal clustering.
Revised: To the best of our knowledge, this is the ﬁrst work on the application of deep learning in temporal clustering.

Original: The main contribution of our study is the formulation of an end-to-end deep learning algorithm that implements objective formulation to achieve meaningful temporal clustering.
Revised: The main contribution of our study is the formulation of an end-to-end deep learning algorithm that implements objective formulation to achieve meaningful temporal clustering.

Original: We carefully formulated the objective to encompass two crucial aspects essential for high clustering accuracy: an effective latent representation and a similarity metric which can be integrated into the learning structure.
Revised: We carefully formulated the objective to encompass two crucial aspects essential for high clustering accuracy: an effective latent representation and a similarity metric which can be integrated into the learning structure.

Original: We demonstrate that the end-to-end optimization of our network for both reconstruction loss and clustering loss offers superior performance compared to cases where these two objectives are optimized separately.
Revised: We demonstrate that the end-to-end optimization of our network for both reconstruction loss and clustering loss offers superior performance compared to cases where these two objectives are optimized separately.

Original: We also show that DTC outperforms current state-of-theart, k-Shape (Paparrizos & Gravano, 2015) and hierarchical clustering with complete linkage, when evaluated on various real world time series datasets.
Revised: We also show that DTC outperforms current state-of-theart, k-Shape (Paparrizos & Gravano, 2015) and hierarchical clustering with complete linkage, when evaluated on various real world time series datasets.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Much of the existing research in temporal clustering methods has focused on addressing one of two core issues: an effective dimensionality reduction and choosing an appropriate similarity metric.
Revised: Much of the existing research in temporal clustering methods has focused on addressing one of two core issues: an effective dimensionality reduction and choosing an appropriate similarity metric.

Original: One class of solutions use application dependent dimensionality reduction to ﬁlter out high frequency noise.
Revised: One class of solutions use application dependent dimensionality reduction to ﬁlter out high frequency noise.

Original: Examples include adaptive piecewise constant approximation (Keogh et al., 2001) and nonnegative matrix factorization (Brockwell & Davis, 2013).
Revised: Examples include adaptive piecewise constant approximation (Keogh et al., 2001) and nonnegative matrix factorization (Brockwell & Davis, 2013).

Original: One drawback of such approaches is that the dimensionality reduction is performed independent of the clustering criterion, resulting in the potential loss of long range temporal correlations as well as ﬁltering out of relevant features.
Revised: One drawback of such approaches is that the dimensionality reduction is performed independent of the clustering criterion, resulting in the potential loss of long range temporal correlations as well as ﬁltering out of relevant features.

Original: Other limitations include hand crafted and application speciﬁc nature of the transformations which require extensive domain knowledge of the data source and are difﬁcult to design.
Revised: Other limitations include hand crafted and application speciﬁc nature of the transformations which require extensive domain knowledge of the data source and are difﬁcult to design.

Original: A second class of solutions have focused on the creation of a suitable similarity measure between two time series by taking into account features such as complexity (Batista et al., 2011), correlation (Galeano & Pe˜na, 2000; Golay et al., 1998), and time warping (Berndt & Clifford, 1994).
Revised: A second class of solutions have focused on the creation of a suitable similarity measure between two time series by taking into account features such as complexity (Batista et al., 2011), correlation (Galeano & Pe˜na, 2000; Golay et al., 1998), and time warping (Berndt & Clifford, 1994).

Original: These similarity measures were then incorporated into traditional clustering algorithms such as k-means or hierarchical clustering.
Revised: These similarity measures were then incorporated into traditional clustering algorithms such as k-means or hierarchical clustering.

Original: Montero et al.
Revised: Montero et al.

Original: (2014) conducted a detailed study on various similarity metrics and showed that the choice of similarity measure has a signiﬁcant impact on the results.
Revised: (2014) conducted a detailed study on various similarity metrics and showed that the choice of similarity measure has a signiﬁcant impact on the results.

Original: However, a good similarity measure, in the absence of proper dimensionality reduction, may not be sufﬁcient to obtain optimal clustering results due to the complexity and high dimensional nature of time series data.
Revised: However, a good similarity measure, in the absence of proper dimensionality reduction, may not be sufﬁcient to obtain optimal clustering results due to the complexity and high dimensional nature of time series data.

Original: The studies mentioned have shown that casting the original time series data into a low dimensional latent space is well suited for temporal clustering.
Revised: The studies mentioned have shown that casting the original time series data into a low dimensional latent space is well suited for temporal clustering.

Original: These approaches lack a general methodology for the selection of an effective latent space that captures the properties of time series data.
Revised: These approaches lack a general methodology for the selection of an effective latent space that captures the properties of time series data.

Original: Another key step in achieving meaningful clustering results is ensuring the similarity metric is compatible with the temporal feature space.
Revised: Another key step in achieving meaningful clustering results is ensuring the similarity metric is compatible with the temporal feature space.

Original: Recent research in clustering methods for static data achieved superior performance by jointly optimizing a stacked autoencoder for dimensionality reduction and a k-means objective for clustering (Xie et al.
Revised: Recent research in clustering methods for static data achieved superior performance by jointly optimizing a stacked autoencoder for dimensionality reduction and a k-means objective for clustering (Xie et al.

Original: (2016), Yang et al.
Revised: (2016), Yang et al.

Original: However, those approaches were designed for static data, and are not well suited for time series data clustering.
Revised: However, those approaches were designed for static data, and are not well suited for time series data clustering.

Original:  P ROPOSED M ETHOD : D EEP T EMPORAL C LUSTERING
Revised:  P ROPOSED M ETHOD : D EEP T EMPORAL C LUSTERING

Original: Consider n unlabeled instances, x 1 , ..., x n , of a temporal sequence x .
Revised: Consider n unlabeled instances, x 1 , ..., x n , of a temporal sequence x .

Original: The goal is to perform unsupervised clustering of these n unlabeled sequences into k ≤ n clusters, based on the latent high-level features of x .
Revised: The goal is to perform unsupervised clustering of these n unlabeled sequences into k ≤ n clusters, based on the latent high-level features of x .

Original: 3. E FFECTIVE L ATENT R EPRESENTATION
Revised: 3. E FFECTIVE L ATENT R EPRESENTATION

Original: Effective latent representation is a key aspect of the temporal clustering.
Revised: Effective latent representation is a key aspect of the temporal clustering.

Original: We achieve this by making use of a temporal autoencoder (TAE) as shown in Figure 1.
Revised: We achieve this by making use of a temporal autoencoder (TAE) as shown in Figure 1.

Original: The ﬁrst level of the network architecture consists of a 1D convolution layer, which extracts key short-term features (waveforms), followed by max pooling layer of size P .
Revised: The ﬁrst level of the network architecture consists of a 1D convolution layer, which extracts key short-term features (waveforms), followed by max pooling layer of size P .

Original: Leaky rectifying linear units (L-ReLUs) are used.
Revised: Leaky rectifying linear units (L-ReLUs) are used.

Original: The ﬁrst level thus casts the time series into a more compact representation while retaining most of the relevant information.
Revised: The ﬁrst level thus casts the time series into a more compact representation while retaining most of the relevant information.

Original: This dimensionality reduction is crucial for further processing to avoid very long sequences which can lead to poor performance.
Revised: This dimensionality reduction is crucial for further processing to avoid very long sequences which can lead to poor performance.

Original: First-level activations are then fed to the second level (Bidirectional LSTM) to obtain the latent representation.
Revised: First-level activations are then fed to the second level (Bidirectional LSTM) to obtain the latent representation.

Original: We use BI-LSTM to learn temporal changes in both time directions.
Revised: We use BI-LSTM to learn temporal changes in both time directions.

Original: This allows to collapse the input sequences in all dimensions except temporal, and to cast the input into a much smaller latent space.
Revised: This allows to collapse the input sequences in all dimensions except temporal, and to cast the input into a much smaller latent space.

Original: Finally, the clustering layer assigns the
Revised: Finally, the clustering layer assigns the

Original: BI-LSTM latent representation of sequences x i , i = 1 ...n , to clusters.
Revised: BI-LSTM latent representation of sequences x i , i = 1 ...n , to clusters.

Original: Learning in both 1D CNN and BI-LSTM is driven by interleaved minimization of two cost functions.
Revised: Learning in both 1D CNN and BI-LSTM is driven by interleaved minimization of two cost functions.

Original: First cost function is provided by the mean square error (MSE) of the input sequence reconstruction from the BI-LSTM latent representation; this ensures that the sequence is still well represented after the dimensionality reduction in levels 1 and 2.
Revised: First cost function is provided by the mean square error (MSE) of the input sequence reconstruction from the BI-LSTM latent representation; this ensures that the sequence is still well represented after the dimensionality reduction in levels 1 and 2.

Original: Reconstruction is provided by an upsampling layer of size P followed by a deconvolutional layer to obtain autoencoder output.
Revised: Reconstruction is provided by an upsampling layer of size P followed by a deconvolutional layer to obtain autoencoder output.

Original: The second cost function is provided by the clustering metric (e.g. KL divergence, see below) of level 3; this ensures that the high-level features that deﬁne the subspace spanned by the cluster centroids indeed separate the sequences x i , i = 1 ...n into k clusters of distinct spatio-temporal behavior.
Revised: The second cost function is provided by the clustering metric (e.g. KL divergence, see below) of level 3; this ensures that the high-level features that deﬁne the subspace spanned by the cluster centroids indeed separate the sequences x i , i = 1 ...n into k clusters of distinct spatio-temporal behavior.

Original: The clustering metric optimization modiﬁes the weights in the BI-LSTM and in the CNN.
Revised: The clustering metric optimization modiﬁes the weights in the BI-LSTM and in the CNN.

Original: As the result, the high-level features encoded by the BI-LSTM optimally separate the sequences into clusters, thus disentangling the spatio-temporal manifolds of the x dynamics.
Revised: As the result, the high-level features encoded by the BI-LSTM optimally separate the sequences into clusters, thus disentangling the spatio-temporal manifolds of the x dynamics.

Original: We emphasize that the end-to-end optimization of our network for both objectives (reconstruction loss and clustering loss) efﬁciently extracts spatio-temporal features that are best suited to separate the input sequences into categories (clusters), that is, to disnetangle the complicated highdimensional manifolds of input dynamics.
Revised: We emphasize that the end-to-end optimization of our network for both objectives (reconstruction loss and clustering loss) efﬁciently extracts spatio-temporal features that are best suited to separate the input sequences into categories (clusters), that is, to disnetangle the complicated highdimensional manifolds of input dynamics.

Original: This in contrast to the traditional approaches where dimensionality reduction (e.g., truncated PCA or stacked autoencoders) only optimizes reconstruction, while clustering only optimizes separation.
Revised: This in contrast to the traditional approaches where dimensionality reduction (e.g., truncated PCA or stacked autoencoders) only optimizes reconstruction, while clustering only optimizes separation.

Original: This results, in the traditional approaches, in separation being carried out in the space of latent features that are not best suited to make the data more separable.
Revised: This results, in the traditional approaches, in separation being carried out in the space of latent features that are not best suited to make the data more separable.

Original: Direct comparison (see Section 4) shows marked improvement in unsupervised categorization when using end-to-end optimization relative to disjointly optimized dimensionality reduction and clustering.
Revised: Direct comparison (see Section 4) shows marked improvement in unsupervised categorization when using end-to-end optimization relative to disjointly optimized dimensionality reduction and clustering.

Original: Forgoing initial dimensionality reduction and applying the traditional clustering approaches (e.g., k-means, dbscan, t-sne) directly to the spatio-temporal data x usually results in severe overﬁtting and poor performance.
Revised: Forgoing initial dimensionality reduction and applying the traditional clustering approaches (e.g., k-means, dbscan, t-sne) directly to the spatio-temporal data x usually results in severe overﬁtting and poor performance.

Original: We further emphasize that our approach not only provides the effective end-to-end optimization, but also makes use of temporal continuity of the spatio-temporal data x to extract and encode the informative features on all time scales in the latent representation of the BI-LSTM.
Revised: We further emphasize that our approach not only provides the effective end-to-end optimization, but also makes use of temporal continuity of the spatio-temporal data x to extract and encode the informative features on all time scales in the latent representation of the BI-LSTM.

Original: 3. T EMPORAL C LUSTERING L AYER
Revised: 3. T EMPORAL C LUSTERING L AYER

Original: The temporal clustering layer consists of k centroids w j , j ∈ 1 ..k .
Revised: The temporal clustering layer consists of k centroids w j , j ∈ 1 ..k .

Original: To initialize these cluster centroids, we use the latent signals
Revised: To initialize these cluster centroids, we use the latent signals

Original: z i obtained by feeding input x i through the initialized TAE.
Revised: z i obtained by feeding input x i through the initialized TAE.

Original: z i are then used to perform hierarchical clustering with complete linkage in the feature space Z through a similarity metric (discussed in the next section 3.2.2).
Revised: z i are then used to perform hierarchical clustering with complete linkage in the feature space Z through a similarity metric (discussed in the next section 3.2.2).

Original: We perform k cut to obtain the clusters and then average the elements in each cluster to get initial centroids estimates w j , j = 1 ...k .
Revised: We perform k cut to obtain the clusters and then average the elements in each cluster to get initial centroids estimates w j , j = 1 ...k .

Original: 3.2. C LUSTERING C RITERION
Revised: 3.2. C LUSTERING C RITERION

Original: After we obtain an initial estimate of the centroids w j , we train the temporal clustering layer using an unsupervised algorithm that alternates between two steps.
Revised: After we obtain an initial estimate of the centroids w j , we train the temporal clustering layer using an unsupervised algorithm that alternates between two steps.

Original: First, we compute the probability of assignment of input x i belonging to the cluster j .
Revised: First, we compute the probability of assignment of input x i belonging to the cluster j .

Original: The closer the latent representation z i of input x
Revised: The closer the latent representation z i of input x

Original: i is to the centroid w j , the higher the probability of x i belonging to cluster j 2.
Revised: i is to the centroid w j , the higher the probability of x i belonging to cluster j 2.

Original: Second, we update the centroids by using a loss function, which maximizes the high conﬁdence assignments using a target distribution p , eq. 5, discussed in subsequent sections.
Revised: Second, we update the centroids by using a loss function, which maximizes the high conﬁdence assignments using a target distribution p , eq. 5, discussed in subsequent sections.

Original: 3.2. C LUSTERING A SSIGNMENT
Revised: 3.2. C LUSTERING A SSIGNMENT

Original: When an input z i is fed to the temporal clustering layer, we compute distances d ij from each of the centroids w j using a similarity metric.
Revised: When an input z i is fed to the temporal clustering layer, we compute distances d ij from each of the centroids w j using a similarity metric.

Original: Then we normalize the distances d ij into probability assignments using a Student’s t distribution kernel (Maaten & Hinton, 2008).
Revised: Then we normalize the distances d ij into probability assignments using a Student’s t distribution kernel (Maaten & Hinton, 2008).

Original: The probability assignment of latent signal belonging to k th cluster is as follows:
Revised: The probability assignment of latent signal belonging to k th cluster is as follows:

Original: [Equation]
Revised: [Equation]

Original: Here q ij is the probability of input i belonging to cluster j , z i corresponds to the signal in the latent space Z, obtained from temporal autoencoder after encoding the input signal x i ∈ X .
Revised: Here q ij is the probability of input i belonging to cluster j , z i corresponds to the signal in the latent space Z, obtained from temporal autoencoder after encoding the input signal x i ∈ X .

Original: The parameter α is the number of degrees of freedom of the Students t distribution.
Revised: The parameter α is the number of degrees of freedom of the Students t distribution.

Original: In an unsupervised setting we can set α = 1 as suggested by (Maaten & Hinton, 2008).
Revised: In an unsupervised setting we can set α = 1 as suggested by (Maaten & Hinton, 2008).

Original: Lastly siml () is the temporal similarity metric which is used to compute the distance between the encoded signal z i and centroid w j .
Revised: Lastly siml () is the temporal similarity metric which is used to compute the distance between the encoded signal z i and centroid w j .

Original: We illustrate a 2 cluster example in Figure 2, where the latent input z is used to compute distances dand d 2 from centroids w 1 and w 2 using a similarity metric, Later converted into probabilities p 1 and p 2 using a Students t distribution kernel.
Revised: We illustrate a 2 cluster example in Figure 2, where the latent input z is used to compute distances dand d 2 from centroids w 1 and w 2 using a similarity metric, Later converted into probabilities p 1 and p 2 using a Students t distribution kernel.

Original: In this study, we experiment with various similarity metrics as follows:
Revised: In this study, we experiment with various similarity metrics as follows:

Original: Complexity Invariant Similarity(CID) proposed by Batista et al.
Revised: Complexity Invariant Similarity(CID) proposed by Batista et al.

Original: (2011) will compute the similarity based on the euclidean distance ED which is corrected by complexity estimation CF of the two series x, y .
Revised: (2011) will compute the similarity based on the euclidean distance ED which is corrected by complexity estimation CF of the two series x, y .

Original: This distance is calculated as follows:
Revised: This distance is calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: where CF ( x, y ) is a complexity factor deﬁned as follows: max ( CE ( x ) ,CE ( y ))
Revised: where CF ( x, y ) is a complexity factor deﬁned as follows: max ( CE ( x ) ,CE ( y ))

Original: min ( CE ( x ) ,CE ( y )) and CE ( x ) and CE ( y ) are the complexity estimates of a time series x and y , respectively.
Revised: min ( CE ( x ) ,CE ( y )) and CE ( x ) and CE ( y ) are the complexity estimates of a time series x and y , respectively.

Original: The core idea of CID is that with increase in the complexity differences between the series, the distance increases.
Revised: The core idea of CID is that with increase in the complexity differences between the series, the distance increases.

Original: Further, if both input sequences have the same complexity then the distance simply is the euclidean distance.
Revised: Further, if both input sequences have the same complexity then the distance simply is the euclidean distance.

Original: The complexity of each sequence is deﬁned as:
Revised: The complexity of each sequence is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where N is length of the sequence 2.
Revised: where N is length of the sequence 2.

Original: Correlation based Similarity(COR) as used by Golay et al.
Revised: Correlation based Similarity(COR) as used by Golay et al.

Original: (1998), computes similarities using the estimated pearsons correlation ρ between the latent representation
Revised: (1998), computes similarities using the estimated pearsons correlation ρ between the latent representation

Original: z i and the centroids w j .
Revised: z i and the centroids w j .

Original: In this study we compute the COR as follows:
Revised: In this study we compute the COR as follows:

Original: [Equation]
Revised: [Equation]

Original: where ρ is the pearson’s correlation, given by ρ x,y = cov( x,y ) σ x σ y and cov is the covariance.
Revised: where ρ is the pearson’s correlation, given by ρ x,y = cov( x,y ) σ x σ y and cov is the covariance.

Original: Auto Correlation based Similarity(ACF) as used in Galeano & Pe˜na (2000), computes the similarity between the latent representation
Revised: Auto Correlation based Similarity(ACF) as used in Galeano & Pe˜na (2000), computes the similarity between the latent representation

Original: z i and the centroids w j using autocorrelation (ACF) coefﬁcients and then performs weighted euclidean distance between the autocorrelation coefﬁcients.
Revised: z i and the centroids w j using autocorrelation (ACF) coefﬁcients and then performs weighted euclidean distance between the autocorrelation coefﬁcients.

Original: Euclidean(EUCL) is the euclidean distance between the input signal and each of the centroids.
Revised: Euclidean(EUCL) is the euclidean distance between the input signal and each of the centroids.

Original: C LUSTERING L OSS C OMPUTATION
Revised: C LUSTERING L OSS C OMPUTATION

Original: To train the temporal clustering layer iteratively we formulate the objective as minimizing the KL divergence loss between q ij and a target distribution p ij .
Revised: To train the temporal clustering layer iteratively we formulate the objective as minimizing the KL divergence loss between q ij and a target distribution p ij .

Original: Choice of p is crucial because it needs to strengthen high conﬁdence predictions and normalize the losses in order to prevent distortion of latent representation.
Revised: Choice of p is crucial because it needs to strengthen high conﬁdence predictions and normalize the losses in order to prevent distortion of latent representation.

Original: This is realized using
Revised: This is realized using

Original: [Equation]
Revised: [Equation]

Original: where f j = (cid:80) ni
Revised: where f j = (cid:80) ni

Original: =1 q ij .
Revised: =1 q ij .

Original: Further empirical properties of this distribution were discussed in detail in
Revised: Further empirical properties of this distribution were discussed in detail in

Original: Hinton et al.
Revised: Hinton et al.

Original: (2006) and Xie et al.
Revised: (2006) and Xie et al.

Original: Now using this target distribution, we compute the KL divergence loss:
Revised: Now using this target distribution, we compute the KL divergence loss:

Original: [Equation]
Revised: [Equation]

Original: where n and k are number of samples in dataset and number of clusters respectively.
Revised: where n and k are number of samples in dataset and number of clusters respectively.

Original: 3. DTC O PTIMIZATION
Revised: 3. DTC O PTIMIZATION

Original: We perform batch-wise joint optimization of clustering and auto encoder by minimizing KL divergence loss and mean squared error loss, respectively.
Revised: We perform batch-wise joint optimization of clustering and auto encoder by minimizing KL divergence loss and mean squared error loss, respectively.

Original: This optimization problem is challenging, and an effective initialization of the cluster centroids is highly important.
Revised: This optimization problem is challenging, and an effective initialization of the cluster centroids is highly important.

Original: Cluster centroids reﬂect the latent representation of the data.
Revised: Cluster centroids reﬂect the latent representation of the data.

Original: To make sure initial centroids represent the data well, we ﬁrst pretrain parameters of the autoencoder to start with a meaningful latent representation.
Revised: To make sure initial centroids represent the data well, we ﬁrst pretrain parameters of the autoencoder to start with a meaningful latent representation.

Original: After pretraining, the cluster centers are then initialized by hierarchical clustering with complete linkage on embedded features of all datapoints.
Revised: After pretraining, the cluster centers are then initialized by hierarchical clustering with complete linkage on embedded features of all datapoints.

Original: Later we update autoencoder weights and cluster centers using gradients dL
Revised: Later we update autoencoder weights and cluster centers using gradients dL

Original: c dz i and dL ae dz , see eqs.
Revised: c dz i and dL ae dz , see eqs.

Original: 7 and 8 below, using backpropagation mini-batch SGD.
Revised: 7 and 8 below, using backpropagation mini-batch SGD.

Original: We also update target distribution p during every SGD update using eq. 5.
Revised: We also update target distribution p during every SGD update using eq. 5.

Original: [Equation]
Revised: [Equation]

Original: Similar approaches have been used in Xie et al.
Revised: Similar approaches have been used in Xie et al.

Original: (2016) and Yang et al.
Revised: (2016) and Yang et al.

Original: This helps prevent any problematic solutions from drifting too far away from the original input signal.
Revised: This helps prevent any problematic solutions from drifting too far away from the original input signal.

Original: As reconstruction of the original signal (reconstruction MSE reduction) is a part of the objective, the latent representation will converge at a suitable representation so as to minimize both the clustering loss and the MSE loss.
Revised: As reconstruction of the original signal (reconstruction MSE reduction) is a part of the objective, the latent representation will converge at a suitable representation so as to minimize both the clustering loss and the MSE loss.

Original: 3. V ISUALIZING  H EAT M AP
Revised: 3. V ISUALIZING T HE H EATMAP

Original: In most applications it is important to identify and localize the main data feature(s) contributing to the ﬁnal classiﬁcation.
Revised: In most applications it is important to identify and localize the main data feature(s) contributing to the ﬁnal classiﬁcation.

Original: To do so, we create a heatmap-generating network following the approach used in (Hwang & Kim, 2016) to localize tumors in medical images using only image-level labels.
Revised: To do so, we create a heatmap-generating network following the approach used in (Hwang & Kim, 2016) to localize tumors in medical images using only image-level labels.

Original: Brieﬂy, we use the cluster labels generated by our DTC network to train a new supervised hierarchical convolutional network to classify the inputs x .
Revised: Brieﬂy, we use the cluster labels generated by our DTC network to train a new supervised hierarchical convolutional network to classify the inputs x .

Original: Activity in this new network allows us to generate heatmaps showing the parts of spatio-temporal inputs x that are most relevant for assigning inputs to clusters.
Revised: Activity in this new network allows us to generate heatmaps showing the parts of spatio-temporal inputs x that are most relevant for assigning inputs to clusters.

Original: Example of such heatmap for NASA data (see Section 4) is shown in Figure 3.
Revised: Example of such heatmap for NASA data (see Section 4) is shown in Figure 3.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our networks were implemented and tested using Python, TensorFlow 1.3 and Keras 2.0 software on
Revised: Our networks were implemented and tested using Python, TensorFlow 1.3 and Keras 2.0 software on

Original: Nvidia GTX 1080Ti graphics processor.
Revised: Nvidia GTX 1080Ti graphics processor.

Original: For the baseline algorithms the TSclust package (Montero et al., 2014) R implementations were used except for k-Shape where Python implementation 1 was used.
Revised: For the baseline algorithms the TSclust package (Montero et al., 2014) R implementations were used except for k-Shape where Python implementation 1 was used.

Original: [Figure]
Revised: [Figure]

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: The performance of our DTC algorithm is evaluated on a wide range of real world datasets.
Revised: The performance of our DTC algorithm is evaluated on a wide range of real world datasets.

Original: • We make use of a few publicly available UCR Time series Classiﬁcation Archive datasets (Chen et al., 2015).
Revised: • We make use of a few publicly available UCR Time series Classiﬁcation Archive datasets (Chen et al., 2015).

Original: Table 1 includes the properties of these datasets: the number of samples N , the time steps in each sequence L , and the class distribution ratio r .
Revised: Table 1 includes the properties of these datasets: the number of samples N , the time steps in each sequence L , and the class distribution ratio r .

Original: Since we are using these datasets as unlabeled data, we combine the training and test datasets for all UCR datasets for all experiments in this study.
Revised: Since we are using these datasets as unlabeled data, we combine the training and test datasets for all UCR datasets for all experiments in this study.

Original: • In addition, we use spacecraft magnetometer data from the recent NASA Magnetospheric Multiscale (MMS) Mission.
Revised: • In addition, we use spacecraft magnetometer data from the recent NASA Magnetospheric Multiscale (MMS) Mission.

Original: The magnetospheric plasma environment exhibits many transient structures and waves.
Revised: The magnetospheric plasma environment exhibits many transient structures and waves.

Original: Here we are interested in automated detection of spacecraft crossings of the so-called ﬂux transfer events (FTEs) in an otherwise turbulent medium.
Revised: Here we are interested in automated detection of spacecraft crossings of the so-called ﬂux transfer events (FTEs) in an otherwise turbulent medium.

Original: FTEs (Russell & Elphic, 1979) are characterized by bipolar signature in the B N component of the magnetic ﬁeld.
Revised: FTEs (Russell & Elphic, 1979) are characterized by bipolar signature in the B N component of the magnetic ﬁeld.

Original: The B N data contains a total of 104 time series, with each sequence having 1440 time steps.
Revised: The B N data contains a total of 104 time series, with each sequence having 1440 time steps.

Original: 4. B ASELINE M ETHODS
Revised: 4. B ASELINE M ETHODS

Original: We compare the results of our DTC algorithm against two clustering methods, the hierarchical clustering with complete linkage, and k-Shape (Paparrizos & Gravano, 2015).
Revised: We compare the results of our DTC algorithm against two clustering methods, the hierarchical clustering with complete linkage, and k-Shape (Paparrizos & Gravano, 2015).

Original: k-Shape is the current state-of-the-art temporal clustering algorithm.
Revised: k-Shape is the current state-of-the-art temporal clustering algorithm.

Original: It is a partitional clustering approach which preserves the shapes of time series and computes centroids under the scale and shift invariance.
Revised: It is a partitional clustering approach which preserves the shapes of time series and computes centroids under the scale and shift invariance.

Original: We have considered four similarity metrics in our experiments: 1) Complexity Invariant Distance(CID) , 2) Correlation based Similarity(COR) , 3) Auto Correlation based Similarity(ACF) , and 4) Euclidean Based Similarity(EUCL) .
Revised: We have considered four similarity metrics in our experiments: 1) Complexity Invariant Distance(CID) , 2) Correlation based Similarity(COR) , 3) Auto Correlation based Similarity(ACF) , and 4) Euclidean Based Similarity(EUCL) .

Original: [Table]
Revised: [Table]

Original: ProximalPhalanx
Revised: ProximalPhalanx

Original: OutlineCorrect (891,80,2.12,4) 0.
Revised: OutlineCorrect (891,80,2.12,4) 0.

Original: ShapeletSim (200,500,1.00,10) 0.
Revised: ShapeletSim (200,500,1.00,10) 0.

Original: SonyAIBO
Revised: SonyAIBO

Original: RobotSurfaceII (980,65,1.61,5) 0.
Revised: RobotSurfaceII (980,65,1.61,5) 0.

Original: SonyAIBO
Revised: SonyAIBO

Original: RobotSurface (621,70,0.78,5) 0.
Revised: RobotSurface (621,70,0.78,5) 0.

Original: ItalyPower
Revised: ItalyPower

Original: Demand (1096,24,1,4) 0.
Revised: Demand (1096,24,1,4) 0.

Original: WormsTwoClass (258,900,1.37,10) 0.
Revised: [Table] [Table]

Original: 4. E VALUATION M ETRICS
Revised: 4. E VALUATION M ETRICS

Original: We have expert labels for our datasets.
Revised: We have expert labels for our datasets.

Original: However, our entire training pipeline is unsupervised and we only used the labels to measure the performance of the models as a classiﬁer.
Revised: However, our entire training pipeline is unsupervised and we only used the labels to measure the performance of the models as a classiﬁer.

Original: To this end, we use the
Revised: To this end, we use the

Original: Receiver Operating Characteristics (ROC) (Fawcett, 2006) and area under the curve (AUC) as our evaluation metrics.
Revised: Receiver Operating Characteristics (ROC) (Fawcett, 2006) and area under the curve (AUC) as our evaluation metrics.

Original: We used bootstrap sampling (Singh & Xie, 2008) and averaged the ROC curves over 5 trials.
Revised: We used bootstrap sampling (Singh & Xie, 2008) and averaged the ROC curves over 5 trials.

Original: 4. I MPLEMENTATION AND P ARAMETER I NITIALIZATION
Revised: 4. I MPLEMENTATION AND P ARAMETER I NITIALIZATION

Original: Parameter optimization using cross-validation is not feasible in unsupervised clustering.
Revised: Parameter optimization using cross-validation is not feasible in unsupervised clustering.

Original: Hence we use commonly used parameters for DTC and avoid dataset speciﬁc tuning as much as possible.
Revised: Hence we use commonly used parameters for DTC and avoid dataset speciﬁc tuning as much as possible.

Original: The convolution layer has 50 ﬁlters with kernel size 10 and two Bi-LSTM’s have ﬁlters 50 and 1 respectively.
Revised: The convolution layer has 50 ﬁlters with kernel size 10 and two Bi-LSTM’s have ﬁlters 50 and 1 respectively.

Original: The pooling size P is chosen such that it makes the latent representation size < 100 for faster experimentation.
Revised: The pooling size P is chosen such that it makes the latent representation size < 100 for faster experimentation.

Original: P for each experiment is listed in the ﬁrst column in Table 1.
Revised: P for each experiment is listed in the ﬁrst column in Table 1.

Original: The deconvolutional layer has kernel size 10.
Revised: The deconvolutional layer has kernel size 10.

Original: We initialize all weights to a zero-mean Gaussian distribution with a standard deviation of 0.01.
Revised: We initialize all weights to a zero-mean Gaussian distribution with a standard deviation of 0.01.

Original: Autoencoder network is pre-trained, using the Adam optimizer, over 10 epochs.
Revised: Autoencoder network is pre-trained, using the Adam optimizer, over 10 epochs.

Original: Temporal clustering layer centroids are initialized using hierarchical clustering with complete linkage and using the chosen metric.
Revised: Temporal clustering layer centroids are initialized using hierarchical clustering with complete linkage and using the chosen metric.

Original: The entire deep architecture is jointly trained for clustering and auto encoder loss until convergence criterion of 0.1% change in cluster assignment is met.
Revised: The entire deep architecture is jointly trained for clustering and auto encoder loss until convergence criterion of 0.1% change in cluster assignment is met.

Original: The mini-batch size is set to 64 for both pretraining and end-to-end ﬁne tuning of the network and the starting learning rate is set to 0.1.
Revised: The mini-batch size is set to 64 for both pretraining and end-to-end ﬁne tuning of the network and the starting learning rate is set to 0.1.

Original: These parameters are the same for all experiments and are held constant across all datasets.
Revised: These parameters are the same for all experiments and are held constant across all datasets.

Original: The baseline algorithms we use are parameter free.
Revised: The baseline algorithms we use are parameter free.

Original: 4. Q UALITATIVE A NALYSIS
Revised: 4. Q UALITATIVE A NALYSIS

Original: We show in Fig.
Revised: We show in Fig.

Original: 3 results of DTC for three distinct time series from the MMS dataset.
Revised: 3 results of DTC for three distinct time series from the MMS dataset.

Original: The top panels shows the three input time series.
Revised: The top panels shows the three input time series.

Original: The bottom panels show the activation map as generated by the algorithm.
Revised: The bottom panels show the activation map as generated by the algorithm.

Original: Activation map proﬁles correlate well with the location of the bipolar signatures of the events.
Revised: Activation map proﬁles correlate well with the location of the bipolar signatures of the events.

Original: Events are highlighted by the dashed ovals for readers’ convenience.
Revised: Events are highlighted by the dashed ovals for readers’ convenience.

Original: Note that the second time series has two events and the heatmap identiﬁes both of the events correctly.
Revised: Note that the second time series has two events and the heatmap identiﬁes both of the events correctly.

Original: The third time series (right panels) is a non-event and the algorithm correctly identiﬁes it as such (heatmap activation is low).
Revised: The third time series (right panels) is a non-event and the algorithm correctly identiﬁes it as such (heatmap activation is low).

Original: [Figure]
Revised: [Figure]

Original: i v
Revised: i v

Original: [Figure]
Revised: [Figure]

Original: 4. Q UANTITATIVE A NALYSIS
Revised: 4. Q UANTITATIVE A NALYSIS

Original: One of the main points of the paper is that the joint training of the two objectives, reconstruction loss and clustering loss, yields superior performance compared to the disjointed training where each loss is optimized independently.
Revised: One of the main points of the paper is that the joint training of the two objectives, reconstruction loss and clustering loss, yields superior performance compared to the disjointed training where each loss is optimized independently.

Original: Direct comparison between joint end-to-end training of the DTC vs disjoint DTC training (training the temporal autoencoder to minimize reconstruction loss and then training the clustering to minimize clustering loss) on the MMS dataset, all other network parameters being identical, results in the average AUC of 0.93 for joint training vs. average AUC of 0.88 for disjointed training.
Revised: Direct comparison between joint end-to-end training of the DTC vs disjoint DTC training (training the temporal autoencoder to minimize reconstruction loss and then training the clustering to minimize clustering loss) on the MMS dataset, all other network parameters being identical, results in the average AUC of 0.93 for joint training vs. average AUC of 0.88 for disjointed training.

Original: Next, we include in Table 1 a detail comparison of results from DTC and the two baseline clustering techniques for 4 different similarity metrics and over 13 different datasets.
Revised: Next, we include in Table 1 a detail comparison of results from DTC and the two baseline clustering techniques for 4 different similarity metrics and over 13 different datasets.

Original: We can see that our algorithm was able to improve the baseline performance in all of the datasets over all metrics.
Revised: We can see that our algorithm was able to improve the baseline performance in all of the datasets over all metrics.

Original: Although k-Shape performed well on a few datasets, chosen the right metric DTC outperforms k-Shape for all the datasets considered.
Revised: Although k-Shape performed well on a few datasets, chosen the right metric DTC outperforms k-Shape for all the datasets considered.

Original: We also show the comparison of ROCs in Fig. 4.
Revised: We also show the comparison of ROCs in Fig. 4.

Original: In order to keep the ﬁgure readable, we only show the ROCs for 4 datasets over 4 similarity metrics.
Revised: In order to keep the ﬁgure readable, we only show the ROCs for 4 datasets over 4 similarity metrics.

Original: These results illustrate DTC to be robust and provide superior performance over existing techniques across datasets from different domains, with different dataset sizes N , different lengths L , as well as a range of data imbalances denoted by r .
Revised: These results illustrate DTC to be robust and provide superior performance over existing techniques across datasets from different domains, with different dataset sizes N , different lengths L , as well as a range of data imbalances denoted by r .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we addressed the question of unsupervised learning of patterns in temporal sequences, unsupervised event detection and clustering.
Revised: In this work we addressed the question of unsupervised learning of patterns in temporal sequences, unsupervised event detection and clustering.

Original: Post-hoc labeling of the clusters, and comparison to ground-truth labels not used in training, reveals high degree of agreement between our unsupervised clustering results and human-labeled categories, on several types of datasets.
Revised: Post-hoc labeling of the clusters, and comparison to ground-truth labels not used in training, reveals high degree of agreement between our unsupervised clustering results and human-labeled categories, on several types of datasets.

Original: This indicates graceful dimensionality reduction from inputs with extensive and complex temporal structure to one or fewdimensional space spanned by the cluster centroids.
Revised: This indicates graceful dimensionality reduction from inputs with extensive and complex temporal structure to one or fewdimensional space spanned by the cluster centroids.

Original: As most natural stimuli are time-continuous and unlabeled, the approach promises to be of great utility in real-world applications.
Revised: As most natural stimuli are time-continuous and unlabeled, the approach promises to be of great utility in real-world applications.

Original: Generalization to multichannel spatio-temporal input is straightforward and has been carried out as well; it will be described in more detail in a separate paper.
Revised: Generalization to multichannel spatio-temporal input is straightforward and has been carried out as well; it will be described in more detail in a separate paper.