Original: D EEP T EMPORAL C LUSTERING : F ULLY UNSUPERVISED LEARNING OF TIME - DOMAIN FEATURES
Revised: D EEP T EMPORAL C LUSTERING : F ULLY UNSUPERVISED LEARNING OF TIME - DOMAIN FEATURES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Unsupervised learning of timeseries data is a challenging problem in machine learning.
Revised: Unsupervised learning of time series data, also known as temporal clustering, is a challenging problem in machine learning.

Original: Here, we propose a novel algorithm, Deep Temporal Clustering (DTC),a fully unsupervised method, to naturally integrate dimensionality reduction and temporal clustering into a single end to end learning framework.
Revised: Here we propose a novel algorithm, Deep Temporal Clustering (DTC), to naturally integrate dimensionality reduction and temporal clustering into a single end-to-end learning framework, fully unsupervised.

Original: The algorithm starts with an initial cluster estimates using an autoencoder for  dimensionality reduction and a novel temporal clustering layer for cluster assignment.
Revised: The algorithm utilizes an autoencoder for temporal dimensionality reduction and a novel temporal clustering layer for cluster assignment.

Original: Then it jointly optimizes the clustering objective and the dimensionality reduction objective.
Revised: Then it jointly optimizes the clustering objective and the dimensionality reduction objective.

Original: Based on requirement and application, the temporal clustering layer can be customized with any temporal similarity metric.
Revised: Based on requirement and application, the temporal clustering layer can be customized with any temporal similarity metric.

Original: Several similarity metrics  are considered and compared.
Revised: Several similarity metrics and state-of-the-art algorithms are considered and compared.

Original: To gain insight into  features that the network has learned for its clustering, we apply a visualization method that generates a heat map of regions of interest in the timeseries.
Revised: To gain insight into temporal features that the network has learned for its clustering, we apply a visualization method that generates a region of interest heat map for the time series.

Original: The viability of the algorithm is demonstrated using timeseries data from diverse domains, ranging from earthquakes to  sensor data from spacecraft.
Revised: The viability of the algorithm is demonstrated using time series data from diverse domains, ranging from earthquakes to spacecraft sensor data.

Original: In each case, we show that our algorithm outperforms traditional methods.
Revised: In each case, we show that the proposed algorithm outperforms traditional methods.

Original: This performance is attributed to  fully integrated temporal dimensionality reduction and clustering criterion.
Revised: The superior performance is attributed to the fully integrated temporal dimensionality reduction and clustering criterion.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning has become the dominant approach to supervised classiﬁcation problems where the network is trained on expert labeledstatic data (LeCun et al., 2015; Schmidhuber, 2015).
Revised: Deep learning has become the dominant approach to supervised learning of labeled data (LeCun et al., 2015; Schmidhuber, 2015).

Original: However, in a large class of problems, either the expert labels are not reliable or they do not exist at all.
Revised: However, in many applications the data labels may not be available or be reliable.

Original: To address such problems, a long list of clustering algorithms have been developed to organize data into separate categories. While much of  the research has  been devoted totechniques dealing with static data, there is a lack of machine learning algorithms for unsupervised timeseries data. 
Revised: A variety of techniques have been developed for unsupervised learning where the algorithms draw inferences from unlabeled data. However, the progress in learning of complex structure in the data has so far been largely restricted to labeled datasets, while relatively little attention was paid to learning of complex, high-level structure and features of unlabeled data. The standard unsupervised techniques include clustering approaches which organize similar objects into clusters.

Revised: Such techniques differ in the method for organizing the data as well as the metrics to measure similarity.

Original: Accurate temporal clustering is essential in  many  science and engineeringapplications, such asevent detection, ﬁnancial trading,and medical applications (Aghabozorgi et al., 2015).However, standard clustering algorithms were designed for static data and do not account for temporal structure in the data. As such, they cannot be directly applied to timeseries.
Revised: While clustering techniques have been successfully applied to static data, their extension to time series data remains an open problem. This has left a gap in technology for accurate unsupervised learning of time series data which encompass many areas of science and engineering such as ﬁnancial trading, medical monitoring, and event detection (Aghabozorgi et al., 2015).

Original: A number of factors make the clustering of the temporal data a particularly challenging task.
Revised: The problem of unsupervised time series clustering is particularly challenging.

Original: Timeseries from different domains exhibit considerable variations in their properties, features, and  temporal scales, and are generally high-dimensional.
Revised: Time series data from different domains exhibit considerable variations in important properties and features, temporal scales, and dimensionality.

Original: Further, timeseries data from real world applications often have data gaps as well as high ﬂuctuating noise due to the data acquisition defects or the inherent nature of the data (Antunes & Oliveira, 2001).
Revised: Further, time series data from real world applications often have temporal gaps as well as high frequency noise due to the data acquisition method and/or the inherent nature of the data (Antunes & Oliveira, 2001).

Original: To address these issues, we present a novel algorithm called deep temporal clustering (DTC) which transforms the timeseries into a latent space, that is compatible with any temporal similarity metric using a learnable deep network.
Revised: To address the issues and limitations of using standard clustering techniques on time series data, we present a novel algorithm called deep temporal clustering (DTC).

Original: This enables theneural net to learn simultaneously a compatible latent space and a clustering criterion. The transformation of timeseries into a low dimensional latent space in DTC is achieved through a deep autoencoder  that isin turn fully integrated with a novel temporal clustering layer.
Revised: A key element of DTC is the transformation of the time series data into a low dimensional latent space using a trainable network, which here we choose to be a deep autoencoder network that is fully integrated with a novel temporal clustering layer.

Original: The overview of our method is illustrated in Figure 1.
Revised: The overview of our method is illustrated in Figure 1.

Revised: The latent representation is compatible with any temporal similarity metric.

Original: The network described in this work was designed based on several key observations.
Revised: The proposed DTC algorithm was designed based on the observation that time series data have informative features on all time scales.

Original: First, natural signals in time domain are continuous (i.e., nearby points in time are predictive of each-other) and can be decomposed into a relatively low number of dominant waveforms on a short time scale. Hence we use the convolutional layer (CNN) as a ﬁrst component of our network to reduce dimensionality and to cast the temporal input into a more convenient spatio-temporal representation without losing much relevant information.
Revised: To disentangle the data manifolds, i.e., to uncover the latent dimension(s) along which the temporal or spatio-temporal unlabeled data split into two or more classes, we propose the following three-level approach. The ﬁrst level, implemented as a CNN, reduces the data dimensionality and learns the dominant short-time-scale waveforms. The second level, implemented as a BI-LSTM, reduces the data dimensionality further and learns the temporal connections between waveforms across all time scales.

Original: Second, natural signals in time domain may contain information over long time scale, potentially much longer than the time extent of  the CNN ﬁlters. Hence we use a bidirectional LSTM to extract and represent those long-term features in the input.
Revised: The third level performs non-parametric clustering of BI-LSTM latent representations, ﬁnding one or more spatio-temporal dimensions along which the data split into two or more classes.

Original: By performing joint end-to-end optimization of CNN and LSTM we achieve best clustering performance along relevant temporal or spatio-temporal dimensions for a broad range of time scales of relevant temporal features. Please note that here, for clarity, we only show results for single-channel temporal inputs. In a separate work we will present the results for multichannel inputs, replacing 1D
Revised: The unique ability of this approach to untangle the data manifolds without discarding the information provided by the time course of the data (e.g., in contrast to PCA-based methods) allows our approach to achieve high performance on a variety of real-life and benchmark datasets without any parameter adjustment. DTC also includes an algorithm to visualize the cluster-assignment activations across time, a feature not available in traditional clustering algorithms.

Original: CNN with 2D CNN.
Revised: This allows the localization of events in unlabeled time series data, and provides explanation (as opposed to black-box approaches) regarding the most informative data features for class assignment.

Original: To the best of our knowledge, this is the ﬁrst work on the application of deep learning in temporal clustering.
Revised: To the best of our knowledge, this is the ﬁrst work on the application of deep learning in temporal clustering.

Original: In the remainder of the paper, we demonstrate how DTC addresses the following questions:

Original: What feature space to use?

Original: How to integrate temporal clustering functionality into deep learning?

Original: Can one visualize the key features in the timeseries that the network learned to use in its clustering result?

Original: How to achieve an optimal result? The main contribution of our study involves formulating an end-to-end deep learning algorithm, and implements objective formulationso as to achieve a temporal clustering.
Revised: The main contribution of our study is the formulation of an end-to-end deep learning algorithm that implements objective formulation to achieve meaningful temporal clustering.

Original: We carefully formulated the objective encompassing two crucial aspects essential for high clustering accuracy, i.e., an effective latent representation and a similarity metric which can be integrated into the learning structure.
Revised: We carefully formulated the objective to encompass two crucial aspects essential for high clustering accuracy: an effective latent representation and a similarity metric which can be integrated into the learning structure.

Original: Joint optimization of  these  objectives together was able to produce a signiﬁcant improvement in comparison with the baseline metrics. This is substantiated by extensive experimentation involving various real world datasets. In autoencoder, we use a 1D CNN to extract temporal features, then use a Bidirectional LSTM to cast the temporal information along both directions into a reduced latent space. We have conducted extensive numerical experiments using various temporal similarity metrics.
Revised: We demonstrate that the end-to-end optimization of our network for both reconstruction loss and clustering loss offers superior performance compared to cases where these two objectives are optimized separately.

Original: Speciﬁcally, we studied 4 different metric combinations on publicly available UCR timeseries datasets as well as recent spacecraft data collected from NASA’s Magnetospheric Multiscale (MMS)

Original: Mission.

Original: Our numerical experiments suggest that our proposed method improves the performance compared to baseline metrics on all datasets. Finally, our proposed method presents some desirable features when applied to real clustering problems. Notably, it can visualize the activations which lead to cluster assignment. Most of the traditional clustering algorithms are not able to do this.
Revised: We also show that DTC outperforms current state-of-theart, k-Shape (Paparrizos & Gravano, 2015) and hierarchical clustering with complete linkage, when evaluated on various real world time series datasets.

Original: This can be particularly useful for datasets where there is a need to localize events without labels.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Much of the existing research to solve temporal clustering  has focused on addressing one ofthe two core issues. First is an effective dimensionality reduction andsecond is choosing an effective similarity metric.
Revised: Much of the existing research in temporal clustering methods has focused on addressing one of two core issues: an effective dimensionality reduction and choosing an appropriate similarity metric.

Original: Several temporal clustering methods based on application dependent transformations have been previously proposed like adaptive piecewise constant approximation (Keogh et al., 2001) and nonnegative matrix factorization (Brockwell & Davis, 2013).Given the high-dimensional nature of timeseries, the transformations consist of dimensionality reduction approaches which can ﬁlter out the high ﬂuctuating noise.
Revised: One class of solutions use application dependent dimensionality reduction to ﬁlter out high frequency noise. Examples include adaptive piecewise constant approximation (Keogh et al., 2001) and nonnegative matrix factorization (Brockwell & Davis, 2013).

Original: However, these methods are inefﬁcient because they perform dimensionality reduction  independent of the clustering criterion. Moreover, these methods may lose long  temporal correlation and even ﬁlter out some relevant feature.
Revised: One drawback of such approaches is that the dimensionality reduction is performed independent of the clustering criterion, resulting in the potential loss of long range temporal correlations as well as ﬁltering out of relevant features.

Original: Furthermore, most of the existing strategies are based on hand crafted  transformations which are meticulously designed for a speciﬁc application. This requires extensive  knowledge of the data, which is not always possible. Recently, Xie et al. (2016) and Yang et al.
Revised: Other limitations include hand crafted and application speciﬁc nature of the transformations which require extensive domain knowledge of the data source and are difﬁcult to design.

Original: (2016) achieved superior performance by jointly optimizing a stacked autoencoder for dimensionality reduction and a kmeans objective for clustering.

Original: However, both the stacked autoencoder and the clustering objective are designed for static data, i.e., they collapse any temporal component and will fail to cluster temporal data.

Original: Another approach has been based on  creation of  suitable similarity measure between two timeseries which take into accountunique features like complexity (Batista et al., 2011), correlation (Galeano & Pe ˜ na, 2000; Golay et al., 1998), and time warping (Berndt & Clifford, 1994).
Revised: A second class of solutions have focused on the creation of a suitable similarity measure between two time series by taking into account features such as complexity (Batista et al., 2011), correlation (Galeano & Pe˜na, 2000; Golay et al., 1998), and time warping (Berndt & Clifford, 1994).

Original: Using these metrics, traditional clustering algorithms such as k-means or hierarchical clustering were implemented.
Revised: These similarity measures were then incorporated into traditional clustering algorithms such as k-means or hierarchical clustering.

Original: Recently, Montero et al.
Revised: Montero et al.

Original: (2014) conducted a detailed study on variousstate-of-the-art similarity metrics. The choice ofthe appropriate similarity metric is a key step in these clustering approaches.
Revised: (2014) conducted a detailed study on various similarity metrics and showed that the choice of similarity measure has a signiﬁcant impact on the results.

Original: However,  similarity metric itself may not be sufﬁcient for good clustering because of the complexity and high dimensional nature of temporal data.
Revised: However, a good similarity measure, in the absence of proper dimensionality reduction, may not be sufﬁcient to obtain optimal clustering results due to the complexity and high dimensional nature of time series data.

Original: Theabove-mentioned studies  have shown that casting the original timeseries into a low dimensional feature space is well suited for timeseries clustering.
Revised: The studies mentioned have shown that casting the original time series data into a low dimensional latent space is well suited for temporal clustering.

Original: However, these approaches lack a general method for the selection of the latent space thatbest captures the properties of a given timeseries data.
Revised: These approaches lack a general methodology for the selection of an effective latent space that captures the properties of time series data.

Original: Another key step in achieving optimal clustering results is ensuring the similarity metric is compatible with the  feature space.
Revised: Another key step in achieving meaningful clustering results is ensuring the similarity metric is compatible with the temporal feature space.

Revised: Recent research in clustering methods for static data achieved superior performance by jointly optimizing a stacked autoencoder for dimensionality reduction and a k-means objective for clustering (Xie et al.

Revised: (2016), Yang et al.

Revised: However, those approaches were designed for static data, and are not well suited for time series data clustering.

Original:  P ROPOSED M ETHOD : D EEP T EMPORAL C LUSTERING
Revised:  P ROPOSED M ETHOD : D EEP T EMPORAL C LUSTERING

Original: Given a dataset of n temporal signals X = x 1 ,  x 2 , .., x  n , we group time series into k clusters based on a similarity metric.
Revised: Consider n unlabeled instances, x 1 , ..., x n , of a temporal sequence x . The goal is to perform unsupervised clustering of these n unlabeled sequences into k ≤ n clusters, based on the latent high-level features of x .

Original: 3.  L ATENT T RANSFORMATION
Revised: 3. E FFECTIVE L ATENT R EPRESENTATION

Original: Latent transformation is a key aspect of the temporal clustering.
Revised: Effective latent representation is a key aspect of the temporal clustering.

Original: We achieve this by making use of a temporal autoencoder (TAE) as shown in Figure 1.
Revised: We achieve this by making use of a temporal autoencoder (TAE) as shown in Figure 1.

Original: The  network architecture consists of a 1D convolution layer, which extracts key features, followed by max pooling which shortens the timeseries.
Revised: The ﬁrst level of the network architecture consists of a 1D convolution layer, which extracts key short-term features (waveforms), followed by max pooling layer of size P .

Revised: Leaky rectifying linear units (L-ReLUs) are used.

Revised: The ﬁrst level thus casts the time series into a more compact representation while retaining most of the relevant information.

Original: This shortening of signal is crucial for further processing to avoid very long sequences which can lead to poor performance.
Revised: This dimensionality reduction is crucial for further processing to avoid very long sequences which can lead to poor performance.

Original: We use leaky rectiﬁed linear units (ReLUs) as activation function. Convolutional features are then fed to Bi-LSTM’s to obtain the latent representation.
Revised: First-level activations are then fed to the second level (Bidirectional LSTM) to obtain the latent representation.

Original: We use Bidirectional LSTM so that they can perceive temporal changes along both directions and can collapse the input sequences in all dimensions except temporal and can cast the input into a much smaller latent space.
Revised: We use BI-LSTM to learn temporal changes in both time directions. This allows to collapse the input sequences in all dimensions except temporal, and to cast the input into a much smaller latent space.

Original: Finally, the latent representation is decoded using upsampling and a deconvolutional layer.
Revised: Finally, the clustering layer assigns the

Original: Architectures having convolutional features followed by LSTM produced consistently good results in supervised speech applications (Hannun et al., 2014) and so we adapt this into our fully unsupervised framework. Note that for a speciﬁc application, one may achieve higher accuracy with some modiﬁcation of our architecture, but the motivation of  this study is focus on a general framework which can encompass a broad range of timeseries data.
Revised: BI-LSTM latent representation of sequences x i , i = 1 ...n , to clusters. Learning in both 1D CNN and BI-LSTM is driven by interleaved minimization of two cost functions. First cost function is provided by the mean square error (MSE) of the input sequence reconstruction from the BI-LSTM latent representation; this ensures that the sequence is still well represented after the dimensionality reduction in levels 1 and 2.

Original: To avoid overﬁtting, we incorporate dropout (Witten et al., 2016) during the optimization.
Revised: Reconstruction is provided by an upsampling layer of size P followed by a deconvolutional layer to obtain autoencoder output.

Original: Training is performed byminimizing the L2 loss.
Revised: The second cost function is provided by the clustering metric (e.g. KL divergence, see below) of level 3; this ensures that the high-level features that deﬁne the subspace spanned by the cluster centroids indeed separate the sequences x i , i = 1 ...n into k clusters of distinct spatio-temporal behavior. The clustering metric optimization modiﬁes the weights in the BI-LSTM and in the CNN. As the result, the high-level features encoded by the BI-LSTM optimally separate the sequences into clusters, thus disentangling the spatio-temporal manifolds of the x dynamics.

Original: 3. T EMPORAL C LUSTERING L AYER
Revised: We emphasize that the end-to-end optimization of our network for both objectives (reconstruction loss and clustering loss) efﬁciently extracts spatio-temporal features that are best suited to separate the input sequences into categories (clusters), that is, to disnetangle the complicated highdimensional manifolds of input dynamics. This in contrast to the traditional approaches where dimensionality reduction (e.g., truncated PCA or stacked autoencoders) only optimizes reconstruction, while clustering only optimizes separation. This results, in the traditional approaches, in separation being carried out in the space of latent features that are not best suited to make the data more separable. Direct comparison (see Section 4) shows marked improvement in unsupervised categorization when using end-to-end optimization relative to disjointly optimized dimensionality reduction and clustering.

Revised: Forgoing initial dimensionality reduction and applying the traditional clustering approaches (e.g., k-means, dbscan, t-sne) directly to the spatio-temporal data x usually results in severe overﬁtting and poor performance.

Revised: We further emphasize that our approach not only provides the effective end-to-end optimization, but also makes use of temporal continuity of the spatio-temporal data x to extract and encode the informative features on all time scales in the latent representation of the BI-LSTM.

Revised: 3. T EMPORAL C LUSTERING L AYER

Original: The temporal clustering layer consists of k centroidshaving weights of w j , j ∈ 1 ..k .
Revised: The temporal clustering layer consists of k centroids w j , j ∈ 1 ..k .

Original: To initialize these cluster centroids, we use the latent signals
Revised: To initialize these cluster centroids, we use the latent signals

Original: z i obtained by feeding input x i∈ X through the initialized TAE.
Revised: z i obtained by feeding input x i through the initialized TAE.

Original: z i are then used to perform hierarchical clustering  in the feature space Z through a similarity metric (discussed in the next section 3.2.2).
Revised: z i are then used to perform hierarchical clustering with complete linkage in the feature space Z through a similarity metric (discussed in the next section 3.2.2).

Original: This yields the  initial centroids estimates w j , j ∈ 1 ..k , where k is number of clusters.
Revised: We perform k cut to obtain the clusters and then average the elements in each cluster to get initial centroids estimates w j , j = 1 ...k .

Original: 3.2. C LUSTERING C RITERION
Revised: 3.2. C LUSTERING C RITERION

Original: After we obtain an initial estimate of the centroids w j , we train the temporal clustering layer using an unsupervised algorithm that alternates between two steps.
Revised: After we obtain an initial estimate of the centroids w j , we train the temporal clustering layer using an unsupervised algorithm that alternates between two steps.

Original: First, we compute the probability of assignment of input x i belonging to the cluster j .
Revised: First, we compute the probability of assignment of input x i belonging to the cluster j .

Original: The closer the  input x i is to the centroid w j , the higher the probability  belonging to cluster j 2.
Revised: The closer the latent representation z i of input x i is to the centroid w j , the higher the probability of x i belonging to cluster j 2.

Original: Second, we update the centroids by using a loss function, which maximizes the high conﬁdence assignments using an external distribution.
Revised: Second, we update the centroids by using a loss function, which maximizes the high conﬁdence assignments using a target distribution p , eq. 5, discussed in subsequent sections.

Original: These two steps are iterated until we reach a stop condition, which in our case is when the change in the centroids is less than 1% or some maximum iterations.

Original:  C LUSTERING A SSIGNMENT
Revised: 3.2. C LUSTERING A SSIGNMENT

Original: When an input z i is fed to the temporal clustering layer, we compute distances d ij from each of the centroids w j using a similarity metric.
Revised: When an input z i is fed to the temporal clustering layer, we compute distances d ij from each of the centroids w j using a similarity metric.

Original: Note that the similarity computation is inside the neural network and fully integrated in the learning process.

Original: Then we normalize the distances d ij into probability assignments using a Student’s t distribution kernel (Maaten & Hinton, 2008).
Revised: Then we normalize the distances d ij into probability assignments using a Student’s t distribution kernel (Maaten & Hinton, 2008).

Original: The probability assignment of latent signal belonging to k th cluster is as follows:
Revised: The probability assignment of latent signal belonging to k th cluster is as follows:

Original: [Equation]
Revised: [Equation]

Original: Here q ij is the probability of input i belonging to cluster j , z i corresponds to the signal in the latent space Z, obtained from temporal autoencoder after encoding the input signal x i ∈ X .
Revised: Here q ij is the probability of input i belonging to cluster j , z i corresponds to the signal in the latent space Z, obtained from temporal autoencoder after encoding the input signal x i ∈ X .

Original: The parameter α is the number of degrees of freedom of the Students t distribution.
Revised: The parameter α is the number of degrees of freedom of the Students t distribution.

Original: In an unsupervised setting, since there is no actual learning, we can set α = 1 as suggested by (Maaten & Hinton, 2008).
Revised: In an unsupervised setting we can set α = 1 as suggested by (Maaten & Hinton, 2008).

Original: Lastly siml () is the temporal similarity metric which is used to compute the distance between the encoded signal z i and centroid w j .
Revised: Lastly siml () is the temporal similarity metric which is used to compute the distance between the encoded signal z i and centroid w j .

Original: We illustrate a 2 cluster example in Figure 2, where the latent input z is used to compute distances d 1 and d 2 from centroids w 1 and w 2 using a similarity metric, Later converted into probabilities p 1 and p 2 using a Students t distribution kernel.
Revised: We illustrate a 2 cluster example in Figure 2, where the latent input z is used to compute distances dand d 2 from centroids w 1 and w 2 using a similarity metric, Later converted into probabilities p 1 and p 2 using a Students t distribution kernel.

Original: In this study, we experiment with various similarity metrics as follows:
Revised: In this study, we experiment with various similarity metrics as follows:

Original:  Invariant Similarity(CID) proposed by Batista et al.
Revised: Complexity Invariant Similarity(CID) proposed by Batista et al.

Original: (2011) will compute the similarity based on the euclidean distance  which is corrected by complexity estimation  of the two series 
Revised: (2011) will compute the similarity based on the euclidean distance ED which is corrected by complexity estimation CF of the two series x, y .

Original: This distance is calculated as follows:
Revised: This distance is calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: where CF ( x, y ) is a complexity factor deﬁned as follows: max ( CE ( x ) ,CE ( y ))
Revised: where CF ( x, y ) is a complexity factor deﬁned as follows: max ( CE ( x ) ,CE ( y ))

Original: min ( CE ( x ) ,CE ( y )) and CE ( x ) and CE ( y ) are the complexity estimates of a timeseries x and y , respectively.
Revised: min ( CE ( x ) ,CE ( y )) and CE ( x ) and CE ( y ) are the complexity estimates of a time series x and y , respectively.

Original: The core idea of CID is that with increase in the complexity differences between the series, the distance increases.
Revised: The core idea of CID is that with increase in the complexity differences between the series, the distance increases.

Original: Further, if both input sequences have the same complexity then the distance simply is the euclidean distance.
Revised: Further, if both input sequences have the same complexity then the distance simply is the euclidean distance.

Original: The complexity of each sequence is deﬁned as:
Revised: The complexity of each sequence is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where N is length of the sequence 2.
Revised: where N is length of the sequence 2.

Original: Correlation based Similarity(COR) as used by Golay et al.
Revised: Correlation based Similarity(COR) as used by Golay et al.

Original: (1998), computes similarities using the estimated pearsons correlation ρ between the latent representation
Revised: (1998), computes similarities using the estimated pearsons correlation ρ between the latent representation

Original: z i and the centroids w j .
Revised: z i and the centroids w j .

Original: In this study we compute the COR as follows:
Revised: In this study we compute the COR as follows:

Original: [Equation]
Revised: [Equation]

Original: where ρ is the pearson’s correlation, given by ρ X,Y = cov( X,Y )[Equation] X σ Y and cov is the covariance.
Revised: where ρ is the pearson’s correlation, given by ρ x,y = cov( x,y ) σ x σ y and cov is the covariance.

Original: Auto Correlation based Similarity(ACF) as used in Galeano & Pe˜na (2000), computes the similarity between the latent representation
Revised: Auto Correlation based Similarity(ACF) as used in Galeano & Pe˜na (2000), computes the similarity between the latent representation

Original: z i and the centroids w j using autocorrelation (ACF) coefﬁcients and then performs weighted euclidean distance between the autocorrelation coefﬁcients.
Revised: z i and the centroids w j using autocorrelation (ACF) coefﬁcients and then performs weighted euclidean distance between the autocorrelation coefﬁcients.

Original: Euclidean(EUCL) is the euclidean distance between the input signal and each of the centroids.
Revised: Euclidean(EUCL) is the euclidean distance between the input signal and each of the centroids.

Original: 3.2. C LUSTERING L OSS C OMPUTATION
Revised: C LUSTERING L OSS C OMPUTATION

Original: Finally, to train the temporal clustering layer iteratively weneed to compute a loss. After we obtain the probability assignments, we formulate the objective as minimizing the KL divergence loss between q ij and ahigh conﬁdence target distribution so as to maximize high conﬁdence assignments.
Revised: To train the temporal clustering layer iteratively we formulate the objective as minimizing the KL divergence loss between q ij and a target distribution p ij . Choice of p is crucial because it needs to strengthen high conﬁdence predictions and normalize the losses in order to prevent distortion of latent representation.

Original: This is realized usinga target distribution p ij
Revised: This is realized using

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where f j = (cid:80) ni
Revised: where f j = (cid:80) ni

Original: =1 q ij .
Revised: =1 q ij .

Original: Further empirical properties of this distribution were discussed in detail in
Revised: Further empirical properties of this distribution were discussed in detail in

Original: Hinton et al.
Revised: Hinton et al.

Original: (2006) and Xie et al.
Revised: (2006) and Xie et al.

Original: Now using this high conﬁdence distribution, we compute the KL divergence loss using the following equation:
Revised: Now using this target distribution, we compute the KL divergence loss:

Original: [Equation]
Revised: [Equation]

Original: where n and k are number of samples in dataset and number of clusters respectively.
Revised: where n and k are number of samples in dataset and number of clusters respectively.

Original: Since we are maximizing the high conﬁdence assignments, this distribution p works robustly when the initial estimates are reliable.

Original: Therefore, the choice of metric used for clustering becomes so important.

Original: 3. DTC O PTIMIZATION
Revised: 3. DTC O PTIMIZATION

Original: We perform batch wise joint optimization of clustering and autoencoder by minimizing KL divergence loss and mean squared error loss, respectively.
Revised: We perform batch-wise joint optimization of clustering and auto encoder by minimizing KL divergence loss and mean squared error loss, respectively.

Revised: This optimization problem is challenging, and an effective initialization of the cluster centroids is highly important.

Revised: Cluster centroids reﬂect the latent representation of the data.

Revised: To make sure initial centroids represent the data well, we ﬁrst pretrain parameters of the autoencoder to start with a meaningful latent representation.

Revised: After pretraining, the cluster centers are then initialized by hierarchical clustering with complete linkage on embedded features of all datapoints.

Original: Similar approaches have been used in Xie et al.
Revised: Later we update autoencoder weights and cluster centers using gradients dL c dz i and dL ae dz , see eqs. 7 and 8 below, using backpropagation mini-batch SGD. We also update target distribution p during every SGD update using eq. 5.

Revised: [Equation]

Revised: Similar approaches have been used in Xie et al.

Original: (2016) and Yang et al.
Revised: (2016) and Yang et al.

Original: This helps prevent any problematical solutions from drifting too far away from the original input signal.
Revised: This helps prevent any problematic solutions from drifting too far away from the original input signal.

Original: Because there is also an objective to reconstruct the original signal, the latent representation will converge at  suitable representation so as to minimize both the clustering loss and the dimensionality reduction loss.
Revised: As reconstruction of the original signal (reconstruction MSE reduction) is a part of the objective, the latent representation will converge at a suitable representation so as to minimize both the clustering loss and the MSE loss.

Original: 3. V ISUALIZING 
Revised: 3. V ISUALIZING H EAT M AP

Original: [Table] EAT

Original: [Table]

Original: AP

Original: [Table]

Original: One Event [Table] [Figure] In this work, we had a kin interest in being able to  localize the main  feature(s) that triggers the ﬁnal classiﬁcation.
Revised: In most applications it is important to identify and localize the main data feature(s) contributing to the ﬁnal classiﬁcation.

Original: For example, in the NASA data set 4.1, we had the knowledge that the bi-polar feature is the main trigger for a time series to be classiﬁed as an event 3.

Original: Adapting a similar heat map structure that has been used to localize tumors using image level labels (Hwang & Kim, 2016), we were able to localize bi-polar features successfully.
Revised: To do so, we create a heatmap-generating network following the approach used in (Hwang & Kim, 2016) to localize tumors in medical images using only image-level labels.

Original: The most important difference here is our network is fully unsupervised and for the training wemake useof the loss generated by temporal cluster layer as discussed in 3.2.3. In the visualization networkshown by the dashed lines in Fig.1, we feed the latent representation z i to the upsampler, followed by a 1D deconvolutional layer, a dense layer and the results ﬁnally added to  the temporal clustering output. During training, this layer is optimized along with the temporal clustering using the same loss as Eq.
Revised: Brieﬂy, we use the cluster labels generated by our DTC network to train a new supervised hierarchical convolutional network to classify the inputs x .

Original: It is worth reiterating that the loss used in (Hwang & Kim, 2016) is computed using a weighted addition of classiﬁcation and localization loss. Whereas inour work, we add the visualization and temporal clustering outputs to compute the loss L using Eq. 6, thereby integrating visualization to our unsupervised framework. Optimizing this structure by incorporating various merge techniques and hyper-parameters can improve the localization performance and that can be addressed in a separate work.
Revised: Activity in this new network allows us to generate heatmaps showing the parts of spatio-temporal inputs x that are most relevant for assigning inputs to clusters. Example of such heatmap for NASA data (see Section 4) is shown in Figure 3.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: DTC Experiments are performed using tensorﬂow 1 . 3 and keras 2 . 0 software on Nvidia GTX 1080Ti graphics processor.
Revised: Our networks were implemented and tested using Python, TensorFlow 1.3 and Keras 2.0 software on Nvidia GTX 1080Ti graphics processor.

Original: The Baseline algorithms are performed in R using publicly available TSclust Package (Montero et al., 2014).
Revised: For the baseline algorithms the TSclust package (Montero et al., 2014) R implementations were used except for k-Shape where Python implementation 1 was used. [Figure]

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: The performance of the proposed algorithm is evaluated on a wide range of real world datasets.
Revised: The performance of our DTC algorithm is evaluated on a wide range of real world datasets.

Revised: • We make use of a few publicly available UCR Time series Classiﬁcation Archive datasets (Chen et al., 2015).

Revised: Table 1 includes the properties of these datasets: the number of samples N , the time steps in each sequence L , and the class distribution ratio r .

Revised: Since we are using these datasets as unlabeled data, we combine the training and test datasets for all UCR datasets for all experiments in this study.

Original: • One dataset is from measurements by spacecraft magnetometer  from the recent NASA Magnetospheric Multiscale (MMS) Mission.
Revised: • In addition, we use spacecraft magnetometer data from the recent NASA Magnetospheric Multiscale (MMS) Mission.

Original: The magnetospheric plasma environment consists of many transient structures and waves.
Revised: The magnetospheric plasma environment exhibits many transient structures and waves.

Original: Here we are interested in automated detection of spacecraft crossings of the so-called ﬂux transfer events (FTEs) in an otherwise turbulent medium.
Revised: Here we are interested in automated detection of spacecraft crossings of the so-called ﬂux transfer events (FTEs) in an otherwise turbulent medium.

Original: FTEs (Russell & Elphic, 1979) are characterized by bipolar signature in the B N component of the magnetic ﬁeld.
Revised: FTEs (Russell & Elphic, 1979) are characterized by bipolar signature in the B N component of the magnetic ﬁeld.

Original: The dataset contains a total of 104 time series, with each sequence having 1440 time steps.
Revised: The B N data contains a total of 104 time series, with each sequence having 1440 time steps.

Original: • We also make use of a few publicly available UCR Time series Classiﬁcation Archive datasets (Chen et al., 2015).

Original: Table 1 includes the properties of these datasets where we describe three attributes for each dataset.

Original: This includes the number of samples N , the time steps in each sequence L , and the class distribution ratio r .

Original: Since we are using these datasets as unlabeled data, we combine the training and test datasets for all UCR datasets for all experiments in this study.

Original: 4. B ASELINE M ETHODS
Revised: 4. B ASELINE M ETHODS

Original: We compare the results of  themethod, hierarchical clustering with complete linkage, against our algorithm (DTC) for the following metrics:
Revised: We compare the results of our DTC algorithm against two clustering methods, the hierarchical clustering with complete linkage, and k-Shape (Paparrizos & Gravano, 2015).

Original: Complexity Invariant Distance(CID) 2.
Revised: k-Shape is the current state-of-the-art temporal clustering algorithm. It is a partitional clustering approach which preserves the shapes of time series and computes centroids under the scale and shift invariance.

Original:  Correlation based Similarity(COR) 3.
Revised: We have considered four similarity metrics in our experiments: 1) Complexity Invariant Distance(CID) , 2) Correlation based Similarity(COR) , 3) Auto Correlation based Similarity(ACF) , and 4) Euclidean Based Similarity(EUCL) . [Table] ProximalPhalanx

Original: Auto Correlation based Similarity(ACF) 4.
Revised: OutlineCorrect (891,80,2.12,4) 0. ShapeletSim (200,500,1.00,10) 0. SonyAIBO

Original: Euclidean Based Similarity(EUCL)
Revised: RobotSurfaceII (980,65,1.61,5) 0. SonyAIBO RobotSurface (621,70,0.78,5) 0. ItalyPower

Revised: Demand (1096,24,1,4) 0.

Revised: WormsTwoClass (258,900,1.37,10) 0.

Original: 4. E VALUATION M ETRICS
Revised: 4. E VALUATION M ETRICS

Original: Since we have  labels for the timeseries, we can compare the results of our clustering with the expert labels. This provides a gauge of accuracy of our unsupervised algorithm as a classiﬁer.
Revised: We have expert labels for our datasets. However, our entire training pipeline is unsupervised and we only used the labels to measure the performance of the models as a classiﬁer.

Original: To this end, we use the Receiver Operating Characteristics (ROC) (Fawcett, 2006) and area under the curve (AUC) to assess the efﬁcacy of our approach.
Revised: To this end, we use the Receiver Operating Characteristics (ROC) (Fawcett, 2006) and area under the curve (AUC) as our evaluation metrics.

Original: We emphasize that our entire training pipeline is unsupervised and we only use the labels to measure the performance of our algorithm on real world datasets.

Original: We used bootstrap sampling (Singh & Xie, 2008) and averaged the ROC curves overtrials.
Revised: We used bootstrap sampling (Singh & Xie, 2008) and averaged the ROC curves over 5 trials.

Original: 4.  P ARAMETER I NITIALIZATION
Revised: 4. I MPLEMENTATION AND P ARAMETER I NITIALIZATION

Revised: Parameter optimization using cross-validation is not feasible in unsupervised clustering.

Revised: Hence we use commonly used parameters for DTC and avoid dataset speciﬁc tuning as much as possible.

Revised: The convolution layer has 50 ﬁlters with kernel size 10 and two Bi-LSTM’s have ﬁlters 50 and 1 respectively.

Revised: The pooling size P is chosen such that it makes the latent representation size < 100 for faster experimentation.

Revised: P for each experiment is listed in the ﬁrst column in Table 1.

Original:  Autoencoder network is initialized with 10 epochs and temporal clustering layer centroids are initialized using hierarchical clustering with complete linkage and using the respective metrics obtained from the TSclust package.
Revised: The deconvolutional layer has kernel size 10. We initialize all weights to a zero-mean Gaussian distribution with a standard deviation of 0.01. Autoencoder network is pre-trained, using the Adam optimizer, over 10 epochs. Temporal clustering layer centroids are initialized using hierarchical clustering with complete linkage and using the chosen metric.

Original: We use the Adam optimizer for our network training.
Revised: The entire deep architecture is jointly trained for clustering and auto encoder loss until convergence criterion of 0.1% change in cluster assignment is met. The mini-batch size is set to 64 for both pretraining and end-to-end ﬁne tuning of the network and the starting learning rate is set to 0.1. These parameters are the same for all experiments and are held constant across all datasets. The baseline algorithms we use are parameter free.

Original: 4. Q UALITATIVE A NALYSIS
Revised: 4. Q UALITATIVE A NALYSIS

Original: In Fig.
Revised: We show in Fig.

Original: 3, each column shows the results obtained in various stages after feeding the input signal through the network.
Revised: 3 results of DTC for three distinct time series from the MMS dataset.

Original: The ﬁrst row represents the input signal, the second row represents the autoencoder output. As illustrated in Fig. 3 the autoencoder gets rid of the chronic noise present in the  input signal and smooths the signal.
Revised: The top panels shows the three input time series.

Original: The last row represents the activation map  generated by the algorithm (green and red represent an event prediction and non-event prediction, respectively). As shown in the Fig.
Revised: The bottom panels show the activation map as generated by the algorithm. Activation map proﬁles correlate well with the location of the bipolar signatures of the events.

Original: 3, the algorithm was able to capture the bipolar signal in column 1 and column 2. Further, column 2 represents a multi-event input, in which our algorithm was able to capture both events.
Revised: Events are highlighted by the dashed ovals for readers’ convenience. Note that the second time series has two events and the heatmap identiﬁes both of the events correctly.

Original: The column 3 is a non-event and is correctly identiﬁed as such by our algorithm (no green output).
Revised: The third time series (right panels) is a non-event and the algorithm correctly identiﬁes it as such (heatmap activation is low). [Figure] i v [Figure]

Original: 4. Q UANTITATIVE A NALYSIS
Revised: 4. Q UANTITATIVE A NALYSIS

Original: We have chosen 4 different metrics over a variety of13 datasets. The Fig. 4 illustrates the comparison of clustering performance on 4 different datasets over 4 different metrics. We can see thatour algorithm was able to improve the baseline performance in all of the datasets over all metrics.
Revised: One of the main points of the paper is that the joint training of the two objectives, reconstruction loss and clustering loss, yields superior performance compared to the disjointed training where each loss is optimized independently.

Original: More detailed comparison is shown in table 1.
Revised: Direct comparison between joint end-to-end training of the DTC vs disjoint DTC training (training the temporal autoencoder to minimize reconstruction loss and then training the clustering to minimize clustering loss) on the MMS dataset, all other network parameters being identical, results in the average AUC of 0.93 for joint training vs. average AUC of 0.88 for disjointed training.

Original: Our algorithm shows to be robust and obtained improved results across timeseries with different dataset sizes N , different lengths L , as well as a range of data imbalances denoted by r . [Figure] T r u e [Figure]
Revised: Next, we include in Table 1 a detail comparison of results from DTC and the two baseline clustering techniques for 4 different similarity metrics and over 13 different datasets.

Original: o

Original: [Figure] i t i v e R a t e [Figure] [Table]
Revised: We can see that our algorithm was able to improve the baseline performance in all of the datasets over all metrics.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: SonyAIBO

Original: RobotSurfaceII (

Original: [Table] [Table] [Table] [Table]
Revised: Although k-Shape performed well on a few datasets, chosen the right metric DTC outperforms k-Shape for all the datasets considered.

Original: SonyAIBO RobotSurface ( [Table] [Table]
Revised: We also show the comparison of ROCs in Fig. 4.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: ItalyPowerDemand (

Original: [Table] [Table] [Table] WormsTwoClass
Revised: In order to keep the ﬁgure readable, we only show the ROCs for 4 datasets over 4 similarity metrics.

Original: [Table]

Original: D ISCUSSION

Original: Our work can be extended in several directions. First, not every temporal metric can be efﬁciently computed in the deep learning pipeline. For instance, metrics like Dynamic Time Warping whose computation time is O ( N2 ) , will be expensive to integrate in our temporal clustering layer. Effective implementation strategies can be explored.
Revised: These results illustrate DTC to be robust and provide superior performance over existing techniques across datasets from different domains, with different dataset sizes N , different lengths L , as well as a range of data imbalances denoted by r .

Original: The proposed algorithm is sensitive to initialization.

Original: If the baseline metric performs poorly, the improvement using DTC may not be high.

Original: Since we maximize the high conﬁdence assignments, we have to start with at least a few high conﬁdence assignments.

Original: Finally, the aim of this study was to propose a general network for fully unsupervised learning of time-domain features and the associated clustering.

Original: It may be possible to modify the network or the optimization of the temporal autoencoder component and the visualization component, depending on the size and properties of the dataset.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we addressed the question of unsupervised learning of patterns in temporal sequences, unsupervised event detection and clustering.
Revised: In this work we addressed the question of unsupervised learning of patterns in temporal sequences, unsupervised event detection and clustering.

Original: Post-hoc labeling of the clusters, and comparison to ground-truth labels not used in training, reveals high degree of agreement between our unsupervised clustering results and human-labeled categories, on several types of datasets.
Revised: Post-hoc labeling of the clusters, and comparison to ground-truth labels not used in training, reveals high degree of agreement between our unsupervised clustering results and human-labeled categories, on several types of datasets.

Original: This indicates graceful dimensionality reduction from inputs with extensive and complex temporal structure to one or fewdimensional space spanned by the cluster centroids.
Revised: This indicates graceful dimensionality reduction from inputs with extensive and complex temporal structure to one or fewdimensional space spanned by the cluster centroids.

Original: As most natural stimuli are time-continuous and unlabeled, the approach promises to be of great utility in real-world applications.
Revised: As most natural stimuli are time-continuous and unlabeled, the approach promises to be of great utility in real-world applications.

Original: Generalization to multichannel spatio-temporal input is straightforward and has been carried out as well; it will be described in  a separate paper.
Revised: Generalization to multichannel spatio-temporal input is straightforward and has been carried out as well; it will be described in more detail in a separate paper.