Paper:
Computing distances between examples is at the core of many learning algorithms for time series. Consequently, a great deal of work has gone into designing effective time series distance measures. We present Jiffy, a simple and scalable distance metric for multivariate time series. Our approach is to reframe the task as a representation learning problem-rather than design an elaborate distance function, we use a CNN to learn an embedding such that the Euclidean distance is effective. By aggressively max-pooling and downsampling, we are able to construct this embedding using a highly compact neural network. Experiments on a diverse set of multivariate time series datasets show that our approach consistently outperforms existing methods.

Review:
This paper presents a solid empirical analysis of a simple idea for learning embeddings of time series: training a convolutional network with a custom pooling layer that generates a fixed size representation to classify time series, then use the fixed size representation for other tasks. The primary innovation is a custom pooling operation that looks at a fraction of a sequence, rather than a fixed window. The experiments are fairly thorough (albeit with some sizable gaps) and show that the proposed approach outperforms DTW, as well as embeddings learned using Siamese networks. On the whole, I like the line of inquiry and the elegant simplicity of the proposed approach, but the paper has some flaws (and there are some gaps in both motivation and the experiments) that led me to assign a lower score. I encourage the authors to address these flaws as much as possible during the review period. If they succeed in doing so, I am willing to raise my score.
QUALITY
I appreciate this line of research in general, but there are some flaws in its motivation and in the design of the experiments. Below I list strengths (+) and weaknesses (-):
+ Time series representation learning is an important problem with a large number of real world applications. Existing solutions are often computationally expensive and complex and fail to generalize to new problems (particularly with irregular sampling, missing values, heterogeneous data types, etc.). The proposed approach is conceptually simple and easy to implement, faster to train than alternative metric learning approaches, and learns representations that admit fast comparisons, e.g., Euclidean distance.
+ The experiments are pretty thorough (albeit with some noteworthy gaps) -- they use multiple benchmark data sets and compare against strong baselines, both traditional (DTW) and deep learning (Siamese networks).
+ The proposed approach performs best on average!
- The custom pooling layer is the most interesting part and warrants additional discussion. In particular, the "naive" approach would be to use global pooling over the full sequence [4]. The authors should advance an argument to motivate %-length pooling and perhaps add a global pooling baseline to the experiments.
- Likewise, the authors need to fully justify the use of channel-wise (vs. multi-channel) convolutions and perhaps include a multi-channel convolution baseline.
- There is something incoherent about training a convolutional network to classify time series, then discarding the classification layer and using the internal representation as input to a 1NN classifier. While this yields an apples-to-apples comparison in the experiments, I am skeptical anyone would do this in practice. Why not simply use the classifier (I am dubious the 1NN would outperform it)? To address this, I recommend the authors do two things: (1) report the accuracy of the learned classifier; (2) discuss the dynamic above -- either admit to the reader that this is a contrived comparison OR provide a convincing argument that someone might use embeddings + KNN classifier instead of the learned classifier. If embeddings + KNN outperforms the learned classifier, that would surprise me, so that would warrant some discussion.
- On a related note, are the learned representations useful for tasks other than the original classification task? This would strengthen the value proposition of this approach. If, however, the learned representations are "overfit" to the classification task (I suspect they are), and if the learned classifier outperforms embeddings + 1NN, then what would I use these representations for?
- I am modestly surprised that this approach outperformed Siamese networks. The authors should report the Siamese architectures -- and how hyperparameters were tuned on all neural nets -- to help convince the reader that the comparison is fair.
- To that end, did the Siamese convolutional network use the same base architecture as the proposed classification network (some convolutions, custom pooling, etc.)? If not, then that experiment should be run to help determine the relative contributions of the custom pooling layer and the loss function.
- Same notes above re: triplet network -- the authors should report results in Table 2 and disclose architecture details.
- A stronger baseline would be a center loss [1] network (which often outperforms triplets).
- The authors might consider adding at least one standard unsupervised baseline, e.g., a sequence-to-sequence autoencoder [2,3].
CLARITY
The paper is clearly written for the most part, but there is room for improvement:
- The %-length pooling requires a more detailed explanation, particularly of its motivation. There appears to be a connection to other time series representations that downsample while preserving shape information -- the authors could explore this. Also, they should add a figure with a visual illustration of how it works (and maybe how it differs from global pooling), perhaps using a contrived example.
- How was the %-length pooling implemented? Most deep learning frameworks only provide pooling layers with fixed length windows, though I suspect it is probably straightforward to implement variable-width pooling layers in an imperative framework like PyTorch.
- Figure 1 is not well executed and probably unnecessary. The solid colored volumes do not convey useful information about the structure of the time series or the neural net layers, filters, etc. Apart from the custom pooling layer, the architecture is common and well understood by the community -- thus, the figure can probably be removed.
- The paper needs to fully describe neural net architectures and how hyperparameters were tuned.
ORIGINALITY
The paper scores low on originality. As the authors themselves point out, time series metric learning -- even using deep learning -- is an active area of research. The proposed approach is refreshing in its simplicity (rather than adding additional complexity on top of existing approaches), but it is straightforward -- and I suspect it has been used previously by others in practice, even if it has not been formally studied. Likewise, the proposed %-length pooling is uncommon, but it is not novel per se (dynamic pooling has been used in NLP [5]). Channel-wise convolutional networks have been used for time series classification previously [6].
SIGNIFICANCE
Although I identified several flaws in the paper's motivation and experimental setup, I think it has some very useful findings, at least for machine learning practitioners. Within NLP, there appears to be gradual shift toward using convolutional, instead of recurrent, architectures. I wonder if papers like this one will contribute toward a similar shift in time series analysis. Convolutional architectures are typically much easier and faster to train than RNNs, and the main motivation for RNNs is their ability to deal with variable length sequences. Convolutional architectures that can effectively deal with variable length sequences, as the proposed one appears to do, would be a welcome innovation.
REFERENCES
[1] Wen, et al. A Discriminative Feature Learning Approach for Deep Face Recognition. ECCV 2016.
[2] Fabius and van Amersfoort. Variational Recurrent Auto-Encoders. ICLR 2015 Workshop Track.
[3] Tikhonov and Yamshchikov. Music generation with variational recurrent autoencoder supported by history. arXiv.
[4] Hertel, Phan, and Mertins. Classifying Variable-Length Audio Files with All-Convolutional Networks and Masked Global Pooling. 
[5] Kalchbrenner, Grefenstette, and Blunsom. A Convolutional Neural Network for Modelling Sentences. ACL 2014.
[6] Razavian and Sontag. Temporal Convolutional Neural Networks for Diagnosis from Lab Tests. arXiv.