Paper:
Multiple Instance Learning for Efficient Sequential Data Classification on Resource-constrained Devices
We study the problem of fast and efficient classification of sequential data (such as time-series) on tiny devices, which is critical for various IoT related applications like audio keyword detection or gesture detection. Such tasks are cast as a standard classification task by sliding windows over the data stream to construct data points. Deploying such classification modules on tiny devices is challenging as predictions over sliding windows of data need to be invoked continuously at a high frequency. Each such predictor instance in itself is expensive as it evaluates large models over long windows of data. In this paper, we address this challenge by exploiting the following two observations about classification tasks arising in typical IoT related applications: (a) the "signature" of a particular class (e.g. an audio keyword) typically occupies a small fraction of the overall data, and (b) class signatures tend to be discernible early on in the data. We propose a method, EMI-RNN, that exploits these observations by using a multiple instance learning formulation along with an early prediction technique to learn a model that achieves better accuracy compared to baseline models, while simultaneously reducing computation by a large fraction. For instance, on a gesture detection benchmark [26], EMI-RNN requires 72x less computation than standard LSTM while improving accuracy by 1%. This enables us to deploy such models for continuous real-time prediction on devices as small as a Raspberry Pi0 and Arduino variants, a task that the baseline LSTM could not achieve. Finally, we also provide an analysis of our multiple instance learning algorithm in a simple setting and show that the proposed algorithm converges to the global optima at a linear rate, one of the first such result in this domain. The code for EMI-RNN is available at [14].

Review:
The authors propose a method for fast and efficient classification of sequential data, motivated by the deployment of such methods on embedded or resource-constrained devices. The method has two key components: (1) it is formulated as a multiple instance learning problem, where each time series is broken down into shorter ones and labels are generated automatically for them, and (2) an early classification formulation that allows the network to emit outputs before processing the whole input sequence. The model is evaluated on five datasets; when compared to the standard LSTM, the proposed solution achieves similar accuracy (sometimes better, sometimes worse) while providing important computational savings.   PROS - Efficient model for sequence classification is specific tasks, like wake word detection in voice-driven assistans (e.g. Google Home or Alexa). Speed and low-consumption ML models are key to the development of such systems. - Results on the evaluated datasets are good, as the model can provide computational savings while matching or even improving the accuracy of the baseline LSTM. - The authors show the advantages of the method in resource-constrained devices, namely Raspberry Pi0 and 3.  CONS - The model looks somewhat niche, as it seems very tailored for the tasks in the paper. Nevertheless, these tasks are important and interesting to the community. - Although the text is generally well written and motivated, some parts are hard to follow and lack some detail. The manuscript needs a final discussion/conclusion section as well. Please see more extensive comments below.   TL;DR: The paper addresses an interesting task and provides motivation for most choices. Some parts of the experimental setup need to be clarified so that readers can understand the contributions more easily and reproduce the results.    COMMENTS  - The model introduces some new hyperparameters. How sensitive is it to the particular choice of these parameters? Please comment on this, as it is very helpful for practitioners trying to replicate/extend this work. - Equation 3.1.1 is very hard to follow, as most variables have not been defined in the text (e.g. s_i or n). Please clarify this part of the manuscript, it is one of the main contributions of the paper. - What is “train-LSTM”? One SGD step, or a full training? If it is a full training, how many times is this process repeated before convergence?  - The description of the evaluation protocol could be more thorough. Why can MI-RNN provide savings when it needs to evaluate some tokens in the time series more than once when the windows overlap, as shown in Figure 1?  - There is no description about how the method was implemented in Raspberry Pi. Did the authors use some particular framework? - Please cite the following related paper: Ma et al., “Learning Activity Progression in LSTMs for Activity Detection and Early Detection”, CVPR 2016 - The format of the references section is not consistent. Please select between full conference name or acronym (e.g. [2] or [3]), follow a common format for arxiv pre-prints (e.g. [10] and [31]). Some pre-prints have already been published (e.g. [7]). - Do not attach the main paper with the supplementary material. - There are some typos in the text, please proof-read:    11: tend to (be) discernible    33: predict (the) label    58: We [we]    69: it’s -> its    79: missing comma before “etc”    88: please do not use “etc” and list all the tasks    118: missing whitespace    124-125: use consistent format (with or without comma) for x_{i,T}    144: “models process”    199: are there the same number of positives and negatives (n) in Z, or is this a typo?    213: “noise the bags”?    Figure 4: this is a table and should be labeled accordingly   ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------  The results in the paper are good, but I had concerns regarding clarity and reproducibility. The author feedback helped with this, and I believe that the paper will be better once the authors introduce these clarifications in the manuscript. They will also release code so that other can easily reproduce their experiments. After reading the author feedback, I would lean towards acceptance. I increased the overall and reproducibility scores accordingly.