S KIP W: R ESOURCE ADAPTABLE RNN WITH STRICT UPPER COMPUTATIONAL LIMIT
Anonymous authors
Paper under double-blind review
A
BSTRACT
We introduce Skip-Window, a method to allow recurrent neural networks (RNNs) to trade off accuracy for computational cost during the analysis of a sequence.
Similarly to existing approaches, Skip-Window extends existing RNN cells by adding a mechanism to encourage the model to process fewer inputs.
Unlike existing approaches, Skip-Window is able to respect a strict computational budget, making this model more suitable for limited hardware like edge devices.
We evaluate this approach on two datasets: a human activity recognition task and adding task.
Our results show that Skip-Window is able to exceed the accuracy of existing approaches for a lower computational cost while strictly limiting said cost.
I NTRODUCTION
Since Recurrent Neural Networks (RNN) have been introduced Williams et al.
(1986), they have become one of the reference methods to process sequences.
A typical architecture is the Long-ShortTerm-Memory neural network (LSTM) which allowed improvement in natural language processing such as large-vocabulary speech recognition (Sak et al., 2014; Li & Wu, 2014).
Used with CNNs they have also reached state of the art in automatic image captioning (Vinyals et al., 2014).
Deep learning models are now brought closer to the user rather than running in a distant cloud, helping to reduce latency, network congestion, and improving data security and privacy.
However, smartphones and user devices impose additional constraints such as limited computation or energy.
Handling these constraints has become an active research topic (Zhang et al., 2017; 2018;
Howard et al., 2019; Wu et al., 2019; Cai et al., 2020).
User devices can also host multiple processes running at the same time and starting or stopping abruptly, modifying the constraints affecting the processes.
Few works have considered models that can be modiﬁed at run time to adapt to an evolving computational limit (Yu et al., 2019; Yu & Huang, 2019;
Guerra et al., 2020; Jin et al., 2020).
However, none of these focus on sequences and therefore none address the problem of adapting the model in the middle of a sequence.
In this context, this paper introduces Skip-Window (SkipW), a ﬂexible recurrent neural network architecture: its computational cost can be dynamically adapted during a sequence analysis to meet real time constraints changes.
The proposed architecture can be combined with any RNN cell and allows to strictly limit the computational resources used to avoid exceeding a given budget.
Furthermore, empirical experiments on two data sets (Adding Task and HAR-2D-POSE) demonstrate that
Skip-Window is able to exceed the accuracy of existing approaches for a lower computational cost.
In addition, measurements on speciﬁc processors highlight that SkipW produces computational and energy savings.
R ELATED WORK
Typically, RNNs maintain a “state”, a vector of variables, over time.
This state is supposed to accumulate relevant information and is updated recursively.
Each input of the sequence is typically a) processed by some deep layers and b) then combined with the previous state through some other deep layers to compute the new state.
Hence, the RNN can be seen as a function taking a sequence of inputs x = ( x 1 , ..., x T ) and recursively computing a set of states s =
( s 1 , ..., s T ) .
Each state s t is computed from s t − 1 and x t by a cell S of the RNN.
As neural networks are increasingly run on limited hardware, recent research has focused on controlling their computational cost.
2. F LEXIBLE N EURAL N ETWORKS
A few architectures have recently been designed to adapt the computational complexity of a Deep
Neural Network (DNN) without reloading the whole model.
This can be achieved by removing/adding neurons (Yu et al., 2019; Yu & Huang, 2019) or by modifying the quantization of the weights (Guerra et al., 2020; Jin et al., 2020).
An efﬁcient embedding of a mixture of Convolutional Neural Network (CNNs) also allows to add or remove several models at the same time, hence changing the computational cost (Ruiz & Verbeek, 2019).
2.1. T HR R NN
For RNNs speciﬁcally, ThrRNN (Lambert et al., 2020) aims to control computation time by not processing some inputs.
This is controlled by an update gate u t .
The tradeoff between the average accuracy and the average number of updates can be modiﬁed during inference by changing a single parameter thr .
ThrRNN can wrap any RNN cell S :
[Equation]
[Equation] [Equation]
[Equation]
When an input is processed, an update gate computes the quantity ∆˜ u t that determines how many inputs will be skipped.
In practice the ∆˜ u t are accumulated in ˜ u t until ˜ u t ≥ thr .
2. R ECURRENT N EURAL N ETWORK WITH L OW C OMPUTATIONAL C OMPLEXITY
Several architectures have been proposed to limit or reduce the computational cost of RNNs, but this cost cannot be adapted at inference.
A ﬁrst class of architectures dynamically reduces computation based on the input.
SkipRNN (Campos et al., 2017) predates and is similar to ThrRNN, except that the binarization function does not change.
A similar mechanism has been proposed by Zhang et al.
Other architectures directly select the next input to process (Yeung et al., 2016; Yu et al., 2017; Hansen et al., 2019; Song et al., 2018).
Early exit has also been investigated by Dennis et al.
Tao et al.
(2019) also use x t as input to an update gate.
So do Seo et al.
; Jernite et al.
(2017); Li et al.
However, they do not skip any input but perform partial state updates.
A second class of architectures focuses on reducing the overal cost of the RNN.
FastRNN is an
RNN augmented with a residual connection with two extra scalar parameters and FastGRNN is an improved FastRNN: the residual connection is extended to a gate and RNN matrices are low rank, sparse and quantized (Kusupati et al., 2018).
Other architectures reduce the RNN length.
Chan et al.
(2016) train an encoder to reduce the input length.
Yeung et al.
; Shan et al.
; Chen et al.
(2018) propose various mechanisms to summarize subsequences of windows of inputs.
2. R ECURRENT N EURAL N ETWORK WITH H IERARCHICAL -D EPENDENT C OMPLEXITY
A class of architectures focuses on hierarchy level concept to reduce the complexity.
These methods are mainly used in the context of multi-layer RNNs where each layer is supposed to model a different level in the hierarchy (e.g. for a corpus the levels could be documents, paragraphs, sentences, words, letters).
These approaches are based on the fact that a hierarchical separation exists within a sequence of inputs, which might not always be the case.
In (Koutn´ık et al., 2014), the hidden state is partitioned into different modules, each module has its own clock period, meaning that they will be updated at different times.
Skipping updates of part of the hidden state decreases the computational cost.
In (Koutn´ık et al., 2014), the update periods are chosen arbitrarily, for example using an exponential series.
For stacked-RNNs, Chung et al.
; Chang et al.
(2017) conditionally update each layer based on a feature level criterion, or by dilating a skip connection.
Layers close to the inputs would model lower feature levels and be updated more frequently.
Further layers would model higher level features.
In (Chung et al., 2016), a layer modeling sentences would be updated only when a word is processed entirely (in a model fed character by character), from the layer modeling words.
Before the end of a word is reached, the state of the former layer is copied across input steps.
2. R ELATIONSHIP TO OUR WORK
ThrRNN is the closest model to SkipW. Both are ﬂexible RNNs and skip some inputs.
However,
ThrRNN optimizes computational cost on average over sequences.
This induces two variabilities: a) inter-sequence variability: the model will not use the same number of updates for every sequence; and b) intra-sequence variability: the number of updates will not be uniform across time steps, updates may be concentrated in a certain part of the sequence.
These two variabilities can cause the model to exceed its computational budget and, therefore, to either shut down or delay the output.
SkipW does not have this problem as it strictly enforces a computational constraint over each window of inputs.
Other strategies for ﬂexible models are not straightforward to apply to RNN.
They require specialized training algorithms.
They have never been applied to models processing inputs of an RNN or to make an RNN ﬂexible and it is not clear how they would need to be modiﬁed.
Furthermore, these models adapt between independent inputs whereas, for sequences, adaptation is necessary between time steps.
RNN architectures with low complexity are orthogonal to our approach.
They do not offer ﬂexibility.
They could be combined with and beneﬁt from our approach.
However, SkipRNN (which we are based on) and related methods have one big advantage over others: by skipping inputs, they also skip any modiﬁcation of an input, such as processing by an expensive CNN for images.
As SkipW makes decision over a window of inputs, it has some superﬁcial similarity to methods summarizing windows or hierarchical RNNs.
However, SkipW a) does not summarize windows and b) does not even look at these inputs before deciding what to skip.
S KIP -W INDOWS ARCHITECTURE
Skip-Windows (SkipW) is a wrapper for a RNN cell S .
It uses a conditional computation mechanism to skip some updates.
Rather than at each input x t , update gates are computed at the beginning of windows of inputs, that is, every L time steps (Figure 1).
In other words, before any new L-size window of inputs, a L-size vector ˜ u W is computed.
˜ u W [ i ] can be seen as the importance of input i in the window.
Then, the architecture includes a select K mechanism.
This function takes as input the vector ˜ u W and outputs the vector ˜ u KW , setting L − K elements to a value that ensures the associated inputs are not processed (0 in Figure 2).
Therefore, it ensures that at most K out of every L inputs will be processed.
In other words, it forces the RNN cell to skip ( L − K ) out of every L inputs.
This ensures a strict upper bound on the computational cost of the model for a sequence and for each window, therefore alleviating the inter-sequence variability and intra-sequence variability issue.
Similarly to other works, the binary state update L-size vector, u W , is then obtained by binarizing the remaining values as in equation 1.
For example, by setting all values below a threshold to a value that ensures the associated inputs are not processed (0 in Figure 2).
An example of the Skip-Window cell implementation is represented in Figure 2.
In this case, select K is implemented as a top K function.
This enforces the strict constraint on the number of updates.
The top K operation keeps unchanged the K highest value in ˜ u W,t , and sets the ( L − K ) others to 0.
The corresponding architecture can be characterized as follows:
[Equation]
where W w is a weight matrix of size ( N + 1) × L , N is the number of hidden units as deﬁned by the RNN cell S,
b W is a L-vector bias, σ is the sigmoid function and mod is the modulo operation.
Instead of a top K function for select K , it is also possible to use a stochastic sampling mechanism that randomly selects (without replacement)
K out of L elements of ˜ u W where the probability of selecting each element of index i is proportional to ˜ u W [ i ] .
Including the time step t in equation 6 is also optional and can be replaced by a value ensuring the state is not static if no update is made in a window.
For example, the number of inputs since the last update or the number of windows already processed.
Training the model The model is trained to minimize a two-part loss, similarly to Campos et al.
The ﬁrst term measures the accuracy of the task, and the second one penalizes inputs used:
[Equation]
where λ is the cost associated to the use of a single input.
Error gradients The model is differentiable except for the f binarize and top K function.
To train the model using standard backpropagation the straight-through estimator is used as done in Campos et al.
(2017) for f binarize .
Other alternatives might involve reinforcement learning such as REINFORCE (Williams, 1992) or, in the case of the top K function, the usage of a differentiable top K as proposed in Xie et al.
[Figure]
K=
[Figure] [Figure]
K
[Figure]
K=[
[Figure]
16] thr=
[Figure]
trade
[Figure]
off SkipRNN
SkipW =1e
[Figure]
L
[Figure]
SkipW =1e
[Figure]
L
[Figure]
ThrRNN
[Figure]
Early experiments using a differentiable top K (Xie et al., 2020) have shown worse results than the straight-through estimator.
This suggests that constraining computation may be an interesting approximation for a top K operation.
Adapting computational cost at inference During inference, adapting the tradeoff between model performance and computational cost can be done using two factors : the K in equation 9 and the thr in equation 10.
These two parameters can be modiﬁed together or one at a time.
Increasing/lowering thr parameter in [0 , 1] encourages the model to process fewer/more inputs.
Changing
K in { 0 . . .
L } forces the model to process at most K/L of the window.
Choice of the window size hyper-parameter By the nature of the model, the task can inﬂuence the choice of L .
It can be hand tuned or computed using typical hyper-parameter search methods such as grid search.
Choosing small L allows the model to make update decisions for the near future only but offers less choice in operating points.
Similarly a bigger L requires the model to predict its update decisions for a bigger time span but offers more ﬂexibility.
At the extreme when L = 1 , each window consists of a single input.
E XPERIMENTS
We empirically show that SkipW a) learns accuracy / computational tradeoffs on par with or, surprisingly, sometimes better than our baselines SkipRNN and ThrRNN; b) upper bounds computational cost; c) enables changing the tradeoff in the middle of a sequence.
To evaluate the computational beneﬁt induced by skipping part of the input sequence we use the number of inputs processed by the model, as this measure is independent from the hardware and the framework used to implement the models.
In addition, we also show how a reduction of the number of inputs processed translate into computational savings on tiny devices.
We evaluate our approach on two data sets.
Human Activity Recognition (HAR) (Oﬂi et al., 2013) consists of 2D-position sequences of 18 joints across a 32-frames videos recorded at 22Hz ( ≈ 1. seconds).
Each sequence is labeled by one of the 6 actions performed by 12 subjects ﬁlmed from 4 angles.
Adding Task Hochreiter & Schmidhuber (1997), consists of tuples-sequences (value, marker) ∈ R × { 0 , 1 } , where the expected output is (cid:80) i marker × value .
For HAR, two stacked
RNNs are used, but only one for Adding Task.
More details can be found in Appendix A.
4. A CCURACY AND COMPUTATIONAL TRADEOFFS
In Figures 3 and 4, the baselines SkipRNN and ThrRNN are compared to SkipW with various values for L and K .
Small L values may not offer good tradeoffs when varying K .
See for example L =in Figure 3.
That’s not surprising: often the number of inputs processed can be signiﬁcantly reduced without affecting accuracy.
For example, SkipRNN on HAR can process as few as 15% inputs without affecting accuracy.
Using L = 4 and K = 1 limits complexity at 20% only.
So, for such a
[Figure]
trade
[Figure]
off SkipRNN
Random sampling limit
ThrRNN
[Figure]
SkipW
[Figure]
2e
[Figure]
[Figure]
small L , varying K has almost no impact on average.
Larger window sizes, on the other hand, offer more tradeoffs when changing K .
For example, on HAR, for L ∈ { 8 , 16 } , K can take more values, and for small K computational gains are traded off for accuracy.
For example, for L = 16 , reducing
K from 4 to 1 more than halves the number of inputs processed for a drop of 3.6% in accuracy.
A similar behavior can be observed on Adding Task.
SkipW enforces a K/L upper bound on the number of inputs processed in each window.
Even when the average number of updates is not impacted much by a change in K , individual sequences can be impacted.
Figure 5 illustrates this impact with a violin plot of the number of inputs processed per sequence for various K and L = 8 on HAR.
Decreasing K below 3 impacts the maximum observed number of updates over the whole sequence.
However, even changes above K = 3 have an impact.
Even though the observed maximum number of updates does not change, the violin changes as some sequences have fewer inputs processed.
This is due to the fact that the computational constraint is enforced over each window.
So, for K ≤ 5 , some sequences are prevented from exceeding the computational budget in some windows of inputs.
It is perhaps surprising that SkipW achieves both a higher accuracy and lower computational cost than SkipRNN and ThrRNN, as the former decides on the next K inputs to process at once whereas the baselines decide on the next input only and so decide more often and with more information.
We have observed that the baselines tend to process one input regularly, even if there is no information in the signal.
This increases the computational cost without beneﬁting accuracy.
We believe this is due to the increment mechanism (Equation 3).
To skip C inputs, ∆˜ u t (Equation 2) must be smaller than 0 . 5 /C .
We conjecture that at some point, for large C , the model is unable to differentiate between these small values and so no more than C skips are performed. [Figure]
SkipW is able to learn where relevant inputs are and tends not to use irrelevant inputs.
This is best illustrated on AddingTask.
Let m 1 be the index of the ﬁrst marker, and m 2 the index of the second one.
The inputs can be divided into four parts p i , i ∈ { 1 , 2 , 3 , 4 } :
[Equation]
Ideally the model should skip all inputs in p 2 and p 4 where there is no marker, while using all the inputs in p 1 and in p 3 where a marker can appear.
Figure 6 illustrates that SkipW skips inputs in different parts of the sequence for various K .
For
K =
L (training point), almost no inputs are skipped in p 1 and p 3 , where the markers are, but some are skipped in p 2 and p 4 , where there is no marker.
Not all inputs are skipped in p 2 and p 4 .
That’s not surprising: decisions to skip are taken at the start of a window, so even if a marker is present in the ﬁrst input of the window, SkipW will not update its decision to process other inputs in the window.
Values of K smaller than but close to L force the model to skip some inputs.
Skips are still mainly made in part p 2 and p 4 for these K values but start increasing in p 3 as the model can no longer process all inputs.
Finally, when K < 5 , SkipW can no longer process the ﬁrst 5 inputs and skips in p 1 start happening.
Note that both the binarization function and the inclusion of t in the inputs of the update gate are necessary for this behavior.
Please see the ablation study in Appendix
C.4.1 for more information.
Modifying thr impacts the accuracy / processing tradeoff as well.
Increasing thr gradually reduces both accuracy and average computational cost.
This is similar to ThrRNN.
Changing thr does not affect the upper bound on computational cost but can provide more accuracy / processing tradeoff points.
Some examples are provided in Appendix B.
4. M ODIFYING THE TRADEOFF DURING INFERENCE
Modifying ( thr, K ) is also possible during the analysis of a sequence.
Results of a change between two different pairs of ( thr, K ) values are shown in Figure 7.
The average accuracy and computational cost of the analysis of the sequences in the test set are reported as a function of the position of the change.
The position at the bottom left (resp.
top right) of the graph corresponds to the operating point of tradeoff 2 (resp.
trade˙off 1 ) when it processes the whole sequence.
The accuracy typically remains between the accuracy of the individual models and increases almost monotonically from the values of tradeoff 2 to the values of tradeoff 1 .
This suggests that SkipW can switch between operating points during the analysis of a sequence.
This behavior was observed on all data sets, for various training runs and ( thr, K ) conﬁgurations.
More details are provided in the supplementary material (C and D).
4. R EAL TIME EXPERIMENTS
We evaluate the performance of SkipW on small hardware to showcase its performance in the setting it is designed for.
We implement the full service, from images to activity recognition, on a Nvidia
[Figure]
I n p u t s p r o c e ss e d ( % )
[Figure]
Jetson Nano platform.
In the HAR-2D-POSE data set, OpenPose is used to compute poses on images, but the embedded GPU is not powerful enough to run this OpenPose.
Instead, we choose to use PoseNet (MobileNetV1 architecture with a 0.75 multiplier) model which is much more efﬁcient to run on such a small platform.
In our implementation, PoseNet model running on the embedded
GPU takes roughly 120 ms to compute a pose on a 640x480 pixel image, while the SkipW model takes around 30 ms to perform all state updates for a sequence.
Figure 8 presents the individual inference time of 1000 HAR image sequences processed by SkipW coupled with the PoseNet detection model on a Jetson Nano platform.
As expected, when K decreases, the run time decreases sharply.
This gain is largely due to the lower number of pose detections that need to be computed.
By drastically limiting the number of images used as input, SkipW saves a large amount of time, computation and energy.
Results on Jetson TX2 and Raspberry Pi lead to similar conclusions (Appendix E).
C ONCLUSION
We presented Skip-Window, a novel RNN cell architecture that dynamically controls the number of inputs processed during the analysis of a sequence.
SkipW allows to adapt the computational cost during and between sequence analysis without reloading any weight.
Unlike existing ﬂexible RNNs,
SkipW can enforce a strict upper bound on the number of updates performed over a time interval.
Therefore, SkipW can operate without exceeding a given computational budget.
Furthermore, our results show that Skip-Window is able to exceed the accuracy of existing ﬂexible RNNs for a lower computational cost.
The strict limit over computational resources across a time interval makes SkipW well suited for shared hardware with real time constraints, as we have shown that SkipW indeed reduces computation on such hardware.
A E XPERIMENTAL SETTINGS DETAILS
We evaluate our approaches on two data sets:
• Human Activity Recognition (HAR): HAR is an action recognition dataset consisting of 2D pose estimations extracted from videos (Eiffert, 2018).
The 2D pose estimation was performed on a subset of the Berkeley Multimodal Human Action Database (MHAD) dataset
(Oﬂi et al., 2013).
Six actions (Jumping in place, Jumping jacks, Punching (boxing), Waving two hands, Waving one hand, Clapping hands) are considered for a total ofvideos recorded at 22Hz.
Each action must then be identiﬁed based on a sequence offrames ( ≈ 1.5 sec).
Each video gives several sequences by applying an overlapping ratio.
A pose is extracted from each frame and estimated by 18 body joints expressed in a 2D coordinate system.
Following the parameter setting deﬁned in (Eiffert, 2018), the dataset is split into 2 independent partitions : 22625 sequences for training and 5751 for validation.
The model architecture consists of a two-stacked RNN of 60 GRU cells each followed by a fully connected layer with a RELU activation function.
The model is trained with batches of 512 sequences using a decaying learning rate for 600 epochs.
The best performance was obtained when the model was initialized with a model trained with a smaller λ .
This architecture was selected by performing an optimization over the hyperparameters.
The following parameters were included in the search:
– Batch size: 4096 and– λ ∈ { 1 e − 4 , 1 e − 3 , 1 e − 2 } – Cell type: LSTM or GRU – Number of cells ∈ { 30 , 40 , 50 , 60 } per layer (identical number of cells in each layer) – Window size L ∈ { 4 , 8 , 16 } (SkipW only)
The search was performed both for ThrRNN (baseline) and SkipW (our approach).
Several combinations of parameters resulted in models that are Pareto optimal with respect to accuracy and computational cost.
To evaluate SkipW and compare it to ThrRNN, we selected one parameter conﬁguration that led to models that were Pareto optimal (or close
to Pareto optimal) for ThRNN and for several values of L for SkipW. • Adding Task: This is a synthetic dataset consisting of sequences of (value, marker) tuples, where the expected output is (cid:80) i marker × value .
This dataset is commonly used to evaluate the performance of RNN models [(Hochreiter & Schmidhuber, 1997), (Neil et al., 2016), (Campos et al., 2017)].
The value elements are uniformly sampled in the range (-0.5, 0.5) and the marker elements are equal to 1 or 0.
We reproduced the experimental setup deﬁned in (Neil et al., 2016) with a sequence of length 50.
Two values are marked for addition with the ﬁrst marker randomly placed in the ﬁrst 10% of the sequence and the second one in the last half of the sequence.
The model architecture consists of a single layer of 110 LSTM cells and a fully connected layer regressing the scalar output.
We train the model with a batch size of 256 and a learning rate of 1 e − 4 .
As with HAR dataset, the best performance was obtained when the model was initialized with a model trained with a smaller λ .
B I MPACT OF THR
Modifying thr impacts the accuracy / processing tradeoff as well.
However, changing thr does not affect the upper bound on computational cost.
Increasing thr gradually reduces both accuracy and average computational cost.
This is similar to ThrRNN.
Figure 9 illustrates the impact of changing thr for various L on HAR.
While this is less interesting in the context of a strict limit on computation, it increases the variety of the possible tradeoff’s points.
That being said, the thresholding mechanism allows the model to skip uninteresting inputs and is therefore necessary.
For example, if this mechanism is removed on adding task, SkipW processes inputs in parts of the sequence where there is no marker.
On the other tasks, removing it similarly increases computational cost without improving accuracy.
This is better highlighted in Appendix C.4.1.
[Figure]
[Figure]
[Figure]
A cc u r a c y trade-off SkipRNN
SkipW =1e-2, L= SkipW =1e-2, L=
SkipW =2e-2, L=
ThrRNN =
1e-
C D ETAILED RESULTS FOR A DDING TASK
The Adding task dataset penalizes the model when skipping markers.
Therefore, a successful model will learn to process as many inputs as possible where the makers can be placed and to ignore the parts of the sequence where the markers are never found.
C. A CCURACY AND COMPUTATIONAL TRADEOFFS
As we see in Figure 4, both SkipW and ThrRNN offer operating points that are better than the ones obtained with individual SkipRNN models (dotted line in the ﬁgure).
SkipW offers results with a lower number of updates and lower error than ThrRNN while adapting thr especially for K between 5 and 10.
Both models also achieve a lower error than a model randomly sampling inputs.
The lowest error such a model can achieve is equal to the number of markers times (2) times the variance of the value (1/12) times the probability to miss a marker.
C.
M ODIFYING TRADEOFF DURING INFERENCE
We test SkipW adaptability during the analysis of a sequence; switching between a model with a low MSE, high update rate and another model with a higher MSE and lower update rate.
As illustrated by Figure 10, the evolution of the MSE is not as smooth as in the HAR experimentation due to the nature of the task - skipping a marker greatly penalizes the MSE.
We can note that changes in MSE and number of inputs processed happen when the switch is performed in the last 40% of the sequence.
This is explained by the fact that the distribution of the inputs processed is identical for
K = 5 and K = 10 at the beginning of the sequence.
Both models a) process the ﬁrst 5 inputs, to ﬁnd the ﬁrst marker, b) skip other inputs up to 50% of the sequence, as they never contain a marker and c) process the last 5 inputs of the window containing the middle of the sequence, as the second marker may be there.
This is illustrated in Figure 13 for K = 10 and Figure 15 for K = 5 .
As the loss in MSE stays reasonable proportionally to the number of updates skipped, SkipW can be used to reduce the number of updates during a sequence to fulﬁll computational restriction.
C. C OMPUTATIONAL T RADEOFFS
Figure 11 illustrates the ability of SkipW to strictly limit the number of inputs processed.
The dotted lines corresponds to the K/L theoretical limit imposed.
It is worth noting that the model uses a maximum of 60% of inputs for K =
L .
This can be explained by the nature of the task where 40% of the sequence does not contain information (no marker) and is also illustrated by the distribution of inputs processed in Figure 13 (SkipW, top row).
[Figure]
[Figure]
When a computational constraint is enforced and K is set to a lower value, some skip patterns are forbidden and skipW falls back to other patterns.
These restrictions inevitably cause SkipW to miss some markers. This is illustrated in Figure 12. Note that the algorithm has learned where interesting inputs are located: the ﬁrst marker starts to be skipped only when K < 5 .
This can also been seen by looking at sampling patterns, summarized in Figure 15 for some values of K [Figure]
Position in the sequence
ThrRNN
SkipWno t SkipW 0.
P r o b .
t
o p r
o
c
e ss i n p u t ﬁrst half
[Table]
As discussed in Section 3, including the index ( t ) in the input of the update gate (Equation 6) is necessary to allow the model to skip all inputs in a window. We show here the impact of removing it by modifying equation 6: ˜ u W,
[Equation]
Figure 13 shows that, without t , the model is forced to process an additional input in the second window even though this part of the sequence contains no marker.
Processing this input enables the model to change its hidden state and therefore to obtain a different activation vector ˜ u W,t +1 in the third window and to continue processing the second part of the sequence where the second marker is found.
Without this update, the model would stop processing the inputs at the end of the ﬁrst window since the state and therefore the activation vector would not change.
Including t in the input of the activation gate ensures that a different activation vector may be computed even without processing an input in a window.
Figure 13 also compares the SkipW models with ThrRNN.
Notice that ThrRNN is unable to stop processing inputs for a long period, as shown by a non-zero probability of using inputs between position 5 and 25.
In addition, ThrRNN sometimes skip the ﬁrst potential location of the second marker.
We believe this explains why SkipW can achieve both a higher accuracy and a lower computational cost than SkipRNN and ThrRNN.
On the other hand, contrary to SkipW models who determine the inputs to process at the beginning of each window, the probability of processing an input gradually lowers after position 25 for ThrRNN making it more efﬁcient for the second part of the sequence as shown in Table 1.
The same behavior can be observed for SkipW, but only from one window to another.
It is also worth noting that the probability that SkipW processes inputs in a window is very close to the probability that ThrRNN processes the ﬁrst input in the window.
C.4. A BLATION OF THRESHOLD ( THR = 0)
A second experiment was done to show the importance of the threshold parameter.
When the threshold is set to 0 the inputs are selected sorely on the basis of the parameter
K. Therefore exactly K inputs are selected in each window using equation 9 and equation 10 becomes u t = f binarize (˜ u K W,t
[ i ]) =
if ˜ u K W,t [ i ] > 0 , 0 otherwise. [Figure] P [Figure]
o p
[Figure] o c e ss i n p u t [Figure] p
[Figure]
Figure 14 shows the distribution of the inputs in each window for different values of K when thr =and Figure 15 when thr = 0 .
The importance of thr is clearly highlighted when comparing the two ﬁgures.
For example, for K = 5 and thr =
(Figure 14), there is no difference in the ﬁrst window and the model ﬁnds the ﬁrst marker.
However, as the model is forced to use 5 inputs per window, it processes inputs in the second window and this impacts the choice made in the third window as the model processes inputs at the beginning of this window instead of the last 5 which could contain the second marker.
This behavior is to be compared with Figure 15 ( thr = 0 .
5 ) where the inputs processed are effectively located in the second half of the third window.
It should be noted that for K = 1 , thr has no discernible effect.
The model selects only one input in the ﬁrst window and this affects the performance (the model will miss the ﬁrst marker 80% of the time) but also which inputs are selected for processing in subsequent windows.
D A DDITIONAL RESULTS FOR HAR D. M ODIFYING TRADEOFF DURING INFERENCE
Figure 16 and 17 show the adaptability of SkipW during the analysis of a sequence when switching between different tradeoffs.
We present here results for a SkipW wih L = 16 and λ = 2 e −with different tradeoffs.
As with previous experiments (Figure 7 and Figure 10), the performance is bounded by the individual performance of the two conﬁgurations evaluated. [Figure] [Figure]
A cc u r a c y thr=0.5, k= thr=0.5, k= Accuracy Inputs (%) I n p u t s p r o c e ss e d ( % )
[Figure]
u r
[Figure]
c y thr=
[Figure]
[Figure]
[Figure]
thr=
[Figure]
[Figure]
, k=
Accuracy
Inputs (%) 0.
I n p u t s p r o c e ss e d ( % )
[Figure]
E R EAL TIME EXPERIMENTS E. L ATENCY EXPERIMENTS
In order to explore the impact of the HW platform on the latency of HAR Services based on
SkipW/PoseNet
[Figure] we measure the latency of the HAR Service on 2 other small platforms: a Jetson TX2 (similar to the Jetson Nano platform (Figure 8) [Figure] but more powerful) and a Raspberry Pi(better CPU, but lacking a compatible GPU for AI model acceleration).
The hardware speciﬁcation of these different devices is provided in Table 2.
Figure 18 compares the performances of the HAR
Services on the 3 selected platforms.
It is worth noting that the impact of the different tradeoffs ( [ K, thr ] ) on the latency are very similar.
E. E NERGY CONSUMPTION
When SkipW coupled with a CNN model is set appropriately, it can enable a huge reduction in the amount of computation.
This can have an immediate impact on the energy consumed to deliver a Service, an important feature for battery powered systems.
As an illustration, we measure the power consumed by the HAR analysis when running on the Jetson boards.
This analysis is designed to run at 22Hz.
The energy consumption is measured by pulling periodically (every 100 ms) the instantaneous power consumption indicator available at the board level.
We report here an average of these instantaneous values over a 8 minutes period.
Figure 19 provides those energy consumption for the Jetson Nano and the Jetson TX2.
Those two platforms display the same behavior: when K decreases, the consumption decreases signiﬁcantly.
The gain is especially noticeable for the Jetson
Nano platform, where it can reach up to 50% of energy gain.
[Figure]