A SSEMBLE N ET : S EARCHING FOR M ULTI -S TREAM N EURAL C ONNECTIVITY IN V IDEO A RCHITECTURES
A BSTRACT
Learning to represent videos is a very challenging task both algorithmically and computationally.
Standard video CNN architectures have been designed by directly extending architectures devised for image understanding to include the time dimension, using modules such as 3D convolutions, or by using two-stream design to capture both appearance and motion in videos.
We interpret a video CNN as a collection of multi-stream convolutional blocks connected to each other, and propose the approach of automatically ﬁnding neural architectures with better connectivity and spatio-temporal interactions for video understanding.
This is done by evolving a population of overly-connected architectures guided by connection weight learning.
Architectures combining representations that abstract different input types (i.e., RGB and optical ﬂow) at multiple temporal resolutions are searched for, allowing different types or sources of information to interact with each other.
Our method, referred to as AssembleNet, outperforms prior approaches on public video datasets, in some cases by a great margin.
We obtain 58.6% mAP on Charades and 34.27% accuracy on Moments-in-Time.
I NTRODUCTION
Learning to represent videos is a challenging problem.
Because a video contains spatio-temporal data, its representation is required to abstract both appearance and motion information.
This is particularly important for tasks such as activity recognition, as understanding detailed semantic contents of the video is needed.
Previously, researchers approached this challenge by designing a two-stream model for appearance and motion information respectively, combining them by late or intermediate fusion to obtain successful results: Simonyan & Zisserman (2014); Feichtenhofer et al.
(2016b;a; 2017; 2018).
However, combining appearance and motion information is an open problem and the study on how and where different modalities should interchange representations and what temporal aspect/resolution each stream (or module) should focus on has been very limited.
In this paper, we investigate how to learn feature representations across spatial and motion visual clues.
We propose a new multi-stream neural architecture search algorithm with connection learning guided evolution , which focuses on ﬁnding higher-level connectivity between network blocks taking multiple input streams at different temporal resolutions.
Each block itself is composed of multiple residual modules with space-time convolutional layers, learning spatio-temporal representations.
Our architecture learning not only considers the connectivity between such multi-stream, multi-resolution blocks, but also merges and splits network blocks to ﬁnd better multi-stream video CNN architectures.
Our objective is to address two main questions in video representation learning: (1) what feature representations are needed at each intermediate stage of the network and at which resolution and (2) how to combine or exchange such intermediate representations (i.e., connectivity learning).
Unlike previous neural architecture search methods for images that focus on ﬁnding a good ‘module’ of convolutional layers to be repeated in a single-stream networks (Zoph et al., 2018; Real et al., 2019), our objective is to search for higher-level connections between multiple sequential or concurrent blocks to form multi-stream architectures.
We propose the concept of AssembleNet, a new method of fusing different sub-networks with different input modalities and temporal resolutions.
AssembleNet is a general formulation that
[Figure]
allows network connectivity the weights to guide evolution addition combining sub network AssembleNet is a of learnable architectures they a generic approach to learn connectivity among representations input modalities being optimized for the the i research stream connections for
Figure 1 shows an example learned AssembleNet.
The proposed algorithm for learning video architectures is very effective: it outperforms all prior work and baselines on two very challenging benchmark datasets, and establishes a new state-of-the-art.
AssembleNet models use equivalent number of parameters to standard two-stream (2+1)D ResNet models.
P REVIOUS WORK
A video is a spatio-temporal data (i.e., image frames concatenated along time axis), and its representation must abstract both spatial and temporal information.
Full 3D space-time (i.e., XYT) convolutional layers as well as (2+1)D convolutional layers have been popularly used to represent videos (Tran et al., 2014; Carreira & Zisserman, 2017; Tran et al., 2018; Xie et al., 2018).
Researchers studied replacing 2D convolutional layers in standard image-based CNNs such as Inception (Szegedy et al., 2016) and ResNet (He et al., 2016), so that it can be directly used for video classiﬁcation.
Two-stream network designs, which combine motion and appearance inputs, are commonly used (e.g.,
Simonyan & Zisserman, 2014; Feichtenhofer et al., 2016a; 2017; 2016b).
Combining appearance information at two different temporal resolutions (e.g., 24 vs. 3 frames per second) with intermediate connections has been proposed by Feichtenhofer et al.
Late fusion of the two-stream representations or architectures with more intermediate connections (Diba et al., 2019), have also been explored.
However, these video CNN architectures are the result of careful manual designs by human experts.
Neural Architecture Search (NAS), the concept of automatically ﬁnding better architectures based on data, is becoming increasingly popular (Zoph & Le, 2017; Zoph et al., 2018; Liu et al., 2018).
Rather than relying on human expert knowledge to design a CNN model, neural architecture search allows the machines to generate better performing models optimized for the data.
The use of reinforcement learning controllers (Zoph & Le, 2017; Zoph et al., 2018) as well as evolutionary algorithms (Real et al., 2019) have been studied, and they meaningfully outperform handcrafted architectures.
Most of these works focus on learning architectures of modules (i.e., groupings of layers and their connections) to be repeated within a ﬁxed single-stream meta-architecture (e.g., ResNet) for image-based object classiﬁcation.
One-shot architecture search to learn differentiable connections (Bender et al., 2018;
Liu et al., 2019) has also been successful for images.
However, it is very challenging to directly extend such work to ﬁnd multi-stream models for videos, as it requires preparing all possible layers and interactions the ﬁnal architecture may consider using.
In multi-stream video CNNs, there are many possible convolutional blocks with different resolutions, and fully connecting them requires a signiﬁcant amount of memory and training data, which makes it infeasible.
Our work is also related to Ahmed & Torresani (2017) which used learnable gating to connect multiple residual module branches, and to the RandWire network (Xie et al., 2019), which showed that randomly connecting a sufﬁcient number of convolutional layers creates performant architectures.
However, similar to previous NAS work, the latter focuses only on generating connections between the layers within a block.
The meta-architecture is ﬁxed as a single stream model with a single input modality.
In this work, our objective is to learn high-level connectivity between multi-stream blocks for video understanding driven by data.
We conﬁrm experimentally that in the multi-stream video
CNNs, where multiple types of input modalities need to be considered at various resolutions, randomly connecting blocks is insufﬁcient and the proposed architecture learning strategy is necessary.
A SSEMBLE N ET
We propose a new principled way to ﬁnd better neural architectures for video representation learning.
We ﬁrst expand a video CNN to a multi-resolution, multi-stream model composed of multiple sequential and concurrent neural blocks, and introduce a novel algorithm to search for the optimal connectivity between the blocks for a given task.
We model a video CNN architecture as a collection of convolutional blocks (i.e., sub-networks) connected to each other.
Each block is composed of a residual module of space-time convolutional layers repeated multiple times, while having its own temporal resolution.
The objective of our video architecture search is to automatically (1) decide the number of parallel blocks (i.e., how many streams to have) at each level of the network, (2) choose their temporal resolutions, and
(3) ﬁnd the optimal connectivity between such multi-stream neural blocks across various levels.
The highly interconnected convolutional blocks allow learning of the video representations combining multiple input modalities at various temporal resolutions.
We introduce the concept of connection-learningguided architecture evolution to enable multi-stream architecture search.
We name our ﬁnal architecture as an ‘AssembleNet’, since it is formulated by assembling (i.e., merging, splitting, and connecting) multiple building blocks.
3. G RAPH FORMULATION
In order to make our neural architecture evolution consider multiple different streams with different modalities at different temporal resolutions, we formulate the multi-stream model as a directed acyclic graph.
Each node in the graph corresponds to a sub-network composed of multiple convolutional layers (i.e., a block), and the edges specify the connections between such sub-networks.
Each architecture is denoted as G i =
( N i , E i )
where N i = { n 0 i , n 1 i , n 2 i , · · · } is the set of nodes and E i is the set of edges deﬁning their connectivity.
Nodes.
A node in our graph representation is a ResNet block composed of a ﬁxed number of interleaved 2D and (2+1)D residual modules.
A ‘2D module’ is composed of a 1x1 conv.
layer, one 2D conv.
layer with ﬁlter size 3x3, and one 1x1 convolutional layer.
A ‘(2+1)D module’ consists of a temporal 1D convolutional layer (with ﬁlter size 3), a 2D conv.
layer, and a 1x1 conv. layer.
In each block, we repeat a regular 2D residual module followed by the (2+1)D residual module m times.
Each node has its own block level, which naturally decides the directions of the edges connected to it.
Similar to the standard ResNet models, we made the nodes have a total of four block levels (+ the stem level).
Having multiple nodes of the same level means the architecture has multiple parallel ‘streams’.
Figure 1 illustrates an example.
Each level has a different m value: 1.5, 2, 3, and 1.5.
m
= 1 . 5 means that there is one 2D module, one (2+1)D module, and one more 2D module.
As a result, the depth of our network is 50 conv.
layers.
We also have a batch normalization layer followed by a ReLU after every conv.
layer.
There are two special types of nodes with different layer conﬁgurations: source nodes and sink nodes.
A source node in the graph directly takes the input and applies a small number of convolutional/pooling layers (it is often referred as the ‘stem’ of a CNN model).
In video CNNs, the input is a 4D tensor (XYT + channel) obtained by concatenating either RGB frames or optical ﬂow images along the time axis.
Source nodes are treated as level-0 nodes.
The source node is composed of one 2D conv.
layer of ﬁlter size 7x7, one 1D temporal conv.
layer of ﬁlter size 5, and one spatial max pooling layer.
The 1D conv. is omitted in optical ﬂow stems.
A sink node generates the ﬁnal output of the model, and it is composed of one pooling, one fully connected, and one softmax layer.
The sink node is also responsible for combining the outputs of multiple nodes at the highest level, by concatenating them after the pooling.
More details are provided in Appendix.
Each node in the graph also has two attributes controlling the convolutional block: its temporal resolution and the number of channels.
We use temporally dilated 1D convolution to dynamically change the resolution of the temporal convolutional layers in different blocks, which are discussed more below.
The channel size (i.e., the number of ﬁlters) of a node could take arbitrary values, but we constrain the sum of the channels of all nodes in the same block level to be a constant so that the capacity of an AssembleNet model is equivalent to a ResNet model with the same number of layers.
Temporally Dilated 1D Convolution.
One of the objectives is to allow the video architectures to look at multiple possible temporal resolutions.
This could be done by preparing actual videos with different temporal resolutions as in Feichtenhofer et al.
(2018) or by using temporally ‘dilated convolutions as we introduce here.
Having dilated ﬁlters allow temporal 1D conv.
layers to focus on different temporal resolution without losing temporal granularity.
This essentially is a 1D temporal version of standard 2D dilated convolutions used in Chen et al.
(2018) or Yu & Koltun (2016):
Let k be a temporal ﬁlter (i.e., a vector) with size 2 d
+ 1 .
The dilated convolution operator ∗ r is similar to regular convolution but has different steps for the summation, described as:
[Equation]
where t , t 1 , and t 2 are time indexes.
r indicates the temporal resolution (or the amount of dilation), and the standard 1D temporal convolution is a special case where r = 1 .
In the actual implementation, this is done by inserting r − 1 number of zeros between each element of k to generate k (cid:48) , and then convolving such zero-inﬂated ﬁlters with the input: F ∗ r k
=
F ∗ k (cid:48) .
Importantly, the use of the dilated convolution allows different intermediate sub-network blocks (i.e., not just input stems) to focus on very different temporal resolutions at different levels of the convolutional architecture.
Note that our temporally dilated convolution is different from the one used in Lea et al.
(2017), which designed a speciﬁc layer to combine representations from different frames with various step sizes.
Our layers dilate the temporal ﬁlters themselves.
Our dilated convolution can be viewed as a direct temporal version of the standard dilated convolutions used in Chen et al.
; Yu & Koltun (2016).
Edges.
Each directed edge speciﬁes the connection between two sub-network blocks, and it describes how a representation is transferred from one block to another block.
We constrain the direction of each edge so that it is connected from a lower level block to a higher level block to avoid forming a cycle and allow parallel streams.
A node may receive inputs from any number of lower-level nodes (including skip connections) and provide its output to any number of higher-level nodes.
Our architectures use a (learnable) weighted summation to aggregate inputs given from multiple connected nodes.
That is, an input to a node is computed as F in = (cid:80) i sigmoid ( w i ) · F outi , where
F outi are output tensors (i.e., representations) of the nodes connected to the node and w i are their corresponding weights.
Importantly, each w i is considered as a variable that has to be learned from training data through back propagation.
This has two key advantages compared to conventional feature map concatenation : (i) The input tensor size is consistent regardless of the number of connections.
(ii) We use learned connection weights to ‘guide’ our architecture evolution algorithm in a preferable way, which we discuss more in Section 3.
If the inputs from different nodes differ in their spatial size, we add spatial max pooling and striding to match their spatial size.
If the inputs have different channel sizes, we add a 1x1 conv.
layer to match the bigger channel size.
Temporal sizes of the representations is always consistent in our graphs, as there is no temporal striding in our formulation and the layers in the nodes are fully convolutional. [Figure]
3. E VOLUTION
We design an evolutionary algorithm with discrete mutation operators that modify nodes and edges in architectures over iterations.
The algorithm maintains a population of P different architectures,
P = { G 1 , G 2 , · · · , G | P | } , where each architecture G is represented with a set of nodes and their edges as described above.
The initial population is formed by preparing a ﬁxed number of randomly connected architectures (e.g., | P | = 20 ).
Speciﬁcally, we (1) prepare a ﬁxed number of stems and nodes at each level (e.g., two per level), (2) apply a number of node split/merge mutation operators which we discuss more below, and (3) randomly connect nodes with the probability p = 0 .
5 while discarding architectures with graph depth < 4.
As mentioned above, edges are constrained so that there is no directed edge reversing the level ordering.
Essentially, a set of overly-connected architectures are used as a starting point.
Temporal resolutions are randomly assigned to the nodes.
We use the tournament selection algorithm (Goldberg & Deb, 1991) as the main evolution framework:
At each evolution round, the algorithm updates the population by selecting a ‘parent’ architecture and mutating (i.e., modifying) it to generate a new ‘child’ architecture.
The parent is selected by randomly sampling a subset of the entire population P (cid:48)
⊂ P , and then computing the architecture with the highest ‘ﬁtness’:
G p = argmax G i ∈ P (cid:48)
f ( G i ) where f ( G ) is the ﬁtness function.
Our ﬁtness is deﬁned as a video classiﬁcation accuracy of the model, measured by training the model with a certain number of initial iterations and then evaluating it on the validation set as its proxy task.
More speciﬁcally, we use top-1 accuracy + top-5 accuracy as the ﬁtness function.
The child is added into the population, and the model with the least ﬁtness is discarded from the population.
A child is evolved from the parent by following two steps.
First, it changes the block connectivity (i.e., edges) based on their learned weights: ‘connection-learning-guided evolution’.
Next, it applies a random number of mutation operators to further modify the node conﬁguration.
The mutation operators include (1) a random modiﬁcation of the temporal resolution of a convolutional block (i.e., a node) as well as (2) a merge or split of a block.
When splitting a node into two nodes, we make their input/output connections identical while making the number of channels in their convolutional layers half that of the node before the split (i.e., C = C p / 2 where C p is the channel size of the parent).
More details are found in Appendix.
As a result, we maintain the total number of parameters, since splitting or merging does not change the number of parameters of the convolutional blocks.
Connection-Learning-Guided Mutation.
Instead of randomly adding, removing or modifying block connections to generate the child architecture, we take advantage of the learned connection weights from its parent architecture.
Let E p be the set of edges of the parent architecture.
Then the edges of the child architecture E c are inherited from E p , by only maintaining high-weight connections while replacing the low-weight connections with new random ones.
Speciﬁcally, E c =
E 1 c ∪ E [Equation] c :
[Equation]
where X ∼ unif (0 , 1) and E ∗ is the set of all possible edges.
E 1 c corresponds to the edges the child architecture inherits from the parent architecture, decided based on the learned weight of the edge W e . is possible
because our
B , which controls whether or not to keep an edge from the parent architecture, could either be a constant threshold or a random variable following a uniform distribution: B = b or B = X B ∼ unif (0 , 1) .
E 2 c corresponds to the new randomly added edges which were not in the parent architecture.
We enumerate through each possible new edge, and randomly add it with the probably of | E p −
E 1 c | / | E − E p | .
This makes the expected total number of added edges to be | E p − E 1 c | , maintaining the size of E c .
Figure 2 shows an example of the evolution process and Figure 3 shows ﬁnal architectures.
Evolution Implementation Details.
Initial architectures are formed by randomly preparing either { 2 or 4 } stems, two nodes per level at levels 1 to 3, and one node at level 4.
We then apply 1 ∼random number of node split operators so that each initial architecture has a different number of nodes.
Each node is initialized with a random temporal resolution of 1, 2, 4, or 8.
As mentioned, each possible connection is then added with the probability of p = 0 .
At each evolution round, the best-performing parent architecture is selected from a random subset of 5 from the population.
The child architecture is generated by modifying the connections from the parent architecture (Section 3.2).
A random number of node split, merge, or temporal resolution change mutation operators (0 ∼ 4) are then applied.
Evaluation of each architecture (i.e., measuring the ﬁtness) is done by training the model for 10K iterations and then measuring its top-1 + topaccuracy on the validation subset.
The Moments-in-Time dataset, described in the next section, is used as the proxy dataset to measure ﬁtness.
The evolution was run for ∼ 200 rounds, although a good performing architecture was found within only 40 rounds (e.g., Figure 3-right).
Figure 1 shows the model found at the 165th round.
10K training iterations of each model during evolution took 3 ∼hours; with our setting, evolving a model for 40 rounds took less than a day with 10 parallel workers.
E XPERIMENTS 4. D ATASETS
Charades Dataset.
We ﬁrst test on the popular Charades dataset (Sigurdsson et al., 2016) which is unique in the activity recognition domain as it contains long sequences.
It is one of the largest public datasets with continuous action videos, containing 9848 videos of 157 classes (7985 training and 1863 testing videos).
Each video is ∼ 30 seconds.
It is a challenging dataset due to the duration and variety of the activities.
Activities may temporally overlap in a Charades video, requiring the model to predict multiple class labels per video.
We used the standard ‘Charades v1 classify’ setting for the evaluation.
To comply with prior work (e.g. Feichtenhofer et al., 2018), we also report results when pre-training on Kinetics (Carreira & Zisserman, 2017), which is another large-scale dataset.
[Table] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
We note that Kinetics is shrinking in size ( ∼ 15% videos removed from the original Kinetics-400) and the previous versions are no longer available from the ofﬁcial site. Moments in Time (MiT) Dataset.
The Moments in Time (MiT) dataset (Monfort et al., 2018) is a large-scale video classiﬁcation dataset with more than 800K videos ( ∼ 3 seconds per video).
It is a very challenging dataset with the state-of-the-art models obtaining less than 30% accuracy.
We use this dataset for the architecture evolution, and train/test the evolved models.
We chose the MiT dataset because it provides a sufﬁcient amount of training data for video CNN models and allows stable comparison against previous models.
We used its standard classiﬁcation evaluation setting.
4. R ESULTS
Tables 1 and 2 compare the performance of AssembleNet against the state-of-the-art models.
We denote AssembleNet more speciﬁcally as AssembleNet-50, since its depth is 50 layers and has an equivalent number of parameters to ResNet-50.
AssembleNet-101 is its 101 layer version having equivalent number of parameters to ResNet-101.
AssembleNet is outperforming prior works on both datasets, setting new state-of-the-art results for them.
Its performance on MiT is the ﬁrst above 34%.
We also note that the performances on Charades is even more impressive at 58 .
6 whereas previous known best results are 42 .
5 and 45 .
For these experiments, the architecture search was done on the
MiT dataset, and then the found models are trained and tested on both datasets, which demonstrates that the found architectures are useful across datasets.
weight learning.
Four-stream models are reported here for the ﬁrst time, and are very effective.
All these models have a similar number of parameters
[Table]
[Table]
In addition [Table] we compare the proposed connection-learning-guided evolution with random architecture search and the standard evolutionary algorithm with random connection mutations.
We made the standard evolutionary algorithm randomly modify 1/3 of the total connections at each round, as that is roughly the number of edges the connection-learning-guided evolution modiﬁes.
Figure 4 shows the results, visualizing the average ﬁtness score of the three top-performing models in each pool.
We observe that the connection-learning-guided evolution is able to ﬁnd better architectures, and it is able to do so more quickly.
The standard evolution performs similarly to random search and is not as effective.
We believe this is due to the large search space the approach is required to handle, which is exponential to the number of possible connections.
For instance, if there are N nodes, the search space complexity is 2 O ( N 2 ) just for the connectivity search.
Note that the initial ∼ 30 rounds are always used for random initialization of the model population, regardless of the search method.
4. A BLATION STUDIES
We conduct an ablation study comparing the evolved AssembleNet to multiple (2+1)D two-stream (or multi-stream) architectures which are designed to match the abilities of Assemblenet but without evolution.
We note that these include very strong architectures that have not been explored before, such as the four-stream model with dense intermediate connectivity.
We design competitive networks having various connections between streams, where the connection weights are also learned (see the supplementary material for detailed descriptions and visualizations).
Note that all these models have equivalent capacity (i.e., number of parameters).
The performance difference is due to network structure.
Table 3 shows the results, demonstrating that these architectures with learnable interconnectivity are very powerful themselves and evolution is further beneﬁcial.
The Moments in Time models were trained from scratch, and the Charades models were pre-trained on MiT. In particular, we evaluated an architecture with intermediate connectivity from the ﬂow stream to RGB, inspired by Feichtenhofer et al.
(2016b; 2018)
(+ connection weight learning).
It gave 30 . 2 % accuracy on MiT and 49 .
5 % on Charades, which are not as accurate as AssembleNet.
Randomly generated models (from 50 rounds of search) are also evaluated, conﬁrming that such architectures do not perform well.
Further, we conduct another ablation to conﬁrm the effectiveness of our search space.
Tablecompares the models found with our full search space vs. more constrained search spaces, such as only using two stems and not using temporal dilation (i.e., ﬁxed temporal resolution).
4. G ENERAL FINDINGS
As the result of connection-learning-guided architecture evolution, non-obvious and non-intuitive connections are found (Figure 3).
As expected, more than one possible “connectivity” solution can yield similarly good results.
Simultaneously, models with random connectivity perform poorly compared to the found AssembleNet.
Our observations also include: (1) The models prefer to have only one block at the highest level, although we allow the search to consider having more than one block at that level.
The ﬁnal block prefers simple connections gathering all outputs of the blocks in the 2nd to last level.
(3) Many models use multiple blocks with different temporal resolutions at the same level, justifying the necessity of the multi-stream architectures.
(4) Often, there are 1 orblocks heavily connected to many other blocks.
(5) Architectures prefer using more than 2 streams, usually using 4 at many levels.
C ONCLUSION
We present AssembleNet, a new approach for neural architecture search using connection-learningguided architecture evolution.
AssembleNet ﬁnds multi-stream architectures with better connectivity and temporal resolutions for video representation learning.
Our experiments conﬁrm that the learned models signiﬁcantly outperform previous models on two challenging benchmarks.
S UPER - GRAPH VISUALIZATION OF THE CONNECTIVITY SEARCH SPACE
Figure 5 visualizes all possible connections and channel/temporal resolution options our architecture evolution is able to consider.
The objective of our evolutionary algorithm could be interpreted as ﬁnding the optimal sub-graph (of this super-graph) that maximizes the performance while maintaining the number of total parameters.
Trying to directly ﬁt such entire super-graph into the memory was infeasible in our experiments.
A. C HANNEL SIZES OF THE LAYERS AND NODE SPLIT / MERGE MUTATIONS
As we described in the paper, each node (i.e., a convolutional block) has a parameter C controlling the number of ﬁlters of the convolutional layers in the block.
When splitting or merging blocks, the number of ﬁlters are split or combined respectively.
Figure 6 provides a visualization of a block with number of ﬁlter speciﬁed to the right and a split operation.
While many designs are possible, we design the blocks and splitting as follows.
The size of 1x1 convolutional layers and 1D temporal convolutional layers are strictly governed by C , having the channel size of C (some 4 C ).
On the other hand, the number of 2D convolutional layer is ﬁxed per level as a constant D v where v is the level of the block.
D 1 = 64 , D 2 = 128 , D 3 = 256 , and D 4 = 512 .
The layers in the stems havechannels if there are only two stems and 32 if there are four stems.
When a node is split into two nodes, we update the resulting two nodes’ channel sizes to be half of their original node.
This enables us to maintain the total number of model parameters before and after the node split to be identical.
The ﬁrst 1x1 convolutional layer will have half the parameters after the split, since its output channel size is now 1/2.
The 2D convolutional layer will also have exactly half the parameters, since its input channel size is 1/2 while the output channel size is staying ﬁxed.
The next 1x1 convolutional layer will have the ﬁxed input channel size while the output channel size becomes 1/2: thus the total number of parameters would be 1/2 of the original parameters.
Merging of the nodes is done in an inverse of the way we split.
When merging two nodes into one, the merged node inherits all input/output connections from the two nodes: we take a union of all the connections.
The channel size of the merged node is the sum of the channel sizes of the two nodes being merged.
The temporal dilation rate of the merged node is randomly chosen between the two nodes before the merge.
A. H AND - DESIGNED MODELS USED IN THE ABLATION STUDY
Figure 7 illustrates the actual architectures of the hand-designed (2+1)D CNN models used in our ablation study.
We also show the ﬁnal learned weights of the connections, illustrating which connections the model ended up using or not using.
We note that these architectures are also very enlightening as the connectivity within them are learned in the process.
We observe that stronger connections tend to be formed later for 2-stream architectures.
For 4-stream architectures, stronger connections do form early, and, not surprisingly, a connection to at least one node of a different
2D conv.
1x 1x 1D conv.
2D conv.
1x + + ...
+ 2D conv.
1x 1x 1D conv.
2D conv.
1x + + ...
+ 2D conv.
1x 1x 1D conv.
2D conv.
1x + + ...
+ 4*C 4*C
C
D v
C=
D v
C
C= C= ...
[Figure] [Figure]
modality is established, i.e. a node stemming from RGB will connect to at least one ﬂow node at the next level and vice versa.
Below is a more detailed description of the networks used in the paper: “Two-stream (late fusion)” means that the model has two separate streams at every level including the level 4, and the outputs of such two level 4 nodes are combined for the ﬁnal classiﬁcation.
“Fusion at lv. 4” is the model that only has one level 4 node to combine the outputs of the two level 3 nodes using a weighted summation.
“Two-stream (fully)” means that the model has two nodes at each level 1-3 and one node at level 4, and each node is always connected to every node in the immediate next level.
“Flow → RGB” means that only the RGB stream nodes combine outputs from both RGB and ﬂow stream nodes of the immediate lower level.
[Table]
A. A SSEMBLE N ET MODEL / LAYER DETAILS
We also provide the ﬁnal AssembleNet model in table form in Table 5.
In particular, the 2nd element of each block description shows the list of where the input to that block is coming from (i
[Table]
e., the connections).
As already mentioned, 2D and (2+1)D residual modules are repeated in each block
[Table]
The number of repetitions m are 1.5, 2,
[Table]
and 1.5 at each level.
m
= 1 . 5 means that we have one 2D residual module, one (2+1)D module, and one more 2D module.
This makes the number of convolutional layers of each block at levels 1-4 to be 9, 12, [Table] and 9.
In addition, a stem has at most 2 convolutional layers.
The total depth of our network is 50, similar to a conventional (2+1)D
ResNet-50.
For AssembleNet-101, we use m = 1.5, 2, 11.5, and 1.5 at each level.
If a block has a spatial stride of 2, the striding happens at the ﬁrst 2D convolutional layer of the block.
In the stem which has the spatial stride of 4, the striding of size 2 happens twice, once at the 2D convolutional layer and at the max pooling layer.
As mentioned, the model has a batch normalization layer followed by ReLU after every convolutional layer regardless of its type (i.e., 2D, 1D, and 1x1).
2D conv.
ﬁlter sizes are 3x3, and 1D conv.
ﬁlter sizes are 3.
A. S INK NODE DETAILS
When each evolved or baseline (2+1)D model is applied to a video, it generates a 5D (BTYXC) tensor after the ﬁnal convolutional layer, where B is the size of the batch and C is the number of channels.
The sink node is responsible for mapping this into the output vector, whose dimensionality is identical to the number of video classes in the dataset.
The sink node ﬁrst applies a spatial average pooling to generate a 3D (BTC) tensor.
If there are multiple level 4 nodes (which rarely is the case), the sink node combines them into a single tensor by averaging/concatenating them.
Averaging or concatenating does not make much difference empirically.
Next, temporal average/max pooling is applied to make the representation a 2D (BC) tensor (average pooling was used for the MiT dataset and max pooling was used for Charades), and the ﬁnal fully connected layer and the soft max layer is applied to generate the ﬁnal output.
A. T RAINING DETAILS
For the Moments in Time (MiT) dataset training, 8 videos are provided per TPU core (with 16GB memory): the total batch size (for each gradient update) is 512 with 32 frames per video.
The batch size used for Charades is 128 with 128 frames per video.
The base framerate we used is 12.5 fps for
MiT and 6 fps for Charades.
The spatial input resolution is 224x224 during training.
We used the standard Momentum Optimizer in TensorFlow.
We used a learning rate of 3.2 (for MiT) and 25.6 (for
Charades), 12k warmup iterations, and cosine decay.
No dropout is used, weight decay is set to 1eand label smoothing set to 0.2.
Training a model for 10k iterations (during evolution) took 3 ∼ 5 hours and fully training the model (for 50k iterations) took ∼ 24 hours per dataset.
We used the TV-L1 optical ﬂow extraction algorithm
(Zach et al., 2007) implemented with tensor operations by Piergiovanni & Ryoo (2019) to obtain ﬂow input.
A. E VALUATION DETAILS
When evaluating a model on the MiT dataset, we provide 36 frames per video.
The duration of each MiT video is 3 seconds, making 36 frames roughly correspond to the entire video.
For the
Charades dataset where each video duration is roughly ∼ 30 seconds, the ﬁnal class labels are obtained by applying the model to ﬁve random 128 frame crops (i.e., segments) of each video.
The output multi-class labels are max-pooled to get the ﬁnal label, and is compared to the ground truth to measure the average precision scores.
The spatial resolution used for the testing is 256x256.