A S PECTRAL N ONLOCAL B LOCK FOR N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
The nonlocal network is designed for capturing long-range spatial-temporal dependencies in several computer vision tasks.
Although having shown excellent performances, it needs an elaborate preparation for both the number and position of the building blocks.
In this paper, we propose a new formulation of the nonlocal block and interpret it from the general graph signal processing perspective, where we view it as a fully-connected graph ﬁlter approximated by Chebyshev polynomials.
The proposed nonlocal block is more efﬁcient and robust, which is a generalized form of existing nonlocal blocks (e g. nonlocal block, nonlocal stage).
Moreover, we give the stable hypothesis and show that the steady-state of the deeper nonlocal structure should meet with it.
Based on the stable hypothesis, a full-order approximation of the nonlocal block is derived for consecutive connections.
Experimental results illustrate the clear-cut improvement and practical applicability of the generalized nonlocal block on both image and video classiﬁcation tasks.
I NTRODUCTION
Capturing the long-range spatial-temporal dependencies is crucial for the Deep Convolutional Neural Networks (CNNs) to extract discriminate features in vision tasks such as image and video classiﬁcation.
However, the traditional convolution operator only focuses on processing local neighborhood at a time.
This makes the CNNs need to go deeper with convolutional operations to enlarge the receptive ﬁelds, which lead to higher computation and memory.
Moreover, going deeper cannot always increase the effective receptive ﬁelds due to the Gaussian distribution of the kernel weight (Luo et al.
To eliminate this limitation, some recent works focus on designing the network architecture with wider and well-designed modules to catch the long-range dependencies such as (Peng et al.
(2017), Chen et al.
(2017), Zhao et al.
Although having larger receptive ﬁelds, these modules still need to be applied recursively to catch the dependencies of the pairs in large distances.
Inspired by the classical non-local means method in image denoising, Wang et al.
(2018) proposes the nonlocal neural network which uses the nonlocal (NL) block to concern the “full-range” dependencies in only one module by exploring the correlations between each position and all other positions.
In the NL block, the afﬁnity matrix is ﬁrst computed to represent the correlations between each position pair.
Then the weight means of features are calculated based on the afﬁnity matrix to reﬁne the feature representation.
Finally, the residual connection is added to the reﬁned feature map.
Due to its simplicity and effectiveness, the nonlocal block has been widely used in image and video classiﬁcation (Wang et al.
; Yue et al.
; Tao et al.
; Chen et al.
image segmentation (Huang et al.
; Yue et al.
; Wang et al.
and person re-identiﬁcation
(Liao et al.
; Zhang et al.
recently.
However, due to the complexity of the afﬁnity matrix, the nonlocal block 1 needs much more computational effort and is sensitive to its number and position in the neural network (Tao et al.
Some works solve the ﬁrst problem by simplifying the calculation of the afﬁnity matrix such as
Huang et al.
, He et al.
(2019), Yue et al.
(2018), Chen et al.
Only a few works try to solve the second problem which limits the robustness of the nonlocal network 2 .
Tao et al.
(2018) proposes the nonlocal stage (NS) block which concerns the diffusion nature and maintains the same afﬁnity matrix for all the nonlocal units in the NS block.
Comparing with the NL block, the NS block is insensitive to the numbers and allows deeper nonlocal structure.
However, the deeper nonlocal structure of NS block increases the complexity and do not have a remarkable improvement.
In this work, we focus on elaborating a robust nonlocal block which is more ﬂexible when using in the neural network.
We prove that the nonlocal operator in the nonlocal block is equivalent to the
Chebyshev-approximated fully-connected graph ﬁlter with irrational constraints that limits its liberty for learning.
To remove these irrational constraints, we propose the Spectral-based Nonlocal (SNL) block which is more robust and can degrade into the NL and NS with speciﬁc assumptions.
We also prove that the deeper nonlocal structure satisﬁes the stable hypothesis with the help of steadystate analysis.
Based on this hypothesis, we give the full-order approximated spectral nonlocal (gSNL) block which is well-performed for deeper nonlocal structure.
Finally, we add our proposed nonlocal blocks into the deep network and evaluate them on the image and video classiﬁcation tasks.
Experiments show that the networks with our proposed blocks are more robust and have a higher accuracy than using other types of nonlocal blocks.
To summarize, our contributions are threefold:
• We propose a spectral nonlocal (SNL) block as an efﬁcient, simple, and generic component for capturing long-range spatial-temporal dependencies with deep neural networks, which is a generalization of the classical nonlocal blocks.
• We propose the stable hypothesis, which can enable the deeper nonlocal structure without an elaborate preparation for both the number and position of the building blocks.
We further extend SNL into generalized SNL (gSNL), which can enable multiple nonlocal blocks to be plugged into the existing computer vision architectures with stable learning dynamics.
• Both SNL and gSNL have outperformed other nonlocal blocks across both image and video classiﬁcation tasks with a clear-cut improvement.
P RELIMINARY
Nonlocal block
The NL block consist of NL operator with residual connection and is expressed as:
[Equation]
where X ∈ R N × C 1 is the input feature map, F ( A , Z ) is the NL operator,
Z ∈ R N × C s is the transferred feature map that compresses the channels of X ∈ R N × C 1 by a linear transformation with kernel W g ∈ R C 1 × C s .
Here N is the number of positions.
The afﬁnity matrix A ∈ R N × N is composed by pairwise correlations between pixels.
In the NL block, the NL operator explores the “full-range” dependencies by concerning the relationships between all the position pairs:
[Equation]
where W ∈ R C s × C 1 is the weight matrix of a linear transformation.
f ( · ) is the afﬁnity kernel which can adopt the “Dot Product”, “Traditional Gasuassian”, “Embedded Gasussian” or other kernel matrix with a ﬁnite Frobenius norm.
Nonlocal stage To make the NL operator follow the diffusion nature that allows deeper nonlocal structure (Tao et al.
, the nonlocal stage (NS) operator uses the graph laplacian L = D
A − A to replace the afﬁnity matrix A in the NL operator:
[Equation]
where ¯ F ( A , Z ) is the NS operator.
d i = (cid:80)
j a ij is the degree of node i .
Moreover, when adding multiple blocks with the same afﬁnity matrix A and replacing the NL operator by the NS operator, these consecutively-connected blocks become the NS block.
We called these nonlocal blocks in the
NS block as the NS units.
M ETHOD
The nonlocal operator can be divided into two steps: calculating the afﬁnity matrix A to represent the correlations between each position pairs and reﬁning the feature map by calculating the weighted means based on A .
In this section, a fully-connected graph ﬁlter is utilized for explaining the nonlocal operator.
With the Chebyshev approximation, we propose the SNL operator which is proved to be a generalized form of NL and NS operator and is more robust with higher performance in computer vision tasks.
Furthermore, based on the stable hypothesis that deeper nonlocal structure tends to learn a stable afﬁnity matrix, we extend our SNL operator into a full-order Chebyshev approximation version, i.e. the gSNL.
3. T HE P ROPOSED S PECTRAL N ONLOCAL O PERATOR
Nonlocal operator in the graph view The nonlocal operator F ( A , Z ) is a ﬁlter that computes a weighted mean of all the positions in the feature map Z based on the afﬁnity matrix A and then conduct the feature transformation with the kernel W .
This is the same as ﬁltering the signal Z by a graph ﬁlter Ω in the graph domain deﬁned by the afﬁnity matrix
A (Shuman et al.
Based on this perspective (Shuman et al.
, we further deﬁne the nonlocal operator as:
Theorem 1.
Given an afﬁnity matrix A ∈ R N × N and the signal Z ∈ R N × C s , the nonlocal operator is the same as ﬁltering the signal Z in the graph domain of a fully-connected weighted graph G :
[Equation]
with L =
D
L − A =
U T ΛU
[Equation]
(4) where the graph ﬁlter Ω ∈ R N × N is a diagonal parameter matrix,
i.e. Ω = diag ( ω ) , ω = ( ω 1 , ω 2 , ..., ω n ) .
G =
( V , A ) is a fully-connected graph with the vertex set V and afﬁnity matrix
A .
Λ = diag ( { λ 1 , λ 2 , ..., λ i , ..., λ N } ) and U = { u 1 , u 2 , ..., u i , ..., u N } are the eigenvectors and eigenvalues of the graph laplacian L .
This deﬁnition requires that the graph laplacian L has non-singular eigenvalue and eigenvector, so the afﬁnity matrix A should be a symmetric, non-negative, row-normalized matrix.
To meet this requirement, the afﬁnity matrix A can be obtained by the following steps.
First, the afﬁnity kernel is used to calculate the matrix A (we use the dot product with embeded weight matrix W φ ∈ R C 1 × C s and W ϕ ∈ R C 1 × C s as the afﬁnity kernel, i.e. A = ( XW φ )( XW ϕ ) ).
Then we make the matrix
A symmetric: ¯A =
A T + A.
We normalize the row of ¯A to make it satisfy d i = 1 and having ˇA =
D − 1 A ¯A .
For the simplicity, in the following sections the symmetric, non-negative, row-normalized matrix ˇA is denoted as A .
The proposed spectral nonlocal operator The graph ﬁlter Ω in Eq.
(4) contains N parameters.
To simplify it, we use the Chebyshev polynomials which can reduce the N parameters into k ( k (cid:28) N ).
For simplicity, we ﬁrstly assume that the input Z , the output F ( A , Z ) and the output F ( A , Z ) have only one channel.
Following the similar method as Defferrard et al.
(2016), the k st -order Chebyshev polynomials is used to approximate the graph ﬁlter function g θ ( Λ ) :
[Equation]
Due to L is a random walk laplacican, the maximum eiginvalue λ max satisﬁes λ max = 2 which makes L (cid:48)
=
A (Shuman et al.
Then Eq.
(5) becomes:
[Equation]
If k = 1 , the ﬁrst-order Chebyshev approximation of Eq.
(6) becomes:
[Equation]
where θ 0 and θ 1 are the coefﬁcients for the ﬁrst and second term which are approximated by learning with SGD.
Then, extending Eq.
(7) into multi-channel conditions, we can get the formation of our
SNL operator:
[Equation] [Equation]
where F s ( A , Z ) is the SNL operator, W 1 ∈ R C s × C 1 , W 2 ∈ R C s × C 1 .
Finally, a residual connection is added with the SNL operator to form the SNL block:
[Equation]
Relation with other nonlocal operators As shown in ﬁg. 1, our SNL operator can degrade into the
NL operator by setting W 1 = 0 , i.e. θ 0 = 0 .
However, its analytic solution: θ 0 = 2 N (cid:80) Nj =0 ω j controls the total ﬁltering intensity, which cannot be guaranteed to be 0.
This setting will limit the search space when training the network and reduce the robustness of the NL block.
The NL operator cannot magnify features of a large range and damp some discriminative features such as the beak of the waterfowl.
Our SNL operator can also degrade into the NS operator by setting W 1 = − W 2 , i.e. θ 1 + θ 0 = 0 .
However, the analytic solution of this equation is θ
[Equation]
When setting it to zero, the ﬁlter strength of the high-frequency signal (with high λ ) such as the small part or twig is suppressed.
Thus, it still cannot magnify the discriminative part such as the beak of the waterfowl as shown in ﬁg. 1.
Comparing with NL and NS, our SNL does not have these irrational constraints and give these two parameters a liberal learning space.
Thus, θ 0 can control the preserve strength of the discriminative features, while θ 1 can pay more attention to the low-frequency signal to diminish the noise.
3. T HE PROPOSED GENERALIZED S PECTRAL N ONLOCAL O PERATOR
To fully exploit the “full-range” dependencies, the nonlocal block should have the ability to be consecutively stacked into the network to form a deeper nonlocal structure.
However, some types of nonlocal blocks such as the NL and CGNL block cannot achieve this purpose (Tao et al.
To show the robustness of our SNL block when used in the deeper nonlocal structure, we ﬁrstly study the steady-state of deeper nonlocal structure when consecutively adding our SNL block.
We also prove the stable hypothesis that the deeper nonlocal structure tends to learn a stable afﬁnity.
Based on this hypothesis, we can extend our SNL block into a full-order Chebyshev approximation, i.e. the gSNL block which is more applicable for deeper nonlocal structure.
The stable hypothesis The Steady-state analysis can be used to analyze the stable dynamics of the nonlocal block.
Here we give the steady-state analysis of our SNL block when consecutively adds into the network structure and get the Stable Hypothesis:
Lemma 1.
The Stable Hypothesis: when adding more than two consecutively-connected SNL blocks with the same afﬁnity matrix A into the network structure, these SNL blocks are stable when the variable afﬁnity matrix A satisﬁes: A k = A .
Proof.
The stability holds when the weight parameters in W 1 , W 2 and W are small enough such that the CFL condition is satisﬁed (Tao et al.
So we ignore them for simplicity.
The discrete nonlinear operator of our SNL have a similar formulation as the NS operator:
[Equation]
where h is the discretization parameter.
Z N is the input of the N th block in the deeper nonlocal structure with Z 0 =
X .
The stable assumption demands that Z N +
=
Z N , so the steady-state equation of the last SNL block can be written as:
[Equation]
The deeper nonlocal structure has more than one SNL blocks.
So the Z N − 1 and L h Z N − 1 can be used to express Z N :
[Equation]
Finally, the steady-state equation becomes:
[Equation]
This equation can naturally extend to the k-hop afﬁnity matrix A k , i.e. A k = A .
To verify the stable hypothesis, we add ﬁve consecutively-connected SNL blocks (and NS blocks) into the PreResnet56 He et al.
and train this model on the train set of the CIFAR100 dataset with the initial learning rate 0 .
1 which is subsequently divided by 10 at 150 and 250 epochs (total 300 epochs).
A weight decay 1 e − 4 and momentum 0 .
9 are also used.
Then we test the trained model on the test set and output the afﬁnity matrix of each image.
Figure.
2 shows the statistics that reﬂects the strength of the afﬁnity matrix, 2-hop, 3-hop, and 4-hop afﬁnity matrix:
A , A 2 , A 3 , A 4 .
We can see that the number of elements in each histogram bin are nearly the same.
This means that the A , A 2 , A 3 , A 4 have similar distribution of all the elements in k-hop afﬁnity matrixes, which also empirically veriﬁes the stable-state equation: A k = A .
Full-order spectral nonlocal operator
With the stable hypothesis, the Chebyshev polynomials can be simpliﬁed into a piece-wise function (details in Appendix B).
Taking this piece-wise function into the Eq.
7, we can get the full-order approximation of the SNL operator:
[Equation]
where
[Equation]
θ
[Equation]
1 , whose upper bound is less than 1 .
Then, extending it into multi-channel input and output with the residual connection, we can get our gSNL block:
[Equation]
The gSNL block is well-performed when the stable afﬁnity hypothesis is satisﬁed, i.e. adding more than two nonlocal blocks with the same afﬁnity matrix as shown in Table.
3. I MPLEMENTATION D ETAILS
The implementation details of the gSNL block is shown in ﬁg. 3.
The input feature map X ∈ R W × H × C 1 is ﬁrst fed into three 1x1 convolutions with the weight kernel: W φ ∈ R C 1 × C s , W ϕ ∈
R C 1 × C s , W g ∈ R C 1 × C s to subtract the number of channel.
One of the output Z ∈ R W × H × C s is used as the transferred feature map to reduce the calculation complexity, while the other two output Φ ∈ R W × H × C s , Ψ ∈ R W × H × C s are used to get the afﬁnity matrix A .
The sub-channel C s are usually two times less than the input channel C 1 .
The afﬁnity matrix is calculated by the afﬁnity kernel function f ( · ) and then use the operation in Sec3.1 to make it non-negative, symmetric and normalized.
Finally, with the afﬁnity matrix A and the transferred feature map Z , the output of the nonlocal block can be obtained by the equation Eq.
Speciﬁcally, the three weight matrixes
W 1 ∈ R C s × C 1 , W 2 ∈ R C s × C 1 , W 3 ∈ R C s × C 1 are implemented as three 1x1 convolutions.
E XPERIMENT 4. S ETTING
Datasets Our proposed SNL and gSNL blocks have been evaluated across several computer vision tasks, including image classiﬁcation and video-based action recognition.
For the image classiﬁcation, both CIFAR-10 and CIFAR-100 datasets (Krizhevsky & Hinton (2009)) are tested.
The CIFAR10 dataset contains 60 , 000 images of 10 classes, and CIFAR-100 dataset contains 60 , 000 images of 100 classes.
For these two datasets, we use 50 , 000 images as the train set and 10 , 000 images as the test set.
We also generate experiments for the ﬁne-grained classiﬁcation on the Birds-200- (CUB-200) dataset (Welinder et al.
(2010)) which contains 11 , 788 images of 200 bird categories.
For the action recognition, the experiments are conducted on the UCF-101 dataset (Soomro et al.
(2012)), which contains 101 different actions.
Backbones For the image classiﬁcation, the ResNet-50 and the PreResNet variations (including both
PreResNet-20 and PreResNet-56) are used as the backbone networks.
For the video classiﬁcation task, we follow the I3D structure (Hara et al.
(2018)) which uses k × k × k kernels to replace the convolution operator in the residual block.
Setting for the network In the main experiments, we set C s =
C 1 / 2 .
Without loss of the generality, we use the “Dot Product” as the afﬁnity kernel in the experiments.
We add one SNL (or gSNL) block into these backbone networks to construct the SNL (or gSNL) network.
For the ResNet and the I3D (Hara et al.
(2018)), following Wang et al.
we add the SNL block right before the last residual block of res 4 .
For the PreResNet series, we add the SNL block right after the second residual block in res 1 .
For the other nonlocal-base block including the NL (Wang et al.
the
NS (Tao et al.
(2018)), the Compact Generalized Nonlocal Block (CGNL)
(Yue et al.
and
the
Double Attention Block (A2), the settings are all the same as ours.
The difference of these blocks are shown in Table.
1, in which the Approximated Condition shows the strategy for the Chebyshev approximation and Channel-wise reﬂect the consideration of the channel relations.
Setting for the training For the image classiﬁcation on CIFAR-10 dataset and CIFAR-100 dataset, we train the models end-to-end without using pretrained model.
The initial learning rate 0 .
1 is used for these two datasets with the weight decay 1 e − 4 and momentum 0 .
The learning rate is divided by 10 at 150 and 250 epochs.
The models are trained for total 300 epochs.
For the ﬁne-grained classiﬁcation on CUB-200 dataset, we use the models pretrained on ImageNet (Russakovsky et al.
to initialize the weights.
We train the models for total 200 epochs with the initial learning rate 0 .
1 which is subsequently divided by 10 at 31 , 61 , 81 epochs.
The weight decay and momentum are the same as the setting of CIFAR-10 and CIFAR-100.
For the video classiﬁcation on the UCF-101 dataset, the weights are initialized by the pretrained I3D model on Kinetics dataset (Kay et al.
We train the models with the initial learning rate 0 .which is subsequently divided by 10 each 40 epochs.
The training stops at the 100 epochs.
The weight decay and momentum are the same as the setting of CIFAR-10 and CIFAR-100.
4. A BLATION E XPERIMENT
[Table]
The number of channels in transferred feature space The nonlocal-based block ﬁrstly reduces the channels of original feature map C 1 into the transferred feature space C s by the 1 × 1 convolution to reduce the computation complexity.
When C s is too large, the feature map will contain redundant information which introduces the noise when calculating the afﬁnity matrix A .
However, if C s is too small, it is hard to reconstruct the output feature map due to inadequate features.
To test the robustness for the number of the C s , we generate three types of models with different number of the transferred channels with the setting: “Sub 1” ( C s = C 1 ), “Sub 2” ( C s = C 1 2 ), “Sub 4” ( C s = C 1 4 ) as shown in Table.
Other parameters of the models and the training steps are the same as the setting in Sec.4.1.
Table.
2 shows the experimental results of the three types of models with different nonlocal blocks.
Our SNL and gSNL blocks outperforms other models proﬁted by their ﬂexible for learning.
Moreover, from Table.
2, we can see that the performances of the CGNL steeply drops when the number of the transferred channels increases.
This is because the CGNL block concerns the relationship between channels, when the number of the sub-channel increases, the relationship between the redundant channels seriously interferes its effects.
Overall, our proposed nonlocal block is the most robust for the large number of transferred channels (our model rise 1 .
1% in Top1 while the best of others only rise 0 .
4% compared to the baseline).
[Table] [Table]
The stage for adding the nonlocal blocks The nonlocal-based blocks can be added into the different stages of the preResNet (or the ResNet) to form the Nonlocal Net.
In Tao et al.
, the nonlocalbased blocks are added into the early stage of the preResNet to catch the long-range correlations.
Here we experiment the performance of adding different types of nonlocal blocks into the three stages (the ﬁrst, the second and the third stage of the preResNet) and train the models on CIFAR dataset with the same setting discussed in Sec.5.2.
The experimental results are shown in Table.
We can see that the performances of the NL block is lower than the backbones when adding into the early stage.
However, our proposed SNL block has 0 .
81% improvement compared with the backbone when respectively adding into all the three stages, which is much higher than the other type nonlocal blocks (only 0 .
42% for the best case).
To intuitively show the stability and robustness of our SNL, we give the spectrum analysis for the estimated weight matrices (Tao et al.
We extract the self-attention weight matrix: W g , W of the NL block and the NS block, W g , W 2 of our proposed SNL block.
The dimension of the weight matrix satisﬁes: W g ∈ R C 1 × C s , W ∈ R C s × C 1 W 2 ∈ R C s × C 1 .
To make all the eigenvalues real, we let: (cid:102) W
=
( W g W )
+( W g W )
T.
We do the same to the W 2 .
Figure.
5 shows the top thirtytwo eigenvalues of the weight matrix of (cid:102) W on the models in Table.
We can see that the density of the negative eigenvalues is higher than the positive eigenvalues of the NL block when adding into all three stages.
This phenomenon makes the NL operator F ( A , Z ) in Eq.
(1) less than zero.
So the output feature map is less than the input feature map, i.e. Y < X (more detail of this phenomenon can be seen in Tao et al.
The NS block can avoid “the damping effect” to some extent by concerning the diffusion nature.
However, when adding into the early stage, only six eigenvalues of the nonlocal stage are not equal to zero.
This phenomenon makes the nonlocal stage cannot effectively magnify the discriminated feature.
Comparing with these two models, our proposed SNL block has more positive eigenvalues which takes effect to enhance the discriminated features and also avoids the “damping effect”.
The number of the nonlocal blocks We test the robustness for adding multiple nonlocal blocks into the backbone network which forms the three type network “Different Position 3 (DP 3)”, “Same
Position 3 (SP 3)”
“Same Position 5 (SP 5)” as shown in Table. 4.
The result are shown in Table.
For the model “DP3”, three blocks are added into the stage 1 , stage 2 , and stage 3 (right after the second residual block).
We can see that adding three proposed nonlocal operators into different stages of the backbone generate a larger improvement than the NS operator and NL operator ( 2 . 4% improvement).
This is because when adding NS and NL into the early stage, these two models cannot better aggregate the low-level features and interfere the following blocks.
For the model “SP 3” (“SP 5”), we add three (ﬁve) consecutively-connected nonlocal blocks into the stage 1 .
Note that different from the experiment in Tao et al.
(2018) and Wang et al.
, these consecutivelyconnected nonlocal blocks have the same afﬁnity matrix.
From Table.
4, we can see that proﬁted by concerning the stable hypothesis discussed in Sec 3.3, our gSNL outperform all other models when adding consecutively-connected nonlocal blocks (rises average 0 .
72 % to the backbone and 0 .
41 % higher than the best performance of other type nonlocal blocks) and has a relatively stable performance.
However, one drawback is that our gSNL may interfere the learning when adding only one nonlocal block (the stable hypothesis is not satisﬁed).
4. M AIN R ESULTS
We test the networks with the Nonlocal Block (NL), the Nonlocal Stage (NS), the Compact Generalized Nonlocal block (CGNL), the Double Attention Block (A2) and our SNL (gSNL) blocks in the different visual learning tasks.
The experiment settings are discussed in Sec.4.1.
Our models outperform other types of the nonlocal blocks across several standard benchmarks.
Table.
11 shows the experimental results on the CIFAR10 dataset, we can see that by adding one proposed block, the
Top1 rises about 0 .
65% , which is higher than adding other type nonlocal blocks (0.3%).
As the experiments on CIFAR100 dataset shown in Table.
13, using our proposed block brings improvement about 1 . 8% with ResNet50.
While using a more simple backbone PreResnet56, our model can still generate 1 . 1% improvement as shown in Table.
Table.
9 shows the experimental results on the ﬁne-grained image classiﬁcation task on CUBdatasets.
Our model outperforms other non-channel-concerning blocks and generate ( 0 .
im-
[Table]
provement.
Comparing with the channel-wise concerning CGNL block, our model is only a bit lower in Top1.
Fig.
4 also shows the visualized feature map which is formed by adding the upsampled feature output with the source image.
We can see that the feature maps of our proposed block can cover more critical area of the birds.
For example, both the left and right wings (red square) of the birds can be focused proﬁted by the better long-range concerning of our SNL.
Moreover, beneﬁted from the ﬂexibility of the W 1 , our proposed SNL can also catch a relatively large range of the discriminative parts.
Table.
8 shows the experimental results on the action recognition task.
The network with our proposed block can generate 1 . 8% improvement than the I3D model and outperforms all other nonlocal models on the UCF-101 dataset.
[Table]
[Table] [Table]
[Table] [Table]
C ONCLUSION
In this paper, we explain the nonlocal block in the graph view and propose the spectral nonlocal (SNL) block which is more robust and well-behaved.
Our SNL block is a generalized version of the NL and NS block and having more liberty for the parameter learning.
We also give the stable hypothesis for deeper nonlocal structure and extend the SNL to gSNL that can be applied to the deeper nonlocal structures.
The experiments on multiple computer vision tasks show the high robustness and performance of our proposed nonlocal block.
Feature works will focus on using the
SNL block into different vision task and its roubustness for the other type of neural network such as the Generative Adversarial Networks (GAN).
A A NALYTIC S OLUTION OF THE C HEBYSHEV A PPROXIMATE
Here we give the analytic solution for the coefﬁcients in Chebyshev polynomials (Phillips (2003)):
Theorem 2.
Giving a function f ( x ) , x = { x 1 , x 2 , ..., x N } , it can be optimally approximated by Chebyshev polynomials: f ( x ) ≈ (cid:80) K − 1 k =
a k T k ( x ) , only when a k satisﬁes: a k = 2 N (cid:80) Nj =0 f ( x j ) T k ( x j ) .
We call the a k as the analytic solution of the Chebyshev coeffcients.
Based on these theorem, we can get the analytic solution of the parameter θ for Eq.
Lemma 2.
The spectral nonlocal operator can be best approximated when the function g ( λ ) =
ω can be best approximated by the Chebyshev polynomials, i.e. the analytic solutions of the Chebyshev coeffcients satisfy:
[Equation]
B T HE P IECEWISE C HEBYSHEV P OLYNOMIALS
Taking A k
=
A into the Chebyshev polynomials of the afﬁnity matrix A , the Chebyshev polynomials becomes:
[Equation]
This cyclic form of Chebshev polynomials T k
( A ) can be reformulated as a piecewise function:
[Equation]
C E XPERIMENT OF S EMANTIC S EGMENTATION ON VOC2012 D ATASET
For the semantic segmentation tasks, we generate experiment on the VOC2012 dataset with the model proposed by Chen et al.
(2017).We add different types of nonlocal blocks on right before the last residual block in res 4 of the ResNet50.
The models are trained for 50 epochs with the SGD optimize algorithm.
The learning rate is set 0 .
007 with the weight decay 5 e − 4 and momentum 0 .
Experimental results show that the model with our proposed block can the best results.
[Table]
D T HE E XAMPLE OF THE AFFINITY MATRIX ON CUB DATASETS
Experiments to verify the stable hypothesis is also generated on the CUB datasets, we add three consecutively-connected SNL blocks (and NS blocks) into the ResNet50 (right before the last residual block of res 4 ) and train this model on the train set of the CUB dataset with the initial learning rate 0 .
1 which is subsequently divided by 10 at 31 , 61 and 81 epochs (total 200 epochs).
A weight decay 1 e − 4 and momentum 0 .
9 are also used.
Figure.
6 shows the histogram of the strength statistics of the afﬁnity matrix A .
We can see that although using different backbone and dataset, the distribution of the k-hop afﬁnity matrixes are corresponded with the experiments on CIFAR100.
E E XPERIMENTS ON V IDEO - BASED P ERSON R E - IDENTIFICATION
Experiments are also conducted on the challenging datasets on Video-based Person Re-identiﬁcation task including the Mars, ILID-SVID and PRID2011.
For the backbone, we follow the strategy of
Gao & Nevatia (2018) that uses the pooling (RTMtp) and attention (RTMta) to fuse the spatialtemporal features.
Note that the models are totally trained on ilidsvid and prid2011 rather than ﬁnetuning the pre-trained model on Mars dataset.
The experimental results are shown in Table.11, 12, 13.
We can see that in these datasets, our proposed block can still generate consistent improvements.
[Table] [Table]
[Table]
F A DDITIONAL E XPERIMENTS ON A CTION C LASSIFICATION
Ours SNL can also improve the performance of other network structures such as the Pseudo 3D Convolutional Network (P3D) (Qiu et al. (2017)), the Motion-augmented RGB Stream (MARS) (Crasto et al. (2019)), the Slow-Fast Network (Slow-Fast)
(Feichtenhofer et al.
and the Video Transformer Network (VTN) (Kozlov et al. For P3D and MARS, our SNL block is inserted right before the last residual layer of the res3.
For the Slow-Fast, we replace its original NL block with our SNL block.
For the VTN, we replace its multi-head self-attention blocks (paralleledconnected NL blocks) with our SNL blocks.
The Slow-Fast network are trained end-to-end on the
UCF-101 dataset while others use the model pretrained on Kinetic400 dataset and ﬁnetuning on the
UCF-101 dataset.
From Table.
14, We can see that all the performances are improved when adding our proposed SNL model.
Experiments on Kinetics-400 dataset are also given in Table.
We can see that inserting SNL block into the Slow-Fast Network can generate 2.1% improvement.
[Table] [Table]