S IAM CAN:S IMPLE YET E FFECTIVE M ETHOD TO ENHANCE S IAMESE S HORT -T ERM T RACKING
Anonymous authors
Paper under double-blind review
A BSTRACT
Most traditional Siamese trackers are used to regard the location of the max response map as the center of target.
However, it is difﬁcult for these traditional methods to calculate response value accurately when face the similar object, deformation, background clutters and other challenges.
So how to get the reliable response map is the key to improve tracking performance.
Accordingly, a simple yet effective short-term tracking framework (called SiamCAN),by which bridging the information ﬂow between search branch and template branch, is proposed to solve the above problem in this paper.
Moreover, in order to get more accurate target estimation, an anchor-free mechanism and specialized training strategy are applied to narrow the gap between the predicted bounding box and groundtruth.
The proposed method achieves state-of-the-art performance on four visual tracking benchmarks including UAV123, OTB100, VOT2018 and VOT2019, outperforming the strong baseline, SiamBAN, by 0.327 → 0.331 on VOT2019 and 0.
→ 0.638 success score, 0.833 → 0.850 precision score on UAV123.
I NTRODUCTION
Visual object tracking is the fundamental task of computer vision, aiming at tracking unknown object of which the information is given by the ﬁrst frame.
Although great progress has been achieved in recent years, a robust tracker is still in desperate demand due to tricky challenge such as scale variation, appearance deformation and similar object with complex background which can deteriorate tracking performance (Wu et al.
; Zhang et al.
Recently, Siamese Network based trackers have taken a vital place in SOT ﬁeld due to its accuracy and speed.
Since (Tao et al.
and (Bertinetto et al.
(2016)) introduced Siamese networks in visual tracking, Siamese structure has been adopted as baseline for researchers to design efﬁcient trackers (Li et al.
Zhu et al.
(2018a)
; Zhang & Peng (2019); Li et al.
; Xu et al.
Chen et al.
After siamRPN (Li et al.
(2018)) being proposed to gain more accurate anchor boxes, region proposal network has become an essential part of tracker.
However, the anchor scales are manual-set which go against the fact that the tracking target is unknown.
Besides, the performance of the Siamese based trackers depends greatly on ofﬂine training by using massive frame pairs.
Therefore, it highly increases the risk of tracking drift when facing signiﬁcant deformation, similar object distractors, or complex background, due to the undiscriminating feature learned from the target when the category of the target is excluded from the training dataset.
In these years, the attention mechanism has become the spotlight in computer vision which inspires the relative works not only in detection task but also in visual tracking (He et al.
; Abdelpakey et al.
; Wang et al.
; Zhu et al.
(2018b)).
The attention mechanism includes channel attention and spatial attention, the former tends to generate a set of channel-weights for modeling interdependencies between channels while the latter focuses on ﬁnding the informative part by utilizing the inter-spatial relationship of features.
Considering these beneﬁts, Siamese based trackers try to introduce attention module to distinguish target from complex background.
Nevertheless, the performance of these trackers is not satisfactory for exploiting the expressive power of the attention mechanism inappropriately.
Based on the limitations discussed above, we design a simple Cross-attention Guided Siamese network (SiamCAN) based tracker with anchor-free strategy which performs better than the state-ofthe-art trackers when facing the similar object challenge.
SiamCAN takes template channel attention to guide the feature extraction of search image by which can strengthen the ability of tracker to overcome distractors and complex backgrounds, performing better than most of Siamese-based trackers, as shown in Figure 1.
The main contributions of this work are:
• We formulate a cross-attention guided Siamese framework (SiamCAN) including crosschannel attention and self-spatial attention.
The cross-channel attention builds an interactive bridge between the target template and search frame to share the identical channel weights.
The self-spatial attention focuses on the discriminative part of the correlated feature map, which is complementary to the cross-channel attention.
• The proposed tracker is adaptive box regression, without numerous hyper-parameters setting.
In order to get more accurate bounding box, we adopt the proper strategy to utilize the merits of anchor-free at the stage of training.
• SiamCAN achieves state-of-the-art results on four large tracking benchmarks, including OTB100 (Wu et al.
, UAV123 (Mueller et al.
(2016)), VOT
(Kristan et al.
and VOT2019 (Kristan et al.
The speed of tracker can also achieveFPS.
R ELATED W ORK
In this section, we brieﬂy review the recent Siamese based trackers, the anchor-free approaches and attention mechanism in both tracking and detection ﬁled.
2. S IAMESE NETWORK BASED TRACKER
The pioneering works, SINT (Tao et al. (2016)) and SiamFC (Bertinetto et al.
, ﬁrst introduce the siamese network in tracking ﬁled.
Due to its fast speed with light structure, Siamese network draws great attention from the visual tracking community.
SiamFC tries to use siamese network to learn the feature of both target template and search frame, and compare the similarity of them to ﬁnd the most conﬁdent candidates.
Although tracks fast, it cannot handle the scale variation problem by applying several scales of feature map.
Inspired by Faster-RCNN (Ren et al.
from object detection, SiamRPN (Li et al.
(2018)) draws on the region proposal network to get more various scale ratio bounding boxes.
Since then, the RPN module has become an essential part of the tracker (Zhu et al.
(2018a)
; Zhang & Peng (2019); Li et al.
; Dong & Shen (2018);
Fan & Ling (2019)).
However, the complexity of anchor design makes the performance of trackers depend greatly on the effect of anchor training.
2. A NCHOR - FREE A PPROACHES
In recent time, Anchor-free approaches have developed fast.
The achor-free work can be divided into two categories.
The ﬁrst one (Kong et al.
; Law & Deng (2018)) aims to estimate the keypoints of the objects, while, the other (Redmon et al.
; Tian et al.
(2019))tends to predict the bounding box for each pixel which can avoid presetting the scale ratio of anchors.
Not only is anchor-free approach popular in detection ﬁeld, but it is suitable for target estimation in tracking ﬁeld due to its high efﬁciency.
SiamFC++ takes example by FCOS (Tian et al.
to design regression subnetwork and add centerness branch to eliminate the low quality samples.
SiamCAR (Guo et al.
(2020)) changes the basic network structure additionally, merging the multi-layers features before correlation.
Different from SiamCAR, SiamBAN (Chen et al.
(2020)) puts emphasis on the label assignment which improves the tracking performance.
Our method differs from the above trackers in details (Section4.3).
2. A TTENTION M ECHANISM
Attention mechanism has been the focus of the detection ﬁled, on account to its powerful ability of enhancing deep CNNs.
SE-Net
(Hu et al.
(2018)) ﬁrstly puts forward the mechanism to generate channel weights in return to direct the learning of channel attention.
After that, CBAM (Woo et al.
(2018)) utilizes both max-pooling and average-pooling to generate the merged attention, includes channel and spatial attention.
Recently, ECA-Net (Wang et al.
(2020b)) ﬁnds that avoiding dimensionality is of great importance for channel attention learning, and propose a cross-channel interaction strategy which performs better than SE-Net.
In the tracking ﬁeld, the recent trackers began to equip with the attention mechanism to get better performance.
SA Siam (He et al.
(2018)) simply combines the SE-Net and SiamFC to get both discriminative and general features which boost the tracking performance.
RASNet (Wang et al.
(2018)) designs residual attention, general attention and channel attention to learn target feature better.
SATIN (Gao et al.
(2020)) uses hourglass network as backbone and designs a cross-attention module for exemplar branch to combine the channel and spatial attention from shallow and deep layers.
However, these trackers only calculate alongside each branch and neglect the information ﬂow between them, as a result, the ability of attention mechanism can not be fully utilized.
O UR APPROACH
[Figure]
As shown in Figure 2, the proposed framework mainly consists of three components, the feature extracted Siamese network, the cross-attention module and anchor-free bounding box regression subnetwork with foreground classiﬁcation subnetwork.
3. F EATURE EXTRACTED S IAMESE N ETWORK
Like the most Siamese based tracker, we adopt the fully convolution network without padding, which guarantees the accurate location calculation.
Feature extracted network composes of two parts, template branch and search branch.
Both of them share the same layer parameters of backbone, by this mean, CNNs can learn the relative feature for them to calculate similarity in the subsequent operation.
The template branch intends to encode the exemplar feature in the ﬁrst frame while the other branch aims to encode the candidates feature which may involve target in the follow-up frames.
Set the input in template branch as I t , the subsequent frames in search branch as I s .
We feed the
I t and I s into backbone, and get the feature φ l ( I t ) and φ l ( I s ) from different l-th backbone layers.
Next, the given features are sent to the corresponding branch after having a convolution with a neck layer to reduce feature channel size to 256 and get the template feature ψ t ( I t ) with the search feature ψ s ( I s ) .
At last, crop the 7 × 7 patch from the center of template feature.
3. C ROSS - ATTENTION N ETWORK
Attention mechanism is created for the purpose that enforce CNNs to focus on the special parts which is of great importance, i.e., channels information and spatial information.
The channel attention is designed to explore the interdependencies between channels while the spatial attention tends to make CNNs pay more attention to the most critical areas of the feature.
Different from (He et al.
; Wang et al.
, the channel attention is used between two branches not being applied as self-attention.
Moreover, SATIN (Gao et al.
(2020)) designs a module also called cross-attention , but the ’cross’ means the combination of different layers which is different from our method.
In this paper, the target branch feature ψ t ( I t ) is sent to global average pooling to get aggregated feature Y t , i.e.,
[Equation]
Given the aggregated feature, the channel weight is obtained by performing a 1D convolution of size k, i.e.,
[Equation]
Where σ is a Sigmoid function, ω indicates the parameters of 1D convolution and Ω ki indicates the set of k adjacent channels of y i .
To let search branch learns the information from target template, we multiply the channel weights with the search feature, i.e.,
[Equation]
3. C LASSIFICATION AND ANCHOR - FREE REGRESSION SUBNETWORK
As shown in Figure 2, the correlation feature map is calculated by the depth-wise correlation operation between (cid:102) ψ s ( I s ) and ψ t ( I t ) , i.e.,
[Equation]
where (cid:63) denotes depth-wise convolution operation.
Then, we apply self-spatial attention to the feature map in order to focus on discriminative part automatically, i.e.,
[Equation] [Equation]
[Equation]
After that, we use two convolution layers with kernel size 1 × 1 to reduce the number of channel from 256 to 2 and 4 respectively for each branch and concatenate the feature maps from different layers of backbone by the trainable weights α , i.e.,
[Equation] [Equation]
where N denotes the total number of the backbone layers we use.
The classiﬁcation feature map has two channels, the one represents the foreground and the points ( i, j ) on P clsw × h × 2 (0 , i, j ) refer to the probability scores of target, the other represents the background and the points ( i, j ) on
P clsw × h × 2 (1 , i, j ) refer to the probability scores of background.
The regression feature map has four channels, each of them represents the four direction distances from the points location in search branch input to the four sides of the bounding box respectively, that is to say, each point ( i, j ) in
P regw × h × 2 (: , i, j ) is a vector which can denoted as ( l, r, t, b ) .
Classiﬁcation label and regression label.
For anchor based methods, the positive sample and the negative one are classiﬁed by the value of Intersection over Union between anchor and groundtruth.
In this paper, We use ellipse and circle ﬁgure region to design label for points ( i, j ) in feature map, which is inspired by (Chen et al.
The ellipse E 1 center and axes length are set by groundtruth center ( g x c , g y c ) of groundtruth size ( g w 2 , g h 2 ) , We also get the circle C 2 with 0 .
5 g w ∗ 0 .
5 g h
[Equation] [Equation]
radius
[Equation]
i.e.
[Equation]
[Equation]
where B denotes the calculation for the location of points ( i, j ) in feature map P clsw × h × 2 back to the search frame.
If the point B ( p i , p j ) falls within the C 2 region, it will be given a positive label, and if it falls outside the E 1 area, it will be given a negative label, i.e.,
[Equation]
For regression branch, the regression targets can be deﬁned by:
[Equation]
[Equation]
14) where ( g x 0 , g y 0 ) , ( g x 1 , g y 1 ) denote the left-top and right-bottom coordinates location of the groundtruth.
Loss function.
We employ cross entropy loss to train the classiﬁcation network.
To predict more accurate bounding box, we adopt the DIoU loss (Zheng et al.
to train the regression network, i.e.,
[Equation]
(15) where ρ ( . ) is the Euclidean distance, p and p gt denote the central points of predicted box and groundtruth and c is the diagonal length of the smallest enclosing box covering the two boxes.
For regression branch training, DIoU loss can optimize the bounding faster than GIoU loss (Rezatoﬁghi et al.
The overall loss function is:
[Equation]
16) where constants λ 1 and λ 2 weight the classiﬁcation loss and regression loss.
During model training, we simply set λ 1 = 1 , λ 2 = 1 without hyper-parameters searching.
3. T RAINING AND INFERENCE
Training.
We train our model by using image pairs, a 127 × 127 pixels template patch and a 255 ×pixels search patch.
The training datasets include ImageNet VID (Russakovsky et al.
, COCO (Lin et al.
, YouTube-Bounding Boxes (Real et al.
(2017)), ImageNet Det (Real et al.
and GOT10k (Huang et al.
Due to the numbers of negative samples are more than the positive samples, we set at most 16 positive samples and 48 negative samples from the search image.
Besides, in order to get more accurate regression information, we adopt the DIoU loss to optimize the regression branch.
Inference.
We feed the cropped ﬁrst frame and the subsequent frames to the feature abstract network as template image and search image.
Next, the features are sent to the cross-attention module and pass the classiﬁcation branch and regression branch.
After that, we get the classiﬁcation map.
The location of the highest score represents the most probable center of the tracking target.
Then, we use the scale change penalty and the cosine window as that introduced in (Li et al.
to guarantee the smooth movements of target.
According to the location p of the ﬁnal score, we can get the predicted boxes B , i.e.,
[Equation]
where d regl,r,t,b denotes the predicted values of the regression targets on the regression map, ( b x 1 , b y 1 ) and ( b x 2 , b y 2 ) are the top-left corner and bottom-left corner of the predicted box.
E XPERIMENTS 4. I MPLEMENTATION DETAILS
Our approach is implemented in Python with Pytorch on 1 RTX2080Ti.
The backbone is modiﬁed
ResNet-50 as in (He et al. (2016)), and the weights are pre-trained on ImageNet (Russakovsky et al.
During the training phase, the model is optimized by the stochastic gradient descent (SGD), at the meantime, the total number of epochs is 20 and the batch size is set as 28.
For the ﬁrstepochs, we frozen the parameters of the backbone and only train the heads structures, for the lastepochs, we unfrozen the last 3 blocks of backbone to be trained together.
Besides, we warm up the training during the ﬁrst 5 epoch with a warmup learning rate of 0.001 to 0.005, and in the lastepochs, the learning rate exponentially decayed from 0.005 to 0.00005.
4. R ESULTS ON THREE BENCHMARKS
To afﬁrm the effect of our method, we evaluate our tracker performance with the recent trackers
MAML (Wang et al.
(2020a))
, PriDiMP (Danelljan et al.
, SiamBAN (Chen et al.
SiamCAR (Guo et al.
, ROAM (Yang et al.
, SiamFC++ (Xu et al.
, STN (Tripathi et al.
, ARTCS (Kristan et al. (2019)), SiamRPN++ (Li et al.
(2019)),SATIN (Gao et al.
, ATOM (Danelljan et al.
, DIMP-18 (Bhat et al.
, DaSiamRPN (Zhu et al.
(2018a))
, SPM (Wang et al.
(2019))ECO (Danelljan et al.
, CFNet (Valmadre et al.
(2017)), SiamRPN (Li et al.
(2018)), DeepSRDCF (Danelljan et al.
(2015a))
, SRDCF (Danelljan et al.
(2015b))
on four benchmarks UAV123 (Mueller et al.
(2016)), VOT
(Kristan et al.
VOT2019 (Kristan et al.
and OTB100 (Wu et al.
(2013))(details in Appendix A.1).
4.2. R ESULTS ON UAV
UAV123 contains 123 challenging video sequences, which can be divided into 11 categories according to their attributes.
The performance of the tracker is evaluated by two evaluation metrics, the precision scores and the AUC scores.
The AUC scores reﬂect the overlap between the predict bounding box and ground-truth box, while the precision scores are relative with the distance between the center of the predict bounding box and ground-truth box.
In Figure 3, our method achieves the best performance in precision scores, i.e., 0.850, and the second best AUC scores 0.678.
As for the 11 categories of the challenge, SiamCAN ranks 1st or 2nd in Scale Variation, Similar Object, Fast
Motion and Low Resolution, in Appendix A.2.
The results demonstrate that our tracker can handle the similar object and scale change challenge, due to the learning of cross-attention subnetwork and anchor-free mechanism.
4.2. R ESULTS ON VOT
VOT2018 consists of 60 challenging videos.
The evaluation metrics to rank the performance of the tracker based on EAO (Expected Average Overlap) which depends on the accuracy and the robustness.
As shown in Table 1, our tracker outperforms SiamRPN++ by 2.8 points by introducing
[Table]
the anchor-free regression network.
Compared with SiamFC++, we achieve EAO improvements of 4.5 points.
Although our method fails to achieve the best EAO, the robustness scores of listed
Siamese-based trackers are higher than our method, which affect the EAO scores, in other words,
SiamCAN is a robust Siamese-based tracker.
4.2. R ESULTS ON VOT
VOT2019 video sequences are 20% different from VOT2018, adding more fast motion and similar object videos.
Table 2 reﬂects the evaluation results on VOT2019 compared with the recent trackers.
We can see that the recent proposed MAML gets the highest accuracy scores, while our SiamCAN surpassing MAML by 2.5 points in terms of EAO.
Besides, our robustness scores also rank 2nd.
4. A NALYSIS OF THE P ROPOSED M ETHOD
Discussion on effective sample selection.
Anchor-free method has the weakness that network may produce low-quality predicted bounding boxes far away from the center of target, even though the predicted box is accurate.
To address this issue, SiamFC++ and SiamCAR introduce centerness branch to get high quality sample, forcing the network focus on the target center.
While, SiamBAN uses ellipse ﬁgure region to design label which has the same effect.
Accordingly, we do several experiments to ﬁnd which method performs better.
As shown in Table 3, baseline tracker consists of cross-attention module and anchor-free network.
Ellipse label does better than Circle label ( 2 (cid:13) vs 1 (cid:13) ),while the centerness branch with ellipse label even have worse effects ( 2 (cid:13) vs 3 (cid:13) ).
Based on
[Table] [Table]
[Table]
the performance of 2 (cid:13) , we visualize the tracking of 2 (cid:13) and compare with 4 (cid:13) , details in Figure 4.
At the training stage, 2 (cid:13) gives the positive label to the points fall within E 2 region, while 4 (cid:13) gives the positive label to the points fall within the C 2 region, the more central position.
In this aspect, the comparision in Table 3 ( 2 (cid:13) vs 4 (cid:13) ) can be explained.
[Figure]
Discussion on components of tracker.
To verify the role of each components of our tracker, we use the results of components on VOT2018 to analyze the details.
As shown in Table 4, the baseline model can get an EAO of 0.351, which consists of a regular backbone, i.e., ResNet50, a classiﬁcation and an anchor-free regression network.
Exchanging DIoU Loss for GIoU Loss during training can get higher scores, due to the more accurate predicted bounding box ( 2 (cid:13) vs 1 (cid:13) ).
Adding crossattention module can obtain a large improvement, i.e., 4.3 points on EAO ( 4 (cid:13) vs 2 (cid:13) ).
This demonstrates the information interaction between the template branch and the search branch is of great signiﬁcance.
Finally, the tracker utilizes the self-spatial attention can gain another improvement of 2.4 points on EAO ( 4 (cid:13) vs 5 (cid:13) ).
Feature visualization.
We visualize the features extracted from tracker 1 (cid:13) , tracker 3 (cid:13) and tracker 5 (cid:13) in Figure 5.
On the left side of Figure 5, tracker 1 (cid:13) vs tracker 3 (cid:13) shows the tracker not equipped with cross-attention module is easier to lose target and focus on the wrong object when appear similar object and the visualized feature demonstrates that cross-attention module can enable tracker to tell target from similar distactors.
On the right side of Figure 5, tracker 5 (cid:13) shows the power of crossattention module combined with proper training strategy.
[Figure]
①
[Figure]
C ONCLUSION
In this paper, we propose a simple Siamese-based tracker called SiamCAN, which combined with cross-attention and anchor-free mechanism.
The cross-attention module utilizes the target template channel attention to guide the feature learning of search frame, bridging the information ﬂow between each branch.
The anchor-free regression discards the fussy design of the anchors, and adjusts the scale ratio of the bounding box automatically.
In order to use them to their fullest potential, choosing appropriate label assignment strategy and suitable loss funtion to boost the tracking performance with limited laboratory equipments.
Extensive experiments are conducted on four benchmarks, which demonstrate our trackers with light structure yet achieves the state-of-the-art performance, especially in scale variation, background cluster, deformations and similar distractors challenges.
A A PPENDIX A. E XPERIMENT RESULTS ON OTB
OTB100 contains 100 challenging video sequences, and the evaluation metrics are same with
UAV123.
In Figure 5, our method achieves the best performance in precision scores, i.e., 0.913, and the second best success scores 0.684.
As for the 9 categories of the challenge, SiamCAN ranks 1st or 2nd in Deformation, Background Clutters, Scale Varation and Out-of-Plane Rotation.
The results demonstrate that our trackers can handle the deformation, background clutters, scale variation and out-of-plane rotation, in Figure 6.
A. M ORE EXPERIMENT RESULTS ON UAV