SwinTrack: A Simple and Strong Baseline for Transformer Tracking
Abstract
Recently Transformer has been largely explored in tracking and shown state-of-theart (SOTA) performance.
However, existing efforts mainly focus on fusing and enhancing features generated by convolutional neural networks (CNNs).
The potential of Transformer in representation learning remains under-explored.
In this paper, we aim to further unleash the power of Transformer by proposing a simple yet efficient fully-attentional tracker, dubbed SwinTrack , within classic Siamese framework.
In particular, both representation learning and feature fusion in SwinTrack leverage the Transformer architecture, enabling better feature interactions for tracking than pure CNN or hybrid CNN-Transformer frameworks.
Besides, to further enhance robustness, we present a novel motion token that embeds historical target trajectory to improve tracking by providing temporal context.
Our motion token is lightweight with negligible computation but brings clear gains.
In our thorough experiments,
SwinTrack exceeds existing approaches on multiple benchmarks.
Particularly, on the challenging LaSOT, SwinTrack sets a new record with 0.713 SUC score.
It also achieves SOTA results on other benchmarks.
We expect SwinTrack to serve as a solid baseline for Transformer tracking and facilitate future research.
Our codes and results are released at https://github.com/LitingLin/SwinTrack .
Introduction
Visual tracking has seen considerable progress since deep learning.
In particular, the recent Transformer [38] has significantly pushed the state-of-the-art in tracking owing to its ability in modeling long-range dependencies.
However, existing methods usually leverage Transformer for fusing and enhancing features generated from convolutional neural networks (CNNs), e.g. , ResNet [18].
The potential of exploiting Transformer for feature representation learning is largely under-explored.
Recently, Vision Transformer (ViT)
[9] has exhibited great potential in robust feature representation learning.
Particularly, its extension Swin Transformer [28] has achieved state-of-the-art (SOTA) results on multiple tasks.
Taking inspiration from this, we argue, besides the feature fusion, the representation learning in tracking can also benefit from Transformer via attention.
Thus motivated, we propose to develop a fully attentional tracking framework based on Siamese architecture.
Specifically, both the feature representation learning and the feature fusion of template and search region are realized by Transformer.
More concretely, we borrow the architecture of the powerful
[Figure]
cce ss
[Figure]
c o r e (
[Figure]
SwinTrack
[Figure]
384 (ours)
SwinTrack
[Figure] [Figure]
STARK
[Figure]
Swin Transformer [28] and adapt it to Siamese tracking.
Note that, other Transformer architectures can be used.
For feature fusion, we introduce a simple homogeneous concatenation-based fusion architecture, without a query-based decoder.
Moreover, taking into consideration that tracking is a temporal task, we propose a novel motion token to improve robustness.
Inspired by that the target usually moves smoothly in a short period, motion token is represented by the historical target trajectory within a local temporal window.
We incorporate the (single) motion token in the decoder of feature fusion to leverage motion information during tracking.
Despite being conceptually simple, our motion token can effectively boost tracking performance, with negligible computation.
We name our framework SwinTrack.
As a pure Transformer framework, SwinTrack enables better interactions inside the feature learning of template and search region and their fusion compared to pure CNN-based [1, 25] and hybrid CNN-Transformer [5, 40, 46] frameworks, leading to more robust performance (see Fig. 1).
Fig.
2 demonstrates the architecture of SwinTrack.
We conduct extensive experiments on five large-scale benchmarks to verify the effectiveness of SwinTrack, including LaSOT [11], LaSOT ext [10], TrackingNet [33], GOT-10k [19] and TNL2k [42].
On all benchmarks, SwinTrack achieves promising results and meanwhile runs fast at 45 fps .
In particular, on the challenging LaSOT, SwinTrack sets a new record of 71.3 SUC score, surpassing the strongest prior tracker [46] (to date) by 3.1 absolute percentage points and crossing the 0.7 SUC threshold for the first time (see Fig.
1 again).
It also achieves 49.1 SUC, 84.0 SUC, 72.4 AO and 55.9 SUC scores on LaSOT ext , TrackingNet, GOT-10k and TNL2k respectively, which are better than or on par with state-of-the-arts (SoTAs).
In addition, we provide a lighter version of SwinTrack that obtains comparable results to SoTAs but runs much faster at around 98 fps .
In summary, our contributions are as follows: ( i )
We propose SwinTrack, a simple and strong baseline for fully attentional tracking; ( ii ) We present a simple yet effective motion token, enabling the integration of rich motion context during tracking, further boosting the robustness of SwinTrack, with negligible computation; ( iii ) Our proposed SwinTrack achieves state-of-the-art performance on multiple benchmarks.
We believe SwinTrack further shows the potential of Transformer and expect it to serve as a baseline for future research.
Related Work
Siamese Tracking.
The Siamese tracking methods formulate tracking as a matching problem and aim to offline learn a generic matching function for this task.
The seminal method of [1] introduces
C
S w i n T r a n s f o r m e r
S t a g e s
S w i n T r a n s f o r m e r
S t a g e s
S w i n T r a n s f o r m e r
S t a g e s
S w i n T r a n s f o r m e r
S t a g e s
S w i n T r a n s f o r m e r
S t a g e s
S w i n T r a n s f o r m e r
S t a g e s
[Figure]
P a t c h P a r titi on P a t c h P a r titi on
[Figure]
Transformer in Vision.
Transformer [38] originates from natural language processing (NLP) for machine translation and has been introduced to vision recently and shows great potential.
The work of [3] first uses Transformer for object detection and achieved promising results.
To explore the capability of Transformer in representation learning, the work of [9] applies Transformer to construct backbone network, and the resulting Vision Transformer (ViT) attains excellent performance compared to convolutional networks while requiring fewer training resources, which encourages many extensions upon ViT[37, 4, 49, 41, 28].
Among them, the Swin Transformer [28] has received extensive attention.
It proposes a simple shifted window strategy to replace the fixed-patch method in
ViT, which significantly improves efficiency and meanwhile demonstrates state-of-the-art results on multiple image tasks.
Our work is inspired by Swin Transformer, but differently, we focus on the video task of visual tracking.
Transformer in Tracking.
Inspired by the success in other fields, researchers have leveraged
Transformer for tracking.
The method of [5] applies Transformer to enhance and fuse features in the Siamese tracking for improvement.
The approach of [40] uses Transformer to exploit temporal features to improve tracking robustness.
The work of [46] introduces a new transformer architecture dedicated to visual tracking, explores the Spatio-temporal Transformer by integrating the model updating operations into a Transformer module.
Our SwinTrack is related to but significantly different from the above Transformer-based trackers.
Specifically, the aforementioned methods mainly apply Transformer to fuse convolutional features and belong to the hybrid CNN-Transformer architecture.
Unlike them, SwinTrack is a pure Transformerbased tracking architecture where both representation learning and feature fusion are realized with
Transformer, enabling the exploration of better features for robust tracking.
Tracking via Vision-Motion Transformer
We present SwinTrack, a vision-motion integrated Transformer for object tracking, in Fig.
The proposed framework contains three main components, i.e. , the Swin-Transformer backbone for feature extraction, the encoder-decoder network for mixing vision-motion cues, and the head network for localizing targets.
In the following sections, we first shortly describe the Swin backbone network, then elaborate on the proposed vision-motion encoder-decoder.
Afterward, we give a discussion about our method and shortly describe the network head and training loss.
3.1 Swin-Transformer for Feature Extraction
The deep convolutional neural network has significantly improved the performance of trackers.
Along with the advancement of trackers, the backbone network has evolved twice: AlexNet [22] and
ResNet [18].
Swin-Transformer [28], in comparison to ResNet, can give a more compact feature representation and richer semantic information to assist succeeding networks in better localizing the target objects (demonstrate in the ablation study demonstrated in the ablation study), which is thus chosen for basic feature extraction in our model.
Our tracker, following Siamese tracking framework [1], requires a pair of image patches as inputs, i.e. , template image z ∈ R H
z × W z × 3 and search region image x ∈ R H x × W x × 3 .
As in the typical SwinTransformer procedure, template and search region images are divided to non-overlapped patches and sent to the network, which generates template tokens (dubbed T-tokens ) φ ( z ) ∈ R
Hzs Wzs × C and search region tokens (dubbed S-tokens ) φ ( x ) ∈ R Hxs Wxs × C .
s is the stride of the backbone network.
Since there is no dimension projection in our model, C is the hidden dimension of the whole model.
3.2 Vision-Motion Representation Learning
The essential step for matching-based visual tracking is injecting the template information into the search region.
In our framework, we adopt an encoder to fuse the features from the template and the search region, meanwhile, a decoder is arranged to achieve vision-motion representation learning, as illustrated in Fig.
Encoder for fusing template and search tokens.
The encoder contains a sequence of Transformer blocks where each consists of a multi-head self-attention (MSA) module and a feed-forward network (FFN).
FFN contains a two-layers multi-layer perceptron (MLP), GELU activation layer is inserted after the first linear layer.
Layer normalization (LN) is always conducted before every module (MSA and FFN).
Residual connection is applied to MSA and FFN modules.
Before feeding the features into the encoder, the template and search region tokens are concatenated along spatial dimensions to generate a mixing representation f m .
For each block, the MSA module computes self-attention over mixing union representation, which equals to separately conducting self-attention on T-tokens/S-tokens and meanwhile performing cross-attention between T-tokens and
S-tokens, but more efficient.
FFN refines the features generated by MSA.
When the tokens get out of the encoder, a de-concatenation operation is arranged to decouple the template and search region tokens.
The process of encoder can be expressed as:
[Equation]
where l denotes the l -th layer and L denotes the number of blocks.
Decoder for fusing vision and motion information.
Before describing the architecture of decoder, we first detail how to generate a motion token (dubbed M-token ).
Motion token is the embedding of the historical trajectory of the target object.
The past object trajectory is represented as a set of target object box coordinates, T = { o 1 , o 2 , ..., o t } , where t represents the frame index, o is the bounding box of target object.
o is defined by the top-left and bottom-right corners of the target object, denotes as o t =
( o x 1 t , o y 1 t , o x 2 t , o y 2 t ) .
For flexible modeling, a sampling process is required to ensure the following properties: 1) fixed length , 2) focusing on the latest trajectories and 3) reducing redundancy .
In our method, we sample object trajectory as:
[Equation]
n is the number of sampled object trajectories, ∆ is the fixed sampling interval.
For Siamese tracker, the search region is cropped from the input image.
In detail, a cropping with resizing operation can be used to describe the process.
Giving the point in the input image as ( x i , y i ) , the corresponding point in the search region as ( x o , y o ) , we can formulate the cropping process employed in pre-processing of the Siamese Tracker as x o =
( x i − i x ) s x + o x
and y o
=
( y i − i y )
s y
+ o y , where
( i x , i y ) is the center of the cropping window in the input image, ( s x , s y ) is the scaling factor, (
o x , o y ) is the center of cropped and scaled window in the search region.
We apply the same transformation on the sampled object trajectory to make the object trajectory invariant to the cropping, denoting ¯
T = { ¯ o s (1) , ¯ o s (2) , ..., ¯ o s ( n ) } as the result.
Then, to embed the transformed object trajectory into the network, we adopt four embedding matrices to embed the elements in box coordinates separately.
We denotes the embedding matrix as
W ∈ R ( g +1)
× d , g controls the embedding granularity of the object trajectory, d is the size of each embedding vector.
The last entry of the embedding matrix is used as the padding vector, indicating an invalid state, like object absence or out of the search region.
Thus, we normalize the sampled target object box coordinates in the range [1 , g ] , and quantize to integers to get the index of embedding vector:
[Equation]
where ˆ o
[Equation]
[Equation]
= [ n (¯ o
[Equation]
[Equation]
[Equation]
, n (
[Equation]
o
[Equation]
[Equation]
n
[Equation]
=
[Equation]
valid
[Equation]
where ( w, h ) is the size of search region feature map.
Finally, the motion token E motion ∈ R 1 × d is given by a concatenation of all box coordinate embedding of the sampled object trajectory.
FLOPs is negligible because the construction of motion token is just a composition of embedding lookups and token concatenation.
The decoder consists of a multi-head cross-attention(MCA) module and a feed-forward network(FFN).
The decoder takes the outputs from the encoder and the motion token as input, generating the final vision-motion representation f vm ∈ R Hxs × Wxs × C of by computing cross-attention over f Lx and
Concat( E motion , f Lz , f Lx ) .
The decoder is akin to a layer in the encoder, except that the correlation between the template tokens and the search tokens is dropped since we do not need to update the features from the template image in the last layer.
The process of the decoder is formulated as:
[Equation]
f vm will feed to the head network to generate a classification response map and a bounding box regression map.
Positional encoding.
Transformer requires a positional encoding to identify the position of the current processing token[38] because the self-attention module is permutation-invariance.
We adopt the untied positional encoding [20] as our positional encoding method.
The untied positional encoding enhances the expressiveness of the model through untie the positional embeddings from token embeddings with an isolated positional embedding matrix.
It also considers the case of special tokens, like the motion token in this paper.
We generalize the untied positional encoding to multi-dimensions multi-sources data to comply with concatenated-based fusion in our tracker.
See the appendix for the details.
3.3 Discussion
Why concatenated attention?
To simplify the description, we call the method described above concatenation-based fusion .
To fuse and process features from multiple sources, it is intuitive to perform self-attention on the feature from each source separately and then compute cross-attention across features from different sources.
We call this method cross-attention-based fusion .
Transformer makes fewer assumptions about the spatial structure of data, which provides great modeling flexibility.
In comparison to cross-attention-based fusion , concatenation-based fusion can save computation cost through operation sharing and reduce model parameters through weight sharing.
From the perspective of metric learning, weight sharing is an essential design to ensure the metric between two branches of data is symmetric.
Through concatenation-based fusion , we implement this property not only in the feature extraction stage but also in the feature fusion stage.
In general, concatenation-based fusion improves both efficiency and performance.
Why not window-based self/cross-
attention?
Since we select stage 3 of the Swin-Transformer as the output, the number of tokens involved is significantly reduced, window-based attention cannot save too many FLOPs.
Furthermore, considering the extra latency introduced by the window partition and window reverse operations, window-based attention may even be the slower one.
Why not a query-based decoder?
Derivated from vanilla Transformer decoder, many transformerbased models in vision tasks leverage a learnable query to extract the desired objective features from the encoder, like object queries in [3], target query in [46].
However, in our experiment, a query-based decoder suffers from slow convergence and inferior performance.
Most Siamese trackers [25, 44, 16] formulate tracking as a foreground-background classification problem, which can better exploit the background information.
The vanilla Transformer decoder is a generative model, the generative approaches are considered not suitable for the classification tasks.
In another aspect, learning a general target query for any kind of object might cause a bottleneck.
In terms of vanilla Transformer encoder-decoder architecture, SwinTrack is an "encoder" only model.
Furthermore, quite a little domain knowledge can be easily applied on a classic Siamese tracker to improve the performance, like introducing the smooth movement assumption by using Hanning penalty window on the response map.
Are other forms of motion token feasible?
Other forms to construct motion token are possible, such as constructing motion token by summing up the past box coordinate embeddings or representing past object trajectories by one token per box.
In our early experiments, we find that the proposed motion token is more effective with the best performance.
Summing up the past box coordinate embeddings may result in over-parameterization on the coordinate embeddings.
While adding temporal motion representation along with visual features to the single-layer decoder in a multi-token form is ineffective, precise temporal modeling may be required in this form.
3.4 Head and Loss
Head.
The head network is split into two branches: classification and bounding box regression.
Each of them is a three-layer perceptron.
And both of them receives the feature map from the decoder as input to predict the classification response map r cls ∈ R
( H x × W x ) × 1 and bounding box regression map r reg ∈ R ( H x × W x ) × 4 , respectively.
Classification loss.
In classification branch, we employ the IoU-aware classification score as the training target and the varifocal loss [50] as the training loss function.
IoU-aware design has been very popular recently, but most works consider IoU prediction as an auxiliary branch to assist classification or bounding box regression [52, 2, 44].
To remove the gap between different prediction branches, [50] and [26] replace the hard classification target from the ground-truth value, (i.e., 1 for positive samples, 0 for negative samples), to the IoU between the predicted bounding box and the ground-truth one, which is named the IoU-aware classification score (IACS).
IACS can help the model select a more accurate bounding box prediction candidate from the pool by trying to predict the quality of the bounding box prediction in another branch at the same position.
Along with the IACS, the varifocal loss was proposed in [50] to help the IACS approach outperform other IoU-aware designs.
The classification loss can be formulated as:
[Equation]
where p denotes the predicted IACS, b denotes the predicted bounding box, and ˆ b denotes the ground-truth bounding box.
Regression loss.
For bounding box regression, we employ the generalized IoU loss[35].
The regression loss function can be formulated as:
[Equation]
The GIoU loss is weighted by p to emphasize the high classification score samples.
The training signals from the negative samples are ignored.
Experiments 4.1 Implementation
Model.
We design two variants of SwinTrack with different configurations as follows:
• SwinTrack-T-224 .
Backbone: Swin Transformer-Tiny [28], pretrained with ImageNet-1k; Template size: [112 × 112] ; Search region size: [224 × 224] ; C = 384 ; N = 4 ; • SwinTrack-B-384 .
Backbone: Swin Transformer-Base [28], pretrained with ImageNet-22k; Template size: [192 × 192] ; Search region size: [384 × 384] ; C = 512 ; N = 8 ;
where C and N are the channel number of the hidden layers in the first stage of Swin Transformer and the number of encoder blocks in feature fusion, respectively.
In all variants, we use the output after the third stage of Swin Transformer for feature extraction.
Thus, the backbone stride s is 16.
For motion token, the number of sampled object trajectory n is set to 16, the fixed sampling interval ∆ is set to 15.
If the frame rate of the video sequence is available, the sampling interval is adjusted according to the frame rate.
Suppose the frame rate is f , the new sampling interval is getting by ∆30 f , 30 fps is the standard frame rate we assumed.
g , which controls the embedding granularity, is set to the same size as the search region feature map, like 14 for SwinTrack-T-224, and 24 for
SwinTrack-B-384.
For the model for GOT-10k sequences, n is set to 8, ∆ is set to 8, and no frame rate adjustment is applied.
Training.
We train SwinTrack using the training splits of LaSOT [11], TrackingNet [33], GOT10k [19] (1,000 videos are removed following [46] for fair comparison) and COCO 2017 [27].
In addition, we report the results of SwinTrack-T-224 and SwinTrack-B-384 with GOT-10k training split only to follow the protocol described in [19].
The model is optimized with AdamW [29], with a learning rate of 5e-4, and a weight decay of 1e-4.
The learning rate of the backbone is set to 5e-5.
We train the network on 8 NVIDIA V100 GPUs for 300 epochs with 131,072 samples per epoch.
The learning rate is dropped by a factor of 10 after 210 epochs.
A 3-epoch linear warmup is applied to stabilize the training process.
DropPath [23] is applied on the backbone and the encoder with a rate of 0.1.
For the models trained for the GOT-10k evaluation protocol, to prevent over-fitting, the training epoch is set to 150, and the learning rate is dropped after 120 epochs.
For the motion token, the object trajectory for the Siamese training pair is generated with the method described above.
The frames that object annotated as absent or out of the video sequence are marked as invalid, the corresponding box coordinates set to −∞ .
Since the coarse granularity of the coordinate embedding in our setting is already can be seen as an augmentation of historical object trajectory, no additional data augmentation is applied.
Inference.
We follow the common procedures for Siamese network-based tracking [1].
The template image is cropped from the first frame of the video sequence.
The target object is in the center of the image with a background area factor of 2.
The search region is cropped from the current tracking frame, and the image center is the target center position predicted in the previous frame.
The background area factor for the search region is 4.
Our SwinTrack takes the template image and search region as inputs and output classification map r cls and regression map r reg .
To utilize positional prior in tracking, we apply hanning window penalty on r cls , and the final classification map r ′ cls is obtained via r ′ cls
=
(1 − γ ) × r cls + γ × h , where γ is the weight parameter and h is the Hanning window with the same size as r cls .
The target position is determined by the largest value in r ′ cls and the scale is estimated based on the corresponding regression results in r reg .
For the motion token, the predicted confidence score and bounding box are collected on the fly.
A confidence threshold θ conf is applied, if the confidence score given by the classification branch of the
[Table]
head is lower than the threshold, the target object in the current frame is marked as lost by setting the collected bounding box to −∞ .
θ conf is set to 0 .
4 for LaSOT, the rests are set to 0 .
4.2 Comparisons to State-of-the-arts
We conduct experiments and compare SwinTrack with SoTA trackers on five benchmarks.
LaSOT. LaSOT
[11] consists of 280 videos for test.
Tab. 1 shows the results and comparisons with
SoTAs.
From Tab. 1, we can observe that SwinTrack-T-224 with light architecture reaches SoTA performance with 0.672 SUC and 0.708 PRE scores, which is competitive compared with other
Transformer-based trackers, including STARK-ST101 (0.671 SUC score) and TransT (0.649 SUC), and other trackers using complicated designs such as KeepTrack (0.671 SUC) and SiamR-CNN (0.648 SUC score).
With a larger backbone and input size, our strongest variant SwinTrack-Bsets a new record with 0.713 SUC score, surpassing START-ST101 and KeepTrack by 4.2 absolute percentage points.
LaSOT ext .
The recent LaSOT ext [10] is an extension of LaSOT by adding 150 extra videos.
These new sequences are challenging as many similar distractors cause difficulties for tracking.
The results of our tracker related to this dataset are an average of three times.
KeepTrack uses a complex association technique to handle distractors and achieves a promising 0.482 SUC score as in Tab. 1.
Compared with complicated KeepTrack, SwinTrack-T-224 is simple and neat, yet shows comparable performance with 0.476 SUC score.
In addition, due to complicated design, KeepTrack runs at less than 20 fps , while SwinTrack-T-224 runs in 98 fps , 5 × faster than KeepTrack.
When using a larger model, SwinTrack-B-384 shows the best performance with 0.491 SUC score.
TrackingNet.
We evaluate different trackers on the test set of TrackingNet [33].
From Tab. 1, we observe that our SwinTrack-T-224 achieves a comparable result with 0.811 SUC score.
Using a larger model and input size, SwinTrack-B-384 obtains the best performance with 0.840 SUC score, better than STARK-ST101 with 0.820 SUC score and TransT with 0.814 SUC score.
GOT-10k.
GOT-10k [19] offers 180 videos for test and it requires trackers to be trained using
GOT-10k train split only.
From Tab. 1, we see that SwinTrack-B-384 achieves the best mAO of 0.724, and SwinTrack-T-224 obtains a mAO of 0.713.
Both models outperform other Transformer-based counterparts significantly, including START-ST101 (0.688 mAO), TransT (0.671 mAO) and TrDiMP (0.671 mAO).
TNL2k.
TNL2k [42] is a newly released tracking dataset with 700 videos for test.
As reported in
Tab. 1, both models surpass the others.
SwinTrack-B-384 set a new state-of-the-art with 0.559 SUC score.
Efficiency comparison.
We report the comparisons of SwinTrack with other Transformer-based trackers in terms of efficiency and complexity.
As displayed in Tab. 2, SwinTrack-T-224 with a small model runs the fastest with a speed of 98 fps .
Especially, compared with STARK-ST101 and
[Table] [Table]
and 42 fps , SwinTrack-T-224 is 3 × and 2 × faster.
Despite using a larger model, our SwinTrack-B-384 is still faster than STARK-ST101 and STARK-ST50.
4.3 Ablation Experiment
Comparison with ResNet backbone.
We compare the Swin-Transformer backbone with popular
ResNet-50 [18].
As shown in Tab. 3 ( ❶ vs. ❷ ).
The Swin Transformer backbone significantly boosts the performance by 2.5% SUC score in LaSOT, 5.1% SUC score in LaSOT ext .
The result shows that the strong appearance modeling capability provided by the Swin Transformer plays a crucial role.
Feature fusion.
As displayed in Tab. 3 ( ❶ vs. ❸ ), compared with the concatenation-based fusion , the cross attention-based fusion runs at a slower speed, occupies much more memory, and also has an inferior performance on all datasets.
Slower speed can be due to the latency brought by the extra operations.
The parameter-sharing strategy not only just reduces the number of parameters but also benefits metric learning.
Comparison with the query-based decoder.
Queries is commonly adopted in the decoder of
Transformer network in vision tasks, e.g. object query [3] and target query [46].
Nevertheless, our empirical results in Tab.
3 ( ❶ vs. ❹ ) show that a target query-based decoder degrades the tracking performance on all benchmarks, even with 2 × training pairs.
As discussed, one possible reason is the generative model is not suitable for classification.
Besides, learning a general target query for any kind of object may also be difficult.
Position encoding.
We compare the united positional encoding used in SwinTrack and the original absolute position encoding in Transformer [38].
Notice, We make a little modification to the original absolute position encoding:
Except for the 2D embedding, the index of token source (e.g. 1 for the tokens from the template patch, 2 for the tokens from the search region patch) is also embedded.
As
[Table]
shown in Tab. 3 ( ❶ vs. ❺ ), our method with united positional encoding obtains improvements with 0.8-1.9 absolute percentage points on the benchmarks with negligible loss in speed (98 vs. 103).
Loss function.
From Tab. 3 ( ❶ vs. ❻ ), we observe that the model trained with varifocal loss significantly outperforms the one with binary cross entropy (BCE) loss without loss of efficiency.
This result indicates that the varifocal loss can assist the classification branch of the head to generate an IoU-aware response map, and thus help the tracker to improve the tracking performance.
Post processing.
One may wonder with highly discriminative Transformer architecture and IoUaware classification loss does the hanning penalty window is still functional, which introduces a strong smooth movement assumption.
In the experiments, we remove the hanning penalty window in post-processing, as shown in Tab. 3 ( ❶ vs. ❼ ), the performance is dropped by 1.0 SUC for LaSOT, 1.3 AO for GOT-10k in absolute percentage, and less than 1% in the SUC metric of other datasets.
This suggests that the strong smooth movement assumption is still applicable for our tracker.
But compared with the former Transformer-based tracker [5], the performance gap between with and without penalty window post-processing is narrowing.
Effectiveness of motion token.
We study the effectiveness of the motion token by conducting comparison experiments.
As shown in Tab. 4 ( ❶ vs. ❸ and ❷ vs. ❹ ), the models with motion token outperforms the models without motion token on all datasets, especially on LaSOT ext and GOT-10k.
The results indicate that the motion token can assist the tracker to handle hard similar distractors in
LaSOT ext and stabilize the short-term tracking like the sequences in GOT-10k test set.
We also study whether the effectiveness of the motion token is simply from the extra embedding vector.
We set up an experiment as in Tab. 4 ( ❺ ), which replaces the motion token with a learnable embedding token.
The result shows that the extra embedding vector has negative impacts indicating the effectiveness of the embedding of object trajectory.
Conclusion
In this work, we present SwinTrack, a simple and strong baseline for Transformer tracking.
In
SwinTrack, both representation learning and feature fusion are implemented with the attention mechanism.
Extensive experiments demonstrate the effectiveness of such architecture.
Besides, we propose the motion token to enhance the robustness of the tracker by providing the historical object trajectory, showing the flexibility of the Transformer model in architectural design.
With the power of sequence-to-sequence model architecture, a context-rich tracker is possible, and more contextual cues can be incorporated.
Finally, We hope this work can inspire and facilitate future research.
Acknowledgments and Disclosure of Funding
This work is supported by Peng Cheng Laboratory Research Project No. PCL2021A07.
Heng Fan and his employer receives no financial support for the research, authorship, and/or publication of this article.
Appendix A Positional Encoding
Transformer requires a positional encoding to identify the position of the current processing token [38].
Through a series of comparison experiments, we choose untied positional encoding , which is proposed in TUPE [20], as the positional encoding solution of our tracker.
In addition, we generalize the untied positional encoding to arbitrary dimensions to fit with other components in our tracker.
The original transformer [38] proposes a absolute positional encoding method to represent the position: a fixed or learnable vector p i is assigned to each position i .
Starting from the basic attention module, we have:
[Equation]
where Q , K , V are the query vector, key vector and value vector, which are the parameters of the attention function, d k is the dimension of key .
Introducing the linear projection matrix and multi-head attention to the attention module (7), we get the multi-head variant defined in [38]:
MultiHead
[Equation]
where head i =
Atten( QW Qi , KW Ki , V W Vi ) , W Qi ∈ R d model × d k , W Ki ∈ R d model × d k , W Vi ∈
R d model ×
d v , W Oi ∈ R hd v × d model and h is the number of heads.
For simplicity, as in [20], we assume that d k = d v = d model , and use the single-head version of self-attention module.
Denoting the input sequence as x = x 1 , x 2 , . .
, x n , where n is the length of sequence, x i is the i -th token in the input data.
Denoting the output sequence as z =
( z 1 , z 2 , . . .
, z n ) .
Self-attention module can be rewritten as
[Equation]
where
[Equation] [Equation]
Obviously, the self-attention module is permutation-invariance.
Thus it can not "understand" the order of input tokens.
Untied absolute positional encoding.
By adding a learnable positional encoding [38] to the singlehead self-attention module, we can obtain the following equation:
[Equation]
The equation (11) is expanded into four terms: token-to-token, token-to-position, position-to-token, position-to-position.
[20] discuss the problems that exist in the equation and proposes the untied absolute positional encoding , which unties the correlation between tokens and positions by removing the token-position correlation terms in equation (11), and using an isolated pair of projection matrices
U Q and U K to perform linear transformation upon positional embedding vector.
The following is the new formula for obtaining α ij using the untied absolute positional encoding in the l -th layer:
[Equation]
(12) where p i and p j is the positional embedding at position i and j respectively, U Q ∈ R d × d and
U K ∈ R
d × d are learnable projection matrices for the positional embedding vector.
When extending to the multi-head version, the positional embedding p i is shared across different heads, while U Q and
U K are different for each head.
Relative positional bias.
According to [36], relative positional encoding is a necessary supplement to absolute positional encoding.
In [20], a relative positional encoding is applied by adding a relative positional bias to equation (12):
[Equation]
(13) where for each j − i , b j − i is a learnable scalar.
The relative positional bias is also shared across layers.
When extending to the multi-head version, b j − i is different for each head.
Generalize to multiple dimensions.
Before working with our tracker’s encoder and decoder network, we need to extend the untied positional encoding to a multi-dimensional version.
One straightforward method is allocating a positional embedding matrix for every dimension and summing up all embedding vectors from different dimensions at the corresponding index to represent the final embedding vector.
Together with relative positional bias , for an n -dimensional case, we have:
[Equation]
Generalize to concatenation-based fusion.
In order to work with concatenation-based fusion , the untied absolute positional encoding is also concatenated to match the real position, the indexing tuple of relative positional bias now appends with a pair of indices to reflect the origination of query and key involved currently.
Take l -th layer in the encoder as the example:
[Equation]
where g and h are the index of the origination of query and key respectively, for instance, 1 for the tokens from the template image, 2 for the tokens from the search image.
The form in the decoder is similar, except that g is fixed.
In our implementation, the parameters of untied positional encoding are shared inside the encoder and the decoder, respectively.
B The Effect of Pre-training Datasets
The two variants of our tracker, SwinTrack-T-224 and SwinTrack-B-384 are using different pretraining datasets, which are derived from the settings from Swin Transformer [28].
Specifically,
SwinTrack-T-224 adopts ImageNet-1k and SwinTrack-B-384 adopts ImageNet-22k.
To analyze the effect of different pre-training datasets, we conduct an experiment on the performance of our tracker with different pre-training datasets.
Other than the pre-training datasets, The experiment follows the same settings in the ablation study in the paper, the motion token is not used and the results on GOT-10k are trained on the full datasets as described in the paper.
From Tab. 5, we can observe that, for smaller model SwinTrack-T-224 (23M # parameters), pre-training on ImageNet-22k brings small improvements on LaSOT (+0.6%) and TrackingNet (+0.4%) but degrades the performance on
[Table] [Table]
GOT-10k (-1.4%).
For larger model SwinTrack-B-384 (91M # parameters), pre-training on ImageNet22k shows significant performance gains on LaSOT (+2.2%) and GOT-10k (+3.0%) but slightly degrades the result on TrackingNet (-0.6%).
On LaSOT ext , ImageNet-22k shows a performance degradation on smaller model
SwinTrack-T-224 (-0.9%) and brings small improvements on larger model
SwinTrack-B-384 (+0.2%).
C Comparison with Newly Released Transformer-based Trackers
We compare our tracker with some newly released Transformer-based trackers, including STARK [46],
SBT [43], ToMP [30], MixFormer [6], AiATrack [15], Unicorn [45], OSTrack [47] in Tab. 6 in four challenging benchmarks.
The result shows our tracker is still competitive.
Fig.
3 and Fig.
4 show the success plot and the precision plot respectively.
The comparison includes our SwinTrack-T-224, our SwinTrack-B-384, TransT[5], STARK[46], MixFormer[6], AiATrack[15] and ToMP[30].
Our tracker obtained the best performance on this benchmark.
By looking into the curves of the figures, there is a significant advantage in the bounding box accuracy compared with other trackers due to our fully attentional architecture.
The success AUC score under different attributes of LaSOT [11] Test set in shown in Fig.
Fig.indicates that our tracker has no obvious shortcomings except the viewpoint change.
D Results on UAV123 and VOT Benchmark
In this section, we report the performance of the tracker on three additional benchmarks, including
UAV123 [32], VOT2020 and VOT-STB2022 [21].
UAV123[32] is an aerial video dataset and benchmark for low-altitude UAV target tracking, containing 123 video sequences.
Our tracker is on par with the state-of-the-art, AiATrack [15], on this benchmark.
The results are shown in Tab. 7.
Finally, we evaluate our tracker on the two versions of the VOT Challenge: VOT2020 and VOTSTB2022.
The VOT2020 dataset contains 60 videos with segmentation masks annotated.
Since our tracker is a bounding box only method, we compare the results with the trackers that produce the bounding boxes as well.
The result in Tab.8 shows that SwinTrack-T-224 has a better performance than the larger SwinTrack-B-384 on this benchmark.
[Table] [Table]
In addition
[Table]
We report the results on VOT-STB2022 in Tab.9.
SwinTrack-T-224 has a better performance on VOT-STB2022 as well.
No comparison is made since VOT-STB2022 is a newly released benchmark.
E Quantitative Analysis of the Effectiveness of Motion Token
To give a further analysis of the effectiveness of motion token, we provide the success plot (Fig. 6) and precision plot (Fig. 7) on LaSOT test set, and the success AUC score under different attributes of
LaSOT test set in Fig.
The success plot and the precision plot show that the motion token improves the performance of the trackers by boosting robustness.
While the Fig.
8 further points out that the motion token can assist the tracker to recover from a failure state when the vision features are not reliable like an object is getting out of view or fully occluded by other objects.
F Response Visualization for Qualitative Analysis
We provide the heatmap visualization of the response map generated by the IoU-aware classification branch of the head in our SwinTrack-B-384 in Fig.
The visualized sequences are from
LaSOT ext [10], with challenges include fast motion, full occlusion, hard distractor, etc .
The results demonstrate the great discriminative power of our tracker.
Many trackers will show a multi-peak on the response map when the target object is occluded or multiple similar objects exist.
With the vision-motion integrated Transformer architecture, our tracker eases such phenomenon.
G Failure Case
We show some typical failure cases of our tracker (SwinTrack-B-384 on LaSOT ext [10] and VOTSTB2022 [21]) in Fig.
The first case suffers from a mixture of low resolution, fast motion, and background clutter.
The second case suffers from a fast occlusion by a distractor.
The third case suffers from the non-semantic target.
[Table]
Overlap threshold
O v e r l a p
P r ec i s i on [ % ]
Success plot
TransT [64.9]
STARK [67.1]
SwinTrack-T-224 [67.2]
ToMP [68.5]
AiATrack [69.0]
MixFormer [70.0]
SwinTrack-B-384 [71.3]
Location error threshold [pixels]
D i s t a n ce
P r ec i s i on [ % ]
Precision plot
TransT [69.0]
SwinTrack-T-224 [70.8]
STARK [72.2]
ToMP [73.5]
AiATrack [73.8]
MixFormer [76.3]
SwinTrack-B-
[Figure]
[Figure]
SwinTrack-T-224-NoMToken [66.7]
SwinTrack-T-224 [67.2]
SwinTrack-B-384-NoMToken [70.2]
SwinTrack-B-384 [71.3]
[Figure]
Location error
[Figure]
[pixels
[Figure]
D i s t a n ce
P r ec i s i on [ % ]
[Figure] [Figure]
Partial Occlusion
Deformation
Motion Blur
Camera Motion
Rotation
Background Clutter
Viewpoint Change
Scale Variation
Full Occlusion
Fast Motion
Out-of-View
Low Resolution
Aspect Ration Change
SwinTrack-T-224-NoMToken
SwinTrack-B-384-NoMToken
SwinTrack-TSwinTrack-B-
[Figure]