Divert More Attention to Vision-Language Tracking
Abstract
Relying on Transformer for complex visual feature learning, object tracking has witnessed the new standard for state-of-the-arts (SOTAs).
However, this advancement accompanies by larger training data and longer training period, making tracking increasingly expensive.
In this paper, we demonstrate that the Transformer-reliance is not necessary and the pure ConvNets are still competitive and even better yet more economical and friendly in achieving SOTA tracking.
Our solution is to unleash the power of multimodal vision-language (VL) tracking, simply using
ConvNets.
The essence lies in learning novel unified-adaptive VL representations with our modality mixer (ModaMixer) and asymmetrical ConvNet search.
We show that our unified-adaptive VL representation, learned purely with the ConvNets, is a simple yet strong alternative to Transformer visual features, by unbelievably improving a CNN-based Siamese tracker by 14.5% in SUC on challenging LaSOT ( 50.7% → 65.2% ), even outperforming several Transformer-based SOTA trackers.
Besides empirical results, we theoretically analyze our approach to evidence its effectiveness.
By revealing the potential of VL representation, we expect the community to divert more attention to VL tracking and hope to open more possibilities for future tracking beyond Transformer.
Code and models are released at https://github.com/JudasDie/SOTS .
Introduction
[Figure]
Transformer tracking recently receives a surge of research interests and becomes almost a necessity to achieve stateof-the-art (SOTA) performance [8, 54, 10].
The success of Transformer trackers mainly attributes to attention that enables complex feature interactions.
But, is this complex attention the only way realizing SOTA tracking?
Or in other words, is Transformer the only path to SOTA?
We answer no , and display a Transformer-free path using pure convolutional neural network (CNN).
Different than complex interactions in visual feature by attention requiring more training data and longer training time, our alternative is to explore simple interactions of multimodal, i.e. , vision and language, through CNN.
In fact, language, an equally important cue as vision, has been largely explored in vision-related tasks, and is not new to tracking.
Prior works [20, 34, 19] have exploited vision-language (VL) multimodal learning for improving tracking.
However, the performance falls far behind current SOTAs.
For instance on LaSOT [17], the gap between current best VL tracker [20] and recent Transformer tracker [8] is absolute 10.9% in SUC (see Fig. 1).
So, what is the bottleneck of VL tracking in achieving SOTA?
The devil is in VL representation.
Feature representation has been shown to be crucial in improving tracking [53, 63, 32, 24, 10].
Given two modalities of vision and language, the VL feature is desired to be unified and adaptive [40, 60].
The former property requires deep interaction of vision and language, while the latter needs VL feature to accommodate different scenarios of visual and linguistic information.
However, in existing VL trackers, vision and language are treated independently and processed distantly until the final result fusion.
Although this fusion may easily get two modalities connected, it does not accords with human learning procedure that integrates multisensory by various neurons before causal inference [3], resulting in a lower upper-bound for VL tracking.
Besides, current
VL trackers treat template and search branches as homoplasmic inputs, and adopt symmetrical feature learning structures for these two branches, inherited from typical vision-only Siamese tracking [20].
We argue the mixed modality may have different intrinsic nature than the pure vision modality, and thus requires a more flexible and general design for different signals.
Our solution.
Having observed the above, we introduce a novel unified-adaptive vision-language representation, aiming for SOTA VL tracking without using Transformer 2 .
Specifically, we first present modality mixer, or ModaMixer, a conceptually simple but effective module for VL interaction.
Language is a high-level representation and its class embedding can help distinguish targets of different categories ( e.g. , cat and dog) and meanwhile the attribute embedding ( e.g. , color, shape) provides strong prior to separate targets of same class ( e.g. , cars with different colors).
The intuition is, channel features in vision representation also reveal semantics of objects [22, 58].
Inspired by this, ModaMixer regards language representation as a selector to reweight different channels of visual features, enhancing target-specific channels as well as suppressing irrelevant both intra- and inter-class channels.
The selected feature is then fused with the original feature, using a special asymmetrical design (analyzed later in experiments), to generate the final unified VL representation.
A set of ModaMixers are installed in a typical CNN from shallow to deep, boosting robustness and discriminability of the unified VL representation at different semantic levels.
Despite simplicity,
ModaMixer brings 6.9% gains over a pure CNN baseline [23] ( i.e. , 50.7% → 57.6%).
Despite huge improvement, the gap to SOTA Transformer tracker [8] remains (57.6% v.s. 64.9%).
To mitigate the gap, we propose an asymmetrical searching strategy (ASS) to adapt the unified VL representation for improvements.
Different from current VL tracking [20] adopting symmetrical and fixed template and search branches as in vision-only Siamese tracking [32], we argue that the learning framework of mixed modality should be adaptive and not fixed.
To this end, ASS borrows the idea from neural architecture search (NAS) [65, 43] to separately learn distinctive and asymmetrical networks for mixed modality in different branches and ModaMixers.
The asymmetrical architecture, to our best knowledge, is the first of its kind in matching-based tracking.
Note, although NAS has been adopted in matching-based tracking [57], this method finds symmetrical networks for single modality.
Differently, ASS is applied on mixed modality and the resulted architecture is symmetrical .
Moreover, the network searched in ASS avoids burdensome re-training on ImageNet [14], enabling quick reproducibility of our work (only 0.625 GPU days with a single RTX-2080Ti).
Our ASS is general and flexible, and together with ModaMixer, it surprisingly shows additional 7.6% gains ( i.e. , 57.6% → 65.2%), evidencing our argument and effectiveness of ASS.
Eventually, with the unified-adaptive representation, we implement the first pure CNN-based VL tracker that shows SOTA results comparable and even better than Transformer-based solutions, without bells and whistles.
Specifically, we apply our method to a CNN baseline SiamCAR [23], and the resulted VL tracker VLT SCAR shows 65.2% SUC on LaSOT [17] while running at 43FPS, unbelievably improving the baseline by 14.5% and outperforming SOTA Transformer trackers [8, 54] (see again Fig. 1).
We observe similar improvements by our approach on other four benchmarks.
Besides empirical results, we provide theoretical analysis to evidence the effectiveness of our method.
Note that, our approach is general in improving vision-only trackers including Transformer-based ones.
We show this by applying it to TransT [8] and the resulted tracker VLT TT shows 2.4% SUC gains ( i.e. , 64.9% → 67.3%), evidencing its effectiveness and generality.
[Figure] We are aware that one can certainly leverage the Transformer [39] to learn a good (maybe better) VL representation for tracking, with larger data and longer training period.
Different than this, our goal is to explore a cheaper way with simple architectures such as pure CNN for SOTA tracking performance and open more possibilities for future tracking beyond Transformer.
In summary, our contributions are four-fold: (i) we introduce a novel unified-adaptive vision-language representation for SOTA
VL tracking; (ii) we propose the embarrassingly simple yet effective ModaMixer for unified VL representation learning; (iii) we present ASS to adapt mixed VL representation for better tracking and (iv) using pure CNN architecture, we achieve SOTA results on multiple benchmarks .
Related Work
Visual Tracking.
Tracking has witnessed great progress in the past decades.
Particularly, Siamese tracking [5, 49], that aims to learn a generic matching function, is a representative branch and has revolutionized with numerous extensions [33, 63, 32, 59, 18, 25, 62, 64, 9].
Recently, Transformer [51] has been introduced to Siamese tracking for better interactions of visual features and greatly pushed the standard of state-of-the-art performance [8, 54, 48, 10, 35].
From a different perspective than using complex Transformer, we explore multimodal with simple CNN to achieve SOTA tracking.
Vision-Language Tracking.
Natural language contains high-level semantics and has been leveraged to foster vision-related tasks [21, 31, 2] including tracking [34, 19, 20].
The work [34] first introduces linguistic description to tracking and shows that language enhances the robustness of vision-based method.
Most recently, SNLT [20] integrates linguistic information into Siamese tracking by fusing results respectively obtained by vision and language.
Different from these VL trackers that regard vision and language as independent cues with weak connections only at result fusion, we propose
ModaMixer to unleash the power of VL tracking by learning unified VL representation.
NAS for Tracking.
Neural architecture search (NAS) aims at finding the optimal design of deep network architectures [65, 43, 37, 26] and has been introduced to tracking [57, 62].
LightTrack [57] tends to search a lightweight backbone but is computationally demanding (about 40 V100 GPU days).
AutoMatch uses DARTS [37] to find better matching networks for Siamese tracking.
All these methods leverage NAS for vision-only tracking and search a symmetrical Siamese architecture.
Differently, our work searches the network for multimodal tracking and tries to find a more general and flexible asymmetrical two-stream counterpart.
In addition, our search pipeline only takes [-0.
-]0.
RTX-2080Ti GPU days, which is much more resource-friendly.
Unified-Adaptive Vision-Language Tracking
This section details our unified-adaptive vision-language (VL) tracking as shown in Fig.
In specific, we first describe the proposed modality mixer for generating unified multimodal representation and then asymmetrical network which searches for learning adaptive VL representation.
Afterwards, we illustrate the proposed tracking framework, followed by theoretical analysis of our method.
3.1 Modality Mixer for Unified Representation
The essence of multimodal learning is a simple and effective modality fusion module.
As discussed before, existing VL trackers simply use a later fusion way, in which different modalities are treated independently and processed distantly until merging their final results [20, 34].
Despite the effectiveness to some extent, the complementarity of different modalities in representation learning is largely unexplored, which may impede the multimodal learning to unleash its power for VL tracking.
In this work, we propose the modality mixer (dubbed ModaMixer ) to demonstrate a compact way to learn a unified vision-language representation for tracking.
For next
ModaMixer
[Figure]
ModaMixer considers language representation as selector to reweight channels of vision features.
In specific, given the language description with N words of a video 3 , a language model [15] is adopted to abstract the sentence to semantic features with size of ( N + 2) ×
d .
The extra “2” denotes the “[CLS][SEP]” characters in language model processing (see [15] for more details).
Notably, descriptions for different videos may contain various length N .
To ensure the ModaMixer applicable for all videos, we first average the features for all words along sequence length dimension “(N+2)” to generate a unique language representation f l ∈ R 1 × d for each description.
Then a linear layer is followed to align the channel number of f l with the corresponding vision feature f v ∈ R H × W × C .
Channel selector is expressed as Hadamard product operator, which point-wisely multiplies language representation ( 1 × C ) to embedding of each spatial position in the vision feature f v .
Finally, a residual connection between the mixed feature f m and vision feature f v is conducted to avoid losing informative vision details.
In a nutshell, the ModaMixer can be formulated as,
[Equation]
where ⊙ denotes Hadamard product, Linear is a linear projection layer with weight matrix size of d × C for channel number alignment, and Block ASS indicates post-processing block before residual connection.
Please note that, to enable adaptive feature modeling for different modalities, we search different Block ASS to process features before and after fusion (see Sec.
3.2 for more details).
The proposed ModaMixer is illustrated in Fig.
Akin to channel attention [46, 22], the high-level semantics in language representation dynamically enhance target-specific channels in vision features, and meanwhile suppress the responses of distractors belonging to both inter- and intra-classes.
3.2 Asymmetrical Search for Adaptive Vision-Language Representation
Besides the fusion module, the other crucial key for vision-language tracking is how to construct the basic modeling structure.
The simplest strategy is to inherit a symmetrical Siamese network from vision-based tracking ( e.g. , [5, 32]), as in current VL trackers [20].
But the performance gap still remains if using this manner, which is mostly blamed on the neglect of the different intrinsic nature between VL-based multimodal and vision-only single modality.
To remedy this, we propose an asymmetrical searching strategy (dubbed ASS ) to learn an adaptive modeling structure for pairing with ModaMixer.
The spirits of network search are originated from the field of Neural Architecture Search (NAS).
We adopt a popular NAS model, in particular the single-path one-shot method SPOS [26], for searching the optimal structure of our purpose.
Although SPOS has been utilized for tracking [57], our work significantly differs from it from two aspects: 1) Our ASS is tailored for constructing an asymmetrical two-stream network for multimodal tracking, while [57] is designed to find a symmetrical Siamese network for vision-only single-modality tracking.
Besides, we search layers both in the backbone network and the post-processing Block ASS in the ModaMixer (see Eq. 1); 2) Our ASS reuses the is the stem convolution layer. ( ) represents the basic ASS unit, where the first three ones indicate Shuffle block [61] with kernel sizes of (3,5,7), respectively, and the last one denotes a Shuffle Xception block [61] with kernel size of 3. [Table] pre-trained supernet from SPOS, which avoids the burdensome re-training on ImageNet [14] (both for the supernet and found subnet) and thus reduces the time complexity of our search pipeline to 1 / 64 of that in LightTrack [57] ( i.e. , 0.625 RTX-2080Ti GPU days v.s. 40 V100 GPU days ).
Due to limited space, please refer to appendix for more details and comparison of our ASS and [57].
The search space and search strategy of ASS are kept consistent with the original SPOS [26].
In particular, the search pipeline is formulated as,
[Equation]
where A represents the architecture search space of the network N , a is a sample from A and W denotes the corresponding network weights.
Notably, the network N includes three components
N = { φ t , φ s , φ m } , where each indicates backbone for the template branch φ t , backbone for the search branch φ s and layers in the ModaMixer φ m .
The whole pipeline consists of training supernet on tracking datasets via random sampling Γ from search space
A (Eq. 2) and finding the optimal subnet via evolutionary algorithms (Eq. 3).
The SUC (success score) on validation data is used as rewards of evolutionary algorithms.
Tab. 1 demonstrates the searched asymmetrical networks in our
VL tracking.
For more details of ASS, we kindly refer readers to appendix or [26].
3.3 Tracking Framework
With the proposed ModaMixer and the searched asymmetrical networks, we construct a new visionlanguage tracking framework, as shown in Fig.
2 and Tab. 1.
Our framework is matching-based tracking.
Both template and search backbone networks contain 4 stages with the maximum stride of 8, the chosen blocks of each stage are denoted with different colors in Tab. 1.
ModaMixer is integrated into each stage of the template and search networks to learn informative mixed representation.
It is worth noting that, the asymmetry is revealed in not only the design of backbone networks, but also the ModaMixer.
Each ModaMixer shares the same meta-structure as in Fig. 3, but comprises different post-processing layers Block ASS to allow adaption to different semantic levels ( i.e. , network depth) and input signals ( i.e. , template and search, pure-vision and mixed feature in each ModaMixer).
With the learned unified-adaptive VL representations from the template and search branches, we perform feature matching and target localization, the same as in our baseline.
3.4 A Theoretical Explanation
This section presents a theoretical explanation of our method, following the analysis in [29].
Based on the Empirical Risk Minimization (ERM) principle [47], the objective of representation learning is to find better network parameters θ by minimizing the empirical risk, i.e. ,
[Equation]
where L denotes loss function, M represents the modality set, n indicates sample number, X i = { x 1 i , x 2 i ...
x |M| i } is the input mutimodal signal, y i is training label, and F demotes optimization space of θ .
Given the empirical risk ˆ r ( θ M ) , its corresponding population risk is defined as,
[Equation]
Following [1, 50, 29], the population risk is adopted to measure the learning quality.
Then the latent representation quality [29] is defined as,
[Equation]
where ∗ represents the optimal case, inf indicates the best achievable population risk.
With the empirical Rademacher complexity R [4], we restate the conclusion in [29] with our definition.
Theorem 1 ([29]) .
Assuming we have produced the empirical risk minimizers ˆ θ M and ˆ θ S , training with the |M| and |S| modalities separately ( |M| > |S| ).
Then, for all 1 > δ > 0 , with probability at least 1 − δ 2 :
[Equation]
where γ
[Equation]
γ D ( M , S ) computes the quality difference learned from multiple modalities M and single modality
S with dataset D .
Theorem 1 defines an upper bound of the population risk training with different number of modalities, which proves that more modalities could potentially enhance the representation quality.
Furthermore, the Rademacher complexity R n ( F ) is proportional to the network complexity, which demonstrates that heterogeneous network would theoretically rise the upper bound of r (cid:16) ˆ θ M (cid:17) − r (cid:16) ˆ θ S (cid:17) , and also exhibits that our asymmetrical design has larger optimization space when learning with |M| modalities compared to |S| modalities ( |M| > |S| ).
The proof is beyond our scoop [Table] and please refer to [29] for details. [Table]
82.6 C-RPN [18] 45. - - - - - - - SiamFC [5] 33. 79.2 ECO [11] 32. - - SiamCAR
[Table]
SNLT [20] 54.
80.4 VLT 0SCAR (Ours) 65.
88.8 VLT t SCAR (Ours) 63.
[Table]
STARK [56] 66.
91.4 TrDiMP [54] 63.
92.5 TransT [8] 64.
[Table]
VLT
0TT
(Ours) 66. 91.2 VLT t TT (Ours) 67.  Experiment 4.1 Implementation Details We apply our method to both CNN-based SiamCAR [23] (dubbed VLT SCAR ) and Transformer-based
TransT [8] (dubbed VLT TT ).
The matching module and localization head are inherited from the baseline tracker without any modifications.
Searching for VLT.
The proposed ASS aims to find a more flexible modeling structure for visionlanguage tracking (VLT).
Taking VLT SCAR as example, the supernet from SPOS [26] is used as feature extractor to replace the ResNet [27] in SiamCAR.
We train the trackers with supernet using training splits of COCO [36], Imagenet-VID [14], Imagenet-DET [14], Youtube-BB [44],
GOT-10k [28], LaSOT [17] and TNL2K [55] for 5 epochs, where each epoch contains 1 . 2 × [-10 
 [Table]-]10template-search pairs.
Once finishing supernet training, evolutionary algorithms as in SPOS [26] is applied to search for optimal subnet and finally obtains VLT SCAR .
The whole search pipeline consumes 15 hours on a single RTX-2080Ti GPU.
The search process of VLT TT is similar to
VLT SCAR .
We present more details in the appendix due to space limitation.
Optimizing VLT SCAR and VLT TT .
The training protocol of VLT SCAR and VLT TT follows the corresponding baselines SiamCAR [23] and TransT [8].
Notably, for each epoch, half training pairs come from datasets without language annotations ( i.e. , COCO [36], Imagenet-VID [14], ImagenetDET [14], Youtube-BB [44]).
The language representation is set as 0-tensor or visual pooling feature under this circumstances (discussed in Sec. 4.4) 4 .
4.2 State-of-the-art Comparison
Tab. 2 presents the results and comparisons of our trackers with other SOTAs on LaSOT [17],
LaSOT Ext [16], TNL2K [55], OTB99-LANG [34] and GOT-10K [28].
The proposed VLT SCAR and
VLT TT run at 43/35 FPS on a single RTX-2080Ti GPU, respectively.
Compared with the speeds of baseline trackers SiamCAR [23]/TransT [8] with 52/32 FPS, the computation cost of our method is small.
Moreover, our VLT TT outperforms TransT in terms of both accuracy and speed.
Compared with SiamCAR [23], VLT SCAR achieves considerable SUC gains of 14.5%/10.8%/14.5% on LaSOT/LaSOT Ext /TNL2K, respectively, which demonstrates the effectiveness of the proposed VL tracker.
Notably, our VLT SCAR outperforms the current best VL tracker SNLT [20] for 11.2%/18.5% on LaSOT/LaSOT Ext , showing that the unified-adaptive vision-language representation is more robust for VL tracking and is superior to simply fusing tracking results of different modalities.
The advancement of our method is preserved across different benchmarks.
What surprises us more is that the CNN-based VLT SCAR is competitive and even better than recent vision Transformer-based approaches.
For example, VLT SCAR outperforms TransT [8] on LaSOT and meanwhile runs faster (43 FPS v.s. 32 FPS) and requires less training pairs ( 2 . 4 × 10 7 v.s. 3 . 8 × 10 7 ).
By applying our method to TransT, the new tracker VLT TT improves the baseline to 67.3% in SUC with 2.4% gains on LaSOT while being faster, showing its generality.
4.3 Component-wise Ablation
We analyze the influence of each component in our method to show the effectiveness and rationality of the proposed ModaMixer and ASS.
The ablation experiments are conducted on VLT SCAR with “0-tensor” setting (discussed in Sec. 4.4), and results are presented in Tab. 3.
By directly applying the ModaMixer on the baseline SiamCAR [23] (“ResNet50+ModaMixer”), it obtains SUC gains of 6 . 9% on LaSOT ( ② v.s. ① ).
This verifies that the unified VL representation effectively improves tracking robustness.
One interesting observation is that ASS improves vision-only baseline for 1 . 4% percents on LaSOT ( ③ v.s. ① ), but when equipping with ModaMixer, it surprisingly further brings 7 . 6% SUC gains ( ④ v.s. ② ), which shows the complementarity of multimodal representation learning (ModaMixer) and the proposed ASS. [Table] [Table]
[Table] Further Analysis
Dealing with videos without language description during training.
As mentioned above, language annotations are not provided in several training datasets ( e.g. , YTB-BBox [44]).
We design two strategies to handle that.
One is to use “0-tensor” as language embedding, and the other is to replace the language embedding with visual features which are generated by pooling template feature in the bounding box (dubbed as “template”).
As shown in Tab. 2, the two strategies perform competitively, but the one with visual feature ( i.e., VLT t SCAR and VLT t TT ) is slightly better in average. No/Pseudo description during inference.
VL trackers require the first frame of a video is annotated with a language description. One may wonder that what if there is no language description? Tab.presents the results by removing the description and using that generated with an recent advanced image-caption method [41] (in ICML2021) based on VLT SCAR and VLT TT with “0-tensor” and “template” settings. The results show that, without language description, tracking performances heavily degrade ( e.g., 63 .
9% → 53 .
→ 61 . 0% SUC on LaSOT of VLT t SCAR and VLT t TT , respectively), verifying that the high-level semantics in language do help in improving robustness.
Even though, the performances are still better than the vision-only baseline.
Surprisingly, when using the generated description, it doesn’t show promising results ( e.g., 51 .
6% of VLT t SCAR and 59 .
7% of
VLT t TT ), indicating that it is still challenging to generate accurate caption in real-world cases and noisy caption even brings negative effects to the model.
Symmetrical or Asymmetrical?
The proposed asymmetrical searching strategy is essential for achieving an adaptive vision-language representation.
As illustrated in Tab.
5a, we experiment by searching for a symmetrical network (including both backbone and Block ASS in the ModaMixer) based on VLT t SCAR , but it is inferior to the asymmetrical counterpart for 3 . 9% / 6 . 2% of success rate (SUC) and precision (P) on LaSOT [17], respectively, which empirically proves our argument. Asymmetry in ModaMixer. The asymmetry is used in not only the backbone network, but also the ModaMixer.
In our work, the post-processing layers for different signals (visual and mixed features) are decided by ASS, which enables the adaption at both semantic levels ( i.e. , network depth) and different input signals ( i.e. , template and search, pure-vision and mixed feature in each [Table] [Table] 8W × 1000Epoch
TransT Four Datasets, TNL2K 62.
[Table]
[Table]
[Table]
[Table]
⑧
VLT TT 3.8W × 1000Epoch
TransT Four Datasets, TNL2K 67.
[Table]
[Table]
[Table]
[Table]
ModaMixer).
As in Tab. 5b, when replacing the post-processing layers with a fixed ShuffleNet block from SPOS [26] ( i.e. , inheriting structure and weights from the last block in each backbone stage), the performance of VLT t SCAR drops from 63 .
9% to 59.1% in SUC on LaSOT.
This reveals that the proposed ASS is important for building a better VL learner.
Residual Connection of ModaMixer The residual connection [27] is a commonly used trick to avoid information loss.
In our VL representation learning, it provides more vision messages for better multimodal fusion.
We experiment by removing the structure based on VLT SCAR with the "template" setting as shown in Tab. 5c.
Compared to the default setting ( i.e., w/. Res), the loss of additional vision details brings decreases for 2 .
1% of SUC on LaSOT/TNL2K, respectively.
Even though, the performance is still much higher than the baseline.
This demonstrates the improvements are mainly attributed to the multimodal fusion.
Volume and Source of Training Data.
As common wisdom, the tracking performance is deeply influenced by training data volume and source.
As illustrated in Tab. 6, we experiment by comparing
VLT t SCAR , VLT t TT and their baselines ( i.e. , SiamCAR and TransT) with different data settings:
(1) We retrain SiamCAR with the same data setting of VLT SCAR ( ③ v.s. ⑤ ).
Compared to the default setting ( ① ), double data volume and three more data sources contain different biases, which affect the trained model to produce biased outcomes, as illustrated in
From
the
results ( ③ )
, the
addition of TNL2K significantly
improves the
default ① with 4 . 4% gains in SUC on TNL2K
, whereas the performance on LaSOT slightly decreases
Compared to VLT SCAR with the same setting ( ⑤ ), ③ is still suppressed for 8 . 6% / 7 .
4% of SUC and P on TNL2K, respectively.
(2) We also retrain VLT SCAR with the only LaSOT ( ④ ), which keeps aligned with ② .
The SUC scores on LaSOT and TNL2K degrade heavily to 57 .
0% compared to the default ⑤ , respectively.
This is caused by the great reduction of the language-annotated training data, i.e., from(LaSOT)+1300 (TNL2K)+9335 (GOT-10k) to 1120 (LaSOT).
Our model is hard to learn a good multimodal representation with the quite less language-annotated training data, which violates our intention. Even though, our VLT SCAR ( ④ ) still outperforms the baseline SiamCAR ( ② ) for 5 . 4% / 4 . 0% of SUC scores on LaSOT/TNL2K. (3) TransT is also retrained with the same data setting as VLT TT ( ⑦ v.s. ⑧ ). More data sources bring similar biases and influence the performance as SiamCAR, compared to default TransT ( ⑥ ). Our VLT TT ( ⑧ ) still achieves superior scores on both LaSOT and TNL2K.
S S S
S S
S S S
M ax .
S c o re
M i n .
S c o re a ft er f u s i o n b e f o re f u s i o n
[Figure]
Channels visualization with max/
[Figure]
selection scores
[Figure]
[Table]
Channel selection by ModaMixer.
ModaMixer translates the language description to a channel selector to reweight visual features.
As shown in Fig.
4, the channel activation maps with maximum selection scores always correspond to the target, while the surrounding distractors are successfully assigned with minimum scores (Fig. 4 (a)-bottom).
Besides, with multimodal fusion (or channel selection), the network can enhance the response of target and meanwhile suppress the distractors (see
Fig.
4 (b)).
This evidences our argument that language embedding can identify semantics in visual feature channels and effectively select useful information for localizing targets.
More visualization results are presented in appendix due to limited space.
Multimodal Vision-Language Tracking with SiamRPN++ [32] We apply our method to another pure CNN-based tracker SiamRPN++ (dubbed VLT RPN++ ) and the results are shown in Tab. 7.
Compared with the baseline SiamRPN++, VLT RPN++ achieves considerable SUC gains of 9 . 4% / 4 .
5% on LaSOT/TNL2K, respectively.
This demonstrates the effectiveness of multimodal representation learning (ModaMixer) and the proposed ASS.
Conclusion
In this work, we explore a different path to achieve SOTA tracking without complex Transformer, i.e., multimodal VL tracking.
The essence is a unified-adaptive VL representation, learned by our
ModaMixer and asymmetrical networks.
In experiments, our approach surprisingly boosts a pure
CNN-based Siamese tracker to achieve competitive or even better performances compared to recent
SOTAs.
Besides, we provide an theoretical explanation to evidence the effectiveness of our method.
We hope that this work inspires more possibilities for future tracking beyond Transformer.
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant (62176020), the National Key Research and Development Program (2020AAA0106800), the Beijing Natural Science Foundation under Grant (Z180006, L211016), CAAI-Huawei MindSpore Open Fund and Chinese Academy of Sciences (OEIP-O-202004). A Appendix The appendix presents additional details of our tracker in terms of design and experiments, as follows.
• A. Volume of Language-Annotated Training Data We analyze how the volume of language-annotated training data affects tracking performance.
• A. Different Language Models
We analyze and compare different language embedding models ( i.e. , BERT [15] and GPT2 [42]) in our method.
• A. Details of The Proposed Asymmetrical Searching Strategy (ASS) We present more details about the pipeline of our proposed ASS.
• A. Comparison of ASS and LightTrack [57] We show efficiency comparison of our ASS and another NAS-based tracker LightTrack [57].
• A. Visualization of Tracking Result and Failure Case We visualize the tracking results and analyze a failure case of our tracker.
• A. Activation Analysis of Different Language Descriptions We study the impact of different language descriptions on tracking performance by visualizing their activation maps.
• A. Attribute-based Performance Analysis We conduct attribute-based performance analysis on LaSOT [17], and the results demonstrate the robustness of our tracker in various complex scenarios.
Language-annotated data is crucial for our proposed tracker in learning the robust vision-language representations.
We analyze the influence of training with different volumes of language-annotated data and the results are presented in Tab.
8a.
The default setting in the manuscript is noted as “100%”. For settings of “50%” and “75%”, the reduced part is filled with the data without language annotation, which keeps the whole training data volume.
It shows that as the language-annotated training pairs reduced, the performance on LaSOT [17] gradually decreases ( 63 . 9% → 61 . 8% → 58 .
9% in SUC), demonstrating that more language-annotated data helps improve model capacity. A.2 Different Language Models
As described in Sec.
3.1 of the manuscript, the language model of BERT [15] is adopted to abstract the semantics of the sentence, which directly relates to the learning of vision-language representation. To show the influence of different language models, we compare the results of using BERT [15] and GPT-2 [42], as shown in Tab. 8b. An interesting finding is that GPT-2 [42] even decreases the performances, which is discrepant with recent studies in natural language processing.
One possible reason is that the bi-directional learning strategy in BERT [15] can better capture the context information of a sentence than the self-regression in GPT-2 [42].
[Table]
As mentioned in Sec. 3.2 of the manuscript, ASS is designed to adapt the mixed modalities for different branches by simultaneously searching the asymmetrical network N = { φ t , φ s , φ m } . The pipeline of ASS consists of two stages.
The first stage is pretraining to search architecture and the second one is to retrain it for our VL tracking, as summarized in Alg. 1. 1: /* Search */ 2: Input : Network N , search space A , max iteration T , random sampling Γ , Train dataset: D train = {X n , y n } Nn =1 , X n = { x vn , x ln ( optional ) } ,
Val dataset: D val = {X m , y m } Mm =1 , For videos without language annotation: x l = “ 0-tensor ” or “ template-embedding ”. 3: Initialize : Initialize the network parameters θ N .
4: for i = 1 : T do 5: for n = 1 : N do 6: if language annotation exists then 7: f ln = BERT ( x ln ) ; 8: else if x ln = “ 0-tensor ” then 9: f ln = zeros _ like [ BERT ( x ln )] ; // Default setting without language annotation 10: else if x ln = “ template-embedding ” then 11: f ln = ROI ( x vn ) ; // Robust setting without language annotation 12: end if 13: a = Γ( A ) , p n = N ( x vn , f ln ; a ) ; 14: end for 15:
Update the network parameters θ N with gradient descent: 16: θ a N ← θ a N − α∂ 1 N (cid:80) Nn =1 L ( p n , y n ; θ a N ) /∂θ a N ; 17: end for 18: a best = EvolutionaryArchitectureSearch( A , D val ; θ N ) ; [26] The pretraining stage (line 3-18 in Alg. 1) contains four steps: 1) ASS first initializes the network parameter θ N of N . Concretely, φ t and φ s reuse the pretrained supernet of SPOS [26], while φ m copies the weight of the last layer in φ t and φ s . This reduces the tedious training on ImageNet [14] and enables quick reproducibility of our work; 2) The language model [15] processes the annotated sentence x l to get corresponding representation f l .
If the language annotations are not provided, two different strategies are designed to handle these cases ( i.e. , “ 0-tensor ” or “ template-embedding ”, illustrated in Sec. 4.4 and Tab. 5c of the manuscript); 3) Then, θ N is trained for T iterations. For each iteration, a subnet a is randomly sampled from search space A by the function Γ and outputs the predictions p .
The corresponding parameters of a would be updated by gradient descent; 4) After pretraining, Evolutionary Architecture Search [26] is performed to find the optimal subnet a best . The rewarding for evolutionary search is the SUC (success score) on validation data D val . The retraining stage (line 19-27 in Alg. 1) is to optimize the searched subnet a best following the training pipeline of baseline trackers [23, 8]. Tab. 9 displays the detailed configurations of the searched asymmetrical architecture, providing a complement to Tab. 1 in the manuscript.
[Table]
Despite greatly boosting the tracking performance, Neural Architecture Search (NAS) brings complicated training processes and large computation costs. Considering the complexity, we ease unnecessary steps of ASS to achieve a better trade-off between training time and performance. Taking another NAS-based tracker ( i
[Table]
e. , LightTrack [57]) as the comparison [Table] we demonstrate the efficiency of our proposed ASS [Table]
As illustrated in Tab. [Table] NAS-based trackers usually need to first pretrain the supernet on ImageNet [14] to initialize the parameters, which results in high time complexity in training. LightTrack even trains the backbone network on ImageNet [14] twice ( i
[Table]
e. , the 1st and 4th steps), which heavily increases the time complexity.
By contrast, our ASS avoids this cost by reusing the pre-trained supernet from SPOS, which is much more efficient.
[Table]
As shown in Fig. [Table]
the proposed VLT SCAR delivers more robust tracking under deformation, occlusion (the first row) and interference with similar objects (the second row). It demonstrates the effectiveness of learned multimodal representation, especially in complex environments. The third row shows the failure case of our tracker. In this case, the target is fully occluded for aboutframes and distracted by similar objects, leading to ineffectiveness of our tracker in learning helpful information.
A possible solution to deal with this is to apply a global searching strategy
[Table]
and we leave this to future work.
Fig.
6 shows that our VLT TT achieves the best performance compared to other
SOTAs. It demonstrates the resilience of our tracker and effectiveness of proposed multimodal VL tracking in complex environments [Table] A.6 Activation Analysis of Different Language Descriptions
Language description provides high-level semantics to enhance the target-specific channels while suppressing the target-irrelevant ones.
As presented in Fig. 7, we show the effect of different words to evidence that language helps to identify targets. The first row shows that the VLT SCAR without language description focuses on two birds (red areas), interfered by the same object class. When introducing the word “bird”, the response of the similar object is obviously suppressed.
With a more detailed “black bird”, the responses of distractors almost disappear, which reveals that more specific annotation can help the tracker better locate the target area. Furthermore, we also try to track the target with only environmental description, i e. , “standing on the ground”. The result in columnshows that the background is enhanced while the target area is suppressed. The comparison evidences that the language description of object class is crucial for the tracker to distinguish the target from the background clutter, while the mere description of the environment (the fourth column) may introduce interference instead.
The last column shows the activation maps with full description, where the tracker can precisely locate the target, demonstrating the effectiveness of the learned unified-adaptive vision-language representation.
Fig. 8 presents the attribute-based evaluation on LaSOT [17]. We compare VLT SCAR and VLT TT with representative state-of-the-art algorithms, as shown in Fig.
8a. It shows that our methods are more effective than other competing trackers on most attributes. Fig. 8b shows the ablation on different components of VLT SCAR , which evidences that the integration of ModaMixer and ASS is necessary for a powerful VL tracker.
[Figure]