Paper: USTAM: UNIFIED SPATIO-TEMPORAL ATTENTION MIXFORMER FOR VISUAL OBJECT TRACKING (ICLR-2024)

Reviewer comment (Novelty):
The proposed approach is an incremental improvement of MixFormer tracker. Many of the components in the paper such as MAM block, asymmetric attention, loss function have already been proposed in MixFormer.

Author response:
Answer of question (1) Thank you for your kind comment. Contrary to what you understood, all blocks in our network are composed of MAM blocks. However, we apply temporal attention using the attention map of the search region from the previous frame, and we introduce spatial attention in the last MAM block to enhance attention for the target. This way, we leverage spatiotemporal information to improve tracking performance.

Reviewer comment (Presentation):
Rearranging the reference section in order of the last name of the first author would enhance searchability.

Author response:
Answer of question (2) Thank you for your kind suggestion. We have crafted the reference section following the guidelines provided by ICLR to enhance searchability. I hope it proves helpful in your searches.

Reviewer comment (Presentation):
Figure 1 appears too small to discern the letters effectively.

Author response:
Answer of question (4) Thank you for your comment. In Figure 1, we have done our best to effectively portray the overall architecture of the proposed network. While it may not be efficient for character identification, we have prioritized the visual representation as the image is more crucial than the characters. We would appreciate it if you could focus more on the visual aspects rather than the characters.

Reviewer comment (Theory):
In equation (6), given that g_i,j is the attention map in equation (5), it follows that the sum of each row of G_f_t should be 1 after applying softmax. This operation essentially subtracts a constant value. Could you elaborate on the rationale behind this step?

Author response:
Answer of question (5) We are very sorry to confuse you with the typo in equation (6). We changed  j=1 to i=1. Softmax was applied to rows and the average was obtained for columns. The score that each token has on all tokens was calculated by averaging the columns.

Reviewer comment (Novelty):
The contribution in this paper is somewhat incremental. It seems that the proposed framework is still similar to the MixFormer framework, although it uses a ViT-based architecture and considers the previous target state by using the temporal attention module.

Author response:
Answer of question (2) Thank you for your comment. There is a problem that the networks are similar because we use MixFormer as a baseline. However, the existing MixFormer uses a score prediction module for online target update to utilize temporal information. However, this requires additional learning. However, we propose a temporal attention module that simply but effectively utilizes temporal information without the need for additional learning.

Reviewer comment (Theory):
The usage of the temporal attention module is a bit similar to using the Cosine Window (e.g., also used in OSTrack), which also makes the tracker object move smoothly in consecutive frames. In this paper, the authors make it in a learnable way by using the attention map in the previous frame. But one unsolved problem is about the reliability of the previous target state. If the previous prediction is noisy, the effectiveness of the proposed approach is also questionable.

Author response:
Answer of question (3) Thank you for your good comment. Since the prediction of the previous target may be noisy, there is a problem that the reliability of the prediction of the previous target state may decrease. To solve this, we apply a spatial attention module to the last layer of the transformer for the previous frame and a temporal attention module to the first layer of the transformer for the current frame to increase the reliability of predictions about the target state.

Reviewer comment (Experiments):
Missing some essential details and unfair comparison. It is not clear whether the proposed tracker uses the pre-trained models, e.g., OSTrack and DropTrack. In Table 2, the authors compare with the OSTrack-384 only trained on GOT-10k training split, while the proposed approach additionally uses more training data, which is not fair. From Table 3, it seems that the proposed USTAM-B-384 trained on GOT-10K is inferior to OSTrack-384. What’s the reason? Does the compared two approaches use the same pre-trained model?

Author response:
Answer of question (4) Thank you for your comment. We prepared a performance table in Table-2 as compared in other papers. However, there was no indication of the GOT-10k model in Table-2, so it was updated. The GOT-10K model is marked with *. The performance of our GOT-10k model was compared in table-3 of the ablation study section.