Paper: MaXTron: Mask Transformer with Trajectory Attention for Video Panoptic Segmentation (ICLR-2024)

Reviewer comment (Novelty):
The major weakness of this paper is lack of novelty, as each component used in the design of MaXTron is mostly brought from existing literatures or with a subtle modification. For instance, Multi-level Deformable Attention, Transformer Decoder, and ASPP are brought from previous works.

Author response:
We thank the reviewer for the question. We respectfully disagree with the reviewer and clarify the concerns of each component as below:

>> W1.1: Multi-scale Deformable Attention

Multi-scale Deformable Attention (MSDeformAttn) have been introduced in [1], and used in most recent works. We introduce MSDeformAttn in our within-clip tracking module for a fair comparison to other state-of-the-art methods. Most importantly, we show below that though MSDeformAttn brings improvement to performance, the most significant improvement is still brought by our own proposed axial-trajectory attention.

Firstly, we provide additional results on VIPSeg with ResNet-50 as backbone and ablate on the effectiveness of multi-scale deformable attention and the proposed axial-trajectory attention in terms of VPQ:

MSDeformAttn | axial-trajectory attention | VPQ |
|:---------:|:----------:|:--------------:|
X     |    X    |      42.7      |
$\checkmark$     |    X    |      44.5      |
X     |    $\checkmark$    |      44.9      |
$\checkmark$     |    $\checkmark$    |      46.1      |

As can be seen, when compared with the baseline Video-kMaX, adding MSDeformAttn only brings 1.8 VPQ improvement, adding axial-trajectory attention brings 2.2 VPQ improvement. When combining together, we get the final 3.4 VPQ improvement. We also emphasize that Video-kMaX with axial-trajectory attention alone already achieves significant improvement compared to other state-of-the-art methods (Video-kMaX with axial-trajectory attention alone 44.9 vs. current state-of-the-art DVIS 39.2). Please refer to Tab. 5 for more comparisons with different within-clip tracking module designs.

Secondly, for video instance segmentation, please note that Tube-Link already contains MSDeformAttn in their own clip-level segmenter design and we simply add the proposed axial-trajectory attention after their MSDeformAttn. As a result, all performance gains on the near-online setting are brought by the proposed axial-trajectory attention. To be concrete, it brings 0.6 AP, 4.4 $AP^\text{long}$, 5.8 AP improvement when evaluating on Youtube-VIS-21, Youtube-VIS-22 and OVIS dataset with ResNet-50, ResNet-50, and Swin-L as backbone, respectively. Please refer to Tab. 2, 8, 9 and 10 for more comparisons.

In summary, MSDeformAttn does help with the within-clip tracking module, but it is not the core component in it and we do not claim it as our novelty as well. Most importantly, even without MSDeformAttn, our model still brings non-trivial improvement.

>> W1.2: Transformer Decoder

We kindly request the reviewer for clarification on the point that *'Transformer Decoder is borrowed from the literature'.* As far as we are concerned, the transformer decoder is an essential and non-removable component for modern video segmentation models and all other works use it. If the reviewer insists on this point, we kindly request the reviewer for more clarifications, e.g., why the follow-up works should not (and never) use the Transformer Decoder.

>> W1.3: ASPP

ASPP is introduced by [2], and originally operates on the dense pixels in the ***spatial*** domain. As far as we are concerned, we are the first to extend it to the ***temporal*** domain. Motivated by its property in capturing local information, we extend it to operate on the clip object query to reason about the temporal information. This is a non-trivial adaptation of ASPP.

Besides, Tab. 6 (6) in the paper shows that even without ASPP, our usage of trajectory attention on clip object query still brings performance improvement due to the capture of global cross-clip interactions. 

[1] Zhu, Xizhou, et al. "Deformable DETR: Deformable Transformers for End-to-End Object Detection". ICLR 2021 (Oral)

[2] Chen, Liang-Chieh, et al. "DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs". TPAMI 2017

Reviewer comment (Novelty):
The used modules with slight changes are 1) Axial-Trajectory Attention and 2) Cross Clip Tracking Module. The Axial-Trajectory Attention manipulates the set of tokens fed into transformer attentions, which has been widely used in lots of different areas.

Author response:
We thank the reviewer for the question. We respectfully disagree with the reviewer and clarify the concerns of each component as below:

>> W2.1: Axial-Trajectory Attention

We respectfully disagree with the reviewer with the point that *'the Axial-Trajectory Attention manipulates the set of tokens fed into transformer attentions, which has been widely used in lots of different areas'.* As far as we are concerned, ***all*** attention mechanisms manipulate the set of tokens thus it is unfair to blame the design on this. Most importantly, it is non-trivial to adapt trajectory attention to video segmentation task. Though different self-attention mechanisms have been explored a lot in **video classification**, few works try to explore it in **video segmentation** due to the intolerant complexity. Our novel design effectively addresses this key challenge and improves within-clip consistency. Please refer to C1 in common concerns for more discussion on this.
Finally, we kindly request the reviewer to re-evaluate our contributions. If the reviewer insists on the point that it is trivial to design ***axial-trajectory attention***, we kindly request the reviewer for any reference papers that have ever successfully done so in literature.

>> W2.2: Cross-clip Tracking Module (Comparison with VITA)

Our cross-clip tracking module is not a simple modification from VITA. We clarify the detailed differences compared to VITA below:

1. VITA introduces an **additional** set of video object queries on top of the existing frame object queries, while MaXTron instead directly manipulates the clip object queries.

2. VITA contains both an encoder and a decoder to encode the information from frame object queries and decode the information from them via video object queries. As a comparison, MaXTron only employs a simple encoder to aggregate information from clip object queries. 

Concretely, the operation flow of VITA is:
frame query self-attention $\rightarrow$ frame query feed-forward network $\rightarrow$ video query cross-attention to frame query $\rightarrow$ video query self-attention $\rightarrow$ video query feed-forward network

The operation flow of cross-clip tracking module in MaXTron is:
clip query trajectory-attention $\rightarrow$ clip query Temporal-ASPP

As can be seen, the proposed cross-clip tracking module enjoys a ***much simpler*** pipeline compared to VITA.

3. VITA uses their proposed video object queries to predict both the whole video mask and class directly, while MaXTron exploits clip object queries to predict clip mask ***separately*** (the final whole video mask is obtained by concatenating the clip masks) and predicts the class by ***weighted mean*** of aligned clip queries, motivated by the fact that the aligned clip queries should have consistent class prediction. 
We argue that the proposed per-clip mask prediction scheme leads to better video mask prediction, since it is challenging to use only one set of video queries to directly predict the mask for the whole video. We provide additional ablation study on this to support our claim by experimenting with MaXTron w/ Video-kMaX with ResNet-50 as backbone on VIPSeg:

mask prediction scheme | VPQ |
|:---------:|:----------:|
per-clip   (adopted by us)  |    46.7    |
per-video (adopted by VITA)    |    46.4    |

4. Finally, VITA introduces additional similarity loss after introducing their video object queries to better supervise the training, while MaXTron does not need that. Besides, VITA takes frame object queries from the last three layers of the transformer decoder as input, while MaXTron only needs the clip object queries from the last layer.

In the end, we provide experimental results when replacing our cross-clip tracking modules with VITA below (all based on the same Video-kMaX baseline to ensure a fair comparison):

cross-clip tracking design | backbone | input size | GPU days (A100) | params | GFlops (cross-clip tracking module only) | VPQ |
|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
MaXTron     |    ResNet-50    |    $24 \times 769 \times 1345$    |    3.5    |    7.7M    |    32    |    46.7    |
VITA     |    ResNet-50    |    $24 \times 769 \times 1345$   |    5.2    |    12.9M    |    47    |    46.3   |

As shown in the table, our cross-clip tracking module is ***faster*** in training, ***more lightweight*** in params and FLOPs and achieves ***better*** performance compared to VITA.

Reviewer comment (Novelty):
The Cross Clip Tracking Module is a simple modification to VITA, using clip-level outputs instead of frame-level outputs.

Author response:
>> W2.2: Cross-clip Tracking Module (Comparison with VITA)

Our cross-clip tracking module is not a simple modification from VITA. We clarify the detailed differences compared to VITA below:

1. VITA introduces an **additional** set of video object queries on top of the existing frame object queries, while MaXTron instead directly manipulates the clip object queries.

2. VITA contains both an encoder and a decoder to encode the information from frame object queries and decode the information from them via video object queries. As a comparison, MaXTron only employs a simple encoder to aggregate information from clip object queries. 

Concretely, the operation flow of VITA is:
frame query self-attention $\rightarrow$ frame query feed-forward network $\rightarrow$ video query cross-attention to frame query $\rightarrow$ video query self-attention $\rightarrow$ video query feed-forward network

The operation flow of cross-clip tracking module in MaXTron is:
clip query trajectory-attention $\rightarrow$ clip query Temporal-ASPP

As can be seen, the proposed cross-clip tracking module enjoys a ***much simpler*** pipeline compared to VITA.

3. VITA uses their proposed video object queries to predict both the whole video mask and class directly, while MaXTron exploits clip object queries to predict clip mask ***separately*** (the final whole video mask is obtained by concatenating the clip masks) and predicts the class by ***weighted mean*** of aligned clip queries, motivated by the fact that the aligned clip queries should have consistent class prediction. 
We argue that the proposed per-clip mask prediction scheme leads to better video mask prediction, since it is challenging to use only one set of video queries to directly predict the mask for the whole video. We provide additional ablation study on this to support our claim by experimenting with MaXTron w/ Video-kMaX with ResNet-50 as backbone on VIPSeg:

mask prediction scheme | VPQ |
|:---------:|:----------:|
per-clip   (adopted by us)  |    46.7    |
per-video (adopted by VITA)    |    46.4    |

4. Finally, VITA introduces additional similarity loss after introducing their video object queries to better supervise the training, while MaXTron does not need that. Besides, VITA takes frame object queries from the last three layers of the transformer decoder as input, while MaXTron only needs the clip object queries from the last layer.

In the end, we provide experimental results when replacing our cross-clip tracking modules with VITA below (all based on the same Video-kMaX baseline to ensure a fair comparison):

cross-clip tracking design | backbone | input size | GPU days (A100) | params | GFlops (cross-clip tracking module only) | VPQ |
|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
MaXTron     |    ResNet-50    |    $24 \times 769 \times 1345$    |    3.5    |    7.7M    |    32    |    46.7    |
VITA     |    ResNet-50    |    $24 \times 769 \times 1345$   |    5.2    |    12.9M    |    47    |    46.3   |

As shown in the table, our cross-clip tracking module is ***faster*** in training, ***more lightweight*** in params and FLOPs and achieves ***better*** performance compared to VITA.

Reviewer comment (Novelty):
As this paper used a number of components that are already proven effective, it is rather expected to see the gain in the accuracy.

Author response:
> > T2.2 I believe the contributions for the overall architectural design is limited, e.g., using the transformer decoder is to be expected.

We are again sorry to hear the subjective comment conerning the transformer decoder. We would like to emphasize that the transformer decoder is not our main design. To clarify, we believe every modern video segmentation model has used the transformer decoder, including our baselines (Video-kMaX and Tube-Link). However, as shown in our results, the performance improvements over the baseslines are purely from our designs.

Specifically, our architectural design, based on an off-the-shelf clip-level segmenter (Video-kMaX or Tube-Link), contains two parts: within-clip tracking module and cross-clip tracking module. This architectural design is quite novel and targets the difficult tracking problem in video segmentation, leading to a significant performance boost over the baseline clip-level segmenter, as shown below.
- 42.7 $\rightarrow$ 46.7 VPQ w/ Video-kMaX R50 on VIPSeg
- 52.7 $\rightarrow$ 57.1 VPQ w/ Video-kMaX ConvNeXt-L on VIPSeg
- 32.1 $\rightarrow$ 37.0 $AP^\text{long}$ w/ Tube-Link R50 on YTVIS-22
- 34.2 $\rightarrow$ 38.9 $AP^\text{long}$ w/ Tube-Link Swin-L on YTVIS-22

Reviewer comment (Evaluation):
How much are the FLOPs and FPS of MaXTron compared to other methods?

Author response:
> > T4.2 GFlops and FPS of other state-of-the-art methods such as TarVIS, VITA, DVIS

The table below provides a summary of the $AP^\text{long}$ on YTVIS-22, GFlops, and FPS comparison with other state-of-the-art methods. GFlops is computed given an input video of size $36 \times 360 \times 640$. FPS is reported based on the inference speed on YTVIS with a single TITAN RTX GPU. As shown in the table, MaXTron remarkably enhances the performances of Tube-Link, while demonstrating comparable GFlops and FPS with other state-of-the-art methods.


model | $AP^\text{long}$ on YTVIS-22 | input size | GFlops | FPS |
|:---:|:---:|:---:|:---:|:---:|
VITA | 31.9 | $36 \times 360 \times 640$ | 2165 | 24.7 |
TarVIS | 33.8 | $36 \times 360 \times 640$ | 2781 | 11.8 |
DVIS Online | 31.2 | $36 \times 360 \times 640$ | 2136 | 23.6 |
DVIS Offline | 35.9 | $36 \times 360 \times 640$ | 2226 | 17.3 |
Tube-Link | 32.1 | $36 \times 360 \times 640$ | 2219 | 19.4 |
Tube-Link + axial-traject attn (MaXTron near-online) | 36.5 | $36 \times 360 \times 640$ | 2530 | 16.8 |
Tube-Link + axial-traject attn + MaXTron cross-clip (MaXTron offline) | 37.0 | $36 \times 360 \times 640$ | 2594 | 12.7 |

Reviewer comment (Evaluation):
What is the statistical significance, i.e., how many runs were executed for reporting the numbers? Are the numbers mean/median of multiple trials?

Author response:
> Q2: Statistical significance.

We thank the reviewer for the question. Yes, all experiments are conducted 3 times and the reported number is the mean of the results. Please note that we do not report this statistical significance as most works in this literature do not report this in their papers.

Reviewer comment (Experiments):
What if we change the number of frames within one video clip? Is there any positive / negative impact on the model performance? Is there any guidances (or suggestion) of how many frames should be selected while spliting the video?

Author response:
We thank the reviewer for bringing up this question. We observe performance change when varying the number of frames within one video clip. Most likely, it is due to the fact that the off-the-shelf clip-level segmenter itself performs differently with different clip sizes. Concretely, for Video-kMaX and MaXTron w/ Video-kMaX on VIPSeg with ResNet50 as backbone in terms of VPQ, we have:

clip size | Video-kMaX | MaXTron online | MaXTron offline |
|:---------:|:----------:|:--------------:|:---------------:|
2     |    42.7    |      46.1      |       46.7      |
3     |    42.1    |      45.1      |       45.5      |
4     |    41.4    |      44.2      |       44.7      |

As we can see, with the increase of clip size, Video-kMaX performance gradually decreases. However, both our MaXTron online (w/ within-clip tracking module) and MaXTron offline (w/ within-clip + cross-clip tracking module) models bring steady and consistent improvements. We hypothesize that the performance difference of Video-kMaX is due to the lack of a temporal module in its framework and the temporal connections are forced to be modeled by the transformer decoder only. As a result, with the increase of clip size, the transformer decoder can not handle the features properly, leading to the decrease in performance. Our axial-trajectory attention alleviates this problem and achieves slightly larger improvement when clip size = 2. Besides, our cross-clip tracking module is less sensitive to clip size as it directly learns to model video-level input.