Paper: Unified Static and Dynamic: Temporal Filtering Network for Efficient Video Grounding (ICLR-2024)

Reviewer comment (Experiments):
What is the motivation behind the implementation of Static Semantic Supplement Network? I am wondering how the cross-modal interaction is performed through the MLP layers. To my understanding, the shared weights across different modalities would extract some common features spanning different modalities. Some analytical experiments on this would be beneficial. Also, the architecture design seems similar to that of Transformer blocks except for the self-attention. What happens if we use the conventional Transformer layers?

Author response:
Thank you for raising the concerns about the motivation of static network. We would like to explain the implementation of our static network from the perspectives of motivation, technology, and experiment.

- **Motivation-wise,** the implementation of Static Semantic Supplement Network is strongly inspired by the **global neural workspace (GNW)** theory in biology [R15], that is, for video understanding, the brain will perform 'global broadcast communication' (interaction) of the multi-source information in the early stage of visual perception for the video, while the cognitive consciousness of the brain will **strengthen perceptual conditions** and **delay conditioned reflexes**, enabling the human brain to understand new conditioned reflexes brought about by perceptual information updates.

    On the premise of the above theory, this inspires us to utilize residual networks structure to update our understanding of video sequences in the delaying conditioned reflex states by aggregating the perceptual information. **Multilayer perceptron (MLP)** is proven to be effective and has strong interpretability in modeling multi-level cognitive processing [R19], thus we use MLP to aggregate the perceptual information. The visual consciousness process is modeled as follows：$x^{n+1}$ = $\text{ResMLP}(x^n)$ = $x^n$+$\text{MLP}(x^n)$, where $x^n$ represents the perception condition in the current state, $x^{n+1}$ denotes the reflection result, and $\text{ResMLP}(x^n)$ can enhance the information of the perception condition $x^n$.

- **Technology-wise**,  we concatenate multiple queries and video clips together as a sequence input into the network for fully connected interaction of all tokens, its main function is to supplement and associate semantics between modalities before the latter graph filtering, provide more video descriptions information and significantly fill the gap between vision-language modalities, aiding in understanding video content.

- **Experiment-wise**, as per your valuable suggestion, we have tested the effect of Transformer as a static network as shown in the table below (**Table R6**), and as you are concerned, Reviewer 'TT77' has also focused on the ablation study of the static module. From the results, in terms of performance and efficiency, Transformer is close to our method, but our results are better. We speculate that the reason is that our network also includes the second stage of graph filtering. The static network uses a lightweight and stable network, which is more conducive to model training. Using Transformer as a static network increases the weight and instability factors [R20] of the network. 

**Table R6: Results of different static networks on the ActivityNet Captions.**

|Static | Infer. Speed (s/query) | R@1, IoU@0.3 | R@1, IoU@0.5 | R@1, IoU@0.7  |  R@5, IoU@0.3 | R@5, IoU@0.5  | R@5, IoU@0.7 | mIoU  | 
| :---------- | :----------: | :----------:  | :----------: | :----------:  | :----------:  | :----------:  | :----------:  | :----------: | 
| Transformer | 0.024 | 75.17 | 59.98 | 38.38 | 91.26 | 85.77 | 74.25 | 54.97| 
|**Our ResMLP** | **0.009** | **75.85**	| **60.75**	| **38.88**	| **91.16** | **85.34** | **74.01**	| **55.47** |

**References:**

[R15] Volzhenin et al., Multilevel development of cognitive abilities in an artificial neural network. Proceedings of the National Academy of Sciences, 119(39), pp. e2201304119, 2022.

[R19] Chavlis et al., Drawing inspiration from biological dendrites to empower artificial neural networks. Current opinion in neurobiology. 70, pp. 1-10, 2021. 

[R20] Hugo et al., Resmlp: Feedforward networks for image classification with data-efficient training. TPAMI, 45(4):5314–5321, 2022.

Reviewer comment (Experiments):
The proposed architecture exploits multiple queries at once, to facilitate the model learning. However, how the number of queries affects the performance is not diagnosed. An ablative study on the number of queries regarding performance and cost would be helpful.

Author response:
Thanks for your kind and valuable suggestion. Following your suggestions, we have conducted ablation studies on the impact of query quantity on model performance on the ActivityNet Captions dataset. An explanation of the distribution of query quantity per video is that, the training set of this dataset contains 10,009 videos, with a minimum of 2 and a maximum of 27 queries per video. Due to the number of queries for each video in this dataset being concentrated in [3, 8], we set the upper limits for the number of queries $M$ per video feed as 1, 3, 5, 8, and no upper limit. In fact, the 25\%, 50\%, and 75\% quantiles are 3, 3, and 4, respectively. When the $M$ = 8, 97.33\% of samples in the dataset are equipped with full queries.

The results are shown in **Table R7** below, and according to the experiments, when the number of training query inputs $M=1$, our model is comparable with state-of-arts. As the query number upper limit increases, the performance of our model improves, which demonstrates the effectiveness of our model in utilizing multimodal information.

**Table R7: The impact of query quantity on ActivityNet Captions dataset for NLVG task.**

|Method (\#Query) | R@1, IoU@0.3 | R@1, IoU@0.5 | R@1, IoU@0.7  |  R@5, IoU@0.3 | R@5, IoU@0.5  | R@5, IoU@0.7 | mIoU  | 
| :---------- | :----------:  | :----------:  | :----------:  | :----------:  | :----------:  | :----------: | :----------: |
| UniSDNet ($M=1$)     | 68.66 | 52.35 | 32.25 | 89.74 | 83.35 | 70.61 | 50.22 |
| UniSDNet ($M\leq 3$) | 72.42 | 57.30 | 36.64 | 89.82 | 83.88 | 72.14 | 53.30 |
| UniSDNet ($M\leq 5$) | 74.41 | 59.27 | 37.70 | 90.83 | 85.07 | 74.17 | 54.58 |
| UniSDNet ($M\leq 8$) | 74.79 | 60.28 | 38.35 | 90.68 | 85.15 | 73.90 | 55.03 |
| **UniSDNet (Full)**  |**75.85**| **60.75** | **38.88** | **91.16** | **85.34** | **74.01** | **55.47** |

Reviewer comment (Experiments):
The proposed method is well validated in the datasets with one-to-one matching between queries and moments. How would it perform for one-to-many matching datasets, such as QVHighlights [1]?

Author response:
Thanks for pointing out the excellent work of QVHighlights and constructive comments. 
QVHighlights is a recently publicized dataset for both moment retrieval (MR) and highlight detection (HD) tasks.  
Following the practice [R21, R22], the commonly used metric is Recall@K, IoU=[0.5, 0.7].
As your valuable suggestion, we have tested our model on this dataset. Current works on the QVHighlights dataset have undergone multi-task learning, including both MR and HD tasks. 
However, due to the urgent rebuttal deadline, we only complete our model evaluation for the MR single task mode on this dataset. 
We have listed the reported results of exiting works for the MR single task in **Table R8** and compared ours with theirs. From the results and facts, there are some conclusions. 

- Our method still performs the best with signle MR task test in one-to-many query matching mode.
This also proves that the good universality of our model for the task setup of one query to multiple moments retrieval, in which each query corresponds to multiple moments retrieval.

- To be applicable to both HD and MR tasks, M-DETR [R21] and UMT [R22] are extended on the basis of the DETR method with the learnable moment queries. We adopt the typical 2D temporal proposal infrastructure and our methods still performs well in solving one-to-many matching MR task. This demonstrates our model's advantage of high precision (R@1, IoU@0.7 is 40.39, much higher than the 33.82 of UMT, and 25.87 of M-DETR).

- In the future, we will continue to collaborate with HD tasks, whose multi-task joint training method has been proven to further promote the MR results [R22],
and will finish this comparison with fair conditions and believe that we will have better performance. 

**Table R8: Comparison with the state-of-art for one-to-many query matching on QVHighlights for single MR task.**
| Method       |        Venue | Task | Video | R@1, IoU@0.5 | R@1, IoU@0.7  | 
| :---------- | :----------  | :----------:  | :----------:  | :----------:  | :----------:  | 
| M-DETR [R21] | NeurIPS 2021 | Single MR task | Slowfast + CLIP | 44.84   |    25.87    | 
| UMT [R22]    | CVPR 2022    | Single MR task | Slowfast + CLIP | 54.14   |    33.82    | 
| **Ours**     |       | **Single MR task** | **Slowfast + CLIP** | **54.58**   |    **40.39**   | 

**References:**

[R21] Lei et al., QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries. NeurIPS, pp. 11846--11858, 2021.

[R22] Liu et al., Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. CVPR, pp. 3042-3051, 2022.

Reviewer comment (Theory):
The inspiration from human visual perception biology is not very motivating. Specifically, it is hard to see why a MLP with residual connection is the way to achieve the “global broadcast communication” of the brain. Either bridge the gap or simply drop the bio-inspiration and go straight into the technical method.

Author response:
Thank you very much for your attention to the design motivation of static networks (MLP with residual connection, ResMLP). Here, we clarify the motivation to bridge the gap between the biological theory 'global broadcast communication' and the technical design of static network ResMLP. 
Sincerely, our work has been strongly inspired by the latest theories of human visual perception [R14, R15]. The **global neuronal workspace (GNW)** theory mentioned in [R15] is the foundation of conscious work in visual understanding [R15, R16, R17], it can achieve the fusion process of multi-source information in the early stage of visual event recognition, that is, the **"global broadcast communication"** of the brain. GNW theory indicates that cognitive consciousness has the function of **strengthening perceptual conditions** and **delaying conditioned reflexes** [R15, R18], enabling the human brain to understand new conditioned reflexes brought about by perceptual information updates [R15, R16].

On the premise of the above theory, this inspires us to utilize residual network structure to update our understanding of video sequences in the delaying conditioned reflex states by aggregating the perceptual information. **Multilayer perceptron (MLP)** is proven to be effective and has strong interpretability in modeling multi-level cognitive processing [R19], thus we use MLP to aggregate the perceptual information. The visual consciousness process is modeled as follows: $x^{n+1} = \text{ResMLP}(x^n) = x^n+\text{MLP}(x^n),$ where $x^n$ represents the perception condition in the current state, $x^{n+1}$ denotes the reflection result, and $\text{ResMLP}(x^n)$ can enhance the information of the perception condition $x^n$.

Technically, in this work, we adopt a unified framework of static (ResMLP) and dynamic (graph filtering) structures for video grounding. In the early stage, this global broadcast mode of ResMLP first perceives all multimodal information, while the latter graph filter purifies key visual information. 
This pre-broadcast learning mode of ResMLP is very necessary as a global biological perception mechanism and has also verified its effectiveness via extensive experiments, as discussed in the paper. We will add the explanation on the rationality of our static network design from a **technical perspective** and incorporate your suggestion into the paper writing by considering both motivation and technical implements for better clarification. 

**References:**

[R14] Barbosa et al., Interplay between persistent activity and activity-silent dynamics in the prefrontal cortex underlies serial biases in working memory. Nature neuroscience, 23(8), pp. 1016-1024, 2020. 

[R15] Volzhenin et al., Multilevel development of cognitive abilities in an artificial neural network. Proceedings of the National Academy of Sciences, 119(39), pp. e2201304119, 2022.

[R16] Cleeremans et al., Consciousness matters: Phenomenal experience has functional value. Neuroscience of consciousness. 2022, pp. niac007, 2022.

[R17] Richards et al., A deep learning framework for neuroscience. Nature neuroscience, 22, pp. 1761-1770, 2019.

[R18] Grover et al., Differential mechanisms underlie trace and delay conditioning in Drosophila. Nature, 603, pp. 302-308, 2022. 

[R19] Chavlis et al., Drawing inspiration from biological dendrites to empower artificial neural networks. Current opinion in neurobiology. 70, pp. 1-10, 2021.

Reviewer comment (Experiments):
Ablation in Fig 5 shows mostly similar results especially on NLVG, indicating that the designs in Dynamic Filter Graph actually do not quite matter.

Author response:
Thank you for your valuable feedback. Fig 5 in the main paper may cause your misunderstanding. Please allow us to provide additional results below (**Table R5**) to address your concerns. 
As shown in **Table R5**, our DTFNet consistently achieves the best performance compared to other graph settings, such as achieving 38.88 at the metric R@1, IoU@0.7. 

*And, why are the results of these graph methods relatively close for NLVG task not as well as the performance improvement of SLVG?*
There may be different characterizations of original textural and audio features. 
In our experiments, we use DistilBERT as textual features and Data2vec as audio features, where DistilBERT is an acknowledged high-quality textual features. 
For the NLVG task, the performance improvement with high-quality features brought by the method development is approaching saturation.
In this case, while the R@1, IoU@0.3 increases by 2.36% and the mIoU increases by 1.92% compared to GAT, they indeed mean a significant improvement for NLVG. 
As for SLVG, the audio-video interaction has the potential to be developed deeply. 

**Table R5: Different dynamic modeling methods on the ActivityNet Captions for NLVG task.**

|Method | R@1, IoU@0.3 | R@1, IoU@0.5 | R@1, IoU@0.7  |  R@5, IoU@0.3 | R@5, IoU@0.5  | R@5, IoU@0.7 | mIoU  |
| :---------- | :----------: | :----------: | :----------: | :----------: | :----------: | :----------: | :----------: |
| GCN | 73.33  | 58.14 | 38.57 | 89.86 | 84.52 | 73.22 | 53.53 |
| GAT | 73.49  | 58.78 | 38.32 | 89.92 | 84.60 | 72.21 | 53.55 |
| D   | 74.40  | 59.16 | 38.46 | 90.42 | 84.62 | 73.26 | 54.06 |
| MLP | 74.29  | 59.52 | 38.07 | 90.52 | 84.48 | 72.88 | 54.12 |
| **Our DTFNet** | **75.85**  | **60.75** | **38.88** | **91.16** | **85.34** | **74.01** | **55.47** |

Reviewer comment (Presentation):
The first page is missing a header.

Author response:
Thank you for pointing out the format issues. We have carefully proofread our paper and fixed them. 

We hope our response addresses your concerns. If there are any further questions, please do not hesitate to let us know. Thank you for your time and valuable feedback.

Reviewer comment (Novelty):
The introduction reads like a related work. It will be great to make more comparison between this work and previous work. Answering what is wrong with previous works? and where the efficiency and performance gain come from in this paper?

Author response:
Thanks for your valuable suggestion. Technology-wise, previous methods mostly focus on solving a certain aspect of the Temporal Video Grounding (TVG) task, such as representation learning of language and video self-modality [R1, R2], multimodal fusion [R3, R4], cross-modal interaction [R5, R6], proposal candidate generation [R7, R8], proposal-based cross-modal matching [R9, R10], target moment boundary regression [R11, R12], and so on.
Our work actually proposes a new paradigm to establish a two-stage unified static and dynamic semantic complementary new architecture. Its unique characteristics are that
- Processing multimodal signals in a unified ResMLP network, while many previous works are independently encoding the language modality and video modality [R1-R7, R9-R12].
- After the implementation of the above static ResMLP, we introduce a Gaussian nonlinear filtering method to learn the semantic associations within the video and combine it with the back-end proposal generation to promote cross-modal semantic alignment, further developing the proposal-based TVG method.

Our work is inspired by visual perception biology. This unified static and dynamic two-stage architecture performs excellent joint learning of language and video, which achieves state-of-the-art performance on NLVG and SLVG tasks. 
The ablation experiments in **Table 4** of our manuscript demonstrate the effectiveness of each proposed component of this work. We will improve our manuscript based on your constructive suggestions.

**Eﬀiciency and performance:** Technology-wise, the inference efficiency of our model comes from a streamlined architecture design, which allows for parallel inference of video and multiple queries in a unified ResMLP, saving time and expenses.
Our promising model performance mainly comes from our model's comprehensive information interaction with query and video (ResMLP, static network), as well as our detailed inference of video content (DTFNet, dynamic graph filtering network), which was confirmed in the first ablation experiment in **Table 4** of the manuscript.

**References:**

[R1] Xia et al., Video-guided curriculum learning for spoken video grounding. ACM MM, pp. 5191–5200, 2022.

[R2] Rodriguez et al., Memory-efficient temporal moment localization in long videos. ECAL, pp. 1901–1916, 2023.

[R3] Li et al., Proposal-free video grounding with contextual pyramid network. AAAI, pp, 1902–1910, 2021. 

[R4] Liu et al., Exploring optical-flow-guided motion and detection-based appearance for temporal sentence grounding. IEEE TMM, 2023.

[R5] Liu et al., Jointly cross-and self-modal graph attention network for query-based moment localization. ACM MM, pp. 4070–4078, 2020.

[R6] Sun et al., Video moment retrieval via comprehensive relation-aware network. IEEE TCSVT, 2023.

[R7] Zhang et al., Learning 2d temporal adjacent networks for moment localization with natural language. AAAI, pp. 12870–12877, 2020.

[R8] Zhang et al., Multi-stage aggregated transformer network for temporal language localization in videos. CVPR, pp. 12669–12678, 2021. 

[R9] Gao et al., Fast video moment retrieval. ICCV, pp. 1523–1532, 2021.

[R10] Zheng et al., Phrase-level temporal relationship mining for temporal sentence localization. AAAI, pp. 3669–3677, 2023.

[R11] Zhang et al., Natural language video localization: A revisit in span-based question answering framework. IEEE TPAMI, 44(8), pp. 4252–4266, 2021.

[R12] Liu et al., Skimming, locating, then perusing: A human-like framework for natural language video localization. ACM MM, pp. 4536–4545, 2022.

Reviewer comment (Writing):
This paper introduces some new/confusing terminologies with their own definition, which hurts the reading experience. For example, 'static semantic supplement network' and 'activity-silent mechanism' are actually the global context interaction.

Author response:
Thanks for your friendly reminder. To clear up your confusion, we clarify and revise some confusing representations as below.

**1) 'static semantic supplement network':** This is the network naming in terms of function, because in our work, we focus on understanding video content in a multimodal environment [R13], and we adopt a unified framework of static and dynamic structures. In the early stage, this global interaction mode first perceives all multimodal information, and then information filtering is performed. In terms of the functionality of the static network for video understanding, it provides more video descriptions information and significantly fills the gap between vision-language modalities, aiding in understanding video content.

**2) 'activity-silent mechanism':** This is the mechanism by which the brain processes information in the early stages of human visual perception for the video, as mentioned in [R14]. It manifests as a static multi-source information interaction, achieving the 'global broadcast communication' of the brain. Because we are strongly inspired by this mechanism in static network design, we specifically mentioned this professional biological term in our manuscript. This network can achieve the effect of 'the global context interaction' from a technical perspective, and this is the basic intention for our network design. 

We will incorporate your suggestion and consider both motivation and technical introduction in the paper.

**References:**

[R13] Lisa et al., Localizing moments in video with natural language. CVPR, pp. 5803-5812, 2017.

[R14] Barbosa et al., Interplay between persistent activity and activity-silent dynamics in the prefrontal cortex underlies serial biases in working memory. Nature neuroscience, 23(8), 1016-1024, 2020.

Reviewer comment (Experiments):
Although the motivation of static and dynamic network is demonstrated, the justification of specific design is not enough. For example, in the static network, transformer architecture or the recent S4[1] architecture can also be used as long-range filter. Some ablation studies regarding either the performance or efficiency would be great to include.

Author response:
Thank you for your valuable suggestion.
For static perception modeling of multi-source information, based on the theoretical guidance of biology, we consider using Multilayer perceptron (MLP) to handle language and video uniformly.
The reviewer suggested we consider more long-range filters in our studies, therefore, we have added the ablation experiments about our static network, and the results are shown in **Table R3**:

**Table R3: Comparison of different static networks on ActivityNet Captions for NLVG task.**

| Static | Infer. Speed (s/query) | R@1,IoU@0.3 | R@1,IoU@0.5 | R@1,IoU@0.7  |  R@5,IoU@0.3 | R@5,IoU@0.5  | R@5,IoU@0.7 | mIoU  | 
| :---------- | :----------: | :----------: | :----------: | :----------: | :----------: | :----------: | :----------: | :----------: |
| Transformer | 0.024 | 75.17 | 59.98 | 38.38 | 91.26 | 85.77 | 74.25 | 54.97 |
| S4          | 0.030 | 70.41 | 55.11 | 34.93 | 89.12 | 83.16 | 70.54 | 51.40 |
| **Our ResMLP** | **0.009** | **75.85** | **60.75** | **38.88** | **91.16** | **85.34** | **74.01** | **55.47**| 

The results indicate that: 
- ResMLP achieves model performance/efficiency trade-offs.
- Note that the S4 model is particularly skilled in handling very long (about 1w) sequence tasks, but in the ActivityNet Captions dataset of NLVG tasks, the total number of video clips and queries does not exceed 100, which may be the reason why it is not suitable for our task.
- The performance of Transformer is close to ResMLP, but its 'Infer. Speed' cost is 2.67 times that of ResMLP. In terms of better performance and efficiency, we choose ResMLP as the implementation of the static network.

Reviewer comment (Writing):
No notation for the 'FNN'. Is this the feedforward network?

Author response:
Yes, it is. 'FNN' is the abbreviation of feedforward network. We have unified the expression of FNN and added a full name explanation in the paper.

Reviewer comment (Presentation):
In the Figure 5, no notation/description for 'D'.

Author response:
The following is a description of **D**, and we will provide a clearer explanation of **D** in the paper.
In the setting of **D**, we use the message aggregation wight $f_{ij}=1/(d_{ij}+1)$ to replace $f_{ij}$=$\mathcal{F}\_{filter}(d_{ij})$, which indicates that we still consider the clue of temporal distance $d_{ij}$ between two nodes but remove the entire Gaussian filtering calculation from our method. 
This replacement results in a decrease of 0.42 and 1.92 on R@1, IoU@0.7 for NLVG and SLVG, respectively.

Reviewer comment (Experiments):
In ResMLP, visual features and multiple query features are concatenated and fed into the network, largely leveraging the information leakage between different queries (because the features incorporate more accurate textual information that describes the video content). If each query is individually input into the network, would this method exhibit a significant performance degradation?

Author response:
Thanks for your recognition of our proposed Dynamic Temporal Filter Network, and your insightful reply is very helpful. We follow your advice and add the new experimental results about comparing multi-query input and single-query input.

> **Q1: In ResMLP, visual features and multiple query features are concatenated and fed into the network, largely leveraging the information leakage between different queries (because the features incorporate more accurate textual information that describes the video content). If each query is individually input into the network, would this method exhibit a significant performance degradation?**

**A1:** Thank you for your valuable feedback. Here, we attempt to clarify our conclusion from the following two aspects. On one hand, regardless multi-query or single-query modes, the role of Multilayer perceptron with residual design (ResMLP) is designed to capture the associations between query and video, simulating the brain's processing of multi-source information in the early stage of visual perception for the video. On the other hand, yes, we agree with your opinion that multiple queries can indeed provide more semantics to the video and significantly fill the gap between vision-language modalities, aiding in understanding video content.  
To eliminate your concern, we have supplemented the relevant experiments and found two facts in **Table R1**:

- Our model still performs best in single-query input mode, compared to other single query methods. For example, our R@1, IoU@0.7 is 32.25, exceeding the current SOTA methods.

- At present, there are some multi-query methods, but their advantages over the single-query methods are not obvious. How to better utilize multiple queries to assist in video grounding is also a challenge. Our method has advantages in handling both single-query and multi-query input modes. 

**Table R1: Model size vs. Infer. Speed vs. Performance comparison in different query input modes.**

|(#Query) | Method | Model Size | Infer. Speed (s/Query) | R@1, IoU@0.3 | R@1, IoU@0.5 | R@1, IoU@0.7  |  R@5, IoU@0.3 | R@5, IoU@0.5  | R@5, IoU@0.7 | mIoU  |
| :---------- | :---------- | :---------- | :----------: | :----------: | :----------: | :----------: | :----------: | :----------: | :----------: | :----------:| 
| Single | 2D-TAN  |  21.62M  |  0.061 | 59.45 | 44.51 | 26.54 | 85.53 | 77.13 | 61.96 | - |
| Single | MS-2D-TAN  | 479.46M  |  0.141 | 61.04 | 46.16 | 29.21 | 87.30 | 78.80 | 60.85 | - |
| Single | MSAT       |  37.19M  |  0.042 |   -   | 48.02 | 31.78 |   -   | 78.02 | 63.18 | - |
| **Single** |**UNiSDNet**|**76.52M**|**0.009** |**68.66**|**52.35**|**32.25**|**89.74**|**83.35**|**70.61**|**50.22**|
| Multi  | MMN       | 152.22M | 0.014 | 65.05 | 48.59 | 29.26 | 87.25 | 79.50 | 64.76 | - |
| Multi  | PTRM      | 152.25M | 0.038 | 66.41 | 50.44 | 31.18 |   -   |   -   |   -   | 47.68 |
| **Multi**  | **UNiSDNet**  |**76.52M**|**0.009**|**75.85**|**60.75**|**38.88**|**91.16**|**85.34**|**74.01**|**55.47**|

Reviewer comment (Experiments):
In the ablation study, individually employing the static network and DTFNet yields significant improvements compared to the baseline. However, the combination of both modules does not exhibit a notably large improvement compared to using either single module. Is there a specific explanation for this phenomenon? The authors should provide more details about the baseline models.

Author response:
> **Q2: In the ablation study, individually employing the static network and DTFNet yields significant improvements compared to the baseline. However, the combination of both modules does not exhibit a notably large improvement compared to using either single module. Is there a specific explanation for this phenomenon? The authors should provide more details about the baseline models.**

**A2:** Thank you for raising the concerns about the main network modules. 
In our manuscript, we propose a method that starts with static global review queries and video content (ResMLP), and then dynamically filters video content to extract important information (DTFNet). 
On the one hand, observing the experimental results of our static module alone (ResMLP) compared with existing works (in Tables 1 and 4 of our manuscript), previous works have overlooked and performed insufficiently in extracting information from global static reviews.
On the other hand, even if the static review information is missed and only basic cross-modal information is obtained, we achieve good video grounding results using the dynamic module (DTFNet) alone. **Effective information filtering method (DTFNet)** can still extract useful information.

For convenience, we restate Table 4 of our manuscript in the following **Table R2**. Relatively speaking, the large performance improvement of a single module through the respective exploration of multimodal information has reached **saturation** (for NLVG task, the R@1, IoU@0.3 of ResMLP and DTFNet are 73.57 and 74.56, respectively.). 
In this case, the R@1, IoU@0.3 of a single module increases from 73.57/74.56 to the final static-dynamic combination of 75.85, which is not a small improvement. 
That's why the effect of combining ResMLP and DTFNet is not as much improved as employing them alone.
This further demonstrates the effectiveness of our static and dynamic modules. 

**Table R2: Ablation studies on the static and dynamic modules on the ActivityNet Captions and ActivityNet Speech datasets for both NLVG and SLVG.**
|Task | Static Module | Dynamic Module | R@1, IoU@0.3 | R@1, IoU@0.5 | R@1, IoU@0.7  |  R@5, IoU@0.3 | R@5, IoU@0.5  | R@5, IoU@0.7 | mIoU  |
| :----------: | :----------: | :----------: | :----------: | :----------: | :----------: | :----------: | :----------: | :----------: | :----------: |
| NVLG | N | N | 61.22 | 44.46 | 26.76 | 87.19 | 78.63 | 63.60 | 43.98 |
| NVLG | Y | N | 73.57 | 58.70 | 37.07 | 91.17 | 85.55 | 73.98 | 54.06 |
| NVLG | N | Y | 74.56 | 59.45 | 37.44 | 90.98 | 85.43 | 73.60 | 54.43 |
| **NVLG** | **Y** | **Y** | **75.85** | **60.75** | **38.88** | **91.16** | **85.34** | **74.01** | **55.47** |
| SLVG | N | N | 53.63 | 35.91 | 20.51 | 84.71 | 74.21 | 55.95 | 38.23 |
| SLVG | Y | N | 69.71 | 53.75 | 31.26 | 90.42 | 84.11 | 70.82 | 50.69 |
| SLVG | N | Y | 71.34 | 54.03 | 31.51 | 89.75 | 82.62 | 68.12 | 50.97 |
|**SLVG** | **Y** | **Y** | **72.27** | **56.29** | **33.29** | **90.41** | **84.28** | **72.42** | **52.22** |


**About the baseline model**, we belong to the scope of the 2D proposal-based method. Specifically, we adopt the same backend decoding method as MMN (including the 2D proposal generation module). For training losses, we use the cross entropy loss between the predicted 2D proposal score map and groundtruth which is consistent with 2D-TAN, and add the contrastive loss proposed in MMN as an auxiliary constraint. That is to say, our method is based on the typical 2D temporal map's backend decoding architecture (2D-TAN and MMN), and by incorporating a static-dynamic module design as the fore-end implementation. 
Our work further develops the 2D proposal-based method in this field. 

We hope our response addresses your concerns. If there are any further questions, please let us know. Thank you for your time and valuable feedback.

Reviewer comment (Experiments):
The authors should provide more details of baseline models in the ablation study and some experimental results about comparing multi-query input and single-query input.

Author response:
Thanks for your recognition of our proposed Dynamic Temporal Filter Network, and your insightful reply is very helpful. We follow your advice and add the new experimental results about comparing multi-query input and single-query input.

> **Q1: In ResMLP, visual features and multiple query features are concatenated and fed into the network, largely leveraging the information leakage between different queries (because the features incorporate more accurate textual information that describes the video content). If each query is individually input into the network, would this method exhibit a significant performance degradation?**

**A1:** Thank you for your valuable feedback. Here, we attempt to clarify our conclusion from the following two aspects. On one hand, regardless multi-query or single-query modes, the role of Multilayer perceptron with residual design (ResMLP) is designed to capture the associations between query and video, simulating the brain's processing of multi-source information in the early stage of visual perception for the video. On the other hand, yes, we agree with your opinion that multiple queries can indeed provide more semantics to the video and significantly fill the gap between vision-language modalities, aiding in understanding video content.  
To eliminate your concern, we have supplemented the relevant experiments and found two facts in **Table R1**:

- Our model still performs best in single-query input mode, compared to other single query methods. For example, our R@1, IoU@0.7 is 32.25, exceeding the current SOTA methods.

- At present, there are some multi-query methods, but their advantages over the single-query methods are not obvious. How to better utilize multiple queries to assist in video grounding is also a challenge. Our method has advantages in handling both single-query and multi-query input modes. 

**Table R1: Model size vs. Infer. Speed vs. Performance comparison in different query input modes.**

|(#Query) | Method | Model Size | Infer. Speed (s/Query) | R@1, IoU@0.3 | R@1, IoU@0.5 | R@1, IoU@0.7  |  R@5, IoU@0.3 | R@5, IoU@0.5  | R@5, IoU@0.7 | mIoU  |
| :---------- | :---------- | :---------- | :----------: | :----------: | :----------: | :----------: | :----------: | :----------: | :----------: | :----------:| 
| Single | 2D-TAN  |  21.62M  |  0.061 | 59.45 | 44.51 | 26.54 | 85.53 | 77.13 | 61.96 | - |
| Single | MS-2D-TAN  | 479.46M  |  0.141 | 61.04 | 46.16 | 29.21 | 87.30 | 78.80 | 60.85 | - |
| Single | MSAT       |  37.19M  |  0.042 |   -   | 48.02 | 31.78 |   -   | 78.02 | 63.18 | - |
| **Single** |**UNiSDNet**|**76.52M**|**0.009** |**68.66**|**52.35**|**32.25**|**89.74**|**83.35**|**70.61**|**50.22**|
| Multi  | MMN       | 152.22M | 0.014 | 65.05 | 48.59 | 29.26 | 87.25 | 79.50 | 64.76 | - |
| Multi  | PTRM      | 152.25M | 0.038 | 66.41 | 50.44 | 31.18 |   -   |   -   |   -   | 47.68 |
| **Multi**  | **UNiSDNet**  |**76.52M**|**0.009**|**75.85**|**60.75**|**38.88**|**91.16**|**85.34**|**74.01**|**55.47**|