Paper: Sparse Mask Representation for Human-Scene Interaction (ICLR-2024)

Reviewer comment (Novelty):
The novelty is limited to sparsifying the input for the human-scene interaction. Using sparse inputs by itself is not new, but it has not been studied for this task before.

Author response:
The use of sparse masks to enhance model speed has been investigated and established as a path within Sparse Coding solutions, as outlined in our Related Work. However, as you mentioned, its effectiveness on the human-scene interaction task has not been studied before. Unlike other methods that mostly apply the sparsity for improving inference speed, our method can increase both accuracy and speed. Despite the simplicity, our method outperforms recent work by a significant margin, for example, we outperform MIME [1] with 2.72% higher accuracy and 60 times faster.

Reviewer comment (Evaluation):
In Figure 6 and Table 4. It seems the model gets worse when using 50 or 10 masks which is strange. Why would using 3 masks be better than 10 or 50 masks? Why would it be even better than using the full dense tensor?

Author response:
We value the feedback provided by the reviewer. The POSA backbone operates on dense tensor inputs. However, ***not all vertices contribute meaningfully*** to predicting human-scene contacts, leading to redundant information that can induce overfitting during the learning process. Sparse masks aim to mitigate this issue by increasing sparsity and selecting useful information. Nonetheless, an excessively large set of sparse masks can introduce a similar problem of redundant information, which adversely affects results when employing 50 or 10 masks.

Among these, the kept 3-mask setup refined by our introduced Sparse Mask Refinement technique yields the best results. This refinement process helps eliminate redundant sparse masks, enabling even better performance than the baseline which still suffers from redundant information issues. For further detailed insights, please refer to Section 4.4 'How do sparse masks help reduce input information?'

Reviewer comment (Novelty):
The paper attributes the COO representation to "Choy 2020". The COO format is much older than that.

Author response:
We appreciate the reviewer for pointing this out and concur that the original COO format has been appropriately cited. To the best of our knowledge, the initial representation of a sparse tensor using the COO format is documented by [2] (Chou et al., 2018), and subsequent extensions to operations on sparse tensors are detailed in [3] (Choy et al., 2020). Please do let us know if we missed any citations.

Reviewer comment (Presentation):
Figure 4 is hard to see. Human bodies are too small.

Author response:
We appreciate your feedback. We have enhanced the size of Figure 4.

Reviewer comment (Experiments):
The method is basically learning mesh subsampling. I wonder about how the method compares to classic subsampling methods.

Author response:
We appreciate the reviewer for mentioning the comparison with mesh subsampling methods. We agree that these methods also enhance the inference speed, with much of the mesh subsampling process occurring at runtime.
    
The table below illustrates the comparison between our proposed SMR and other Mesh Subsampling methods, including typical algorithm-based methods and deep-based ones. Across all cases, our method significantly outperforms other approaches in terms of both speed and accuracy. This result is easily explainable, as most mesh subsampling methods do not adhere to any specific algorithms during the subsampling process to preserve model performance. Additionally, their primary objectives revolve around finding a better discrete representation of a mesh with triangles of equal edge length, rather than catering to human-scene interaction tasks. Besides, some deep-based mesh simplifier methods also take time to finish their sampling process.

|Methods|Mesh Subsampling|Reconstruction Accuracy(%)|Speed (s/sample)|
|:-|:-:|:-:|:-:|
|Baseline (POSA)|Algorithm-based|91.12|0.28|
|Isotropic Remeshing [4]|Algorithm-based |82.18(-8.94)|0.18($\downarrow$ 1.56)|
|Vertex Clustering [5]|Algorithm-based | 78.67 (-12.45)|0.13($\downarrow$ 2.15)|
|Incremental Decimation [6]|Algorithm-based | 79.45 (-11.67) | 0.09 ($\downarrow$ 3.11)|
|Neural Mesh Simplification [7]|Deep-based|85.56(-5.56)|0.47($\uparrow$ 1.68)|
|CoMA [8]|Deep-based|89.22(-1.90) |0.23($\downarrow$ 1.22)|
|**Ours**|-|**93.69**(+2.57)|**0.01**($\downarrow$ 28)|

*Result comparison between SMR and other mesh subsampling methods. Results are benchmarked on the PROXD dataset.*

Reviewer comment (Evaluation):
I understand that the method can be faster than POSA, but why would it be more accurate?

Author response:
The baseline POSA, which involves a comprehensive tensor setup, includes numerous vertices. Many of these vertices are redundant for the human-scene interaction task and could potentially lead to overfitting. However, by utilizing sparse masks and implementing the Sparse Mask Refinement technique, we effectively mitigate this issue, resulting in superior outcomes compared to the baseline POSA. Figure 3 and Table 4 in our main paper demonstrate this scenario.

Reviewer comment (Novelty):
Methodology is incremental and not much novelty by itself.

Author response:
We thank the reviewer for the comment, we verify our contribution as below:
> - We propose a simple yet effective method that focuses on ***the representation of inputs used for human-scene interaction***. Our method reduces processing time while enhancing model effectiveness by eliminating redundant information. Existing works on human-scene interaction have not yet investigated this problem.
> - We not only utilize sparse masks as in previous work but also propose a refinement strategy to create zero-filtered inputs.
> - Despite simplicity, our method outperforms other state-of-the-art methods significantly. For example, our method outperforms MIME [1] with ***2.72% higher accuracy and 60 times faster***.

Reviewer comment (Evaluation):
From Figure 6, it shows optimal performance for k = 3 mask with 90% sparsity, but there is no clear pattern or correlation between sparsity ratio vs accuracy. Could authors give an explanation on this trend?

Author response:
We appreciate the reviewer's feedback. In the original submission, we have shown the correlation between the sparsity ratio, number of masks, accuracy, and running time in Figure 6. That figure shows that using only one mask results in a significant drop in performance due to missing information. Increasing the number of masks helps retain essential information and enhances the overall model performance. However, an excessive number of sparse masks (50 masks) can introduce redundant and inconsistent information, potentially lowering the overall accuracy. By maintaining the sparse masks at appropriate values (which can be easily achieved using Sparse Mask Refinement), we can optimize performance while significantly speeding up the evaluation process.
   
    
>As per your suggestion, we have further added a comprehensive discussion about the mentioned information in Section E.1 of our Appendix, providing a more comprehensive exploration. This includes an in-depth analysis of the correlation between the number of sparse masks and sparsity concerning accuracy. Additionally, another experiment examining inference speed is also conducted within this section.