Paper: ReBaR: Reference-Based Reasoning for Robust Human Pose and Shape Estimation from Monocular Images (ICLR-2024)

Reviewer comment (Novelty):
Technical contribution is limited. Table 4 clearly shows that there are noticeable performance improvements from the baseline, but what the authors did is not very novel actually. Utilizing the body feature (global feature) increases the performance a lot, which shows that using both local and global features is necessary. This makes sense, but is kind of expected.

Author response:
$\textbf{Q1 Technical contribution is limited.}$ We appreciate your feedback on the technical contribution of our work. While it is true that the utilization of both local and global features is an expected approach, we would like to emphasize that our proposed method goes beyond merely combining these features. Our method introduces a novel body-reference attention mechanism and leverages the Transformer architecture to effectively process and integrate both local and global features, resulting in noticeable performance improvements over the baseline, as shown in $\textbf{Table $\color{red}{4}$}$.
The attention mechanism helps our model focus on relevant body parts and reference region in the input images, while the Transformer architecture enables their dependencies. This combination of techniques contributes to the improved performance of our method.
We acknowledge that the core idea of using both local and global features may not be groundbreaking; however, the way we have implemented and integrated these features in our model, along with the body-reference feature learning and Transformer architecture, represents a meaningful contribution to the field. We believe that our work provides valuable insights for handling occlusion and depth ambiguity by learning a reference feature .

Reviewer comment (Presentation):
There are two ‘GAP’ in Figure 2. What does it mean? Does it mean global average pooling? The main manuscript says nothing about GAP.

Author response:
$\textbf{Q3 What does ‘GAP’ mean?}$ We apologize for any confusion caused by the presence of two 'GAP' labels in $\textbf{Figure $\color{red}{2}$}$. You are correct in your assumption that 'GAP' stands for Global Average Pooling. We understand that this detail was not explicitly mentioned in the main manuscript, and we appreciate your attention to this matter. We have ensured that this is clarified in our revised manuscript.

Reviewer comment (Experiments):
How is the part segmentation GT obtained? The authors rendered SMPL meshes by considering visibility as well? For example, the occluded body part has GT part segmentation?

Author response:
$\textbf{Q4 How to generate the body/part attention map?}$ To generate the body/part attention map labels, we employ the inverse weak-perspective projection as described in the work by $\textit{Kissos et al. (2020)}$ [1]. The weak-perspective projection is based on the premise that the focal length and object distance are sufficiently large, allowing for the neglect of variations in the object along the Z-axis. It is a technique for converting three-dimensional camera coordinates into pixel coordinates.
We understand the importance of providing a clear explanation of this process, and we have included a more detailed introduction regarding the generation of body/part attention maps using inverse weak-perspective projection in the revised supplementary document (see $\textbf{Appendix.B. Part/Body Segmentation Maps}$ for more detail). 
Since we generate 3D meshes using the ground truth (GT) SMPL labels and project them onto the pixel coordinate system, all body parts are visible. Even if a certain part is occluded in the image, we still use visible segmentation maps for supervision. This approach helps our attention mechanism effectively tackle the challenges posed by occlusions.

$\textbf{References}$

[1] Kissos I, Fritz L, Goldman M, et al. Beyond weak perspective for monocular 3d human pose estimation[C]//Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020: 541-554.

Reviewer comment (Writing):
This paper should be better organized and improve the writing: The readability can be greatly enhanced if the author can further polish this paper, making long sentences and some of transactions more smooth. There are many typos that need to be corrected. For example: Last paragraph of the introduction: “datasets Then” should be “datasets. Then”. Section 3.3: “ambiguity.Next” should be “ambiguity. Next”, etc.

Author response:
We have taken your comments into consideration and made the following improvements:
1. Restructured the content in the method section for better organization and flow.
2. Revised the writing to improve clarity and conciseness.
3. Enhanced the quality and alignment of figures and tables.
4. Corrected any typos and grammatical errors.
These improvements significantly enhance the overall quality and presentation of our paper, making it more accessible and informative for readers. We truly appreciate your time and effort devoted to reviewing and providing valuable, detailed feedback on our paper.

Reviewer comment (Presentation):
Figures and tables should be clearer and better aligned with the text. For example: In Fig 2, the loss components should be more distinguishable. As indicated in the left corner, the auxiliary loss is marked as orange. However, all the losses are colored in orange.

Author response:
We have taken your comments into consideration and made the following improvements:
1. Restructured the content in the method section for better organization and flow.
2. Revised the writing to improve clarity and conciseness.
3. Enhanced the quality and alignment of figures and tables.
4. Corrected any typos and grammatical errors.
These improvements significantly enhance the overall quality and presentation of our paper, making it more accessible and informative for readers. We truly appreciate your time and effort devoted to reviewing and providing valuable, detailed feedback on our paper.

Reviewer comment (Presentation):
Table 1 would be better positioned adjacent to Section 4: Comparison to the State-of-the-Art. Other improvements, such as those mentioned above, should also be addressed to enhance the paper's organization and clarity.

Author response:
We have taken your comments into consideration and made the following improvements:
1. Restructured the content in the method section for better organization and flow.
2. Revised the writing to improve clarity and conciseness.
3. Enhanced the quality and alignment of figures and tables.
4. Corrected any typos and grammatical errors.
These improvements significantly enhance the overall quality and presentation of our paper, making it more accessible and informative for readers. We truly appreciate your time and effort devoted to reviewing and providing valuable, detailed feedback on our paper.

Reviewer comment (Reproducibility):
While the proposed method achieves excellent performance, it appears to be relatively complex. It would be beneficial to provide insights into the training process. Is the proposed method trained in multiple stages, such as initially training for 2D/3D pose estimation and subsequently training for the final mesh? Additionally, there are multiple loss components involved; it would be helpful to clarify whether these losses are optimized jointly or separately during different training stages.

Author response:
$\textbf{Q1 Providing insights into the training process.}$ To provide insights into the training process, we would like to clarify that our method optimizes the multiple loss components jointly almost throughout the training. However, we employ a small training trick borrowed from PARE to improve the effectiveness of the training process. Specifically, we train the model with all losses for the first 100 epochs. After that, we remove the segmentation supervision loss and continue training the model. This initial training phase helps the model learn to recognize and emphasize relevant regions in the input images. Once the attention mechanism has been guided towards the body parts, we remove the segmentation supervision loss and continue training. During this phase, the attention mechanism adapts further to better recover the human mesh from the images.

Reviewer comment (Experiments):
The proposed method benefits from supervision by ground truth body/part segmentation maps, which some other methods do not utilize. To better understand the impact of this additional annotation, it would be valuable to conduct an ablation study. This would help ensure a fair and comprehensive comparison with other methods.

Author response:
$\textbf{Q2 To better understand the impact of body/part segmentation maps, it would be valuable to conduct an ablation study.}$ We greatly appreciate your meticulous suggestion to conduct an ablation study to better understand the impact of using the additional annotation provided by ground truth body/part segmentation maps.
Therefore, we have supplemented our paper with additional experiments (see $\textbf{Section $\color{red}{4}$}$. Ablation Experiment for more detail) to enhance its rigor and ensure a fair and comprehensive comparison with other methods.
To build a comparing method without using segmentation maps supervision (no region attention learning mechanism), we removed all attention-based learning component, and extend global features extracted from HRNet to align with the number of SMPL parts. Subsequently, we concatenate the T-Pose template SMPL parameters as tokens to the global features and predict SMPL parameters through a self-dimensionality reduction Transformer module. We refer this method as HMR + TF (no att.).
As shown in the following table, the approach, which does not rely on attention-based methods, is at a disadvantage in terms of performance compared to our proposed method. This finding highlights the importance of the attention mechanism and the additional supervision provided by the ground truth body/part segmentation maps in achieving improved performance.
|  Method  | MPJPE  | PA-MPJPE  | MPVPE  |
|  ----  | ----  | ----  | ----  | 
| HMR (no att.) [1] |  85.1 | 52.2 | 118.5 | 
| PARE (+ part att.) [2] |  74.5 | 46.5 | 88.6 | 
| HMR + TF (no att.) |  76.8 | 49.0 |89.9 | 
| PARE + TF (+ part att.) | 73.2 | 44.7 | 85.0 | 
| ReBaR (+ part and body att.)  |  69.1 | 41.8 | 81.9 |

Reviewer comment (Evaluation):
Given the potential computational cost of the proposed method, reporting the total parameters and actual inference times for each frame would be valuable information for readers and practitioners.

Author response:
$\textbf{Q3 Reporting the total parameters and actual inference times for each frame.}$ We provided the requested comparison results as follow, and added this information in our revised supplementary document (see $\textbf{Appendix.B. Inference Time and Parameter Counts}$ for more detail).
|  Method  | PARAMS  | Inference Time  | Training Time  | MPJPE  | MPJPE_z  |
|  ----  | ----  | ----  | ----  | ----  | ----  |
| PARE  |  125.5MB | 1.24s | 16h | 74.5 | 58.2 |
| PARE + TF  | 155.8MB | 1.32s | 19h | 73.2 | 56.6 |
| ReBaR  |  222MB | 1.47s | 28h | 69.1 | 46.6 |
| CLIFF* [3] | 305MB | 1.52s | / | 69.0 | 51.8 |
| ReBaR*  | 361.1MB | 1.81s | 48h | 67.2 | 45.5 |

Reviewer comment (Evaluation):
The proposed method significantly outperforms the baseline method PARE in Table 1 and only slightly outperforms PARE in Table 2. Why does this happen?

Author response:
$\textbf{Q1 Why is the improvement difference between ReBaR and PARE in Table 1 and Table 2 so significant? }$ We appreciate your insightful comment and thank you for pointing out the need for clarification. The observed performance difference between the fine-tuned and non-fine-tuned comparisons can be attributed to the fine-tuning process. In our adjusted $\textbf{Table $\color{red}{1}$}$, we have separated the comparison into fine-tuned and non-fine-tuned results. The non-fine-tuned results of PARE on the AGORA dataset are consistent with those on the publicly available leaderboard, representing the performance of the model without fine-tuning. Consequently, ReBaR significantly outperforms PARE in the non-fine-tuned comparison.
In response to your comment, we have also included the results of ReBaR without fine-tuning, allowing for a fair comparison with other models on the leaderboard that have not undergone fine-tuning. As demonstrated in the non-fine-tuned section of $\textbf{Table $\color{red}{1}$}$, among all methods employing weak-perspective projection, ReBaR still surpasses previous approaches even without fine-tuning.
We also want to clarify that the performance improvement of ReBaR over PARE is significant. PARE achieved a result of PAMJE 46.5, while our method reached 41.8.

Reviewer comment (Theory):
To reduce depth ambiguity, the 3D GT labels such as SMPL parameters and 3D joints also provide the depth information of the human body. If they are accurately regressed, the depth information can be well estimated. Why is the relative depth loss L_{RD} still needed?

Author response:
$\textbf{Q2 Why is the relative depth loss $L_{RD}$ still needed? }$ Thank you for your insightful question. It is true that the 3D ground truth (GT) labels, such as SMPL parameters and 3D joints, provide depth information for the human body, and accurate regression can lead to a good estimation of depth. However, by defining three orthogonal planes and measuring the distance of a point to these planes, the relative depth loss captures the relative relationships between joints and the planes, which can further improve the performance due to the following possible reasons:
1. Local depth information: While regressing 3D GT labels primarily focuses on the global structure and pose of the human body, the relative depth loss emphasizes local depth information by considering the pairwise relationships between body joints and the planes. This helps the model to capture more subtle depth variations and improve the overall depth estimation.
2. Improved generalization: The relative depth loss  encourages the model to learn depth patterns that are more generalizable to different poses and body shapes. Since it focuses on the relative relationships between joints and planes, the model is less likely to overfit to specific training examples and can better adapt to unseen data.

Reviewer comment (Reproducibility):
Some training details are unclear. For example, 1) what are the weight parameters for each loss? Please specify each weight parameter in the training; 2) considering there are so many losses, how to balance these losses in training? 3) how to generate the body/part attention map (M_p, M_B); 4) since some training sets may not have GT SMPL labels, how to ensure supervision? 5) The loss functions in Equation 3 are somewhat confusing. Please consider the consistency throughout the paper (e.g., L_{BS} and L_{PS} in Figure 2 appear to have the same meaning as L_{b_seg} and L_{p_seg} in Equation 3).

Author response:
$\textbf{Q3 Some training details are unclear. }$ We apologize for any confusion regarding the training details. In this work, we follow the implementation of PARE for setting the loss weight parameters, where the weights for segmentation map supervision, reference feature 2D/3D auxiliary constraints, and relative depth supervision are set to 60, and the weights for SMPL parameter and keypoint supervision are five times greater (300). We have updated this information in the revised supplementary material (see $\textbf{Appendix.B. Loss Weight}$ for more detail).

$\textbf{Q4 Since some training sets may not have GT SMPL labels, how to ensure supervision?}$ In order to ensure supervision when some training sets do not have ground truth (GT) SMPL labels, we follow the training practice of PARE. For the COCO dataset, we make use of pseudo-SMPL labels that are consistent with PARE to provide supervision. This allows us to train our model effectively even in the absence of genuine SMPL labels.
For the 3DHP dataset, which SMPL labels is quietly coarse, we adapt our training strategy by not supervising the corresponding SMPL parameters and segmentation maps. Instead, we focus on supervising the keypoints, which are available in the dataset. This approach allows us to make the most of the available data and maintain supervision during the training process. For all other datasets in our training pipeline, we have access to genuine SMPL labels, which we use for supervision as intended.

$\textbf{Q5 How to generate the body/part attention map?}$ To generate the body/part attention map labels, we employ the inverse weak-perspective projection as described in the work by $\textit{Kissos et al. (2020)}$ [1]. The weak-perspective projection is based on the premise that the focal length and object distance are sufficiently large, allowing for the neglect of variations in the object along the Z-axis. It is a technique for converting three-dimensional camera coordinates into pixel coordinates.
We understand the importance of providing a clear explanation of this process, and we have included a more detailed introduction regarding the generation of body/part attention maps using inverse weak-perspective projection in the revised supplementary document (see $\textbf{Appendix.B. Part/Body Segmentation Maps}$ for more detail).

Reviewer comment (Evaluation):
Considering transformers used in the architecture, how does the training/inference time of the proposed method compare to the baseline methods (PARE and CLIFF)?

Author response:
$\textbf{Q6 How does the training/inference time of the proposed method compare to the baseline methods (PARE and CLIFF)?}$ We appreciate your interest in comparing the training and inference time of our proposed method with the baseline methods, PARE and CLIFF. Here is a table summarizing the comparison results:
|  Method  | PARAMS  | Inference Time  | Training Time  | MPJPE  | MPJPE_z  |
|  ----  | ----  | ----  | ----  | ----  | ----  |
| PARE[2]  |  125.5MB | 1.24s | 16h | 74.5 | 58.2 |
| PARE + TF  | 155.8MB | 1.32s | 19h | 73.2 | 56.6 |
| ReBaR[3]  |  222MB | 1.47s | 28h | 69.1 | 46.6 |
| CLIFF*  | 305MB | 1.52s | / | 69.0 | 51.8 |
| ReBaR*  | 361.1MB | 1.81s | 48h | 67.2 | 45.5 |

Please note that since CLIFF does not provide open-source training code, we are unable to provide the training time for this method. We have included more detailed information about this comparison and the discussion in the revised supplementary document (see $\textbf{Appendix.B. Inference Time and Parameter Counts}$ for more detail).

Reviewer comment (Evaluation):
What are the limitations and failure cases of the proposed method? A discussion is suggested to provide inspiration for future research directions.

Author response:
$\textbf{Q7 The limitations and failure cases.}$ We appreciate your suggestion to discuss the limitations and failure cases of our proposed method, ReBaR. We have included failure case examples and a detailed discussion of these aspects in the revised supplementary document (see$\textbf{Appendix.E}$  for more detail) and updated the conclusion section to mention limitations and future works as well.

Reviewer comment (Writing):
Typos.  
(1) On page 4, “R^{6840x3}” should be “R^{6890x3}”.  
(2) On page 7, “Table 7 shows the result where our method still performs …” should be “Table 1”.  
(3) On page 9, “MJE Depth reduced from 58.2 to 48.5” should be “… 58.2 to 47.5”

Author response:
$\textbf{We have rectified all writing errors in the paper, standardized the notation, and prevented any confusion for the readers.
}$

Reviewer comment (Presentation):
The approach section contains too many details and lacks motivations and general pictures. Also, the pipeline is blurry.

Author response:
$\textbf{Q1 Improve the approach section and supplement the motivation and general picture.}$  Thank you for taking the time to review our work and provide valuable feedback. We appreciate your concerns regarding the approach section.
We understand that the approach section may seem dense with details, but we believe that providing a comprehensive description of the methods is essential for the reproducibility of our results. It allows fellow researchers to better understand the nuances of our work and build upon it in the future. However, we acknowledge that motivations and general pictures are crucial to provide context and make our work more accessible.
In light of your feedback, we revised the approach section (see $\textbf{Section $\color{red}{3}$}$ for more detail) to include more motivations and made significant improvements to the pipeline (see $\textbf{Figure $\color{red}{2}$}$) to present the framework more clearly. We believe that these modifications will help readers grasp the overall concept and purpose of our method design. 
Once again, thank you for your constructive feedback. We hope that our $\textbf{revisions }$will address your concerns while maintaining the integrity of our methods and results.

Reviewer comment (Experiments):
Is PARE your baseline? If so, I am interested in whether REBAR and PARE have the same experimental setting and the comparison in FLOPS and PARAMS, especially for the PARAMS in the part branch.

Author response:
$\textbf{Q2 The FLOPS and PARAMS of ReBaR and PARE (especially in part branch).}$ Yes, PARE serves as our baseline, and both REBAR and PARE share the same experimental setting. The part branch in REBAR has identical PARAMS and FLOPS as those in PARE. To provide a clear comparison, we present the FLOPS, PARAMS, and Inference Time for each method in the table below:
|  Method  | FLOPS  | PARAMS  | Inference Time  | MPJPE  | MPJPE_z  |
|  ----  | ----  | ----  | ----  | ----  | ----  |
| PARE[1]  | 14.9G | 125.5MB | 1.24s | 74.5  | 58.2  |
| PARE + TF  | 18.1G | 155.8MB | 1.32s | 73.2  | 56.6  |
| ReBaR  | 26.4G | 222MB | 1.47s | 69.1  | 46.6  |

Our approach enhances performance while maintaining an acceptable inference speed.

Reviewer comment (Evaluation):
It seems not very clear to me why your method achieves better results in depth with your network design.

Author response:
$\textbf{Q3 Why does your method perform better, especially in depth compared with PARE?}$ We appreciate the reviewer's concern about the improved performance in our method compared to PARE. The primary reason for our method's better performance, especially in depth estimation, is the introduction of body-aware features and the use of a transformer to model their dependencies.
Our method incorporates an attention-guided encoder (AGE) and a body-aware regression module (BAR), which are specifically designed to capture both global and local features of the human body and establish relationships between different body parts. The key difference between our method and PARE is the learning of body-aware features and the use of a transformer to model their dependencies.
To better illustrate the impact of body-aware features on the depth estimation, we provided a comparison as follow between PARE, PARE+TF (a modified version of our method without body-aware features), and our method. The results show that the main reason for the performance improvement is the introduction of body-aware features into the transformer learning.
|  Method  | MPJPE  | PAMPJPE  | MPVPE  | MPJPE_z  |
|  ----  | ----  | ----  | ----  | ----  |
| PARE  | 74.5 | 46.5 | 88.6 |58.2  |
| PARE + TF  | 73.2 | 44.7 | 85.0 | 56.6  |
| PARE + Ba-Feat + TF  | 70.4 | 42.4 | 82.2 | 47.5  | 

By incorporating body-aware features, our method can better understand the spatial relationships between body parts and the whole body, leading to a more accurate 3D reconstruction, especially in depth. The use of a transformer allows our model to effectively exploit the spatial relationships between parts and bodies, generating more informative part representations and improving depth estimation.
In summary, the improved depth estimation in our method is primarily attributed to the incorporation of body-aware features and the use of a transformer to model their dependencies. This key difference in our method's design principles plays a significant role in achieving better depth estimation compared to PARE.