Paper: SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmark (ICLR-2024)

Reviewer comment (Experiments):
The reviewer asks how important the biomechanics constraints or other regularizations are on the annotation quality.

Author response:
We thank the reviewer for their considerate feedback. We appreciate that they recognize the impact and significance of the SMPL-X and MANO annotations in our dataset as well as its multi-prompt and cross-language nature. We are delighted that the reviewer finds our system design novel and paper exceptional. We confirm that the reviewer has a good grasp of the contributions of our paper.

Evaluating the impact of biomechanical constraints is indeed a good idea. To this end, we have included a version of our method ablating the biomechanical constratints in Table $\textcolor{red}{6}$ of our $\textbf{revised Appendix}$. This table can also be found in response to Reviewer qy3Y. Furthermore, Figure $\textcolor{red}{12}$ qualitatively depicts the the impact of these constraints, where the reconstructed poses are essentially more realistic and plausible.

Reviewer comment (Evaluation):
The reviewer asks whether there are metrics for biomechanics constraints or other regularizations on annotation quality, or if the evaluation is mostly perceptual.

Author response:
We thank the reviewer for their considerate feedback. We appreciate that they recognize the impact and significance of the SMPL-X and MANO annotations in our dataset as well as its multi-prompt and cross-language nature. We are delighted that the reviewer finds our system design novel and paper exceptional. We confirm that the reviewer has a good grasp of the contributions of our paper.

Evaluating the impact of biomechanical constraints is indeed a good idea. To this end, we have included a version of our method ablating the biomechanical constratints in Table $\textcolor{red}{6}$ of our $\textbf{revised Appendix}$. This table can also be found in response to Reviewer qy3Y. Furthermore, Figure $\textcolor{red}{12}$ qualitatively depicts the the impact of these constraints, where the reconstructed poses are essentially more realistic and plausible.

Reviewer comment (Experiments):
Supplementary video could have been better. There is no narration in the video. Failure analysis and having SLP with different prompts that are corresponding to the same meaning would have strengthened the submission.

Author response:
$\textbf{General.}$ We thank the reviewer for the suggestions. Our new supplementary video on our project page (see general comments) now contains narration. Rapid motions are still difficult for us to handle. Handling ambiguities in the SL videos is in fact an important future work.

Reviewer comment (Novelty):
This manuscript is framed as a dataset paper, however there is no new data that is collected or will be released. What the paper actually presents is derivative data, i.e. 3D pose estimates from existing datasets, which is disappointing as a reader who was hoping to find a new, potentially useful data source.

Author response:
$\textbf{About Dataset.}$ We kindly disagree with the reviewer. There are several components to a dataset, two of which are the input and the labels or annotations. While we source our videos from various online repositories, we do compute our annotations (crucial for supervised learning) which indicates a new dataset. We will in fact be releasing these annotations upon publication.

Reviewer comment (Experiments):
Given one of the main proposed contributions of this paper is the pose estimation framework, I would have expected more qualitative and quantitative results against the state-of-the-art approaches from the model based pose estimation domain. If this is just an application of previously existing approaches, such as (Spurr et al., 2020), then this needs to be clearly stated.

Author response:
$\textbf{About Pose Estimation Framework.}$ We thank the reviewer for bringing this up. Note that our annotation pipeline (pose estimation or reconstruction algorithm) with its temporal-consistency, biomechanical constraints and hierarchical initialization is novel. To demonstrate its advantage, we have now included more qualitative and quantitative results, both regarding our dataset and EHF dataset [1]. Specifically, $\textbf{Appendix B.1}$ provides new evaluations of our reconstruction in comparison to the state-of-the-art. In the existing benchmark of EHF, our annotation method shows a clear advantage (see Table 6, which we also include in response to Reviewer qy3Y). We further illustrate this qualitatively in Figure $\textcolor{red}{10}$. 

[1] Pavlakos, Georgios, et al. "Expressive body capture: 3d hands, face, and body from a single image." CVPR 2019.

Reviewer comment (Writing):
"We compile SignAvatars by synergizing various data sources from public datasets to online videos and form seven subsets, whose distribution is reported in Fig. 2". - As it was used multiple times in the manuscript, what does "synergizing" mean in the context of this paper?

Author response:
$\textbf{Synergizing}.$ We use the term "synergize" in the context where we source multiple datasets from different repositories and domains yielding a whole that is more effective than the sum of its parts. We now rephrased this as "gathering" or "combining" to aid clarity.

Reviewer comment (Presentation):
Figure 2 is not clear. What is "word", which is the "ASL" dataset? It would have been better to have a clear introduction of the terminology and the source dataset that are utilized just after or before Figure 2.

Author response:
$\textbf{word / ASL dataset.}$ The source datasets were introduced in the "Data source" part of Section 3.1 and the source data distribution is actually presented in Table 2, instead of Figure 2. We corrected this typo in the caption of Table 2 by stating $\textbf{``source datasets'' instead of `subsets'}$. For example, the ASL (American Sign Language) and GSL (German Sign Language) datasets are co-articulated but annotated in different sign languages (American and German). The other two groups, HamNoSys and Word subsets, are isolated having HamNoSys and single-word annotations, respectively. We now clarified this.

Reviewer comment (Writing):
"Our SL annotations can be categorized into four common types: HamNoSys, spoken language, word, and gloss, which can be used for a variety of downstream applications such as SLP and SLR". What is the difference between word and gloss in this context?

Author response:
$\textbf{Word vs. GLOSS.}$ Technically, GLOSS is a label for the sign to easy talking about it. It is written all upper case to distinguish from words. It is not the translation and does not convey the sign's meaning itself. Moreover, different words might be written as the same gloss (e.g. swim, swan, swum all correspond to 'SWIM'). Regarding the usage in our work, the "gloss" specifically refers to the continuous and sentence-level gloss annotation to distinguish them from other single signs videos with only HamNoSys or isolated word-level annotation. The "word" specifically refers to the abbreviation of "word-level annotation", the samples with isolated signs and single word/gloss annotation. We have changed "gloss" and its abbreviation "G" to "sentence-level gloss" and "SG", and added more details and descriptions in Section 3.1 to make it clearer.

Reviewer comment (Reproducibility):
"To demonstrate our dedication, we have submitted all source code as part of the appendix." - I was unable to find the source code either in the appendix or the supplementary material.

Author response:
$\textbf{Data and code release.}$ Our dataset and code will be released soon upon acceptance.

Reviewer comment (Novelty):
I am still adamant about my standing of this not being a dataset paper, as I categorically do not consider automatic pose estimations as annotations.

Author response:
$\textbf{About Dataset.}$ We kindly disagree with the reviewer. There are several components to a dataset, two of which are the input and the labels or annotations. While we source our videos from various online repositories, we do compute our annotations (crucial for supervised learning) which indicates a new dataset. We will in fact be releasing these annotations upon publication.

Reviewer comment (Experiments):
Why did the author use the annotation method in Figure 3? Are there other labeling methods that can be compared?

Author response:
$\textbf{1. Are there other labeling methods that can be compared?}$ We agree with the reviewer that other annotation methods can be used and thank the reviewer for bringing this up. We have considered this suggestion and have evaluated our method against other potential annotation strategies, on a dataset where precise ground truth is available, specifically the EHF dataset [1]. In particular, we choose the state-of-the-art methods of OSX [2], PyMafX [3], and Pixie [4], and present a comparison in $\textbf{Table 6}$ of our $\textbf{revised Appendix}$. We also include this table below:

| **Method**                                          | **MPVPE** |       |       | **PA-MPVPE** |       |       | **PA-MPJPE** |       |
| :---------------------------------------------------- | :--------: | :---: | :---: | :------------: | :---: | :---: | :--------------: | :---: |
|                                                       | Holistic   | Hands | Face  | Holistic       | Hands | Face  | Body             | Hands |
| SMPLify-X [1] *                        | -          | -     | -     | 65\.3          | 75\.4 | 12\.3 | 62\.6            | 12\.9 |
| FrankMocap [5]   $\dagger$         | 107\.6     | 42\.8 | -     | 57\.5          | 12\.6 | -     | 62\.3            | 12\.9 |
| PIXIE [4]  $\dagger$                       | 89\.2      | 42\.8 | 32\.7 | 55\.0          | 11\.1 | 4\.6  | 61\.5            | 11\.6 |
| Hand4Whole [6] $\dagger$ | 76\.8      | 39\.8 | 26\.1 | 50\.3          | 10\.8 | 5\.8  | 60\.4            | 10\.8 |
| PyMAF-X [3]    $\dagger$                   | 64\.9      | 29\.7 | 19\.7 | 50\.2          | 10\.2 | 5\.5  | 52\.8            | 10\.3 |
| OSX [2]    $\dagger$                              | 70\.8      | -     | -     | 48\.7          | -     | -     | 55\.6            | -     |
| Motion-X [7]   $\ddagger$               | 44\.7      | -     | -     | 31\.8          | -     | -     | 33\.5            | -     |
| Motion-X w/GT 3Dkpt [7]   $\ddagger$   | 30\.7      | -     | -     | 19\.7          | -     | -     | 23\.9            | -     |
| **Ours (w/o bio)** *                                 | 21\.6      | 12\.5 | 7\.8  | 14\.2          | 5\.4  | 4\.3  | 16\.5            | 6\.2  |
| **Ours**  *                                         | **20\.1**      | **9\.7**  | **7\.8**  | **12\.9**           | **4\.7**  | **4\.3**  | **15\.6**            | **5\.8**  |

We further present in Figures $\textcolor{red}{9}, \textcolor{red}{10}$ and $\textcolor{red}{11}$ of our $\textbf{revised Appendix}$, that our method provides significantly better quality regarding $\textbf{pixel alignment}$, with more natural and plausible hand poses. Please find more results in our updated videos on our anonymous project page, found in the general comments.

Reviewer comment (Presentation):
It is expected that the author can describe the specific structure of the "Autoregressive Transformer" in Figure 4.

Author response:
$\textbf{2. Structure of autoregressive Transformer and code-index vector.}$
Code index vector is the (discrete) latent feature of the motion. Let us consider Motion VAE in Figure 4. Any SL motions can be encoded into a continuous latent space, which can be discretized into a sequence of codebook indices via quantization using a learnable codebook. This is denoted as $X = [x_{1}, ..., x_{T/w}, x_{\mathrm{EOS}}]$, where $\mathrm{EOS}$ (end of sequence) is an extra learnable end token representing the stop signal. By projecting $X$ back to their corresponding codebook entries, we can obtain the $\textit{code index vectors}$ as the latent feature $F^m_{1:(T/w)}=(f_{1}^{m}, \dots , f_{1:(T/w)}^{m})$ and $f_{i}^{m} \in \mathbb{R}^{d_{z}}$.

Autoregressive Transformer serves as an autoregressive code index generator and the objective for training the code index generator can be seen as an autoregressive next-index prediction task. The process can be formulated as follows: given the previous $i-1$ indices and the conditioning code $c$ embedded by fusing the linguistic feature $E^{l}$ and the codebook index vectors, the Transformer predicts the distribution of possible next indices $p(X_{i},c\mid X_{<i})$ as shown in Figure 4.

Reviewer comment (Writing):
What is the specific meaning of "code index vector" in Figure 4? Please clarify.

Author response:
$\textbf{2. Structure of autoregressive Transformer and code-index vector.}$
Code index vector is the (discrete) latent feature of the motion. Let us consider Motion VAE in Figure 4. Any SL motions can be encoded into a continuous latent space, which can be discretized into a sequence of codebook indices via quantization using a learnable codebook. This is denoted as $X = [x_{1}, ..., x_{T/w}, x_{\mathrm{EOS}}]$, where $\mathrm{EOS}$ (end of sequence) is an extra learnable end token representing the stop signal. By projecting $X$ back to their corresponding codebook entries, we can obtain the $\textit{code index vectors}$ as the latent feature $F^m_{1:(T/w)}=(f_{1}^{m}, \dots , f_{1:(T/w)}^{m})$ and $f_{i}^{m} \in \mathbb{R}^{d_{z}}$.

Autoregressive Transformer serves as an autoregressive code index generator and the objective for training the code index generator can be seen as an autoregressive next-index prediction task. The process can be formulated as follows: given the previous $i-1$ indices and the conditioning code $c$ embedded by fusing the linguistic feature $E^{l}$ and the codebook index vectors, the Transformer predicts the distribution of possible next indices $p(X_{i},c\mid X_{<i})$ as shown in Figure 4.

Reviewer comment (Evaluation):
One of the main concerns is the evaluation metrics. It is important to evaluate the semantics of the produced motion. Although the authors claim that the metric of back-translation is not generic for each text prompt, we can divide the dataset into multiple groups, i.e., word-level and sentence-level. Word-level and sentence-level videos should be divided, as they have different co-articulated characteristics.

Author response:
$\textbf{Evaluation metrics.}$ We absolutely agree with the reviewer that, due to the novelty of the task and our benchmark, evaluation needs to be rethought. Unfortunately, we are not aware of a direct 3D back-translation model for motion SLP, which we plan to investigate in a future study. In this work, instead of re-inventing the wheel, we indeed contribute to the available repertoire of metrics. In particular, inspired by the studies in human-motion-generation, we introduce MR-precision, which evaluates how much $\textit{generated motion}$ resembles the $\textit{ground truth motion}$. As we describe in our Appendix C1, it measures the average accuracy at top-$k$ positions of sorted Euclidean distance between a motion feature generated from a sequence joint-rotations and 16 motion samples from dataset. We argue that, as opposed to mean 3D joint position error, this metric is especially suited to the 3D  motion SLP task since: (i) the orientation of hand joints carries particular semantic meaning, (ii) such comparison in the latent space parallels a $\textit{semantic evaluation}$ of signs, without having the need for an explicit continuous-mesh to sign classifier. In the future, we will work on designing more explicit, physical evaluation-models. We have now updated the paper for further clarity.

Reviewer comment (Presentation):
What is PLFG? I cannot find this module in Figure 4.

Author response:
$\textbf{PLFG}$ is our parallel linguistic feature generator, which is the VQVAE based structure illustrated at the top of Figure 4. It is a linguistic feature generator jointly optimized with our SL motion generator (bottom of Figure 4) in a parallel manner. We have polished the description of this part in our revised version.