Paper: It HAS to be Subjective: Human Annotator Simulation via Zero-shot Density Estimation (ICLR-2024)

Reviewer comment (Experiments):
The main weakness is the limited compared method in the experiments. In the Tables in the paper, the state-of-the-art meta-learning methods are missing and the latested human annnotation simulators are also disregarded. Please consider add more SOTA methods for comparsion.

Author response:
Human annotation simulation (HAS) is a new task proposed in our paper, so there is not, to our knowledge, latest human annotation simulator available - all baselines in our paper are adapted from suitable existing methods from other domains. We could not find suitable existing meta-learning techniques to adapt because our method is the first method that aims to meta-learn density estimators, whereas existing meta-learning methods focus on meta-learning (single-output) predictors. We’d be grateful if the reviewer could give citations to suitable prior meta-learning methods for HAS.

Reviewer comment (Evaluation):
The other weakness is that the costing time is not reported in the paper. Since the authors claimed that the proposed method is effective, the training and test time should be list compared to other SOTA in the experiments. Please consider to add more details here.

Author response:
The computational time cost of all of the methods that have been compared for the four tasks studied is shown in Table (i) - (iv) in general response (2). Please refer to general response for more details and analysis.

Reviewer comment (Experiments):
The proposed method is evaluated on only three human evaluation tasks, which may not be sufficient to generalize the effectiveness of the proposed method to other domains.

Author response:
The three tasks used to evaluate the proposed method in our paper are real-world tasks from three very representative domains (emotion class annotation, toxic speech detection, and speech quality assessment) that concern human evaluations. In addition, we’ve added an extra emotion attribute prediction task in Appendix K of the revised manuscript, which, instead of estimating discrete emotion labels such as happy, sad, neutral, predicts emotion attributes (i.e. valence – whether the speaker is positive or negative, arousal – whether the speaker is excited or calm, dominance – whether the speaker is weak or dominant). Please also refer to general response for more detail.

Reviewer comment (Experiments):
The paper does not compare the proposed method with the latest state-of-the-art methods for HAS. The methods compared in this paper include deep ensemble (Ensemble) (Lakshminarayanan et al., 2017), Monte-Carlo dropout (MCDP) (Gal & Ghahramani, 2016), Bayes-by-backprop (BBB) (Blundell et al., 2015), conditional variational autoencoder (CVAE) (Kingma & Welling, 2014), conditional argmax flow (A-CNF) (Hoogeboom et al., 2021), Dirichlet prior network (DPN) (Malinin & Gales, 2018), Gaussian process (GP) (Williams & Rasmussen, 2006), and evidential deep learning (EDL) (Amini et al., 2020). The most recent method A-CNF was proposed two years ago.

Author response:
Human annotation simulation (HAS) is a new task setting proposed in our paper, for which no directly applicable prior method is available. All baselines considered in our paper are adapted from suitable existing methods from other domains. In addition, although we cite (Malinin & Gales, 2018) and (Amini et al., 2020) for DPN and EDL in the main paper, they are actually adapted from variants proposed in recent papers (Wu et al., 2022b) and (Wu et al., 2023) which are cited in Appendix E.

Reviewer comment (Evaluation):
The paper does not provide a detailed analysis of the robustness of the proposed method.

Author response:
The proposed method is evaluated for four different real-world tasks (three in the paper plus one newly added). Results reported in Table 1-3 are the mean and standard error from three independent runs with different random seeds, which shows the robustness of the proposed method.

Reviewer comment (Novelty):
I found the problem address highly similar to the following paper [1,2] that aims to model the label space in each step iteratively using Gibbs sampling (or previous like of work that used MCMC). This work may require annotation process in a more dynamic setting, still very relevant to subjective labelling task and I think, they are needed to discussed.

    [1] Harrison, P., Marjieh, R., Adolfi, F., van Rijn, P., Anglada-Tort, M., Tchernichovski, O., ... & Jacoby, N. (2020). Gibbs sampling with people. Advances in neural information processing systems, 33, 10659-10671. https://proceedings.neurips.cc/paper_files/paper/2020/file/7880d7226e872b776d8b9f23975e2a3d-Paper.pdf.  

    [2] Sanborn, A., & Griffiths, T. (2007). Markov chain Monte Carlo with people. Advances in neural information processing systems, 20. https://papers.nips.cc/paper_files/paper/2007/file/89d4402dc03d3b7318bbac10203034ab-Paper.pdf

Author response:
Thanks for pointing out these related works. We agree that delving into the differences between these tasks and our method is enlightening.

Paper [2] developed a MCMC method for sampling from a particular subjective probability distribution which allows a person to act as an element of an MCMC algorithm by choosing whether to accept or reject a proposed change to an object. Paper [1] generalises paper [2] to a continuous-sampling paradigm. Despite both modelling subjective distributions, our method differs from these methods in the following two aspects.

Firstly, these methods require an online annotation process where the annotators need to label data points based on the current state of the Markov chain, whereas in our setting the annotations can be obtained offline. 

Secondly, in those traditional density estimation frameworks, distributions are estimated based on observations while our meta-learning method does not require observations to be provided. Extending the notation in the paper, we denote an event as $d_i$, which consists of  a descriptor (i.e. an utterance)  $x_i$ and $M_i$ associated observations (i.e. human annotations) $D_i=\{\eta_i^{(m)}\}_{m=1}^{M_i}$. For a test event $d^*$, the test descriptor and observations are denoted as $x^*$ and $D^*$. The target to estimate is the distribution of $D^*$, denoted as $p^*$.

Given an event of interest $d^*$, MCMC-based methods present the descriptor $x^*$ to human participants who are asked to provide sequence of decisions $D^*$ following a Markov Chain Monte Carlo acceptance rule. The distribution $p^*$ is then estimated based on $D^*$. In other words, observations $D^*$ are necessary in order to estimate each subjective probability distribution.  Each Markov chain only targets a specific subjective probability distribution and there is no obvious way to transfer information between different Markov chains. Therefore, these methods cannot be applied to simulate annotation distribution for unlabelled data.

An advantage of our method is that only the descriptor $x^*$ is needed to simulate the distribution of event $d^*$ and no $D^*$ is needed which are often unavailable in real-world settings. As described at the beginning of Section 3.1, the proposed approach meta-learns a conditional density estimator across $D_{meta}=\{D_i\}_{i=1}^N$. 

In other words, given $\{x_i, D_i\}_{i=1}^N$, the model is trained to learn how to learn the underlying distribution of $D_i$ given $x_i$. During testing,  given the test descriptor $x^*$, the model directly estimates $p^*$ in a zero-shot fashion.

We’ve added this discussion in Appendix M in the revised manuscript.

Reviewer comment (Experiments):
How does the proposed approach tackle the highly imbalanced data domains? For instance, in one of the tasks, MSP podcast dataset contains angry, sad, happy, neutral, and other. When continuous labels (valence-arousal annotations of the same dataset) is used, the skewed labeled distribution will be more visible. I suspect the proposed method to cause higher uncertainty (interrupter disagreement in less discovered part of label parameter space).

Author response:
Valence-arousal annotation of the MSP-Podcast dataset uses a 7-point Likert scale, which is similar to the speech quality assessment experiments in Section 5.3 that uses a 5-point Likert scale. I-CNF is proposed in the paper to handle those tasks that involve ordinal annotations. We’ve now added further experiments on emotion attribute prediction of MSP-Podcast in Appendix K. Please also refer to general response for more details.

Reviewer comment (Evaluation):
Different performance metrics are used. Particularly, Fleis' kappa for reliability of categorical labels is good. However, why did not you used Intraclass correlation coefficient (ICC) in continuous labels instead of RMSE and the absolute error of the average standard deviations? In continuous/ordinal subjective labelling tasks, ICC reliability is one of the golden standards that define the labelling quality or difficulty of the task.

Author response:
Thanks for suggesting the use of the intraclass correlation coefficient (ICC). We’ve now included the absolute error between ICC(1,k) of human annotations and simulated annotations as an additional metric in Table 3 and Section 4 of the revised manuscript. Results are also listed in the following table. The ICC of human labels is 0.468 for the SOMOS dataset and 0.702 for the MSP-Podcast emotion attributes.

| E(ICC)   | Speech quality assessment on SOMOS | Emotion attribute annotation on MSP-Podcast |
| -------- | ---------------------------------- | ------------------------------------------- |
| GP       | 0.433$\pm$0.000                   | 0.169$\pm$0.000                            |
| EDL      | 0.107$\pm$0.029                   | 0.172$\pm$0.017                            |
| MCDP     | 0.495$\pm$0.010                   | 0.087$\pm$0.014                            |
| Ensemble | 0.136$\pm$0.028                   | 0.057$\pm$0.003                            |
| BBB      | 0.480$\pm$0.003                   | 0.241$\pm$0.003                            |
| CVAE     | 0.214$\pm$0.028                   | 0.192$\pm$0.003                            |
| I-CNF    | **0.079$\pm$0.015**                   | **0.032$\pm$0.012**                            |


We hope our response fully resolves your concern.

Reviewer comment (Evaluation):
The whole motivation of the paper is that it should be possible to capture the data bias (when majority of labelled samples are wrong), and correct for it in some way. However, in all of the tables, the ensemble (tab 1,2)/gaussian process (tab 3) seem to be performing better than the proposed CNF method. While it is qualitatively visible that CNF is learning a better density distribution, the quantitative numbers don’t justify the intuition. Perhaps, the authors would be well served by explicitly identifying the samples which possess such labelling bias, correct for it (by forcing the distribution to skew), and show better performance than all other methods. Right now, such form of analysis is lacking.

Author response:
First, we’d like to clarify that the motivation of the paper is to model label distribution rather than correcting the labelling bias. Human perception and interpretation is subjective. There may not always exist a single “ground truth” for tasks that involve human evaluation (i.e., how expressive the synthesised speech is? What’s the correct score for an ICLR paper review?). In those tasks, multiple human annotators are commonly employed to label each sample (e.g., each ICLR submission is assigned at least three reviewers). The provided labels can be different. The difference (“bias”) reflects different human perceptions of the same event. No particular perception is correct nor wrong and the difference is valuable. As stated in Section 2.1, we propose modelling annotators’ subjective interpretations rather than seeking to reduce the variability in annotations by enforcing a single correct answer. 

> (1) The whole motivation of paper is that it should be possible to capture the data bias (when majority of labelled samples are wrong), and correct for it in some way. However, in all of the tables, the ensemble (tab 1,2)/gaussian process(tab 3) seem to be performing better than the proposed cnf method. While it is qualitatively visible that CNF is learning a better density distribution, the quantitative numbers dont justify the intuition. Perhaps, the authors would be well served by explicitly identifying the samples which possess such labelling bias, correct for it (by forcing the distribution to skew), and show better performance than all other methods. 

As mentioned in the clarification above, there may not exist a single correct answer for such subjective tasks. We thus frame the task as label distribution estimation. Therefore, accuracy/RMSE is no longer the proper metric for the human annotation simulation (HAS) task as these measures can only measure the majority/mean prediction. In order to better assess the model’s ability to model label distribution, we adopted multiple metrics as discussed in Section 4. 
The quantitative numbers in Table 1-3 indeed justify that the proposed method outperforms all baselines in label distribution estimation. It produces the smallest $\text{NLL}^\text{all}$, indicating better distribution matching, and the smallest  $\text{RMSE}^s$, $\mathcal{E}(\bar{s})$, $\mathcal{E}(\hat{\kappa})$, indicating a superior simulation of inter-annotator variability. Besides, we have added an additional metric ICC following Reviewer js5b’s suggestion, for which our method also outperforms all baselines.

Reviewer comment (Theory):
What is the motivation behind latent-diffusion model? In my understanding, the marginal p(y|z) is already encoding information on different annotators z1, z2… z_m sampled over z. What does additional variable v inject into the model conceptually (apart from additional representational capacity)?

Author response:
> (2) What is the motivation behind latent-diffusion model. in my understanding, the marginal p(y|z) is already encoding information on different annotators z1,z2....z_m sampled over z. what does additional variable v inject into the model conceptually (apart from additional representational capacity).

Regarding variable $v$. This is because the flow-based model is an invertible transformation, as mentioned in the paragraph above Eqn. (2), which cannot directly transform the continuous latent variable $z$ into discrete output $y$. Therefore, $z$ is first transformed into $v$ (continuous) with a flow and then discretized into $y$ (discrete).

Reviewer comment (Experiments):
What is the reason for separate analysis of ordinal and categorical variables? I understand that ordinal categories are ordered, and that probability estimation then reduces to summing over the continuous space of latent variable v. However, I haven’t seen standard classification setups explicitly enforce such ordering constraints. Also, are there any cases where annotations are continuous (for example, annotation by a speech, etc.), which could be used to explore continuous cases? That could act as an interesting toy experiment.

Author response:
> (3) What is the reason for separate analysis of ordinal and categorical variables? I understand that ordinal categories are ordered, and that probability estimation then reduces to summing over the continuous space of latent variable v. However, I haven't seen standard classification setups explicitly enforce such ordering constraints. Also are there any cases where annotations are continuous (for eg, annotation by a speech etc.), which could be used to explore continuous cases? That could act as an interesting toy experiment.

Regarding separate analysis of ordinal and categorical variables. You are correct that the standard classification setups cannot explicitly enforce ordering constraints. That’s why we conduct separate analysis of ordinal and categorical variables. 

Ordinal rating schemes (e.g., the  5-point Likert scale) are commonly employed by human evaluation tasks (e.g.,  the speech quality assessment task in the paper, ICLR review) . Directly applying standard classification setups to ordinal annotations would result in poor performance due to model mis-specification. Therefore, we propose I-CNF to separately handle such cases. 

Continuous annotations can be handled by the identity transformation $p(y|v)=\delta(y-v)$ as discussed at the bottom of page 3. In addition, we’d like to point out that continuous annotations are less common in human evaluation tasks (e.g., ICLR reviewers would not rate papers with a score of 6.12).

Reviewer comment (Novelty):
Possible extensions to cases where only single annotation is available for each sample: how does this work compare to works which aim to filter noisy labels from the network? Perhaps this could be mentioned in the related work.

Author response:
> (4) Possible extensions to cases where only single annotation is available for each sample: - how does this work compare to works which aim to filter noisy labels from the network. perhaps this could be mentioned in the related work.

Thanks for pointing out the task of filtering noisy labels which is a related but different task. It is insightful to discuss the differences between these two tasks. 

Both tasks involve inconsistent labels while the source of inconsistency is different. When filtering noisy labels, it is assumed that there is a ground truth and we want to remove misleading labels. For HAS, the inconsistency stems from subjective perception of humans. No particular label is incorrect and there’s no single  “ground truth” . As discussed in the clarification above, the difference reflects different human interpretations of the same event, which is valuable. Instead of enforcing a ground truth, we propose to model such variability.  We’ve added this discussion in Appendix M.1 of the revised manuscript.

Reviewer comment (Experiments):
The motivation does not reflect better performance on real-world metrics (i.e., accuracy). It feels like a fancy technique (i.e., density estimation) whose practical real-world experiments cannot be seen. Also, there are no experiments on zero-shot learning, which is the main title of the paper.

Author response:
> However, the motivation does not reflect better performance on real-world metric (i.e. accuracy).

Please see our response to your comment (1).

Reviewer comment (Experiments):
This paper seems to present a chicken and egg problem. Most of the datasets in the real world contain labels of only one annotator but might be biased. However, this paper does not work on them. Instead, it requires datasets where each sample has been annotated by many people, but that might not be how labelling happens in the real world. The authors could discuss how to extend their method to single label cases where distributions could not be learnt.

Author response:
> Finally, this paper seems to present a chicken and egg problem. Most of the datasets in the real- world contain labels of only one annotator but might be biased. However, this paper does not work on them. Instead, it requires datasets where each sample has been annotated by many people. But that might not be how labelling happens in real-world. So, perhaps authors could discuss how to extend their method to single label cases where distributions could not be learnt.

It’s common for real-world human evaluation tasks to recruit multiple annotators to evaluate each input (e.g., each ICLR submission is assigned to at least three reviewers). In our paper, the proposed method is evaluated on three representative real-world datasets. Since different people perceive emotion differently, most emotion datasets employ multiple annotators to label each utterance (e.g., [1-5]).  In addition, Mean Opinion Score (MOS) is the standard measure to assess the quality of synthesised speech where each sample is assessed by a group of human annotators (e.g., [6-10]).

Reviewer comment (Experiments):
Lack of clarity on the relevance to the title of the paper. The paper is titled zero-shot density estimation. By zero-shot, I understand that a sample different from the original dataset used for fitting the estimator could be used so that, given a set of ground truth classes, the network dynamically predicts the distribution of labels over annotators. However, there are no such experiments; training and inference are both done over the same dataset of speech, toxicity, and emotion annotations.

Author response:
First, we’d like to clarify that the proposed distribution estimation framework is different from standard supervised learning tasks since no “ground truth” is available for training. The objective is to learn the underlying distribution given observations (annotations) while the true distribution is unknown. By “zero-shot density estimation”, we mean that our meta-learned human annotation simulator can be used to estimate the distribution for a given event (e.g., input utterance) without requiring observations (i.e., annotations). 

Extending the notations in the paper, we denote an event as $d_i$, which consists of  a descriptor (i.e. an utterance)  $x_i$ and $M_i$ associated observations (i.e. human annotations) $D_i=\\{\eta_i^{(m)}\\}_{m=1}^{M_i}$. For a test event $d^*$, the test descriptor and observations are denoted as $x^*$ and $D^*$. The target to estimate is the distribution of $D^*$, denoted as $p^*$.

The first type of approach mentioned in Section 2.2 hand-crafts proxy variables $h_i$ based on each $D_i$, treats the proxy variables as the ground truth, and learns the proxy in a supervised way with paired $\\{x_i, h_i\\}$. During testing, given a descriptor $x^*$, it outputs the prediction of proxy $h^*$ which may not capture the underlying distribution $p_*$. That’s why supervised learning is not suitable for such tasks.

The traditional methods to learn subjective distributions (i.e. Gibbs sampling with people [1] and MCMC with people [2] mentioned by Reviewer js5b), require human annotators to be involved in the process in a dynamic setting. Given an event of interest $d^*$, these methods present the descriptor $x^*$ to human participants who are asked to provide sequence of decisions $D^*$ following a Markov Chain Monte Carlo acceptance rule. The distribution $p^*$ is then estimated based on $D^*$. In other words, observations $D^*$ are necessary in order to estimate each subjective probability distribution and there is no obvious way to transfer information between different Markov chains. Therefore, these methods cannot be applied to simulate annotation distribution for unlabelled data.

An advantage of our method is that only the descriptor $x^*$ is needed to simulate the distribution of event $d^*$ and no $D^*$ is needed since $D^*$ are often unavailable in real-world settings. That is the meaning of “zero-shot” in this density estimation framework. Each event is framed as a dataset in the proposed meta-learning framework. 
The proposed approach meta-learns a conditional density estimator across all datasets  $D_{meta}=\\{D_i\\}_{i=1}^N$. 

It leverages knowledge about the agreements and disagreements among different human annotators across different examples for estimating the label distribution of each input rather than designing the proxy solely based on $D_i$. In other words, given $\\{x_i, D_i\\}_{i=1}^N$, we train the model to learn how to learn the underlying distribution of $D_i$ given $x_i$. During testing,  given the test descriptor $x^*$, the model estimates $p^*$ which can be easily sampled from.

Thanks for bringing attention to this potential confusing point. We’ve added this clarification in Appendix M.3 of the revised manuscript.