Paper:
Eliciting Categorical Data for Optimal Aggregation
Models for collecting and aggregating categorical data on crowdsourcing platforms typically fall into two broad categories: those assuming agents honest and consistent but with heterogeneous error rates, and those assuming agents strategic and seek to maximize their expected reward. The former often leads to tractable aggregation of elicited data, while the latter usually focuses on optimal elicitation and does not consider aggregation. In this paper, we develop a Bayesian model, wherein agents have differing quality of information, but also respond to incentives. Our model generalizes both categories and enables the joint exploration of optimal elicitation and aggregation. This model enables our exploration, both analytically and experimentally, of optimal aggregation of categorical data and optimal multiple-choice interface design.

Review:
The paper describes a special setting for crowdsourcing labelling with multiclass labels. Unlike previous studies, where an agent (worker) choose the most likely class label, in the present study, an agent is asked a number of questions about his/her posterior probabilistic distribution over class labels. The problem setting assumes that the agent is incentivized to provide true answers. Unlike previous papers, the goal of the current study is to optimize the interface of the questionnaire. Some theoretical analysis is provided for simple cases of the general setting and some experiments are provided to compare a proposed heuristic approach with simple baselines.
**Novelty** The idea of optimizing a labelling interface, which elicit confidence, is interesting and novel. The problem setting would be a good contribution to the literature on crowdsourcing. However, I am not sure that paper is ready for publication for the following reasons:  1) the theoretical part looks not solid, 2) the proposed algorithm (HA) looks not grounded, 3) the results of experiments are not significant.  These points are supported below.  **Theoretical part (Section 4):** Though Theorem 2 is interesting, it is very simple. Lemmas 3,4 are reasonable, however, they cover only very special cases. Specifically, Lemma 3 considers only one agent and Lemma 4 assumes that all agents have the same amount of information (they observed exactly $n$ samples). Both lemmas consider only binary case, while it would be interesting if they could be generalized to multiclass setting. In particular, the following statement seems to be provable in a similar way: **Statement.** In the general setting of $k$ classes, among the shadow partitions (and among all $k$-cell partitions), the optimal one is the partition is $D_r = \{p\mid r\in \argmax_x p(x)-p^*(x)\}$, where $p^*(x)=1/k$ is the uniform distribution over classes.  By the way, the proofs are not perfectly compact and sometimes can be simplified. For example, the first two outline equations in Section F are redundant, are never used further, so their appearance confuses the idea of the proof.  **Proposed algorithm (Section 5):**  No approach to solve the optimization problem (1) is proposed in a general case. There is only a heuristic aggregation (HA) method (line 265), which is applicable only in the case when agents have equal accuracy (equal number of observations $n$). Moreover, HA seems to be biased, while I would suggest that some MC-like method would better estimate the posterior probability $P(\teta\mid D^1,\ldots, D^m)$ (see lines 188-189).  **Experiments:**  I don’t appreciate Section 5.2, because only one simplest baseline is considered (Majority voting) and this baseline does not take into account priors on classes. A more appropriate baseline would be one of the consensus models used in crowdsourcing.  In section 5.3, the budget of the two treatments are different and the probabilities of bonus payment are also different. Are the values fitted in a way to guarantee the equality of the expected costs of the two treatments? What were the actual costs in the experiment? If the costs are different in expectation, this makes the results less interpretable. When choosing the threshold and the payment scheme in section 5.3, it is assumed that the principal knows the proportion of examples in categories, however, in practice it is usually unknown, also the proportions of categories for golden set samples is biased towards easy categories. How to choose the threshold in such practical case? What would be if the assumed proportions for categories were wrong?    **Other comments:**  Lines 162-164 tells nothing new.  **Questions:** 49: “If the principal can elicit agents’ entire belief distributions, our framework can achieve optimal aggregation, in the sense that the principal can make predictions as if she has observed the private information of all agents”. Can you reformulate this sentence so that one could understand it before reading the theoretical part?   148 “This assumption can be satisfied by, for example, allowing the principal to ask for an expert opinion outside of the m agents”. Outside expert opinion sounds like a correct answer, which is not noisy (otherwise, what is the difference between the expert and an agent?). Then the distribution $p(x\mid \teta^*)$ of the expert opinion should be deterministic: $p(x\mid \teta^*)=Id(x=\teta^*)$.  252 Does simple majority voting take into account priors?  Figure 2. What are the values of $\epsilon$ and $n$ in the experiment reported?  292 I did not get the sentence.  312. “Performing HA for Baseline and Maj for ProbBased both led to higher aggregation errors” What are the quantitative results? It would be interesting to examine.   **Misspellings:**  23 “Moreover, disregard of whether full or partial information about agents’ beliefs is elicited, aggregating the information into a single belief or answer is often done …” I believe, there should be “disregarding whether”.  163 ”a agent” -> “an agent”  165 “what are all the scores which are truthful for it”. I believe, “scoring rules” is meant instead “scores”?  194 “The expectation is taken over which partition D^i…” I believe, there should be “which cell”, not “partition”.  236 “if the number of agent is one” -> “if the number of agents is one”    In Supplementary file:  387 ”a agent” -> “an agent”  389 “can can”  Figure 3: “theta=0” -> “$\theta=0$”.   ------  **UPDATE on Author Rebuttal**  After reading rebuttal, I still have some additional arguments on some points.  - A: While Theorem 2 might seem simple, it resolves an open question in previous work as it was considered impossible to obtain the global PPD from the PPDs of all agents.  Though Theorem 2 is reasonable, I would avoid to claim that an open question was resolved. In fact, I don’t think Theorem 2 was previously conjectured in the literature, since it resolves a rather particular and quite simple question (unlike general questions considered in [1]). Moreover, I would rather name it “Statement”, not “Theorem”, since the proof is even easier than to find the right statement.  - A: We also share the reviewer's conjecture on the multi-signal case (indeed, Fig 2 provides some evidence), though given the difficulty of proving Lemma 4, we do not think such a result would be easy to establish without new techniques.  I claim that I can generalize Lemma 3 to the case of multiple classes (as I stated). I believe, it is a proved statement (in mi mind), not a conjecture. I also believe that the paper would be more thorough if this statement was included.   - A: It is true that we do not give a fast algorithm to solve the optimization problem. Ffor small n (as in our experiments), it is feasible to just search over the discretized parameter spaces. Ideally, we could give fast optimization algorithms by MC-like methods. In practice, however, we felt that a heuristic grounded in intuition but supported by the theory would likely be more valuable for practitioners; fortunately, such a heuristic performs very well empirically. Moreover, these heuristics give qualitative insights about how to place thresholds and aggregate.  I have two different issues here, which ate not connected: 1. No approach to solve the optimization problem (1) in the general case of workers *with different accuracies*. 2. In the case of equal accuracies, HA seems to be based on *biased* estimates of posterior probability $P(\teta\mid D^1,\ldots, D^m)$, therefore, some sampling methods would be more appropriate.  - A: We assumed the prior is known. In practice, there are often historical data that provide good estimate of the prior. It's a very interesting and important future direction to study the robustness of our approach to incorrect priors.  It is not clear how one can use historical data of workers’ labels to set the prior, since these labels are usually biased.