Paper:
Reputation-based Worker Filtering in Crowdsourcing
In this paper, we study the problem of aggregating noisy labels from crowd workers to infer the underlying true labels of binary tasks. Unlike most prior work which has examined this problem under the random worker paradigm, we consider a much broader class of adversarial workers with no specific assumptions on their labeling strategy. Our key contribution is the design of a computationally efficient reputation algorithm to identify and filter out these adversarial workers in crowdsourcing systems. Our algorithm uses the concept of optimal semi-matchings in conjunction with worker penalties based on label disagreements, to assign a reputation score for every worker. We provide strong theoretical guarantees for deterministic adversarial strategies as well as the extreme case of sophisticated adversaries where we analyze the worst-case behavior of our algorithm. Finally, we show that our reputation algorithm can significantly improve the accuracy of existing label aggregation algorithms in real-world crowdsourcing datasets.

Review:
This work explores the problem of identifying and filtering out *adversarial* workers in crowdsourced classification problems. A worker is adversarial if his/her voting strategy doesn’t follow the standard random process: vote correctly with probability p and incorrectly with probability 1-p, where p is the worker’s reliability.  To solve the problem of adversarial worker filtering, this paper proposes to attach reputation values to workers. The reputation value of a worker is calculated based on if this worker’s votes agree with other workers’ (in particular, this paper proposes to penalize disagreements instead of awarding agreements)  The authors propose two versions of the penalty algorithms and prove their properties with different adversarial worker voting strategies. Experiments on both synthetic and real datasets are conducted.I think this paper addresses a very interesting problem.  Most of other papers on label aggregation assume workers’ votes are i.i.d. samples from some distribution. This paper considers different voting strategies and takes a step towards a more realistic setting. However, the theoretical analysis on hard penalty algorithms are not too satisfactory since 1) it’s hard to interpret the results and 2) the authors make a strong assumption in their analysis (i.e., honest workers are perfect).  I also have some concerns about the experiment results. (Please see comments below.)Some detailed comments:- In the analysis of hard penalty assignment algorithm (and also the simulation), you assume the honest workers are perfect, i.e., give correct votes with probability 1. This seems to be a fairly strong assumption to me. Is this necessary in your analysis? Do you have intuitions on how to relax this assumption?- The results of Theorem 3 and 4 are represented using the workers’ voting graph B_H. It would help the readers to get intuitions by giving examples to illustrate the bounds.- In the experiments results on Table 2, the number of users being filtered out is different in each setting. I think you should show all the results (i.e., set the number of filtered users from 1 to 10) since you won’t know the optimal parameters when running your algorithm in real applications.- In the experiments, your algorithm boosts the performance of ITER (by Karger et. al. 2011) the most. However, ITER assumes all workers complete the same number of tasks. If you want to apply ITER in datasets without this property, you should modify their algorithm by normalizing the messages at every step during the message passing. Otherwise the algorithm performance would just be dominated by the accuracies of workers who complete the most number of tasks.  Without this modification, your algorithm is essentially just filtering out low-accuracy workers completing lots of tasks. It doesn’t provide evidences that your algorithm can filter out adversarial workers.AFTER THE REBUTTAL:Thanks for addressing my concern about the ITER implmentation. I have updated my scores. Please report the updated results if accepted. Overall, I think this is a nice paper addressing a more reasonable setting. I am still a little bit bothered by the perfect-worker assumption, but the empirical evaluations mitigate this shortcoming. I would vote for accept.