Information Elicitation without Veri cation (IEWV) is a classic problem where a principal wants to truthfully elicit high-quality answers of some tasks from strategic agents despite that she cannot evaluate the quality of agents' contributions. e established solution to this problem is a class of peer prediction mechanisms, where each agent is rewarded based on how his answers compare with those of his peer agents. ese peer prediction mechanisms are designed by exploring the stochastic correlation of agents' answers. e prior distribution of agents' true answers is o en assumed to be known to the principal or at least to the agents. In this paper, we consider the problem of IEWV for heterogeneous binary signal tasks, where the answer distributions for di erent tasks are di erent and unknown a priori. A concrete se ing is eliciting labels for training data. Here, data points are represented by their feature vectors x's and the principal wants to obtain corresponding binary labels 's from strategic agents. We design peer prediction mechanisms that leverage not only the stochastic correlation of agents' labels for the same feature vector x but also the (learned) correlation between feature vectors x's and the ground-truth labels 's. In our mechanism, each agent is rewarded by how his answer compares with a reference answer generated by a classi cation algorithm specialized for dealing with noisy data. Every agent truthfully reporting and exerting high e ort form a Bayesian Nash Equilibrium. Some bene ts of this approach include: (1) we do not need to always re-assign each task to multiple workers to obtain redundant answers. (2) A class of surrogate loss functions for binary classi cation can help us design new reward functions for peer prediction. (3) Symmetric uninformative reporting strategy (pure or mixed) is not an equilibrium strategy. (4) e principal does not need to know the joint distribution of workers' information a priori. We hope this work can point to a new and promising direction of information elicitation via more intelligent algorithms.
(5) Other models: REF design peer prediction mechanism in the machine learning setting.
Machine-Learning Aided Peer Prediction
EC '17
Computer Science