Multi-winner Approval Voting Goes Epistemic
Abstract
Epistemic voting interprets votes as noisy signals about a ground truth.
We consider contexts where the truth consists of a set of objective winners, knowing a lower and upper bound on its cardinality.
A prototypical problem for this setting is the aggregation of multi-label annotations with prior knowledge on the size of the ground truth.
We posit noise models, for which we define rules that output a set of winners corresponding to local maxima of the data likelihood function.
We report on experiments on multi-label annotations (which we collected).
INTRODUCTION
The epistemic view of voting assumes the existence of a ground truth which, usually, is either an alternative or a ranking over alternatives.
Votes reflect opinions or beliefs about this ground truth; the goal is to aggregate these votes so as to identify it.
Usual methods define a noise model specifying the probability of each voting profile given the ground truth, and output the alternative that is the most likely state of the world, or the ranking that is most likely the true ranking.
Now, there are contexts where the ground truth does not consist of a single alternative nor a ranking, but of a set of alternatives .
Typical examples are multi-label crowdsourcing (find the items in a set that satisfy some property, e.g. the sport teams appearing on a picture) or finding the objectively k best candidates (best papers at a conference, best performance in artistic sports, k patients with highest probabilities of survival if being assigned a scarce medical resource).
These alternatives that are truly in the ground truth are called ‘winning’ alternatives.
Depending on the context, the number of winning alternatives can be fixed, unconstrained, or more generally, constrained to be in a given interval.
This constraint expresses some prior knowledge on the cardinality of the ground truth .
This prior knowledge is held by the central authority that aggregates the votes, and not necessarily by the voters themselves.
Here are some examples:
• Picture annotation via crowdsourcing : participants are shown a picture taken from a soccer match and have to identify the team(s) appearing in it.
The ground truth is known to contain one or two teams.
• Guitar chord transcription : voters are base classifier algorithms Nguyen et al.
[2020] which, for a given chord, select the set of notes constitute it.
The true set of notes can contain three to six alternatives.
• Jury : participants are members of a jury which has to give an award to three papers presented at a conference: the number of objective winners is fixed to three.
(In a variant, the number of awards would be at most three.)
We assume that voters provide a simple form of information: approval ballots , indicating which alternatives they consider plausible winners.
These approval ballots are not subject to any cardinality constraint: a voter may approve a number of alternatives, even if it does not lie in the interval bearing on the output .
This is typically the case for totally ignorant voters, who may plausibly approve all alternatives.
Sometimes, the aggregating mechanism has some prior information about the likelihood of alternatives and the reliability of voters.
We first study a simple case where this information is specified in the input: in the noise model, each voter has a probability p
i (resp.
q i ) of approving a winning (resp.
non-winning) alternative and each alternative has a prior probability to be winning.
This departs from classical voting, where voters are usually treated equally ( anonymity ), and similarly for alternatives ( neutrality ).
This simple case serves as a building component for the more complex case where these parameters are not known beforehand but estimated from the votes : votes allow to infer information about plausibly winning alternatives, from which we infer information about voter reliabilities, which leads to revise information about winning alternatives, and so on until the process converges to a local optimum of the likelihood function.
Here we move back to an anonymous and neutral setting, since all alternatives (resp.
voters) are treated equally before votes are known.
After discussing related work (Section 2), we introduce the model (Section 3) and give an estimation algorithm
(Section 4), first in the case where the parameters are known, and then in the case where they are estimated from the votes.
In
Section 5 we present a data gathering task and analyse the results of the experiments.
Section 6 concludes.
RELATED WORK
Epistemic social choice Epistemic social choice consists in recovering an objective ground truth from votes seen as noisy reports about the ground truth, using maximum likelihood estimation.
It dates back from Condorcet’s jury theorem [Condorcet, 1785]: n independent, equally reliable voters vote on two alternatives that are a priori equally likely; if every vote is correct with probability p > 12 , then majority outputs the correct alternative with a probability increasing with n and tending to 1 when n grows to infinity.
There are several extensions of Condorcet’s jury theorem: Young [1988] for an arbitrary number of alternatives; Shapley and Grofman [1984] and Drissi-Bakhkhat and Truchon [2004] for voters with various competence degrees; Ben-Yashar and Nitzan [1997] and Ben-Yashar and
Paroush [2001] for nonuniform priors over alternatives; Pivato [2013] and Pivato [2017] for dependent voters.
Conitzer and Sandholm [2005] and Conitzer et al.
[2009] characterize various voting rules as maximum likelihood estimators, each associated with a particular noise model.
See Nitzan and
Paroush [2017] and Elkind and Slinko [2016].
for surveys on recent developments.
Multi-winner voting Multi-winner voting rules map voting profiles into sets of alternatives.
A voting profile can be either a collection of subsets of alternatives (approval ballots) or a collection of ranking over alternatives (ordinal ballots).
The output is often constrained to have a fixed cardinality, but not always: see Kilgour [2016], Faliszewski et al.
There have been a lot of recent developments in the field: see the recent surveys Faliszewski et al.
[2017]) and Lackner and Skowron [2020].
They, however, deal only with the classical (non-epistemic) view of social choice, where votes express preferences.
voting has received only little attention so far.
Procaccia et al.
[2012] assume a ground truth ranking over alternatives, and identify rules that output the k alternatives maximizing the likelihood to contain the best alternative, or the likelihood to coincide with the top- k alternatives.
The last section of [Xia and Conitzer, 2011] defines a noise model where the ground truth is a set of k alternatives (and the reported votes are partial orders).
The only work we know where the noise models produce random approval votes from a ground truth consisting of a set of alternatives is [Caragiannis et al., 2020].
They define a family of distance-based noise models, whose prototypical instance generates approval votes selecting an alternative in the ground truth (resp.
not in the ground truth) with probability p (resp.
1 − p ); as we see further, this is a specific case of our noise model.
Generalizing multiwinner voting, Xia et al.
[2010] study epistemic voting on combinatorial (or multi-attribute ) domains.
Epistemic approval voting Epistemic voting with approval ballots has scarcely been considered.
Procaccia and
Shah [2015] assume that the ground truth is a ranking over alternatives, and identify noise models for which approval voting is optimal given k -approval votes, in the sense that the objectively best alternative gets elected.
Allouche et al.
[2022] continue this line of research but assume instead that the ground truth consists of a single alternative.
They define various noise models and show that those that work best on real datasets are those that give a higher confidence to voters who approve few alternatives.
Caragiannis and Micha [2017] study the number of samples needed to recover the ground truth ranking over alternatives with high enough probability from approval ballots; they show that is is exponential if ballots are required to approve k candidates, but polynomial if the size of the ballots is randomized.
Crowdsourcing and social choice A social choice
- theoretic study of collective annotation tasks was done by
Kruger et al.
[2014] and Qing et al.
Mechanisms for incentive-compatible elicitation with approval ballots in crowdsourcing applications have been designed by Shah and Zhou [2020].
Meir et al.
[2019] define a method to aggregate votes weighted according to their average proximity to the other votes as an estimation of their reliability.
Prelec et al.
[2017] introduce the Bayesian truth serum approach: eliciting, in addition to the voters’ answers, their prediction of the distribution of answers, gives much better results.
This approach was generalized by Hosseini et al.
[2021] to contexts where the ground truth is a ranking.
Beyond social choice, collective multi-label annotation was first addressed by Nowak and Rüger [2010], who study the agreement between experts and non-experts in some multilabelling tasks, and by Deng et al.
[2014], who solve the multi-label estimation problem with a scalable aggregation method.
THE MODEL
Let N = { 1 , . .
. , n } be a set of voters, and A = { a 1 , . .
, a m } a set of alternatives (possible objects in im- ages, notes in chords, papers, patients...).
Consider a set of
L instances : an instance z consists of an approval profile
A z =
( A z 1 , . . .
A zn ) where A zi ⊆
A is an approval ballot for every i ∈ N .
For example, in a crowdsourcing context, a task usually contains multiple questions, and an instance comprises the voters’ answers to one of these questions.
For each instance z ∈ L , there exists an unknown ground truth S ∗ z belonging to S = 2 A , which is the set of objectively correct alternatives in instance z .
It is prior knowledge by the central authority (but not necessarily by voters), that the number of alternatives in each of them lies in the interval [ l, u ] : S ∗ z ∈ S l,u = { S ∈ S , l ≤ | S | ≤ u } , for given bounds 0 ≤ l ≤ u ≤ m .
Our goal is to unveil the ground truth for each of these instance using the votes and the prior knowledge on the number of winning alternatives.
We define a noise model consisting of two parametric distributions, namely, a conditional distribution of the approval ballots given the ground truth, and a prior distribution on the ground truth.
Here we depart from classical noise models in epistemic social choice, as we suppose that the parameters of these distributions may be unknown and thus need to be estimated.
For each voter i ∈ N , we suppose that there exist two unknown parameters ( p i , q i ) in (0 , 1) such that the approval ballot
A zi on an instance z ∈ L is drawn according to the following distribution: for each a ∈ A ,
[Equation]
where p i (resp.
q i ) is the (unknown) probability that voter i approves a correct (resp. incorrect) alternative.
Then we make the following assumptions:
(1) A voter’s approvals of alternatives are mutually independent given the ground truth and parameters ( p i , q i ) i ∈N .
(2) Voters’ ballots are mutually independent given the ground truth.
(3) Instances are independent given the parameters ( p i , q i ) i ∈N and the ground truths.
To model the prior probability of any set S to be the ground truth S ∗ , we define parameters t j = P ( a j ∈ S ∗ ) .
t j can be understood as the prior probability of a j to be in the ground truth set S ∗ before the cardinality constraints are taken into account.
These, together with an independence assumption on the events { a j ∈ S ∗ } , gives P ( S = S ∗ ) =
(cid:81) a j ∈ S t j (cid:81) a j / ∈ S 1 − t j .
Note that the choice of the parameters t j is not crucial when running the algorithm for estimating the ground truth: we will see in Section 4.3 that it converges whatever their values.
The distribution conditional to the prior knowledge on the size of the ground truth ˜ P ( S ) can be seen as a projection on the constraints followed by a normalization:
[Equation]
It follows:
[Equation]
[Equation]
where β ( l, u, t ) =
(cid:80)
S ∈S l,
[Equation]
(cid:81)
[Equation]
j
[Equation]
t j (cid:81)
[Equation]
j [Equation] (1 − t j ) . The ground truths associated with different instances are assumed to be mutually independent given the parameters.
Two particular cases are worth discussing.
First, when ( l, u ) =
(0 , m ) , the problem is unconstrained and we have β (0 , m, t )
=
P ( | S ∗ | ∈ [0 , m ]) = 1 , so ˜ P ( S ) =
P ( S =
S ∗ ) .
In this case the problem degenerates into a series of independent binary label-wise estimations (see Subsection 4.1).
Second, in the single-winner case ( l, u ) = (1 , 1) , we have
[Equation]
, therefore, for any approval profile A :
[Equation]
We recover the same estimation problem if we simply introduce α j = P ( S ∗ = { a j } ) with (cid:80) α
j = 1 as in
Ben-Yashar and Paroush [2001], in which case we have
P ( S ∗ = a j | A ) ∝ α j P ( A | S ∗ = a j ) .
ESTIMATING THE GROUND TRUTH
Our aim is the intertwined estimation of the ground truth and the parameters via maximizing the total likelihood of the instances:
[Equation]
where:
[Equation]
To this aim, we will introduce an iterative algorithm whose main two steps will be presented in sequence, in the next subsections, before the main algorithm is formally defined and its convergence shown.
These two steps are:
• Estimating the ground truths given the parameters.
• Estimating the parameters given the ground truths.
Simply put, the algorithm consists in iterating these two steps until it converges to a fixed point.
4.1 ESTIMATING THE GROUND TRUTH GIVEN THE VOTES AND THE PARAMETERS
Since instances are independent given the parameters, we focus here on one instance with ground truth S ∗ and profile
A =
( A 1 , . . .
, A n ) .
Before diving into maximum likelihood estimation (MLE), we introduce some notions and prove some lemmas.
In this subsection, we suppose that the parameters ( p i , q i ) i ∈N and ( t j ) j ∈A are known (later on, these parameters will be replaced by their estimations at each iteration of the algorithm).
Thus, all in all, input and output are as follows:
• Input: approval profile A ; parameters ( p i , q i ) i ∈N and ( t j ) j ∈A .
• Output: MLE of the ground truth S ∗ .
Definition 1 (weighted approval score) .
Given an approval profile ( A 1 , . . .
, A n ) , noise parameters ( p i , q i ) 1 ≤ i ≤ n and prior parameters ( t j ) 1 ≤ j ≤ m , define:
[Equation]
The scores app w ( a j ) can be interpreted as weighted approval scores for a ( n + m ) -voter profile where:
• for each voter 1 ≤ i ≤ n : i has a weight w i = ln (cid:16) p i (1 − q i )
q i (1 − p i ) (cid:17) and casts approval ballot A i .
• for each 1 ≤ j ≤ m : there is a virtual voter with weight w j = ln (cid:16) t j 1 − t j (cid:17) who casts approval ballot A j = { a j } .
While the weight of each voter i ∈ N depends on her reliability, each prior information on an alternative plays the role of a virtual voter who only selects the concerned alternative, with a weight that increases as the prior parameter increases.
From now on, we suppose without loss of generality that the alternatives are ranked according to their score:
[Equation]
Definition 2 (threshold and partition) .
Define the threshold:
[Equation]
and the partition of the set of alternatives in three sets:
[Equation]
and let k τ n
[Equation]
=
[Equation]
The next result characterizes the sets in S that are MLEs of the ground truth given the parameters.
Theorem 1.
˜ S ∈ arg max S ∈S L ( A, S, p, q, t ) if and only if there exists k ∈ [ l, u ] such that ˜ S is the set of k alternatives with the highest k values of app w and:
[Equation]
So the estimator ˜ S is made of some top- k alternatives, where the possible values of k are determined by Eq.
The first equation imposes that ˜ S includes as many elements as possible from S τ n max (without exceeding the upper-bound u ), whereas the second one imposes that ˜ S includes as few elements as possible from S τ n min (without getting below the lower-bound l ).
An example is included in the appendix.
Proof.
Since ˜ P ( S ) >
⇐⇒ S ∈ S l,u , we have that
[Equation]
,u L ( S ) .
Moreover, we have that for any S ∈ S l,u :
[Equation]
Thus the log-likelihood ℓ ( S ) reads:
[Equation]
) q i (1 − p i ) = (cid:88) a j ∈ S   l ( a j ) (cid:122) (cid:125)(cid:124) (cid:123) ln t j 1 − t j + (cid:88) i : a j ∈ A i ln p i (1 − q i ) q i (1 − p i ) (cid:124) (cid:123)(cid:122) (cid:125) app w ( a j ) − n (cid:88) i =
ln 1 − q i 1 − p
i (cid:124) (cid:123)(cid:122) (cid:125) τ n 

This means that a ∈ S τ n max if and only if ℓ ( a ) > 0 , a ∈
S τ n min if and only if ℓ ( a ) < 0 and a ∈ S τ n tie if and only if ℓ ( a ) = 0 .
Now, let S M be a maximizer of the likelihood.
Since ℓ ( a j ) ≥ ℓ ( a h ) ⇐⇒ app w ( a j ) ≥ app w ( a h ) we have that S M , which maximizes (cid:80) a j ∈ S ℓ ( a j ) , is made of top- k alternatives for some k ∈ [ l . .
u ] .
Furthermore, | S M ∩ S τ n min | = max(0 , l − k τ n tie − k τ n max ) .
Start by noticing that | S M ∩ S τ n min
| ≥ max(0 , l − k τ n tie − k τ n max ) , since | S M ∩ S τ n min | ≥ l −| S M ∩ S τ n max |−| S M ∩ S τ n tie | ≥ l − k τ n max − k τ n tie .
Suppose that | S M ∩ S τ n min | > max(0 , l − k τ n tie − k τ n max ) .
Then we have that | S M | > l because otherwise, if | S M | =
l , then | S M ∩ S τ n max | + | S M ∩ S τ n tie | = l − | S M ∩
S τ n min | < k τ n max + k τ n tie , which would mean that there are elements in S τ n tie and S τ n max which are not in S M , which is a contradiction since | S M ∩ S τ n min | > 0 and S M is a top- k set.
Now consider a ∈ S M ∩ S τ n min , we have that | S M \{ a }| ≥ l and l ( S M )
=
l ( S M \{ a } )
+ l ( a ) <
l ( S M \{ a } ) which is a contradiction.
With the same idea we can prove that | S M ∩ S τ n max | = min( u, k τ n max ) .
Conversely, consider an admissible set S of top- k alternatives that verifies the constraints (1).
Let S M be a MLE which, by the first part of the proof, is a top- k ′ set that also satisfies the same constraints (1).
Thus we have that | S M ∩ S τ n max | = | S ∩ S τ n max | = min( u, k τ n max ) , and since S and S M are top- k and top- k ′ sets, we have that S ∩ S τ n max =
S M ∩ S τ n max .
Similarly we have that S ∩ S τ n min =
S M ∩ S τ n min .
This suffices to prove that ℓ ( S ) =
ℓ ( S M ) is maximal.
Notice that when ( l, u ) = (0 , m ) , the problem degenerates into a collection of label-wise problems, one for each alternative: a j is selected if a j ∈ S τ n max , rejected if a j ∈ S τ n min , and those that are on the fence can be arbitrarily selected or not.
Example 1.
Consider 5 alternatives A
= { a, b, c, d, e } and 10 voters N all sharing the same parameters ( p, q ) = (0 . 7 , 0 . 4) .
We thus have that all voters share the same weight w = ln (cid:16) p (1 − q ) q (1 − p ) (cid:17)
= 1 .
25 and τ n = (cid:80) ni
=1 ln (cid:16) 1 − q 1 − p (cid:17) = 6 .
We consider the constraints ( l, u ) =
First, suppose that t d = 0 .
6 and that t j = 0 .
5 for all the remaining candidates.
Consider also the approval counts (and weighted approval scores) in the table below.
Candidate a b c d e
Approval count     app w 11.
We can easily check, by Theorem 1 that ˜ S = arg max S ∈S P ( S = S ∗ | A )
= { a, b, c } .
We have that
S τ n max = { a, b, c } , S τ n tie = ∅ and S τ n min =
{ d, e } .
We know that there exists some k ∈ [1 , 4] such that ˜ S would consist of the top k alternatives.
We also have that:
[Equation]
So the only possibility is ˜ S
= { a, b, c } .
4.2 ESTIMATING THE PARAMETERS GIVEN THE GROUND TRUTH 4.2.1 Estimating the prior parameters over alternatives
Once the ground truths are estimated at one iteration of the algorithm, the next step consists in estimating the prior parameters ( t j ) j ∈A , with the ground truths being given (in
Subsection 4.3 the ground truth will be replaced by its estimation at each iteration).
The next proposition explicits the closed-form expression of the MLE of the prior parameter of each alternative given the ground truth of each instance
S ∗ z once the prior parameters of all other alternatives are fixed.
Input [Equation] profile ( A 1 , . . .
, A n ) , ground truths S ∗ z , and all but one prior parameters ( t h ) h ̸
= j . • Output: MLE of t j
Proposition 2.
For every a j ∈ A :
[Equation]
where:
[Equation]
Notice that α j = P ( l ≤ | S ∗ | ≤ u | a j ∈ S ∗ ) and α j =
P ( l ≤ | S ∗ | ≤ u | a j / ∈ S ∗ ) so β = α
j t j + α j (1 − t j ) .
occ ( j ) is the number of instances whose ground truth contains a j .
Proof. Fix all sets S z ∈ S l,u and all the noise parameters ( p i , q i ) i and all the prior parameters t h but for one t j for some j ≤ m , and let t ∈ (0 , 1) : [Equation] ∝
L (cid:89)
z
[Equation]
β ( l
[Equation]
u
[Equation]
t ) (cid:89) a h ∈ S
z t h (cid:89) a h / ∈ S
[Equation]
[Equation]
t h ) ∝
[Equation]
(cid:89)
[Equation]
β ( l, u, t
[Equation]
t − j ) (cid:89) a h ∈ S
z t h (cid:89) a h
[Equation]
[Equation]
t h
[Equation]
[Equation]
Taking the log we can write the function as:
[Equation]
Its derivative reads:
[Equation]
Canceling it, we obtain:
[Equation]
The derivative vanishes in a single point in (0 , 1) and lim t → 0 ℓ ( t ) =
lim t → 1 ℓ ( t ) =
−∞ thus ℓ reaches a unique maximum.
We will see later that the algorithm applies Propositionsequentially to estimate the alternatives’ parameters one by one (see Example 2).
4.2.2 Estimating the voter parameters
Once the ground truths are known (or estimated), we can estimate the voters’ parameters ( p, q ) .
Input
Instances [Equation]
A
[Equation]
A
[Equation]
, ground truths ( S ∗ 1 , . . .
, S ∗ L ) .
• Output: MLE of voter reliabilities ( p, q )
The next result simply states that the maximum likelihood estimator of p i of some voter is the fraction of alternatives that the voter approves and that actually belong to the ground truth; the estimation of q i is similar.
See Example 2.
Proposition 3.
Fix sets S z ∈ S l,u and prior parameters t j .
Then:
[Equation]
where:
[Equation]
The (simple) proof is omitted.
4.3 ALTERNATING MAXIMUM LIKELIHOOD ESTIMATION
Now the estimation of the ground truths and that of the parameters are intertwined to maximize the overall likelihood
L ( A, S, p, q, t ) by the Alternating Maximum Likelihood Estimation algorithm . AMLE is an iterative procedure similar to the Expectation-Maximization procedure introduced in Baharad et al. [2011] but with a coordinate-steepest-ascentlike iteration, whose aim is to intertwinedly estimate the voter reliabilities, the alternatives’ prior parameters and the instances’ ground truths.
The idea behind this estimation consists in alternating a MLE of the ground truths given the current estimate of the parameters, and an updating of these parameters via a MLE based on the current estimate of the ground truths.
1 Each of these steps have been discussed
Algorithm
Input:
Approval ballots ( A zi ) 1 ≤ z ≤ L,i ∈N
Initial parameters ˆ θ (0) , Bounds ( l, u ) , error ε
Output: Estimations ( ˆ S z ) , (ˆ p i , ˆ q i ) , (ˆ t j ) repeatfor z = 1 . . .
L do
Compute ˆ S ( v +1)
z = { a 1 , . .
. , a k } with k ∈ [ l, u ] and: | ˆ S ( v +1)
z ∩ S ( v ) max,z | = min( u, k
( v ) max,z )
| ˆ S ( v +1)
z ∩ S ( v ) min,z | = max(0 , l − k
( v ) tie,z
− k ( v ) max,z ) end for for i = 1 . . .
N do
Update the parameters ( p i , q i ) given ˆ S ( v +1) : ˆ p ( v +1) i = (cid:80) z ∈ L | A zi ∩ ˆ S ( v +1)
z | (cid:80) z ∈ L | ˆ S ( v +1)
z | , ˆ q ( v +1) i
=
(cid:80) z ∈ L | A zi ∩ ˆ S ( v +1) z | (cid:80) z ∈ L | ˆ S ( v +1)
z | end for for j = 1 . . .
m do
Update ˆ t ( v +1)
j by: ˆ t ( v +1)
j = occ ( v +1)
( j ) α ( v +1)
j occ ( v +1)
( j ) α ( v +
[Equation]
+ [Equation] L − occ
[Equation]
+ [Equation]
where :
[Equation]
, ˆ
[Equation]
end for until || ˆ θ ( v +1)
− ˆ θ ( v ) || ≤ ε in the previous subsections and are now incorporated into
Algo. 1.
The algorithm continues to run until a convergence criterion is met in the form of a bound on the norm of the change in the parameters’ estimations.
In practice we chose ℓ ∞ , but any other norm could be used in Algorithm 1 as in finite dimensions, all norms are equivalent (if a sequence converges according to one norm then it does so for any norm).
We define the vector of parameters ˆ θ ( v ) =
(ˆ p ( v ) , ˆ q ( v ) , ˆ t ( v ) ) containing the voters’ estimated noise parameters as well as the prior information estimated parameters at iteration v .
In particular ˆ θ (0) is the input initial values.
The choice of the exact initial values depends on the application at hand.
Note that at convergence, only local optimality is guaranteed.
Theorem 4.
For any initial values ˆ θ (0) , AMLE converges to a fixed point after a finite number of iterations.
We only provide a sketch of proof and defer the full proof to the Appendix.
Proof.
First we have by Theorem 1 that:
[Equation]
By Proposition 2 and Proposition 3, we deduce that:
[Equation]
Hence, the likelihood increases at every step.
Since there is a finite number of possible values for the ground truth (namely 2 mL ), the convergence of the algorithm is guaranteed.
Because L ( A, ˆ S ( v +1) , ˆ θ ( v +1) ) ≥ L ( A, ˆ S ( v +1) , ˆ θ ( v ) ) ≥
L ( A, ˆ S ( v ) , ˆ θ ( v ) ) , the likelihood increases at each step of the algorithm.
This guarantees that whenever the execution stops, the likelihood is closer to the maximum than it initially was.
Therefore the algorithm can not only be run until convergence, but it can also be run as an anytime algorithm.
Example 2.
Take n = 3 , m = 5 , l = 1 , u = 2 , L = 4 , and the following profile and initial parameters:
[Equation]
Voter{ a 1 , a 4 } { a 1 } { a 3 } { a 1 }
Voter{ a 2 } { a 5 } { a 4 } { a 1 }
Voter{ a 2 , a 3 , a 4 } { a 2 , a 3 , a 5 } { a 2 , a 3 } { a 3 }   ˆ p (0)1 = 0 .ˆ p (0)2 = 0 .ˆ p (0)3 = 0 .ˆ
q (0)1 = 0 .ˆ
q
(0)2 = 0 .ˆ q
(0)3 =
[Equation]
ˆ t
[Equation] [Equation]
=
[Equation]
Estimating the ground truth: The first step is the application of Theorem 1 to estimate the ground truth of the instances given the initial parameters, yielding ˆ S (1)
[Equation]
{ a 2 , a 4 } , ˆ S (1) =
[Equation]
a 2 , a [Equation]
[Equation]
{ a 1 , a 3 } Estimating the voter reliabilities: In the next step we use these estimates of the ground truths to compute the MLEs of the voter reliabilities.
For instance, voter 1 has 2 false positive labels from a total of 12 negative labels {+so ˆ q (1) 
+}
=12 = 0 .
and she has 3 true positive labels out of 8 positive ones so ˆ p (1)1 = 38 = 0 .
In the end, we get:
[Equation]
Estimating the prior parameters: The final step of this iteration consists in updating the estimations of the prior parameters by applying Proposition 2 sequentially.
First we estimate ˆ t (1) given ˆ S (1) and ˆ t (0)2 , . . .
, ˆ t (0) by maximum likelihood estimation.
We first compute α 1 = β (0 , 1 , t 2 , . . .
, t 5 ) = 0 .
α 1 =
β
(1 , 2 , t 2 , . .
t 5 ) =and occ ( a 1 )
= 1 .
Then the MLE of t 1 is:
[Equation]
The next steps are to estimate ˆ t (1)2 given ˆ t (1)1 , ˆ t (0)3 , ˆ t (0)4 , ˆ t (0) and so on.
Finally, we get: [Equation]
[Equation]
Fix ε
=
10 − 5 .
We repeat all steps until convergence (according to ℓ ∞ ), after 5 full iterations.
In the fixed point, the estimations of the ground truths are: [Equation]
EXPERIMENTS 5.1 EXPERIMENT DESIGN AND DATA COLLECTION
We designed an image annotation task as a football quiz.
We selected 15 pictures taken during different matches between two of the following teams: Real Madrid, Inter Milan,
Bayern Munich, Barcelona, Paris Saint-Germain.
In each picture, it may be the case that players from both teams appear, or players from only one team, therefore l = 1 and u = 2 .
Each participant is shown the instances one by one, and is each time asked to select all the teams she can spot (see Figure 1).
We designed a simple incentive for participants, consisting in ranking them according to the following principle:
• The participants get one point whenever their answer contains all correct alternatives for a picture.
They are then ranked according to their cumulated points.
• To break ties, the participant who selected a smaller number of alternatives overall is ranked first.
We gathered the answers of 76 participants: only two of them spammed by simply selecting all the alternatives.
Figure 2 shows that voters responded well to the incentives by mostly selecting one or two alternatives.
5.2 ANNA KARENINA’S INITIALIZATION
Inspired by the Anna Karenina Principle in Meir et al.
[2019], we assign more weight to voters who are closer to the others on average, initializing the precision parameters ( p i , q i ) accordingly.
This suits our context, where voter competence is highly polarized: some voters are experts and cast similar answers close to the ground truth, the others are less reliable and their answers are dispersed among all combinations.
We use the following heuristics (see Algorithm 2) for the initialization:
Algorithm 2 Initializing ( p i , q i ) i
Input:
Approval ballots ( A zi ) z,i
Output: Initialization (ˆ p (0) i , ˆ q (0) i ) -Compute
w max = n 1+ n , w min = 1 1+ n -Compute d i = (cid:80) j ̸
= i
d Jacc ( A i , A j ) (Jaccard distance) -Compute
d max = max d i , d min = min
d i -Compute w i = ( w max − w min ) (cid:16) 1 / di − 1 / dmax 1 / dmin − 1 / dmax (cid:17) + w min -Fix ˆ p (0) i = 12 and ˆ q (0) i =
[Equation]
Algorithm 2 guarantees that the parameters (ˆ p (0) i , ˆ q (0) i ) of a voter are such that her initial weight is equal to w i , and that w max w min = n : therefore, initially, the voter closest in average to the other voters counts n times more than the voter with the largest average distance.
In the Appendix we give an example illustrating this initialization, and an empirical comparison with other classical initializations. [Figure]
5.3 RESULTS
To assess the importance of prior information on the size of the ground truth, we tested the AMLE algorithm with free bounds ( l, u ) =
(0 , m ) (will be referred to as AMLE f ) and the AMLE c algorithm with ( l, u ) =
We also apply the modal rule Caragiannis et al.
[2020] which outputs the subset of alternatives that most frequently appears as an approval ballot arg max S ∈S | i ∈ N , S
=
A i | , and a variant of label-wise majority rule which outputs the subset of alternatives S such that a ∈ S ⇐⇒ | i ∈ N , a ∈ A i | > n 2 .
If this subset is empty it is replaced by the alternative with highest approval count, and if it has more than two alternatives then we only keep the top-2 alternatives.
We took 20 batches of n = 10 to n = 74 randomly drawn voters and applied the four methods to all of them (see Figure 3a,3b).
As classically done in the literature Nguyen et al.
[2020], we use the Hamming accuracy 1 mL (cid:80)
Lz =1 | S ∗ z
∩ ˆ S z | + | S ∗ z
∩ ˆ S z | and the 0/1 accuracy 1 L (cid:80)
Lz =1 1 { S ∗ z
= ˆ S z } as metrics and report their 0.95 confidence intervals.
We notice that the majority and the modal rule are outperformed by AMLE, which can be explained by the fact that they do not take into account the voters’ reliabilities.
Comparing the performances of AMLE c and AMLE f emphasizes the importance of the prior knowledge on the committee size to improve the quality of the estimation.
We also compared the execution time of AMLE c and
AMLE f (see Figure 4) when run on Intel Core i7-10610U
CPU @1.80Ghz 4 cores, 8 threads and 32Gb RAM.
Unsur- prisingly, AMLE c needs more running time, especially for more than 40 voters.
CONCLUSION
We study multi-winner approval voting from an epistemic point of view.
The specificity of our work is threefold: (a) the ground truth consists of a set of alternatives; (b) the input consists of approval votes; (c) the competence of the various voters is not known a priori but learnt from the input.
We proposed a noise model that incorporates the prior belief about the size of the ground truth.
Then we derived an iterative algorithm to intertwinedly estimate the ground truth labels, the voter noise parameters and the prior belief parameters and we prove its convergence.
Our algorithm is based on a simplification of Expectation-Maximization (EM), and its simple steps are more easily explainable to voters than
EM and other similar statistical learning approaches.
Although we mainly considered a general multi-instance task that fits the collective annotation framework, where each voter answers several questions on the same set of alternatives, we can nonetheless apply the same algorithm to single-instance problems (such as the allocation of scarce medical resources) where only one question is answered.
In this case, the prior parameters cannot be updated and it suffices to fix them once and for all and alternate between the estimation of the ground truth and the voter parameters.
In some contexts ( e.g. , patients in a hospital), alternatives and votes are not observed at once but streamed.
To cope with this online setup we consider extending our AMLE algorithm in the spirit of Cappé and Moulines [2009].
Acknowledgements
This work was funded in part by the French government under management of Agence Nationale de la Recherche as part of the “Investissements d’avenir” program, reference
ANR-19-P3IA-0001 (PRAIRIE 3IA Institute).