  Crowdsourcing has become a common approach for annotating large amounts of
data. It has the advantage of harnessing a large workforce to produce large
amounts of data in a short time, but comes with the disadvantage of employing
non-expert annotators with different backgrounds. This raises the problem of
data reliability, in addition to the general question of how to combine the
opinions of multiple annotators in order to estimate the ground truth. This
paper presents a study of the annotations and annotators' reliability for audio
tagging. We adapt the use of Krippendorf's alpha and multi-annotator competence
estimation (MACE) for a multi-labeled data scenario, and present how MACE can
be used to estimate a candidate ground truth based on annotations from
non-expert users with different levels of expertise and competence.
