k- R ater R eliability: T he Correct Unit of Reliability for Aggregated Human Annotations (2022)

Abstract: Since the inception of crowdsourcing, aggregation has been a common strategy for dealing with unreliable data. Aggregate ratings are more reliable than individual ones. However, many Natural Language Processing (NLP) applications that rely on aggregate ratings only report the reliability of individual ratings, which is the incorrect unit of analysis. In these instances, the data reliability is under-reported, and a proposed k -rater reliability (kRR) should be used as the correct data reliability for aggregated datasets. It is a multi-rater generalization of inter-rater reliability (IRR). We conducted two replications of the WordSim-353 benchmark, and present empirical, analytical, and bootstrap-based methods for computing kRR on WordSim-353. These methods produce very similar results. We hope this discussion will nudge researchers to report kRR in addition to IRR.

## introduction crowdsourcing has become a mainstay for data collection in nlp @xcite . it can produce data in a scalable and cost effective manner. however, these benefits come at a cost: quality. the reliability of crowd workers is always of central concern. one common strategy to increase the data reliability is to collect multiple, independent judgements and to use the aggregated judgements instead. indeed, early papers such as @xcite show that average ratings correlate more strongly with expert judgements. this makes sense, as average ratings are known to have a higher reliability than individual ones @xcite . a number of strategies have been proposed to address data quality issues, e.g. rater modeling, label correction, label pruning @xcite , but aggregation remains very popular @xcite . @xcite present nine crowdsourced datasets across a wide range of nlp tasks to compare different aggregation methods. see @xcite for a recent review of aggregation techniques. in short, aggregation has become the default method for acquiring reliable data from the crowd. interestingly, after we adopted aggregation as a community, we forgot to update our reliability measures correspondingly. the field continues to report data reliability in terms of irr, even when aggregate ratings are used. focusing on irr, we are unable to capture the increase in reliability due to aggregation. the actual data reliability is hence unknown. this has important consequences. reliability is often used as a safeguard for reproducibility. therefore conclusions about the reproducibility of a dataset drawn based the reliability of individual ratings may be different than that based on the reliability of aggregate ratings. by reporting the correct reliability that is actually higher, this may even have a side effect of lessening the stigma on low-irr datasets. as a result, this may create a path forward towards reliable data on subjective tasks, where a high irr is difficult to obtain, such as emotions @xcite and toxicity @xcite . with a reproducibility crisis looming in the background @xcite , more frequent and accurate reporting of reliability is our primary safeguard @xcite . we denote the reliability of aggregate ratings as k-rater reliability (krr), in order to differentiate it from inter-rater reliability. in this paper we present a few methods for computing krr. first, we demonstrate a general, empirical approach that is based on replications. to that end, we conducted two replications of wordsim-353 @xcite , a widely used word similarity dataset. we then discuss two other alternatives that do not require replications. one is a re-sampling-based bootstrap approach @xcite . it is suitable for experiments with a high rating redun-dancy. the other is an existing analytical approach based on intraclass correlation (icc). it is suitable for continuous data where the aggregation is the mean. we conclude with recommendations for reporting reliability of crowdsourced annotations, and novel research questions to expand the usefulness of krr. ## related work various authors have stressed the importance of measuring reliability for the correct unit of analysis. @xcite asks "is it better to estimate the reliability of individual ratings or the reliability of average ratings? if decisions are based upon average ratings, it of course follows that the reliability with which one should be concerned is the reliability of those averages." @xcite and @xcite reiterate similar points. these studies primarily focus on the reliability of the mean, which is just one of many different aggregation methods. there is a reason. not only is the mean a popular choice, it is also the only known choice where the reliability of the aggregate ratings can be computed analytically from the reliability of individual ratings. this is done in the icc framework. icc is typically used to measure the reliability of single ratings, but it actually has a variant that can be used for mean ratings as well. @xcite list several types of icc coefficient, one of which is for mean ratings. they call it icc(k), where k is the number of ratings per item. in this generalized notation, icc(1) is just the reliability of individual ratings, or the irr. note that @xcite use a slightly different notation, icc(1, k), to explicitly denote that it is for a one-way random effects model, where the raters are treated as interchangeable. that is a common assumption in most crowdsourcing experiments done on commercial platforms such as amazon mechanical turk. icc(k) is an established way of measuring the reliability of mean ratings, hence it is readily usable by researchers. however, it has some drawbacks. being part of the icc family, icc(k) is only applicable to continuous data. in addition, icc(k) measures the reliability of mean ratings, therefore it cannot accommodate other aggregation functions. in other words, for other popular data types, such as majority votes of binary data, there is no known coefficients for measuring the reliability of aggregate ratings. other than icc(k), the authors are not aware of any multi-rater generalization for other coefficients such as @xcite kappa or @xcite . we therefore take icc(k) as an inspiration and abstract away from it to define a class of reliability that describes the reliability of aggregate ratings for any data types. we denote it krr. ## k-rater reliability we define krr as the chance-adjusted agreement between replications of aggregate ratings. this definition is very similar to irr. in fact, they only differ in terms of interpretation. krr is identical to irr other than that each individual rating in the irr calculation is replaced by a k-rater aggregate rating. after all, the mathematics in irr are agnostics to how those labels are produced. just like irr, a minimum of two replications is required to calculate krr. given two vectors of aggregate ratings, one can calculate the reliability between them using any irr coefficients that fit the purpose. krr is designed to be analogous to irr so that we can build upon the rich irr literature and the various coefficient choices for different experimental conditions and assumptions. for example, in a binary task, if all the items are rated by two fixed but distinct groups of raters (raters from different locales), @xcite kappa is a suitable choice. whereas if the raters groups are homogeneous, and the rating scale is ordinal (e.g. likert), then krippendorff's alpha @xcite can be used. just like irr, krr is a general concept and is agnostic to the choice of coefficient. this definition of krr can be directly operationalized by creating replications. we call this approach to calculating krr the empirical approach. we demonstrate it in the next section on the wordsim-353 benchmark. the empirical approach is the most direct and most general, with the drawback that a minimum of two replications are required. we later present two narrower alternatives in section 5 that do not require replications. the empirical results will be used as a golden reference to validate them. ## other approaches to computing krr the empirical approach is general, as it can accommodate any choice of rating scale, aggregation function, and reliability coefficient. however, it has a major drawback. as we see in section 4.1, it can be difficult to do a perfect replication postfact. this backward incompatibility will present a challenge to computing krr for existing datasets. below we present two alternatives that can work on existing datasets under some conditions without requiring any additional data collection. one is a re-sampling based bootstrap approach @xcite , the other is icc(k). ## conclusions and discussion we pointed out where aggregated ratings are used, as is the case in many crowdsourced datasets, reliability of aggregate ratings is the correct accounting of data reliability. we introduced k-rater reliability (krr) as a multi-rater extension of irr. we emphasise the reliability of aggregate ratings is higher than that of individual ratings. we present analytical and bootstrap-based methods for computing the krr on the original wordsim dataset. both methods produce similar estimates for 13-rater reliability ranging from 0.940 to 0.953. we conduct two replications of the entire wordsim-353 benchmark to validate these methods. we make our replication data publicly available on github. while aggregation makes it possible to have reliable benchmarks on subjective topics, some readers may feel uneasy about increasing reliability via gathering additional ratings, as opposed to other traditional means such as improving rater guidelines. we suggest to mediate this concern by reporting both irr and krr. in fact, krr is not meant to replace irr, but rather complement it. irr speaks to the reliability of the labeling process, whereas krr quantifies the reliability of the aggregated data we consume. we urge researchers to report both where possible. in fact, @xcite states, "in cases where single measures iccs are low but averagemeasures iccs are high, the researcher may report both iccs to demonstrate this discrepancy." this research also raises interesting questions for future research: we urge researchers to report both irr and krr of aggregated human annotations, and for further inquiry around the above fundamental questions about reliability.