Student’s t-Distribution: On Measuring the Inter-Rater Reliability When the Observations are Scarce (2023)

Abstract: In natural language processing (NLP) we always rely on human judgement as the golden quality evaluation method. However, there has been an ongoing debate on how to better evaluate inter-rater reliability (IRR) levels for certain evaluation tasks, such as translation quality evaluation (TQE), especially when the data samples (observations) are very scarce. In this work, we first introduce the study on how to estimate the confidence interval for the measurement value when only one data (evaluation) point is available. Then, this leads to our example with two human-generated observational scores, for which, we introduce “Student’s t -Distribution” method and explain how to use it to measure the IRR score using only these two data points, as well as the confidence intervals (CIs) of the quality evaluation. We give a quantitative analysis of how the evaluation confidence can be greatly improved by introducing more observations, even if only one extra observation. We encourage researchers to report their IRR scores in all possible means, e.g. using Student’s t -Distribution method whenever possible; thus making the NLP evaluation more meaningful, transparent, and trustworthy. This t -Distribution method can be also used outside of NLP fields to measure IRR level for trustworthy evaluation of experimental investigations, whenever the observational data is scarce.

## introduction human evaluations have been always the gold standard to judge the quality of natural language processing (nlp) system's outputs @xcite @xcite . this applies to many sub-tasks including machine translation (mt) @xcite @xcite ), text summarisation @xcite , question answering @xcite , information extraction @xcite , and prediction @xcite , as well as domain applications such as social media, biomedical and clinical domains knowledge representation @xcite @xcite . nonetheless, human evaluations have been subject to criticisms and debates about their reliability, particularly when conducted without strictly defined procedures. @xcite @xcite . despite the inclusion of factors such as quality controls and clear guidelines, human evaluation results can vary greatly among different individuals due to subjective judgements influenced by factors such as backgrounds, personalities, cultures, and so on. naturally, the confidence levels of human evaluation become the key point to the validity of such work. there have been some efforts made on how to measure the confidence level of human evaluations from a statistical point of view, such as very recent work using monte carlo sampling simulations by @xcite . however, this kind of statistical measurement still needs a good amount of data points, or observations, to be based upon. when there are a limited amount of observations obtained from the experiments, how to measure the confidence level properly is still a challenging question. one of the solutions to address this is to calculate the inter-rater agreement level and inter-rater reliability (irr) scores. there are some historical irr measurement metrics including cohen's kappa @xcite and krippendorff's alpha @xcite . how-ever, as the last issue with statistical sampling, both cohen's kappa and krippendorff's alpha need a certain amount of samples data for probability calculation, which becomes troublesome when the observations are really scarce, e.g. only one, two, or a few data points. in addition, there are existing criticisms regarding the undesired prediction of cohen's kappa, e.g. @xcite gave examples about how kappa produced better scores for worse classifiers. in this study, we examine scenarios in which observations from human evaluations are extremely scarce (e.g., limited to one or two values) and explore potential solutions. this endeavor is motivated by the realities of the translation and localization industry. practitioners often need to determine the reliability of human machine translation (mt) quality evaluations based on the judgments of a single linguist, or at most, two to three linguist evaluations. we start from one observation and introduce a confidence estimation method borrowed from @xcite which was applied to forest study by @xcite . then we discuss how this one observation-based confidence estimation is problematic and not much reliable. following this, we bring an example of mt evaluation where two observations are obtained from the human assessment. in this case study, we introduce how to apply student's t-distribution to measure irr with detailed formula interpretation and guidance. we also further give instructions on how to measure confidence intervals (cis) using this method. we discuss the much improvement achieved by using two observational data points and student's t-distribution regarding narrowed-down cis. finally, we suggest that researchers also apply student's t-distribution to other nlp tasks and even beyond, i.e. outside of nlp tasks. the rest of the paper is organised as below: section 2 surveys the related work to ours on measuring irr and confidence intervals from different fields, section 3 presents a case study with a single observation, section 4 follows up with two and more observations where we introduce student's t-distribution, and section 5 concludes the paper with discussion and future work. ## related work regarding agreement measurement, cohen's kappa metric was defined by @xcite as below: in other words, the kappa value reflects the agreement level (or proportion of agreement) after deducting the chance agreement. in the perfect situation, when the raters all agree with each other, i.e. the p o value equals 1, the kappa value will be 1 @xcite . however, if the raters totally disagree with each other, i.e. the value of p o is almost the same value of agreement by chance p e , the kappa value will be close or equal to 0. however, kappa's value can be a negative number, when the agreement exhibition level is even smaller than by chance, e.g. using the above equations when the value of p op e or f of e is negative. as mentioned in the earlier section, the kappa value requires a certain number of observations to properly estimate the metric scores. looking into the irr measurement in crowdsourcing human evaluation domain, @xcite argued that the traditional krippendorff's alpha or cohen's kappa threshold values, e.g. above 0.6, are not ideal due to the ignorance of cultural and individual differences from crowdsource workers. they proposed a cross-replication reliability method based on cohen's kappa and tested the methods on human judgements of facial expressions using a large amount of 4 million data points. from nlp and mt field, @xcite ; @xcite applied monte carlo simulation analysis method to generate more samples for statistically estimating the confidence intervals of judgements when the samples presented for the human evaluation are small. their experimental outputs demonstrate that not less than 100, and ideally 200 segments are necessary for the test set to produce an unbiased, statistically significant quality evaluation of the mt system output. outside of nlp fields, there are also some efforts made to address similar issues in measuring reliability and confidence intervals. for instance, @xcite gave a tutorial on measuring irr for the psychology domain when multiple coders are involved using case studies using commonly used cohen's kappa and intra-class correlation (icc). similarly in the educational and psychological domain, walker and göc ¸er s ¸ahin (2020) carried out a study on applying differential item functioning (dif) analysis to measure irr, in comparison to the inter-class correlation coefficient and cohen's kappa statistics. from animal behaviour studies, harvey (2021) raised the issues on inter-rater and intra-rater reliability and made a discussion on cohen's kappa and krippendorff's alpha values. from the sociological domain, @xcite reported a systematic survey on reporting irr values from crime studies on multiple coders. they made further discussion on how human factors affect decisionmaking and how important it is to report accuracy, precision and reliability from screening/coding. there is some existing business software integrating irr into their statistical tools such as spss that has been used in different sectors including medical training assessment @xcite . the ibm spss uses interclass correlation coefficient (icc) to measure the irr values among different groups of raters foot_0 . however, to the best of our knowledge, there is no existing work on applying student's t-distribution for measuring irr in nlp applications, especially in translation quality evaluation (tqe) field. ## on single judgement when observational data is very scarce, more than half a century ago, @xcite proved the possibility of measuring confidence intervals on a single data point from a mathematical point of view, and the later work from furnival et al. @xcite further elaborated abbott and rosenblatt's formula with a more narrowed interval generation. we name it the arf interval by taking the initial letters of their names. 2 this method may appear statistically counterintuitive, but it is certainly worth mentioning here, particularly as production decisions are frequently based on a single quality measurement. an intriguing paradox arises: while many statisticians would argue it's impossible to determine a confidence interval from one measurement, project managers often rely on a single tqe (translation quality evaluation) value to make their decisions. in actuality, conclusions about the reliability of a single measurement can be made, but they require supplemental information, e.g., for translation industy, known vendor's past performance. within the arf interval calculation method, this additional data is also derived from an experimentally-based prior knowledge or theoretically-based value that, while external to the measurement, arises from the project context. interestingly, project managers who use a single measurement's value to make their decisions apply a similar intuition. consequently, it's fascinating to explore what mathematical principles can elucidate within this context. the width of the confidence intervals reflects the uncertainty of the experiments, i.e., the wider it is, the less knowledge is available about the setup. a relatively narrowed confidence interval indicates the controlled situations, for instance, the normal distribution in the following formula of standardised transformation: for the situation with one observation, let μ be the independent and fixed value that is known before and outside of the measurement, and y be the experimental measurement value. furnival et al. @xcite gives the following calculation intervals: this arf interval contains the probability of µ that is larger than or equal to 1α, and α meets the following equation with k: the pair value of (k, α) was given by furnival et al. @xcite as in ## on observations of more than one following the last section, we call on more measurement points for nlp evaluation tasks, especially in the language service provider sector where the single observation value is still very common in practice due to the cost concern. 3 for instance, when a single translation quality measurement is not satisfactory for one of the parties, second quality measurement can be made to validate the first measurement. then, how much improvement to the confidence interval can be obtained by introducing extra observational data? to answer this, the obvious problem, of course, is that at least 20-30 data points are required to calculate the mathematical variance for a normal distribution. in settings where the sample size is less than 30, and the standard deviation of the entire population is unknown, student's t-distribution can be used to evaluate standard deviation based only on the number of measurements between one and 30, e.g. 2, 3, etc. ## conclusion and future work when it comes to evaluating translation quality, the ability to measure alone is not sufficient; we also need to know how reliable the measurement is. automatic evaluation of quality quickly produces the same scores if repeated a number of times, which creates an illusion of precision. unfortunately, the results of automatic quality measurement not only depend on the language pair, the machine learning system, the decoder, and the content type, but also vary from dataset to dataset, depending on the way the data have been cleaned and formatted. given these factors, automatic measurement can be very fast and "reliable", but it may be (and often is) invalid, as well as inconsistent. human translation quality evaluation (tqe) is currently the only way to obtain valid measurements of human perception of quality, and considered to be the golden standard of tqe. however, human measurement's inter-rater reliability (irr) should be assessed, even if evaluation has been carried out correctly. even if evaluators are experienced linguists, trained to do evalu-ation according to proper system, and client specifications are clearly defined, the evaluators would still produce close but not the same evaluation results due to the very nature of human perception of quality, which is by definition the function of personal perception. this problem is exacerbated by the fact that in real life production setting there is no time or budget to validate the translation quality measurement even with the second reviewer, and even if there is a second reviewer, the low irr of such measurement makes it difficult to confirm the first measurement. in this paper, we first studied the typical production setting of gold standard human quality measurement, where tqe is performed by only one experienced, trained linguist, according to clearly defined customer specifications, producing a single measurement, and make conclusions about the reliability of such measurement. we then illustrated the results with the case of student's t-distribution analysis of two measurements made by two different reviewers. from the first and second experiments, we can conclude that a single measurement has very low reliability and only has an indicative value. the confidence interval for one measurement is (as shown in section 3) as wide as 25%, and therefore one evaluation cannot be taken as a basis for process decisions, more measurements are required. for instance, the second measurement can narrow down this interval and render it two times smaller, to around 10% (section 4). yet we can say that the middle of the confidence interval lies halfway between the earlier average result and the lower recent measurement. a good rule of thumb is born: if the single measurement deviates from the average, the true value sits, in all probability, halfway between the average and the new measurement. consequently, the recommendation is: please do not over-react on an unusually low new single score, take a middle ground between the older average and the new score, and think about it as the most probable result. the second measurement may improve the confidence interval significantly but is rarely done unless during the arbitration. therefore, it is more practical to obtain additional data points from other evaluations of different samples, in the course of the translation process. subsequent evaluations effectively are placed into two categories: (a) mostly pass with only rare occasional fail, (b) all other cases (mostly fail, or many fails). this strategy is caused by the desire to ensure that a system is reliably well above the pass/fail threshold and thus ensures quality results. multiple evaluations also confirm the validity of quality measurements and allow the application of well-known maths of statistics of normal distribution. however, it is worth noting that proper methods of data analysis are required to analyse quality evaluation data-sets, such as: it should be remembered, that data science only allows obtaining good results if you clean the data properly. incorrect, biased, not properly calibrated, or imprecise conclusions and inferences may result from using uncleaned data.