Detailed Instructions: In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.
Q: We perform a direct comparison of expert and crowd contributors, for 1000 term groups based on the number of total annotations(200 term groups with 2 total annotations, 200 term groups with 3 total annotations, and so on up to term groups with 6 total annotations). Evaluators were given a summary of the annotations received for the term group in the form of:The term group "inequality inequity" received annotations as 50.0% sadness, 33.33% disgust, 16.67% anger. Then, they were asked to evaluate on a scale from 1 to 5, how valid these annotations were considered.
Question: How do they compare lexicons?
A:
Human evaluators were asked to evaluate on a scale from 1 to 5 the validity of the lexicon annotations made by the experts and crowd contributors.