In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.
Q: Human Judgments
Following BIBREF11 , BIBREF12 and the vast amount of previous work on semantic similarity, we ask nine undergraduate subjects to assess the similarity of 360 pairs of relations from a subset of Wikidata BIBREF8 that are chosen to cover from high to low levels of similarity. In our experiment, subjects were asked to rate an integer similarity score from 0 (no similarity) to 4 (perfectly the same) for each pair.
Question: How do they gather human judgements for similarity between relations?
A:
By assessing similarity of 360 pairs of relations from a subset of Wikidata using an integer similarity score from 0 to 4