In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.

[EX Q]: We conducted the human evaluation using Amazon Mechanical Turk to assess subjective quality. We recruit master level workers (who have good prior approval rates) to perform a human comparison between generated responses from two systems (which are randomly sampled from comparison systems). The workers are required to judge each utterance from 1 (bad) to 3 (good) in terms of informativeness and naturalness. Informativeness indicates the extent to which generated utterance contains all the information specified in the dialog act. Naturalness denotes whether the utterance is as natural as a human does. To reduce judgement bias, we distribute each question to three different workers. Finally, we collected in total of 5800 judges.
Question: What was the criteria for human evaluation?
[EX A]: to judge each utterance from 1 (bad) to 3 (good) in terms of informativeness and naturalness

[EX Q]: The input for both tasks consists of news articles in free-text format, collected from 36 propagandist and 12 non-propagandist news outlets and then annotated by professional annotators.
Question: What dataset was used?
[EX A]: news articles in free-text format

[EX Q]: Overall, this resulted in a total of 34,432 user conversations. Together, these users gave Gunrock an average rating of 3.65 (median: 4.0), which was elicited at the end of the conversation ("On a scale from 1 to 5 stars, how do you feel about talking to this socialbot again?").
Question: What is the sample size of people used to measure user satisfaction?
[EX A]:
34,432 user conversations