Manual evaluation of translation quality is generally thought to be excessively time consuming and expensive . We explore a fast and inexpensive way of doing it using Amazon 's Mechanical Turk to pay small sums to a large number of non-expert annotators . For $ 10 we redundantly recreate judgments from a WMT08 translation task . We find that when combined non-expert judgments have a high-level of agreement with the existing gold-standard judgments of machine translation quality , and correlate more strongly with expert judgments than Bleu does . We go on to show that Mechanical Turk can be used to calculate human-mediated translation edit rate ( HTER ) , to conduct reading comprehension experiments with machine translation , and to create high quality reference translations . Conventional wisdom holds that manual evaluation of machine translation is too time-consuming and expensive to conduct . Instead , researchers routinely use automatic metrics like Bleu ( Papineni et al . , 2002 ) as the sole evidence of improvement to translation quality . Automatic metrics have been criticized for a variety of reasons ( Babych and Hartley , 2004 ; Callison-Burch et al . , 2006 ; Chiang et al . , 2008 ) , and it is clear that they only loosely approximate human judgments . Therefore , having people evaluate translation output would be preferable , if it were more practical . In this paper we demonstrate that the manual evaluation of translation quality is not as expensive or as time consuming as generally thought . We use Amazon 's Mechanical Turk , an online labor market that is designed to pay people small sums of money to complete human intelligence teststasks that are difficult for computers but easy for people . We show that : • Non-expert annotators produce judgments that are very similar to experts and that have a stronger correlation than Bleu . • Mechanical Turk can be used for complex tasks like human-mediated translation edit rate ( HTER ) and creating multiple reference translations . • Evaluating translation quality through reading comprehension , which is rarely done , can be easily accomplished through creative use of Mechanical Turk . 2 Related work Snow et al . ( 2008 ) examined the accuracy of labels created using Mechanical Turk for a variety of natural language processing tasks . These tasks included word sense disambiguation , word similarity , textual entailment , and temporal ordering of events , but not machine translation . Snow et al . measured the quality of non-expert annotations by comparing them against labels that had been previously created by expert annotators . They report inter-annotator agreement between expert and non-expert annotators , and show that the average of many non-experts converges on performance of a single expert for many of their tasks . Although it is not common for manual evaluation results to be reported in conference papers , several large-scale manual evaluations of machine translation quality take place annually . These include public forums like the NIST MT Evaluation Workshop , IWSLT and WMT , as well as the project-specific Go / No Go evaluations for the DARPA GALE program . Various types of human judgments are used . NIST collects 5-point fluency and adequacy scores ( LDC , 2005 ) , IWSLT and WMT collect relative rankings ( Callison-Burch et al . , 2008 ; Paul , 2006 ) , and DARPA evaluates using HTER ( Snover et al . , 2006 ) . The details of these are provided later in the paper . Public evaluation campaigns provide a ready source of goldstandard data that non-expert annotations can be compared to . Mechanical Turk is an inexpensive way of gathering human judgments and annotations for a wide variety of tasks . In this paper we demonstrate that it is feasible to perform manual evaluations of machine translation quality using the web service . The low cost of the non-expert labor found on Mechanical Turk is cheap enough to collect redundant annotations , which can be utilized to ensure translation quality . By combining the judgments of many non-experts we are able to achieve the equivalent quality of experts . The suggests that manual evaluation of translation quality could be straightforwardly done to validate performance improvements reported in conference papers , or even for mundane tasks like tracking incremental system updates . This challenges the conventional wisdom which has long held that automatic metrics must be used since manual evaluation is too costly and timeconsuming . We have shown that Mechanical Turk can be used creatively to produce quite interesting things . We showed how a reading comprehension test could be created , administered , and graded , with only very minimal intervention . We believe that it is feasible to use Mechanical Turk for a wide variety of other machine translated tasks like creating word alignments for sentence pairs , verifying the accuracy of document-and sentence-alignments , performing non-simulated active learning experiments for statistical machine translation , even collecting training data for low resource languages like Urdu . The cost of using Mechanical Turk is low enough that we might consider attempting quixotic things like human-in-the-loop minimum error rate training ( Zaidan and Callison-Burch , 2009 ) , or doubling the amount of training data available for Urdu .

Challenge: Because of the high cost required for manual evaluation, most works rely on automatic evaluation metrics although there are several drawbacks.
Approach: They investigate whether judgements by non-experts from Amazon's Mechanical Turk can be a fast and inexpensive means of evaluation for machine translation systems.
Outcome: They found that non-expert judgements with high agreement correlate better with gold standard judgements than BLEU while keeping the cost low.