This work introduces AM-FM , a semantic framework for machine translation evaluation . Based upon this framework , a new evaluation metric , which is able to operate without the need for reference translations , is implemented and evaluated . The metric is based on the concepts of adequacy and fluency , which are independently assessed by using a cross-language latent semantic indexing approach and an n-gram based language model approach , respectively . Comparative analyses with conventional evaluation metrics are conducted on two different evaluation tasks ( overall quality assessment and comparative ranking ) over a large collection of human evaluations involving five European languages . Finally , the main pros and cons of the proposed framework are discussed along with future research directions . Evaluation has always been one of the major issues in Machine Translation research , as both human and automatic evaluation methods exhibit very important limitations . On the one hand , although highly reliable , in addition to being expensive and time consuming , human evaluation suffers from inconsistency problems due to inter-and intraannotator agreement issues . On the other hand , while being consistent , fast and cheap , automatic evaluation has the major disadvantage of requiring reference translations . This makes automatic evaluation not reliable in the sense that good translations not matching the available references are evaluated as poor or bad translations . The main objective of this work is to propose and evaluate AM-FM , a semantic framework for assessing translation quality without the need for reference translations . The proposed framework is theoretically grounded on the classical concepts of adequacy and fluency , and it is designed to account for these two components of translation quality in an independent manner . First , a cross-language latent semantic indexing model is used for assessing the adequacy component by directly comparing the output translation with the input sentence it was generated from . Second , an n-gram based language model of the target language is used for assessing the fluency component . Both components of the metric are evaluated at the sentence level , providing the means for defining and implementing a sentence-based evaluation metric . Finally , the two components are combined into a single measure by implementing a weighted harmonic mean , for which the weighting factor can be adjusted for optimizing the metric performance . The rest of the paper is organized as follows . Section 2 , presents some background work and the specific dataset that has been used in the experimental work . Section 3 , provides details on the proposed AM-FM framework and the specific metric implementation . Section 4 presents the results of the conducted comparative evaluations . Finally , section 5 presents the main conclusions and relevant issues to be dealt with in future research . This work presented AM-FM , a semantic framework for translation quality assessment . Two comparative evaluations with standard metrics have been conducted over a large collection of humangenerated scores involving different languages . Although the obtained performance is below standard metrics , the proposed method has the main advantage of not requiring reference translations . Notice that a monolingual version of AM-FM is also possible by using monolingual latent semantic indexing ( Landauer et al . , 1998 ) along with a set of reference translations . A detailed evaluation of a monolingual implementation of AM-FM can be found in Banchs and Li ( 2011 ) . As future research , we plan to study the impact of different dataset sizes and vector space model parameters for improving the performance of the AM component of the metric . This will include the study of learning curves based on the amount of training data used , and the evaluation of different vector model construction strategies , such as removing stop-words and considering bigrams and word categories in addition to individual words . Finally , we also plan to study alternative uses of AM-FM within the context of statistical machine translation as , for example , a metric for MERT optimization , or using the AM component alone as an additional feature for decoding , rescoring and/or confidence estimation .

Challenge: For evaluating machine translation systems, manual evaluation is costly and inconsistent and the automatic counterpart is unreliable because it requires references.
Approach: They propose a new evaluation framework which assesses adequacy and fluency by using a cross-language latent indexing approach and n-gram based language modelling without references.
Outcome: The proposed method underperforms two standard metrics on a large collection in different languages and shows however it does not require reference translations.