Twenty Flavors of One Text
Dvacet příchutí jednoho textu
Year: 2013
Authors: Daniel Zeman, Ondřej Bojar

Abstract (English):
The relationship between a sentence in a natural language as written down and
its meaning is a very complex phenomenon. Many variations of the sentence
preserve the meaning while other superficially very small changes can distort or
completely reverse it. In order to process and produce sentences
algorithmically, we need to somehow capture the semantic identity and similarity
of sentences.

The issue has been extensively studied from a number of directions, starting
with thesauri and other lexical databases that capture synonymy of indivitual
words (most notably the WordNet \cite{wordnet}), automatic paraphrasing
of longer phrases or even sentences
(e.g.
\cite{bannard-callison-burch-2005},
\cite{kauchak-barzilay},
\cite{denkowski-lavie:2010:WMT}) or
textual entailment \cite{textual-entailment}.
We are still far away from a satisfactory solution.

The field of machine translation
makes the issue tangible in a couple of ways, most importantly within the
task of automatic MT evaluation.
Current automatic MT evaluation techniques rely on the availability of some
reference translation, i.e. the translation as produced by a human translator.
Obtaining such reference translations is relatively costly, so most datasets
provide only one reference translation, see e.g.
\cite{callisonburch-EtAl:2012:WMT}.

While there are many possible
translations of a given input sentence, only a handful of them are available as
reference translations. The sets of hypotheses considered or finally selected by
the MT system can be completely disjoint from the set of reference translations.
\cite{bojar-kos:2010:WMT} observed that only about 5--10\% of reference translations were \emph{reachable} for
Czech-to-English translation , regardless whether the system
would actually search towards them.
Relying mostly on unreachable reference translations is detrimental for MT
system development. Specifically, automatic MT evaluation methods perform worse
and consequently automatic model optimization suffers.

Occasionally there are a few more references available.
For example, in the NIST OpenMT evaluation\footnote{\url{http://www.itl.nist.gov/iad/mig/tests/mt/2006/doc/mt06eval_official_results.html}}
or in the Prague Czech-English Dependency Treebank \cite{pcedt}
there are four.
Difficult as they may be to obtain, even four references fail to provide
an adequate representation of the enormous space of correct translations.

We want to trigger discussion about how many and what sort of references
we actually need for MT, how can we obtain and use them.

Obviously the set of possible translations is exponentional in the length of the sentence.
Dreyer and Marcu \cite{dreyer-marcu} generated translation networks for the HyTER
evaluation metric; the networks provided them with a compact representation
of the large space of possible translations.
Along the same lines, we propose unification-based approach to describe the space
\cite{tsd-hyter}.
Using a Prolog-based annotation tool
and applying a 2-hour upper working limit per sentence,
our translators were able to generate tens of thousands (or more) Czech references
for each English sentence; the record figure was 19 billion translations
of one long sentence.
The way of actually using such data is an open question.
If we just take the \texttt{mteval} script and run it with 19 billion references instead of four,
we will probably not live long enough to see the results.
Randomly selected and manually checked samples from these generated translations
indicated that the translations are indeed plausible.
We used a Levenshtein-based string similarity measure to confirm diversity of the generated set.

We also measured the string similarity between system outputs and various reference translations:
those from the generated set
and also two manually post-edited system outputs.
Unsurprisingly, each of the two post-edited translations were much closer to the
original system outputs than the single reference translation that was
originally available for that test set.
However, our annotators managed to produce references which are almost
exactly as close as the post-edited translations,
even though they did not have access to them or the system outputs.

\cite{wmt2013} showed that with several references (and with the BLUE and NIST metrics)
smaller test set with multiple post-edited references
has the same level of correlation with human judgments
as a much larger test set with just one reference, created independently of MT.
In future we might further investigate that in terms of
\textit{which} system outputs to post-edit (average? good? all?)

During the past 20 years we have accummulated impressive amount of bitext and it keeps growing.
Our current conclusion is that not every bitext is equally valuable.
Diversity of translation alternatives in either language is at least as important.
So where will we be after the next 20 years?
Quite likely, the amount of bitext available will multiply accordingly.
However, we also hope that for every sentence ``pair'', the number of available translation variants
will also multiply.

Abstract (Czech):
Vztah mezi větou v přirozeném jazyce, tak jak je napsaná, a jejím významem je velmi složitý jev. Existuje mnoho variant téže věty, které zachovávají význam, zatímco jiné, povrchově velmi malé změny mohou význam zkreslit nebo zcela otočit. Abychom mohli věty algoritmicky zpracovávat a generovat, musíme najít způsob, jak zachytit jejich sémantickou identitu a podobnost.