Demystifying Learning of Unsupervised Neural Machine Translation
Authors: Guanlin Li, lemao liu, Taro Watanabe, Conghui Zhu, Tiejun Zhao
Keywords: Unsupervised Neural Machine Translation, Marginal Likelihood Maximization, Mutual Information
Abstract: Unsupervised Neural Machine Translation or UNMT has received great attention
in recent years. Though tremendous empirical improvements have been achieved,
there still lacks theory-oriented investigation and thus some fundamental
questions like \textit{why} certain training protocol can work or not under
\textit{what} circumstances have not yet been well understood. This paper
attempts to provide theoretical insights for the above questions. Specifically,
following the methodology of comparative study, we leverage two perspectives,
i) \textit{marginal likelihood maximization} and ii) \textit{mutual information}
from information theory, to understand the different learning effects from the
standard training protocol and its variants. Our detailed analyses reveal
several critical conditions for the successful training of UNMT.