  Recently, universal neural machine translation (NMT) with shared
encoder-decoder gained good performance on zero-shot translation. Unlike
universal NMT, jointly trained language-specific encoders-decoders aim to
achieve universal representation across non-shared modules, each of which is
for a language or language family. The non-shared architecture has the
advantage of mitigating internal language competition, especially when the
shared vocabulary and model parameters are restricted in their size. However,
the performance of using multiple encoders and decoders on zero-shot
translation still lags behind universal NMT. In this work, we study zero-shot
translation using language-specific encoders-decoders. We propose to generalize
the non-shared architecture and universal NMT by differentiating the
Transformer layers between language-specific and interlingua. By selectively
sharing parameters and applying cross-attentions, we explore maximizing the
representation universality and realizing the best alignment of
language-agnostic information. We also introduce a denoising auto-encoding
(DAE) objective to jointly train the model with the translation task in a
multi-task manner. Experiments on two public multilingual parallel datasets
show that our proposed model achieves a competitive or better results than
universal NMT and strong pivot baseline. Moreover, we experiment incrementally
adding new language to the trained model by only updating the new model
parameters. With this little effort, the zero-shot translation between this
newly added language and existing languages achieves a comparable result with
the model trained jointly from scratch on all languages.
