  Multilingual machine translation has attracted much attention recently due to
its support of knowledge transfer among languages and the low cost of training
and deployment compared with numerous bilingual models. A known challenge of
multilingual models is the negative language interference. In order to enhance
the translation quality, deeper and wider architectures are applied to
multilingual modeling for larger model capacity, which suffers from the
increased inference cost at the same time. It has been pointed out in recent
studies that parameters shared among languages are the cause of interference
while they may also enable positive transfer. Based on these insights, we
propose an adaptive and sparse architecture for multilingual modeling, and
train the model to learn shared and language-specific parameters to improve the
positive transfer and mitigate the interference. The sparse architecture only
activates a subnetwork which preserves inference efficiency, and the adaptive
design selects different subnetworks based on the input languages. Evaluated on
multilingual translation across multiple public datasets, our model outperforms
strong baselines in terms of translation quality without increasing the
inference cost.
