Fine-tuning is a major approach for domain adaptation in Neural Machine Translation (NMT) . However, unconstrained fine-tuning requires very careful hyper-parameter tuning otherwise it is easy to fall into over-fitting on the target domain and degradation on the general domain. To mitigate it , we propose PRUNE-TUNE , a novel domain adaptation method via gradual pruning. It learns tiny domain-specific subnetworks for tuning. During adaptation to a new domain, we only tune its corresponding subnetwork. PRUNE-TUNE alleviates the over-fitting and the degradation problem without model modification. Additionally, with no overlapping between domain-specific subnetworks, PRUNE-TUNE is also capable of sequential multi-domain learning . Empirical experiment results show that PRUNE-TUNE outperforms several strong competitors in the target domain test set without  the quality degradation of the general domain in both single and multiple domain settings.
Neural machine translation often adopts the fine-tuning approach to adapt to specific domains . However, nonrestricted fine-tuning can easily degrade on the general domain and over-fit to the target domain. To mitigate the issue , we propose Prune-Tune , a novel domain adaptation method via gradual pruning. It learns tiny domain-specific sub-networks during fine-tuning on new domains. Prune-Tune alleviates the over-fitting and the degradation problem without model modification. Furthermore, Prune-Tune is able to sequentially learn a single network with multiple disjoint domain-specific sub-networks for multiple domains . Empirical experiment results show that Prune-Tune outperforms several strong competitors in the target domain test set without sacrificing the quality on the general domain in both single and multi-domain settings. The source code and data are available at URL
R
Fine-tuning is a major approach for domain adaptation in Neural Machine Translation (NMT)
Neural machine translation often adopts the fine-tuning approach to adapt to specific domains
R
unconstrained
nonrestricted
R
requires very careful hyper-parameter tuning otherwise it is easy to fall into over-fitting on the target domain and degradation on the general
can easily degrade on the general domain and over-fit to the target
R
it
the issue
R
PRUNE-TUNE
Prune-Tune
R
subnetworks for tuning. During adaptation to a new domain, we only tune its corresponding subnetwork. PRUNE-TUNE
sub-networks during fine-tuning on new domains. Prune-Tune
R
Additionally, with no overlapping between
Furthermore, Prune-Tune is able to sequentially learn a single network with multiple disjoint
R
subnetworks, PRUNE-TUNE is also capable of sequential multi-domain learning
sub-networks for multiple domains
R
PRUNE-TUNE
Prune-Tune
A
sacrificing the quality on
D
quality degradation of the
R
multiple domain settings.
multi-domain settings. The source code and data are available at URL