Investigating the Effectiveness of Multiple Expert Models Collaboration (2023)

Abstract: This paper aims to investigate the effectiveness of several machine translation (MT) models and aggregation methods in a multi-domain setting under fair conditions and explore a direction for tackling multi-domain MT. We mainly compare the performance of the single model approach by jointly training all domains and the multi-expert models approach with a particular aggregation strategy. We conduct experiments on multiple domain datasets and demonstrate that a combination of smaller domain expert models can outperform a larger model trained for all domain data.

## introduction the machine translation (mt) research community has been paying much attention to multi-domain evaluation @xcite as well as document-level evaluation these days. this trend is supported by the fact that the main translation task in the wmt competition foot_0 shifted to a multi-domain setting after 2022. we explore a more effective and efficient approach since this is a more realistic and desirable setting in actual mt systems. there are two major approaches for tackling multi-domain adaptation: a multi-domain model (mdm), and domain expert models (dems). in mdm, the model is trained on all domain data; in dems, expert models are trained for each domain (figure 1 ). most previous studies have focused on mdm @xcite since mdm is the identical approach when we do not consider domains. in contrast, dems have yet to be investigated in-depth due to not much attention in the community until recent years. we hypothesize that dems have the potential to achieve superior performance if each expert model is well-trained in the corresponding domain and better combined. moreover, dems may have several advantages over mdms. thus, verifying dems' potential advantages would be worthwhile. ## investigation focus the multi-domain translation task assumes n domains, with data d i (@xmath0, 2, ..., n}) corresponding to each domain. besides the domain data, general domain data d general is assumed to be available. based on the existing domain adaptation approach @xcite , the model is pre-trained in d general , then fine-tuned in the domain data. at test time, no domain information is provided with the source, although it is an instance of one of the n-domains. following is a description of mdms and dems. ## experiments we performed experiments on english-to-japanese (en-ja) and japanese-to-english (ja-en) translations with five domains. ## results, analyses and discussions figures 2 and 3 show the results for the en-ja and ja-en test data, respectively. 5 comparing mdm and dems for the same model size, dems scored higher for almost all methods. in addition, dems-small (90m × 5) achieved competitive performance to mdm-large (1b) by oracle and mbr. we assessed statistical significance through the paired t-test and bootstrap resampling @xcite . the results indicated that oracle and mbr outperformed compared to mdm settings. these findings suggest that effective combinations of small expert mt models can surpass the performance of a larger mt model. comparison of output selection methods and ensemble for dems: across all model sizes and language directions, the five methods ranked @xmath1. the difference between qe and oracle can be ascribed to the performance gap between ms-comet-qe-22 and ms-comet-22. it seems counter-intuitive that mbr scored higher than qe. considering the superior translation performance of the expert model in the target domain, it is reasonable to expect that qe, which selects a single high-quality output, would surpass mbr, which chooses the sentence most semantically similar to others in y. therefore, improving qe methods is crucial for the success of dems setting. even the expert model that matches the input domain can generate low-quality output. in such cases, mbr and qe can select better output from other models. this may account for the dm results being lower than the mbr and qe. ens scored lower than qe and mbr. this result suggests that in the dems setting, ens is negatively impacted by models with poor performance. hence, weighted ens could improve the performance @xcite . however, determining the appropriate weights typically necessitates training or tuning, and any alteration in the number of domains requires a reconfiguration of these weights. qe and mbr would be more flexible than the ens in multi-domain mt tasks. selected outputs in oracle, qe, and mbr: tables 3, 4, and 5 show which expert model's outputs were selected by oracle, qe, and mbr for each domain of input in en-ja, respectively. for oracle, the corresponding expert model's output for the input domain was consistently the most selected, but this was not the case for mbr and qe. even in oracle, outputs from models other than the expert one corresponding to the input were often selected. this suggests that the dm did not lead to high performance. we assume that this is an effect of including instances whose domain is not far distant from the different domain data. since oracle simply selects good-quality sentences without being aware of the domain, it can select the appropriate model's output flexibly. this appears to explain the significant difference in scores between oracle and dm. effects of multiple models and data separating: as discussed in the previous paragraph, we can assume that there are two contributing factors to the comparing dems and mdms, it is intriguing to note that dems outperformed mdms in all methods except avg, even though dems uses less data for fine-tuning than mdms. in addition, none ## conclusion we investigated the effectiveness of mt models in a multi-domain setting in fair conditions and explored a direction for tackling multi-domain mt. we mainly compared the performance of the two main approaches, mdm and dems, on english to/from japanese tasks with five domains. our results revealed that the domain-based data separation strategy is effective in a multi-domain setting. additionally, we found that when we want to use metrics for selection in a multi-domain setting, the metrics must be multi-domain capable. furthermore, combining five 90m dems based on an output selection approach has the potential to perform better than a single 1b mdm, which may suggest a new direction for developing mt models in the multi-domain setting.