On the Economics of Multilingual Few-shot Learning: Modeling the Cost-Performance Trade-offs of Machine Translated and Manual Data (2022)

Abstract: Borrowing ideas from Production functions in micro-economics, in this paper we introduce a framework to systematically evaluate the performance and cost trade-offs between machine-translated and manually-created labelled data for task-specific fine-tuning of massively multilingual language models. We illustrate the effectiveness of our framework through a case-study on the TyDIQA-GoldP dataset. One of the interesting conclusion of the study is that if the cost of machine translation is greater than zero, the optimal performance at least cost is always achieved with at least some or only manually-created data. To our knowledge, this is the first attempt towards extending the concept of production functions to study data collection strategies for training multilingual models, and can serve as a valuable tool for other similar cost vs data trade-offs in NLP.

## introduction transformer based massively multilingual language models (mmlms) such as mbert @xcite , xlm-roberta @xcite and mt5 @xcite are surprisingly effective at zero-shot cross-lingual transfer @xcite . however, while zero-shot transfer is effective, often the performances across different languages is not consistent. low-resource languages @xcite and the languages that are typologically distant from the pivot language @xcite are known to benefit the least from zero-shot transfer, which can often be mitigated by using targetlanguage specific labelled data for the task during fine-tuning. one common approach for collecting such data in the target language is to translate the training data available for the pivot-language to the target by using an off-the-shelf machine translation (mt) system. this is usually referred to as the translatetrain setup @xcite . few-shot transfer is another alternative; as shown by @xcite , a few labelled examples in the target language, that can be obtained cheaply, can lead to substantial improvements over the zeroshot performance. however, there has not been much work on comparing the performance across these two strategies. in one such study, @xcite compare the performance of translate-train with few-shot transfer on tydiqa-goldp @xcite dataset, but they only evaluate the few-shot case with 1000 examples, which does not provide any insight into how the performance varies with increasing dataset sizes for these two approaches. additionally, there are trade-offs related to the data acquisition costs as well. the cost per training instance is expected to be much smaller for an mt-based approach than manual translation or labeling of examples. however, depending on the nature of task, language, and quality of the mt output, the amount of data required to achieve the same performance through these two approaches can be drastically different. more importantly, fine-tuning the mmlms with a combination of the data from the two strategies could be the cheapest alternative for achieving a target accuracy, which, to the best of our knowledge, has not been explored yet. inspired by the above observations and gaps, in this paper, we ask the following question: given a pre-determined budget to fine-tune a multilingual model on a task for which some data is available in a pivot language, what is the best achievable accuracy on a target language by (a) training the model on the pivot-language data, (b) different amounts of machine-translated and (c) manually-collected data in the target language. solving this requires an understanding of the exact nature of the performance and cost trade-offs between the two kinds of target language datasets and their relative costs of acquisition, apart from factors such as the amount of pivot language data, the task, the mmlm, and the languages concerned. this problem of modeling and measuring the trade-offs between different input factors and their costs is well-studied in the field of microeconomics. a sophisticated machinery has been developed in the form of production functions and allied analytical methods @xcite , in order to solve the following generic problem: with the best available technology, how are the inputs to a production process (eg. labor and capital) related to its output, that is the quantity of goods produced. in this paper, we adapt this framework to address the aforementioned question of mmlm fine-tuning trade-offs. the key contributions of our work are threefold. 1. we extend the idea of production functions to performance functions that model the relationship between input data sizes and performance of a system; we propose a possible analytical form for this function and derive the performance trends and optimal data collection strategies under fixed costs. 2. we illustrate the usefulness of this framework through a case study on a q&a task -tydiqa-goldp @xcite and systematically study the various trade-offs for 8 languages. 3. our study provides several important insights such as (a) if the cost of mt data creation is non-zero, then the optimal performance under a fixed budget is always achieved with either only manually-created data or a combination of the two; (b) the ratio of the two datasets for the least cost combination usually remains constant at different levels of performance. to the best of our knowledge, this is the first work that applies the idea of production functions to analyze the cost-performance trade-offs of mmlm fine-tuning. the proposed framework can be extended to a multitude of nlp problems where the trade-offs similar to the ones discussed above, are common (e.g., pre-training vs. fine-tuning data). to encourage reproducible research, we have made our code, the performance data, and a detailed list of the results publicly available 1 . ## theoretical foundations one of the foundational pillars of neoclassical economics is the idea of production functions. simply put, a production function is a mathematical formalization of the relationship between the output of a firm (industry, economy) and the inputs that have been used in obtaining it (khatskevich and 1 github.com/kabirahuja2431/performancefunctionanalysis @xcite . a multifactor production function is defined as a map ## case-study on tydiqa-goldp in order to understand the efficacy of the proposed framework, we conduct a case-study on a popular multilingual question answering task (cf. t) using tydiqa-goldp @xcite dataset and consider mbert as the mmlm m. in the following subsections, we provide the details of the task and training setup for generating the performance π for different combination of the input factors, the procedure for estimating the parameters of the performance functions, and the findings. ## discussion and conclusion in this work, we have proposed a micro-economics inspired framework to study the performance and cost trade-offs between manually annotated and machine-translated data for training multilingual models, and demonstrated its efficacy through a case-study on the tydiqa-goldp dataset. the key findings from this case-study are: 1. some amount of manually collected data in a target language is crucial to attain optimal performance at minimum cost irrespective of how much cheaply mt data can be procured, as long as the cost is non-zero. 2. the ratio of manually collected and machine-translated data at least cost operating point remains nearly uniform at the different levels of performance 3. the usefulness of translated data is higher when the amount of pivot language data is less. there are several other insights that can be drawn from the t-m diagrams and other plots, which could not be presented here due to the paucity of space. this work can be expanded in several ways. in the current work we considered a single-pivot and single-target case. generalizing this to the case where the model is allowed to be trained on multiple pivot languages and then be evaluated on multiple targets is of considerable interest. this implies extension to multiple-output production functions with multiple (> 2) input factors. here, we have not considered the effect of multiple technology on the isoperfs. for our problem, multiple technologies may correspond to the different mmlms such as mbert, xlmr and mt5, different mt systems, and even different training curricula. identifying the optimal allocation policy considering the presence of such multiple technological alternatives would be an interesting exercise. in particular, it will be interesting to explore the impact of translation quality on the trade-offs. an important limitation of the current framework is that it presumes availability of certain amounts of m and t datasets such that the performance function can be estimated. however, in practice, one would like to understand the trade-offs before collecting the data. recently, @xcite showed that it is possible to predict the zero-shot and few-shot performance of mmlms for different languages using linguistic properties and their representation in the pre-training corpus. understanding if there exists a similar dependence of the performance trade-offs with the linguistic properties of different languages can help us generalize our framework to the new languages without the need for explicit data collection. finally, we believe that performance functionbased analysis can be applied to a multitude of three-way trade-offs among technology, cost and data that are commonly encountered in the nlp world. the economics of language data can be a new direction of study with important practical and theoretical applications.