Cost-Performance Optimization for Processing Low-Resource Language Tasks Using Commercial LLM s (2024)

Abstract: Large Language Models (LLMs) exhibit impressive zero/few-shot inference and generation quality for high-resource languages (HRLs). A few of them have been trained on low-resource languages (LRLs) and give decent performance. Owing to the prohibitive costs of training LLMs, they are usually used as a network service, with the client charged by the count of input and output tokens. The number of tokens strongly depends on the script and language, as well as the LLM’s subword vocabulary. We show that LRLs are at a pricing disadvantage, because the well-known LLMs produce more tokens for LRLs than HRLs. This is because most currently popular LLMs are optimized for HRL vocabularies. Our objective is to level the playing field: reduce the cost of processing LRLs in contemporary LLMs while ensuring that predictive and generative qualities are not compromised. As means to reduce the number of tokens processed by the LLM, we consider code-mixing, translation, and transliteration of LRLs to HRLs. We perform an extensive study using the IndicXTREME classification and six generative tasks dataset, covering 15 Indic and 3 other languages, while using GPT-4 (one of the costliest LLM services released so far) as a commercial LLM. We observe and analyze interesting patterns involving token count, cost, and quality across a multitude of languages and tasks. We show that choosing the best policy to interact with the LLM can reduce cost by ~90% while giving better or comparable performance, compared to communicating with the LLM in the original LRL.

## introduction large language models (llms) like @xcite , chatgpt, llama-2 @xcite , palm @xcite , inter alia, are greatly contributing to the advancement of nlp with their exceptional zero/few-shot inference and generation abilities. llms can also reduce dependence on expensive human-generated gold data for finetuning models for various downstream tasks, particularly for lrls where gold data is scarce. however, our experience suggests that this benefit is offset by one problem. commercial llm services like gpt-4 charge by the number of tokens exchanged with the client. the typical message from the client to the llm consists of a task description or instruction, followed by zero or more few-shot "in-context examples", and the payload instance to be solved. the output tokens from the llm express the solution to the payload instance. the number of tokens exchanged between the client and llm service depends on the (subword) vocabulary of the llm, and the language(s) that the client uses. if the client's language is a lrl, chances are, the llm will heavily segment the lrl tokens into subwords, because lrl subwords have minority status in most popular llms today @xcite . consequently, use of the llm will be expensive compared to hrl clients. despite giant leaps in reasoning and instructionfollowing capabilities, gpt-4 effectively discriminates against lrl clients in the same manner, and for the same reason, as its predecessor multilingual models mbert and xlm-r @xcite . this results in lrl clients being disadvantaged in terms of pricing, if not also quality. our goal is to reduce this inequity. our choice of gpt-4 is driven by the fact that it is one of the costliest (yet most popular) commercial blackbox llm-as-a-service. the cost of using the gpt-4 8k context model api is @xmath00.06 per 1,000 output tokens. for the 32k context model, the cost is as high as @xmath10.12 per 1,000 output tokens. thus, a task of summarizing the wikipedia (∼6 billion tokens) to half its size would cost @xmath2360,000 with gpt-4 having context length of 32k and 8k respectively. our initial study shows that gpt-4 generates diverse numbers of tokens for translations of a source sentence in different indian languages. we also get a profile of default lrl task performances in these languages, if directly communicated to the llm. the tool at our disposal is to preprocess the lrl messages from client to llm service, in the face of a black-box service that we cannot influence. specifically, we try to decrease the api cost by reducing the number of tokens exchanged between the client and llm, while trying to maintain task quality. this can be done in various ways, as shown in figure 1 and detailed later. to compare these approaches, we define (in equation 1) a metric called the relative performance to cost ratio (rtpcr), which specifies the task performance we can get, given the tokendriven cost we pay using various preprocessing methods m , relative to using the original lrl text. we desist from directly comparing the cost of llm service access against operating costs of an in-house llm, because the latter depends on too many hard-to-model factors like power and cooling costs (and comparing against the llm service). explicit instance translation using llm before inference will accrue extra cost, compared to freeof-charge translation using an open-source machine translation tool. an alternative is implicit translation, in which we instruct the llm to translate the payload instance to english in the background before solving it, thus saving explicit translation cost. (we do not ask the llm for the translated version.) we observe that the improvement using implicit translation trails behind ex-plicit translation. therefore, using a dedicated lrl-to-hrl translator outside the llm service can save cost and improve lrl task quality. a problem may arise if the lrl does not have a high-quality, open-sourced translator available and falling back on the llm for translation will increase costs and decrease rtpcr. we next strike out in a third direction: can mixing lrl and hrl words help? to reduce cost, we can use a lrl-hrl or lrl-lrl dictionary, because such translations are often context-independent. lastly, as a natural companion to selective wordmixing, we experiment with a transliteration technique, in which we transliterate the whole lrl message into latin script. after all, the giant corpora used to train llms are more likely to contain lrl words heavily transliterated into latin script, wordmixed with native lrl and hrl words, than 'pure' lrls. we summarize our contributions. (1) we identify a key cost-performance disadvantage faced by llm clients that wish to solve lrl problems. (2) we demonstrate that this disadvantage is present in gpt-4 with respect to 15 indian languages for diverse classification and generation tasks, using a new measure we introduce, called rtpcr. (3) we experiment with implicit and explicit client message translation and compare their rtpcr scores, to show that a free translator can reduce cost and improve performance. (4) we experiment with wordmixing and transliteration and show that both help reduce cost, but transliteration may result in modest performance impacts. (5) apart from aggregative performance, we also present many case studies with interesting findings: the number of token counts varies greatly across indian languages; translation to english may both increase and decrease the number of english words, depending on the source lan-guage; and llm performance is rather sensitive to syntax. ## related work recently llms like chatgpt, @xcite , llama-2 @xcite , palm @xcite , bloom @xcite , etc, have shown incredible zeroshot and few-shot capabilities across language and tasks. although their performance in highresource languages (hrls) is impressive, studies @xcite find that the same level of performance is not seen for low-resource languages (lrls). various benchmark datasets like xtreme @xcite , xtreme-r @xcite , and xglue @xcite have been designed to gauge the cross-lingual capabilities of a multilingual llm. benchmarks are also available in lrls, e.g., indicxtreme @xcite for indian languages. benchmarks are also available in other language families like african @xcite and indonesian @xcite . in this work, we work with indicxtreme. mega @xcite witnesses improved task performance when translating the test instance to english. having said that, they experiment from the performance perspective using the bing translator (not an open-sourced tool). unlike them, our focus is not only on performance but also on the cost. as inference using gpt-4 is costly for lrls, we experiment with different techniques to reduce cost without hurting the performance. similarly, a recent study @xcite finds that a prompt written in english performs better than one written in other low-resource languages. there are works like @xcite and @xcite which explore the over-fragmentation of tokenizer in different languages, and later one proposes a metric called parity premium to quantify the tokenization disparity between two languages. we compared the parity premium metric with our rtpcr and found latter more robust. frugal-gpt @xcite tries to address the cost aspect of gpt-4 on a different level, like selecting the fewer but more effective in-context examples for few-shot inference, caching previous queries and responses for future use, or llm-cascading where cheap llms are tried first to check if the response is reasonable and only go for expensive ones if previous responses are not satisfactory. they have experimented with english, where the words are not fragmented much. lrls present a steeper challenge, where first we need to reduce the cost by controlling over-fragmentation of lrl words. frugelgpt can be applied thereafter to further reduce the cost. as per our knowledge, we are first to study the effect on performance when prepossessing the lrl input instance in a variety of different ways and relating it to the llm api cost. ## methodology in this work, we primarily preprocess the original lrl input instances using the techniques below to reduce the number of tokens generated. ## conclusion we study the cost of solving five classification and six generation task datasets across various indian languages using gpt-4, the leading commercial llm. while gpt-4 does well on these tasks, lrl words get highly fragmented, leading to high cost of api calls. we wish to retain the performance advantage of gpt-4 while reducing api cost. to that end, we try different pre-processing techniques involving translation, wordmixing, and transliteration. we find that translating the whole lrl sentence to english reduces the cost and improves performance (given a free @xmath3. in the absence of such an mt system, implicit translation using gpt-4 or even an lrl-hrl dictionary can be used to replace highly fragmented words and reduce costs. we introduce the rtpcr metric that considers both the performance and cost aspects and gives a holistic picture of the gain achieved among the two transformation methods. in future, we wish to explore wordmix more deeply while using other hrls related to the source lrl.