Do LLM s Speak K azakh? A Pilot Evaluation of Seven Models (2024)

Abstract: We conducted a systematic evaluation of seven large language models (LLMs) on tasks in Kazakh, a Turkic language spoken by approximately 13 million native speakers in Kazakhstan and abroad. We used six datasets corresponding to different tasks – questions answering, causal reasoning, middle school math problems, machine translation, and spelling correction. Three of the datasets were prepared for this study. As expected, the quality of the LLMs on the Kazakh tasks is lower than on the parallel English tasks. GPT-4 shows the best results, followed by Gemini and . In general, LLMs perform better on classification tasks and struggle with generative tasks. Our results provide valuable insights into the applicability of currently available LLMs for Kazakh. We made the data collected for this study publicly available: https://github.com/akylbekmaxutov/LLM-eval-using-Kazakh.

## introduction large language models (llms) increase human productivity and eliminate routine tasks in many areas, making them a powerful economic driver @xcite . at the same time, llms can lead to an inequality between different language communities and a widening gap between developed and developing countries @xcite . creating llms requires huge amounts of text and computation, as well as skilled engineers. most llms are trained for high-resource languages with large populations of speakers, primarily english. training language models for low-resource languages can be technically and economically problematic -there is little training data, and it is unclear whether potential users can amortize the cost of collecting data and training the model. although models trained primarily on english data express capabilities in other languages, their quality in these secondary languages is lower than in english @xcite . recently, thanks to the advent of open llms, their adaptations to less-resourced languages are emerging @xcite . evaluating llms in different languages is crucial in this situation. in this study, we make the first attempt to evaluate the quality of available llms in kazakh. kazakh belongs to the turkic language family and is the official language of the republic of @xcite . estimated 10 million kazakh native speakers live in kazakhstan, and about 3 million more abroad, predominantly in north-western china and western mongolia. the language employs an extended cyrillic alphabet with 42 letters. kazakh is an agglutinative language, meaning that words are formed by adding various suffixes to root words. the language's rich inflectional morphology is reflected in the complex interaction of suffixes for number, possession, and case. for instance, the plural form, possessive affixes, and various case endings are layered sequentially onto noun roots. kazakh has eight types of possessive agreements, adding complexity to its morphological structure. kazakh verbs exhibit similar tenses and moods as turkish ones but include unique tenses such as the goal-oriented future tense. kazakh consonant and vowel harmony rules significantly affect its morphological structure. consonant harmony determines the form of suffixes based on the voicing of the final consonant of the root word, while vowel harmony aligns suf-fix vowels with the vowel type (front or back) of the root. kazakh is considered a mid-resourced language @xcite . we experimented with seven models in totalfive closed (gpt 3.5 and 4, gemini 1.5 pro, yan-dexgpt 2 and 3) and two open (llama 2 and aya) ones. 1 we focused on automatic benchmarkbased evaluation, while trying to make the set of tasks diverse. we used a collection of six datasets sourced in different ways: 1) existing multilingual benchmarks that include kazakh data (machine translation and multiple-choice question answering), 2) the recently published monolingual question answering dataset kazqad (both openand closed-book scenarios), 3) machine-translated copa dataset 2 (commonsense causal reasoning), 4) original math school problems in kazakh that we scraped online and post-processed, and 5) a kazakh spelling correction dataset that we created from scratch within this study. based on our experiments, we can conclude that the gpt-4 is the most capable of all the models in the experiment. gemini is the runner-up in the classification tasks. aya is quite competitive, especially if we take into account its relatively small size and a long list of supported languages. all models show a lower quality in the generative tasks. as expected, the quality on kazakh tasks is significantly lower than on english tasks, as we can see on parallel multilingual datasets (multiple-choice question answering, causal reasoning). specialized models may still provide better quality for downstream tasks, such as machine translation or classification tasks. we cannot confirm previous findings that english prompts systematically improve llm quality on non-english tasks: our results are mixed across tasks and models. our findings provide valuable insights into the applicability of currently available llms for kazakh. we also anticipate that the study will contribute to the methodology of evaluating llms and improving the quality of llms in mid-and low-resource languages. the methods introduced 1 mgpt @xcite is another llm that officially supports kazakh. however, only a pre-trained mgpt is available, while the models in the study are instruction tuned. in our work can be used to experiment with other languages and llms. we made the data and evaluation code publicly available. foot_0 2 related work as has been shown by @xcite , multilingual abilities of language models emerge when they are exposed even to a tiny fraction of non-english data in a large pre-training corpus. earlier studies demonstrated that multilingual models learn high-level abstractions common to all languages, which make cross-lingual transfer possible even when languages share no vocabulary @xcite . open llms such as llama @xcite and qwen @xcite can be adapted to other languages by expanding their vocabularies, continual pre-training and subsequent aligning on the data in target language @xcite . another approach is to train a model from scratch: for example, jais model was trained on a mixture of english and arabic data in ratio 2:1 @xcite . despite the development of non-english and multilingual models, many languages remain underrepresented in the modern llm landscape. this situation is partly due to objective reasons (lack of training data), but also to inequalities in economic and technological development. llm evaluation is a complex and multifaceted problem @xcite . llms are truly multitasking, and users can leverage them to solve non-standard and creative problems, for example, brainstorming ideas or generating jokes. for generative tasks, the variety of formulations can be very large, making it difficult to automatically compare the answer to a "gold standard." with the proliferation of llms and their active use, evaluation of models becomes relevant not only at the task level, but also from their safety and security perspectives. the main approach to automatic llm evaluation is based on ensembles of annotated benchmarks covering a wide range of usage scenarios @xcite . popular benchmarks include mmlu (massive multitask language understanding) @xcite that measures llm's knowledge across 57 subjects and gsm8k (grade school math) @xcite , aimed at evaluating multi-step math reasoning. mmlu contains multiple-choice ques-tions, while gsm8k accepts numerical answers. there are multilingual adaptions of these datasets: @xcite employed chatgpt to translate the original mmlu dataset in multiple languages; mgsm dataset contains 250 problems from the gsm8k manually translated into 10 typologically diverse languages @xcite . studies that evaluate llms on non-english tasks are fewer that those targeting english and vary in their scope @xcite . some focus on multilingual datasets @xcite , while others concentrate on a specific language, e.g. arabic @xcite or russian @xcite . our study belongs to the latter type. llms, as expected, are better in solving problems formulated in english than in other languages. moreover, fine-tuned models such as xlm-r @xcite in general outperform llms on specific tasks. the quality on non-english tasks can be improved by preceding actual task formulation with english prompts, or by explicitly stating in the prompt that the task must first be translated into english @xcite . the multilingual abilities of llms also depend on the task type. it can be concluded that llms are better at "understanding" a language other than english than at generating a non-english answer @xcite . thus, models do better in multilingual classification, reasoning, and multiple-choice question answering and struggle with generative tasks. based on experiments with llama 2, @xcite hypothesize that the model first solves the task using english as a pivotal language, then generates the answer in the target language. this process can be seen as an implicit translate-test approach. these observations are partially confirmed by our experiments. recently, several annotated kazakh datasets @xcite and multilingual datasets including kazakh @xcite have been published. however, we are not aware of any studies that have systematically evaluated the quality of existing llms in kazakh. ## data the data used in our experiments is summarized in belebele is a massively multilingual machine reading comprehension dataset that spans 122 languages, including kazakh @xcite . belebele contains 900 multiple-choice questions, each associated with one of 488 distinct passages originating from the flores-200 dataset @xcite . first, the english multiple choice questions and answers were manually created using english passages from the flores dataset. later, questions and answers were translated in other languages and aligned with corresponding passages from flores-200. each question has four answer options, one of which is correct. so, a random guessing would result in accuracy of 0.25. all 900 belebele questions are intended exclusively for testing, there is no training supplement to the dataset. authors report performance of gpt-3.5turbo and llama2-chat 70b in zero-shot fashion on kazakh/english belebele subsets: 35.0/87.7 and 32.4/78.8 accuracy points, respectively. dataset task size metric language class. belebele (bandarkar et al., 2023) multiple-choice qa 900 accuracy human-translated kkcopa * causal reasoning 500 accuracy machine-translated nis math * school math 100 accuracy orig. in kazakh kazqad § (yeshpanov et al., 2024) reading comprehension 1,000 token-level f1 orig. in kazakh gen. kkwikispell * spelling correction 160 token-level jaccard orig. in kazakh kazqad § (yeshpanov et al., 2024) generative qa 1,927 token-level recall orig. in kazakh flores-101 (goyal et al., 2022) machine translation 500 bleu human-translated * datasets prepared within this study. § kazqad data was used both in open-and closed-book scenarios. ## models in our work, we evaluated seven models. since five of the seven models are closed, many of their aspects such as the number of parameters or the data on which they were trained are unknown. gpt 3.5 and 4 are two generations of llms from openai. kazakh is included in the official list of languages that gpts work with. 5 we access the models through their official apis. we use gpt-3.5-turbo-0125 and gpt-4-0125-preview versions in our study. gemini 1.5 pro is the latest publicly available llm from google. kazakh is not on the list of languages officially supported by gemini. 6 this is probably the reason why gemini returns empty results or error messages for a significant share of requests, see details in section 5. we accessed gemini-1.5-pro-preview-0409 model through google cloud's vertex ai studio. llama is a collection of open llms of different sizes. they have been pre-trained on 2t tokens, of which an estimated ∼90% are english. due to limited computational resources, we use an 8bit quantized version of llama 2-chat 7b, an aligned model for dialogue use cases. although the model was mainly trained on english data, it has some multilingual capabilities, as shown by numerous experiments. yandexgpt 2 and 3. few technical details about yandex' language models are disclosed, but the company's blog posts provide results of evaluating models on proprietary benchmarks and comparing yandexgpts side-by-side with chatgpt and llama 2 on tasks in russian. we could not find an official list of supported languages, but our model xmmlu gsm8k release date |v| t/w gpt-3.5-turbo foot_6 c 70.0 † 57.1 11.2022 100k 4 5.80 gpt-4-turbo (achiam et al., 2023) c 86.4 † 92.0 03.2023 llama 2 (touvron et al., 2023) o 45.3 † 56.8 02.2023 32k 4.78 gemini 1.5 pro (reid et al., 2024) c 81.9 † 91.7 02.2024 256k 3.63 aya (üstün et al., 2024) o 37.3 § -02.2024 250k 2.66 yandexgpt 2 foot_7 c 55.0 * -09.2023 ? 3.83 yandexgpt 3 foot_8 c 63.0 * -03.2024 aya is a massively multilingual model based on the 13b mt5-xxl model @xcite that supports 101 languages, including kazakh. the main challenge of the aya project was to prepare a large instruction dataset to cover all supported languages @xcite . we hosted the aya model foot_11 on a cloud gpu. ## conclusion our results provide valuable insights into the applicability of currently available llms for kazakh. gpt-4 shows the best results, followed by gemini and aya. gemini's results are promising, although the proportion of empty answers is quite high. aya is very competitive compared to its supposedly larger closed counterparts. as expected, the quality of the llms on the kazakh tasks is lower than on the parallel english tasks. in general, llms perform better on classification tasks and struggle with generative tasks. english instructions can improve results on some tasks/models. our evaluation showed that there is a steady progress in llms for kazakh (gpt-3.5 vs. gpt-4). we expect the support of kazakh by gemini and yandexgpt to be strengthened, as well as the appearance of a kazakh adaptation of an open llm. we made the datasets prepared for the study and the collected llm responses publicly available. these resources can form the basis for an llm benchmark focused on the kazakh language. in our future work, we plan to expand the list of llms and the set of benchmarks.