Information Parity: Measuring and Predicting the Multilingual Capabilities of Language Models (2024)

Abstract: Large Language Models (LLMs) are increasingly deployed in user-facing applications worldwide, necessitating handling multiple languages across various tasks. We propose a metric called Information Parity (IP) that can predict an LLM’s capabilities across multiple languages in a task-agnostic manner. IP is well-motivated from an information theoretic perspective: it is associated with the LLM’s efficiency of compressing the text in a given language compared to a reference language. We evaluate IP and other popular metrics such as Tokenization Parity (TP) and Tokenizer Fertility (TF) on several variants of open-sourced LLMs (Llama2, Gemma, Mistral). Among all metrics known to us, IP is better correlated with existing task-specific benchmark scores from the literature and thus better predicts such scores in a certain language. These findings show that IP may be useful for ranking multilingual LLMs’ capabilities regardless of the downstream task.

## introduction llms comprehend and generate human language across various domains and tasks, powering applications like virtual assistants and machine translation. as llms become more widely used globally, it is necessary to assess their capabilities in processing and understanding a specific language. ## information parity: theoretical background and definition for a given text w 1:@xmath0, .., w n ) where w i is the i-th token, denote its negative log-likelihood under a language model (lm) by information parity: suppose we have english text w e and its translation to another language w l . we define the ip of w under the lm as in words, ip is the ratio between the information content of the text in english and the information content of the translated text in another language. it aims to measure how efficiently the llm represents information provided by a text in the language l compared to the same information provided in english. a higher ip indicates a higher representation efficiency hence a closer alignment with the ideal language-agnostic compressor. ## limitations instruction tuning: the ideology behind ip is that good multilingual llms act as efficient languageagnostic compressors. namely, they efficiently encode information in text, regardless of the text's language. however, we evaluated instruction-tuned llms via reinforcement learning with human feedback (rlhf) process @xcite and dpo @xcite . this tuning may affect their compression efficiency. additionally, ip does not account for the ability of llms to follow instructions in different languages, which may be relevant for some applications or tasks. dataset contamination: ip relies on parallel corpora that contain the same information in different languages. however, some of these corpora may have been used in pre-training which may inflate their compression performance. machine translation artifacts: some of the task-specific metrics like mmlu, arc, and hel-laswag were machine translated by gpt3.5 in @xcite . this translation may add biases and artificats to data that might alter performance measurements on these benchmarks. ## conclusions we introduced the information parity (ip) metric to provide a task-agnostic evaluation of the multilingual capabilities of llms. ip is easy to evaluate and has a natural information-theoretic interpretation as the efficiency of an llm in representing the same information across different languages. evaluations with publicly available llms reveal strong correlations between ip and a diverse set of downstream tasks, particularly those involving natural language understanding and commonsense reasoning. these properties suggest that ip could enable researchers and practitioners to assess model performance even for low-resource languages, leading to a more comprehensive understanding of llm behavior across all languages.