1+1>2: Can Large Language Models Serve as Cross-Lingual Knowledge Aggregators? (2024)

Abstract: Large Language Models (LLMs) have garnered significant attention due to their remarkable ability to process information across various languages. Despite their capabilities, they exhibit inconsistencies in handling identical queries in different languages, presenting challenges for further advancement. This paper introduces a method to enhance the multilingual performance of LLMs by aggregating knowledge from diverse languages. This approach incorporates a low-resource knowledge detector specific to a language, a strategic language selection process, and mechanisms for answer replacement and integration. Our extensive experiments demonstrate notable performance improvements, particularly in reducing the performance disparity across languages. An ablation study confirms that each component of our method significantly contributes to these enhancements. This research highlights the inherent potential of LLMs to harmonize multilingual capabilities and offers valuable insights for further exploration.

## introduction large language models (llms) are increasingly recognized for their impressive capabilities in natural language processing (nlp). employed across a variety of domains such as the medical sector @xcite , data generation @xcite , scientific research @xcite , and llm-based agents @xcite @xcite , llms have demonstrated significant utility. additionally, recent advancements in llms have expanded research @xcite @xcite , which focuses on enhancing their ability to process multiple languages and thereby increasing their accessibility and relevance across diverse linguistic demographics. ## experiments we chose english and chinese for our experiments primarily due to their broad applicability and the availability of resources. firstly, most llms, particularly open-source ones like the chatglm series, perform significantly in english and chinese. this trend highlights the advanced development and optimization of llms for these languages, making them ideal for rigorous testing. secondly, major llm benchmarks and datasets predominantly focus on these two languages. for instance, besides english benchmarks or datasets, benchmarks such as halluqa and alignbench are primarily designed around english and chinese, providing a robust framework for evaluating our methods. lastly, the linguistic features and data availability in english and chinese ensure comprehensive evaluation and validation of our approaches and suggest that our findings could be extrapolated to other languages. this potential for cross-linguistic application supports the broader relevance and utility of our study, choosing english and chinese as both strategic and impactful. ## discussion on other approach as the confidence of the generated content is related to its entropy during the generation process, a natural idea is to calculate the entropy in different languages and compare them to decide which is the best language to answer the question. this approach is widely used for measuring llms' uncertainty and detecting hallucinations @xcite . however, our trial demonstrates that this approach is infeasible and achieves merely randomguess-level performance when selecting the right language for the given queries. to explore how to leverage entropy-related statistics to select the target language, we train a model f that takes the statistics as input and outputs the selection of the language y . the statistics we use for a language l include the entropy of the query e q l , the entropy of the response e r l , the perplexity of the query p q l , and the perplexity of the response p r l . we adopt an mlp as the classification model f : (e q l , e r l , p q l , p r l ) → y and train the model on the low-resource query dataset we construct. we trained based on svm and random forests in llama2-7b's output. the accuracy is no more than 60%. this is a merely random-guess-level performance when taking the entropy-related statistics as input. we attribute this to the hallucination issue of llms, that the model may become overconfident even with the wrong answer @xcite , which indicates llms are not calibrated well now @xcite . ## conclusion this paper presents a method to improve the multilingual capabilities of llms by leveraging knowledge from various languages, which includes a low-resource knowledge detector, a process for selecting languages, and answer replacement & integration. our experiments show significant enhancements in performance, especially in reducing disparities across languages. moreover, each module in our method contributes to the improvement. overall, this study underscores the potential of llms to unify multilingual functions and provide insights for future research.