L ang B ridge: Multilingual Reasoning Without Multilingual Supervision (2024)

Abstract: We introduce LangBridge, a \textit{zero-shot} approach to adapt language models for multilingual reasoning tasks without multilingual supervision. LangBridge operates by bridging two models, each specialized in different aspects: (1) one specialized in understanding multiple languages (e.g., mT5 encoder) and (2) one specialized in reasoning (e.g., MetaMath). LangBridge connects the two models by introducing minimal trainable parameters between them. Despite utilizing only English data for training, LangBridge considerably enhances the performance of language models on low-resource languages across mathematical reasoning, code completion, logical reasoning, and commonsense reasoning. Our analysis suggests that the efficacy of LangBridge stems from the language-agnostic characteristics of multilingual representations. We publicly release our code and models.

## introduction language models (lms) are known to exhibit inferior performance in solving reasoning tasks such as math or coding in low-resource languages @xcite . this tendency primarily stems from the fact that lms are predominantly trained on corpora comprised of a few highresource languages @xcite . this results in low-resource languages being represented as long-tail knowledge @xcite . prior works have mainly approached this problem by adapting english-centric lms to other languages through continual training on the target language @xcite @xcite . however, scaling this approach to a large number of languages is challenging, as it requires targeted training corpora for each language. this issue is particularly pronounced for lms such as metamath @xcite and orca 2 @xcite , which have undergone continuous domain-specific adaptation from llama 2 @xcite . these specialized, domain-specific datasets are typically in english, complicating multilingual support for the underlying lm. in this paper, we introduce langbridge, a novel approach that adapts lms to solve multilingual reasoning tasks without explicitly training on multilingual data. inspired from the multimodal literature that integrates two independently pretrained modalities @xcite @xcite @xcite , we leverage the encoder from mt5 @xcite and introduce a small number of trainable parameters between the encoder and the target lm. most importantly, our approach does not require multilingual supervision and solely relies on english data while generalizing to multiple languages during test time, resembling zero-shot cross-lingual transfer @xcite @xcite . we demonstrate the effectiveness of lang-bridge by applying our method to lms specialized in diverse reasoning tasks of mathematical reasoning, code completion, logical reasoning. our empirical results show langbridge substantially enhances the multilingual reasoning performance of lms. for example, langbridge applied to metamath-13b leveraging mt5-xl encoder (2.2b) boosts the average accuracy on mgsm @xcite from 40.5% to 53.5%, matching the performance of palm-540b @xcite , which stands at 51.3%. we observe langbridge also significantly boosts lm performance on reasoning datasets that require intrinsic linguistic understanding such as specific subtasks of big-bench hard @xcite and xcopa @xcite . we hypothesize that the effectiveness of lang-bridge is anchored in the language-agnostic characteristics of multilingual representations @xcite . by mapping these representations to the target lm's input space, we conjecture that the lm is able to grasp the semantics of these representations. since these representations are language-neutral, understanding them allows the lm to become less dependent on the specific language of the input, thereby enabling it to tackle tasks in languages it rarely encountered during pretraining. our empirical analysis of langbridge, using principal component analysis (pca) and qualitative methods, supports this hypothesis. ## analysis of langbridge based on the empirical evidence presented in the previous sections, we assert that langbridge effectively enhances lms' capability to address multilingual tasks without multilingual training, especially for low-resource languages. this section presents two fundamental observations that further reinforce the hypothesis outlined in section 3. ## conclusion in this paper, we propose langbridge, a simple yet effective method of extending the capabilities of lms to solve multilingual reasoning tasks without using multilingual supervision. we show that langbridge is surprisingly effective for enhancing multilingual reasoning capabilities for low-resource languages. additionally, we provide analysis that indicates the effectiveness of lang-bridge is due to the language-agnostic nature of the multilingual representations. we hope our findings benefit the low-resource language users and spur further research advancing the development of lms inclusive of the entire global community.