R oman L ens: The Role Of Latent R omanization In Multilinguality In LLM s (2025)

Abstract: Large Language Models (LLMs) exhibit strong multilingual performance despite being predominantly trained on English-centric corpora. This raises a fundamental question: How do LLMs achieve such multilingual capabilities? Focusing on languages written in non-Roman scripts, we investigate the role of Romanization—the representation of non-Roman scripts using Roman characters—as a potential bridge in multilingual processing. Using mechanistic interpretability techniques, we analyze next-token generation and find that intermediate layers frequently represent target words in Romanized form before transitioning to native script, a phenomenon we term Latent Romanization. Further, through activation patching experiments, we demonstrate that LLMs encode semantic concepts similarly across native and Romanized scripts, suggesting a shared underlying representation. Additionally, for translation into non-Roman script languages, our findings reveal that when the target language is in Romanized form, its representations emerge earlier in the model’s layers compared to native script. These insights contribute to a deeper understanding of multilingual representation in LLMs and highlight the implicit role of Romanization in facilitating language transfer.

## introduction the majority of modern large language models (llms) @xcite @xcite are trained predominantly on english-dominated corpora. nonetheless, they exhibit strong multilingual performance across diverse languages @xcite @xcite . this raises a fundamental question: how do llms develop such robust multilingual capabilities despite their english-centric training? to address this, prior work by @xcite suggests that llms encode multilingual information within a shared, language-agnostic latent space, albeit with an inherent bias toward english due to training data composition and architectural choices. building on this perspective, we investigate a complementary mechanism that may underlie multilingual processing, particularly for languages written in non-roman scripts. we hypothesize that llms leverage romanized forms of non-roman script languages as an intermediate bridge between their language-agnostic concept space and language-specific output representations. romanization-the representation of non-roman scripts using roman characters-may facilitate this process by aligning non-english languages more closely with english. supporting this, @xcite demonstrated that explicitly romanizing inputs improves model performance on multilingual tasks, suggesting an inherent alignment between romanized text and english representations. we investigate whether llms indeed use romanization as a bridge between languageagnostic concepts and language-specific outputs given its potential implications for understanding multilingual processing in llms. our primary experiment visualizes next-token generation using the logit lens @xcite , applying the language modeling head to intermediate layers. as illustrated in figure 1 , we prompt the llama-2 7b @xcite model with "francais: porte -ú :" to translate "door" from french to hindi. our results show that in the middle-to-top layers (layers 20-29), romanized hindi subwords intermittently appear before transitioning to native script, suggesting an internal representation of romanized text as an intermediary. ## related work recent studies have explored various aspects of llms' multilingual behavior: examining whether english emerges as a latent language in englishcentric llms @xcite , how the composition of training corpus mixtures influences latent representations @xcite and how llms handle multilingual capabilities @xcite . @xcite describe distinct phases in multilingual information processing: initial layers map language-specific lexical and syntactic representations to a language-independent semantic space, middle layers maintain this semantic abstraction, and final layers transform these representations into language-specific lexical and syntactic forms. interpretability tools relevant to this work include logit lens @xcite , tuned lens @xcite and direct logit attribution @xcite which are key tools for decoding intermediate token representations in transformer models. the logit lens applies the language modeling head to earlier layers without additional training, while the tuned lens in addition to this trains an affine mapping to align intermediate states with final token predictions. direct logit attribution attributes logits to individual attention heads. this work focuses on the logit lens (section 4.1) to investigate whether english-centric decoder only llms when prompted in a non-roman script language, processes via romanized latent states before producing native language text. tuned lens is avoided as its training process might obscure the intermediate romanized states by aligning them to final native script outputs, potentially masking the phenomenon under investigation. activation patching @xcite is a key interpretability technique employed in our study. this technique has been used to draw causal interpretations of llms representations @xcite @xcite @xcite . building on these approaches, we adopt an activation patching-based experimental framework to investigate and compare how concepts are encoded in romanized versus native scripts. previous studies have demonstrated that romanization can serve as an effective approach to interact with llms @xcite . @xcite and @xcite employ an approach based on contrastive learning for post-training alignment, contrasting sentences with their transliterations in roman script to overcome the script barrier and enhance cross-lingual transfer. however, our work distinguishes itself from prior research by exploring the presence of roman-ized representations in the latent layers of an llm during multilingual tasks, an aspect that, to the best of our knowledge, has not yet been investigated. ## background we give a quick background of the transformer's forward pass, romanization and the basics of mechanistic interpretability approaches such as logit lens and activation patching which we leverage in this paper. transformer's forward pass decoder-only transformer models (vaswani, 2017) employ a residual architecture to process input sequences through multiple layers, producing a sequence of hidden states (latents). these latents, whose dimensionality remains the same, are updated iteratively across layers through transformer blocks f j , where @xmath0, k] indicates the layer index and k is the final layer index. for next-token prediction, the final latent h (k) i is transformed by an unembedding matrix @xmath1.f. appendix a). ## methodology we design our analysis setup with the intention of addressing the following research questions: rq1: do llms exhibit latent romanization during multilingual text completion tasks? (section 4.1) rq2: how does the representation of semantic concepts in llms compare between native and romanized scripts of non-roman script languages? (section 4.2) rq3: what are the differences in hidden layer representations when processing the same language in romanized and native scripts? (section 4.3) prompt design. we design prompts that facilitate next-token (x n+1 ) prediction from the given context (x 1 , . . . , x n ). this is adopted across all analysis setups. the prompts are designed around translation, repetition, and cloze tasks, as described below. translation task. we prompt the model to translate a word given five in-context examples. repetition task. we prompt the model to repeat a word in the same language given five in-context examples. cloze task. we prompt the model to predict the masked word in a sentence given two in-context examples. these tasks cover a range of multilingual text completion setups. among these, the repetition task is more syntactic in nature compared to translation and cloze tasks. appendix b provides a hindi example of prompts across each of the three tasks, along with their english translations and romanized forms. ## experimental settings languages: we focus on five indic languages: hindi, gujarati, tamil, telugu, and malayalam, as well as chinese and georgian. among these, hindi and gujarati belong to the indo-aryan branch of the indo-european language family and use scripts derived from the devanagari and gujarati scripts, respectively. tamil, telugu, and malayalam, on the other hand, are part of the dravidian language family and each has its own distinct script. chinese belongs to the sino-tibetan language family and is written using logographic characters. georgian is part of the kartvelian language family and uses the unique georgian script. to examine the generality of latent romanization, we perform qualitative analyses on five additional languages that use different writing systems: greek, ukrainian, amharic, hebrew, and arabic (c.f appendix h). language models: in this study, we focus mainly on gemma-2 9b, gemma-2 9b instructiontuned @xcite , llama-2 7b, llama-2 13b @xcite and mistral-7b (jiang et al., 2023) (c.f. appendix k) language models, some of the best performing open weights englishcentric llms. although the training data for these models are primarily english, these models have high multilingual capabilities @xcite @xcite . romanization: we have used the indicxlit scheme @xcite (c.f. appendix f) for indic languages, pypinyin @xcite for chinese and unidecode @xcite for georgian to romanize native scripts. ## conclusion our findings show that llms implicitly use romanization as a bridge for non-roman scripts, exhibiting latent romanization in intermediate layers before switching to native scripts. layerwise analyses reveal that semantic concepts are encoded similarly across native and romanized inputs, indicating a shared internal representation. moreover, when translating into a romanized script, target words emerge earlier, highlighting romanization as a structural link between language-agnostic concepts and language-specific output. while our study reveals initial insights into latent romanization, future work could focus on applying these findings to develop training strategies that enhance performance across diverse linguistic communities.