J apanese Rule-based Grapheme-to-phoneme Conversion System and Multilingual Named Entity Dataset with International Phonetic Alphabet (2024)

Abstract: In Japanese, loanwords are primarily written in Katakana, a syllabic writing system, based on their pronunciation. However, the transliterated loanwords often exhibit spelling variations, such as the word “Hepburn” being written as “ヘボン (hebon)”, “ヘプバーン (hepubaan)”, “ヘップバーン (heppubaan)”. These orthographical variants pose a bottleneck in multilingual Named Entity Recognition (NER), because named entities (NEs) do not have one-to-one matches. In this study, we introduce a rule-based grapheme-to-phoneme (G2P) system for Japanese based on literature in linguistics and a large-scale multilingual NE dataset with annotations of the International Phonetic Alphabet (IPA), focusing on IPA to address the Katakana spelling variations in loanwords. These rules and dataset are expected to be beneficial for tasks such as NE aggregation, G2P system, construction of cross-lingual language models, and entity linking. We hope our work advances research on Japanese NER with multilingual loanwords by solving the spelling ambiguities.

## introduction japanese orthography consists of three unique writing systems: hiragana, katakana, and kanji. among these, katakana is mainly used for transliteration of loanwords originating from languages outside the sinosphere, in particular, for named entities (nes) such as proper nouns. however, there are no clear rules for this transliteration process, and nes are transliterated into the closest katakana based on the pronunciation of the source language. while some loanwords close to the japanese pronunciation are often transliterated into katakana representing mostly similar sounds (e.g., "obama" to "オバマ (obama)"), this is not the case for other loanwords, leading to ambiguities due to the lack of unified common transliteration (e.g., "hepburn" to "ヘボン (hebon)", "ヘプバーン (hepubaan)", and "ヘップバーン (heppubaan)"). more details are described in appendix a. this ambiguity poses challenges in unifying katakana-written loanword nes within japanese and identifying the original nes. this issue stems from significant differences in phonological and writing systems between japanese and other languages, especially in historical documents written when the reading of foreign words was not customary among general readers, resulting in a wide variety of transliterations based on the sound perception of individuals without established transliteration rules. furthermore, the inconsistent transliteration of loanword nes in japanese can be an obstacle for japanese learners. given these issues, we first developed a rulebased japanese grapheme-to-phoneme (g2p) system. although katakana can be mapped into ipa by rule-based conversion, the development is challenging because constructing precise rules requires knowledge of linguistics and phonetics. currently, only neural-based approaches support japanese g2p. our rule-based g2p system, founded on linguistic principles, accurately represents pronunciation even in cases where ipa is automatically extracted from a source whose phonetic accuracy is not guaranteed. we hope that our g2p system can serve as a useful learning aid for japanese learners and is expected to be applicable to pronunciationbased japanese text analysis methods, such as ner and entity linking. next, we constructed a large-scale multilingual ne dataset with ipa annotations to address the spelling variations such as katakana spellings of loanwords in japanese. this dataset contains over 69 million pairs of ne and ipa, and over 14 million ids used to identify nes from 68 languages. on average, each id is associated with five different pairs of ne and ipa. ours is the largest dataset among the multilingual ne datasets with phonetic annotation. we hope our work advances research dealing with the spelling variations in japanese, including approaches of cross-lingual word alignments such as @xcite ; ren (2023), which utilizes phoneme sequences (though non-ipa) as an intermediate to align english and japanese words. to summarize, our contributions are as follows: ## conclusion in this paper, we introduced the new g2p rules for japanese based on the literature in linguistics and phonetics, and constructed the largest multilingual dataset of ne with ipa using rule-based g2p including our own rules for japanese. these resources will be beneficial for solving ne-related tasks such as ner and entity linking in japanese. the actual application of our g2p tool and dataset to downstream tasks on japanese nes like transliteration, text-to-speech, and so on is left for future work. nonetheless, we have confirmed that g2p conversion is possible with a general level of accuracy. dataset in this paper, our contributions are the development of the ne dataset with ipa annotations and do not include experiments in downstream tasks. however, applying ne datasets with ipa annotations to downstream tasks has been reported in recent studies @xcite , and we intend to apply ours to downstream tasks in future work. additionally, the development of our large-scale dataset, comprising over 69 million nes with associated ipa representations, demanded a significant investment of computational resources and time. we used a total of 96 cpu cores of intel(r) xeon(r) platinum 8160 cpu @ 2.10ghz and 384gb ram, taking nearly two months to complete annotating ipa. the availability of the large-scale dataset enables rapid experimentation, rendering it a highly valuable resource for advancing future research.