Revisiting Syllables in Language Modelling and Their Application on Low-Resource Machine Translation (2022)

Abstract: Language modelling and machine translation tasks mostly use subword or character inputs, but syllables are seldom used. Syllables provide shorter sequences than characters, require less-specialised extracting rules than morphemes, and their segmentation is not impacted by the corpus size. In this study, we first explore the potential of syllables for open-vocabulary language modelling in 21 languages. We use rule-based syllabification methods for six languages and address the rest with hyphenation, which works as a syllabification proxy. With a comparable perplexity, we show that syllables outperform characters and other subwords. Moreover, we study the importance of syllables on neural machine translation for a non-related and low-resource language-pair (Spanish–Shipibo-Konibo). In pairwise and multilingual systems, syllables outperform unsupervised subwords, and further morphological segmentation methods, when translating into a highly synthetic language with a transparent orthography (Shipibo-Konibo). Finally, we perform some human evaluation, and discuss limitations and opportunities.

## introduction in language modelling (lm), we learn distributions over sequences of words, subwords or characters, and the last two can allow an open-vocabulary generation @xcite . we rely on subword segmentation as a widespread approach to generate rare subword units @xcite . however, the lack of a representative corpus, in terms of the word vocabulary, can constrain the unsupervised segmentation (e.g. with scarce monolingual texts @xcite ). as an alternative, we could use character-level modelling, since it also has access to subword information @xcite , but we face long-term dependency issues and require longer training time to converge. similar issues are extended to other generation tasks, such as machine translation (mt). in this context, we focus on syllables, which are speech units: "a syl-la-ble con-tains a sin-gle vowel u-nit". syllables can be defined as a group of segments that is pronounced as a single articulatory movement. syllables are fundamental phonological units since they participate in important word prosodic patterns, such as stress assignment. in this sense, syllables are more linguistically relevant units than characters, and behave as a mapping function to reduce the length of the sequence with a larger "alphabet" or syllabary. their extraction can be rule-based and corpus-independent, but datadriven methods or hyphenation using dictionaries can approximate them as well. we assess whether syllables are useful for encoding and/or decoding a diverse set of languages on two generation tasks. first, for lm, we study 21 languages, to cover different levels of orthographic depth, which is the degree of graphemephoneme correspondence @xcite and a factor that can increase complexity to syllabification @xcite . 1 whereas for mt, we focus on the distant and low-resource language-pair of spanish-shipibo-konibo. we choose shipibo-konibo 2 because it is an endangered language with scarce textual corpora, which limits unsupervised segmentation methods, and has a transparent orthography, which could be beneficial to syllabification. also, we consider multilingual mt systems, as they outperformed pairwise systems for the chosen language pair @xcite . ## related work the closest lm study to ours is from @xcite for subword-grained prediction in english, where they used syllables as a proxy to split words with low frequency, reduce the vocabulary and compress the model size. besides, syllable-aware lm 1 e.g., english has a deep orthography (weak correspondence), whereas finnish is transparent @xcite . for syllable-based mt, there are mostly studies for related paired languages, such as indic languages (in statistical mt without subwordbased baselines: kunchukuttan and bhattacharyya (2016)), tibetan-chinese @xcite , and myanmar-rakhine @xcite . instead, spanish-shipibo-konibo is a non-related language-pair. the only distant pair was english-myanmar @xcite , but they did not compare it with unsupervised subwords. neither of these studies analysed multilingual settings. ## open-vocabulary language modelling with a comparable perplexity open-vocabulary output we generate the same input unit (e.g. characters, syllables or other subwords) as an open-vocabulary lm task, where there is no prediction of an "unknown" or outof-vocabulary word-level token @xcite . we thereby differ from previous works, and refrain from composing the syllable representations into words to evaluate only word-level perplexity. character-level perplexity for a fair comparison across all granularities, we evaluate all results with character-level perplexity: ## low-resource machine translation after observing the positive impact on lm, we focus on syllables for mt, which adds complexity to the process, as there is at least one extra language involved. in contrast to prior work, we (i) study syllable-based mt for a distant and low-resource language-pair, spanish-shipibo-konibo; (ii) compare syllables against the most widespread unsupervised segmentation method (bpe) with automatic metrics and human evaluation; and (iii) anal-yse the applicability of syllables on multilingual translation systems. the last element is significant, as a multilingual setting is the state-of-the-art approach for leveraging low-resource language-pairs performance @xcite . moreover, we decided to apply syllabification only on shipibo-konibo, a highly synthetic foot_3 language with scarce textual data and with a transparent orthography foot_4 . for this reason, we focus in three settings. first, mono, a pairwise system where each source and target is segmented with a different method. second, joint, another pairwise system where the bpe baseline is jointly trained with the source and target data. third, o2m, a multilingual one-tomany foot_5 system, where the bpe baseline is jointly trained with all the languages (we added spanish-english in our experiments). ## limitations and opportunities syllables only cannot offer a universal solution to the subword segmentation problem for all languages, as the syllabification tools are languagedependent. besides, the analysis should be extended to more scripts and morphological types. furthermore, we do not encode any semantics in the syllable-vector space, with a few exceptions like in korean @xcite . nevertheless, our results confirm that syllables are reliable for lm and mt, and building a syllable splitter might require less effort than annotating morphemes to train a robust supervised tool foot_6 . specifically for mt, syllables could be useful when: (i) we are dealing with extremely lowresource data, which affects unsupervised word segmentation, (ii) we are translating into a language with a high synthesis, which has been observed as a factor that impacts on mt performance @xcite , and (iii) we are working with a language with a transparent orthography. this is the scenario for several languages from the americas, where their writing systems have been recently standardised for documentation and revitalisation purposes @xcite , and some resources for mt have been compiled @xcite . ## conclusion we have proved that syllables are valuable for generation tasks such as: (i) open-vocabulary lm, where they behave positively even for languages with deep orthography, and overcome character and subword baselines. (ii) low-resource and multilingual mt, outperforming bpe pieces when we translate into a language with a transparent orthography and complex morphology (high synthesis), even when the language-pair is not related.