Transliteration of J udeo- A rabic Texts into A rabic Script Using Recurrent Neural Networks (2020)

Abstract: We trained a model to automatically transliterate Judeo-Arabic texts into Arabic script, enabling Arabic readers to access those writings. We employ a recurrent neural network (RNN), combined with the connectionist temporal classification (CTC) loss to deal with unequal input/output lengths. This obligates adjustments in the training data to avoid input sequences that are shorter than their corresponding outputs. We also utilize a pretraining stage with a different loss function to improve network converge. Since only a single source of parallel text was available for training, we take advantage of the possibility of generating data synthetically. We train a model that has the capability to memorize words in the output language, and that also utilizes context for distinguishing ambiguities in the transliteration. We obtain an improvement over the baseline 9.5% character error, achieving 2% error with our best configuration. To measure the contribution of context to learning, we also tested word-shuffled data, for which the error rises to 2.5%.

## introduction many great jewish literary works of the middle ages were written in judeo-arabic, a jewish dialect of the arabic language family that adopts the hebrew script as its writing system. prominent authors include maimonides (12th c.), judah halevi (11th-12th c.), and saadia gaon (10th c.). in this work, we develop an automatic transliteration system that converts hebrew-letter judeo-arabic into readable arabic text. generally speaking, given a text, transliteration is a process of converting the original graphemes to a sequence of graphemes in a target script. specifically, transliterating a judeo-arabic text into the arabic script almost invariably results in a text that has a similar number of letters. yet, the correspondence between the letters in the transliteration is not one to one. judeo-arabic hebrew script includes matres lectionis (e.g. some other challenges are: (1) authors of judeo-arabic texts sometimes use different mappings between hebrew and arabic letters. some authors use the hebrew letter ‫ג‬ to transliterate the arabic letter , and others will use it to transliterate the arabic letter . (2) diacritic marks (small dots placed either above or below letters) are often omitted in the hebrew script, another source of ambiguity. for example, the arabic letters and are sometimes represented by the same letter in hebrew. when the diacritic marks are maintained in the original hebrew script, usually they are used in an inconsistent way. even if they are used in the original manuscript, in many cases those marks appear differently or are completely missing in digital editions (such as those we used in this work). figure 1 shows a few examples of this problem. the apostrophe is the only diacritic mark that is used in the digital texts we used. one important diacritic mark that is often missing from digital versions is shadda (gemination), which may be used to easily resolve some lexical ambiguities. for example, the word ("he studied") has a different meaning when it appears with a shadda on the second consonant ("he taught"). the friedberg jewish manuscript society ( https://fjms.genizah.org ) has recently released a collection of hundreds of judeo-arabic works from different periods of time, formatted digitally as plain text and decorated with html tags. hebrew and aramaic citations and borrowings are annotated in those texts by domain experts. in the present work, we use texts from this collection. we noticed that some common borrowings were missed by the annotators; therefore, in this work we focus on the task of the transliteration of judeo-arabic words that originated in arabic words only. detecting the boundaries between arabic-origin words and borrowings from other languages, a task that is known as "code switching", is left for future improvements. since in our approach we consider the context of a word that needs to be transliterated, mostly to resolve lexical ambiguity, we masked those citation and borrowings, allowing the model to handle a continuous sequence of characters. this judeo-arabic-to-arabic transliteration problem has been addressed in one previous work @xcite . we elaborate on this and some other relevant previous works in the next section. in this work, we propose an end-to-end model to handle the transliteration task, by training a flavor of a recurrent neural network (rnn) on relatively short parallel texts, augmented with some synthetically generated data. our model was designed with the capability of memorizing words in the output language. section 3 describes the texts at our disposal. it is followed by a description of the baseline algorithm, and then our rnn solution. ## related work the only previous effort dealing with the transliteration of judeo-arabic texts @xcite employed a method inspired by statistical machine translation, which was state of the art before deep neural nets took over. it consisted of a log linear model where the main component is a phrase table that counts the number of occurrences in the training data. as in @xcite , they also regarded transliteration as translation at the character level, that is, imagining single letters to be words. they improved their results by reranking their model's top predictions using a word-level language model. this is expected to be beneficial since using a character-level mechanism would incur the danger of generating non-words and nonsensical sequences of letters. a word-level language model that is rich enough to avoid high unknown rates would screen those results out. proper noun transliteration is a related task, which has been addressed previously in many works. the input term is usually provided without context, and the task is to rewrite the name in the target script, maintaining the way the name is pronounced. a direct phrase-based statistical approach to transliterating proper nouns from english to arabic script is described in @xcite . it was preceded by largely handcrafted methods for romanization from arabic to english such as @xcite . a deep-belief approach for proper names was taken in @xcite ). an attention-based method for transliteration from arabic named entities to english and vice-versa is taken in @xcite . in @xcite , a transliteration system of names that employs a recurrent neural network (rnn), in two different ways, was described. in one way, they developed a sequence-to-sequence model that elegantly handles the different lengths of the input and output, with ctc alignment (see section 5.1). a minor difference is their use of "epsilon insertion" to deal with input sequences that are shorter than the matching output, which ctc cannot handle. we instead use a similar solution of letter doubling discussed later. the second approach is a model inspired by recent work in the field of machine translation applying an encoder-decoder architecture with an attention mechanism. they report on improved results using rnn compared to previous methods, as is usually the case when employing deep learning techniques with problem previously solved by other means. another closely related problem is the transcription of arabizi, which is a romanized transcription of arabic that emerged naturally as a means for writing arabic in chats, especially in the days of limited keyboards. transcription into arabic graphemes was studied by @xcite . except for the different dialects and code switches from colloquial arabic of different dialects to modern standard arabic (msa), they also had to handle changes in the language induced by the platform it appears in, including emoticons and deliberate manipulation of words, such as repetition of a single letter for emphasis common with internet social media. the same problem was addressed in @xcite , which maps roman letters to arabic scripts to produce a set of possibilities and chooses from them using a language model. in another related task, the goal was to translate written text to the international phonetic alphabet (ipa), capturing the pronunciation of the written text. the work in @xcite tried different rnn models, handling unequal sized input-output pairs by epsilon post-padding. they experimented with time delays, that is, postponing the output by a few timestamps (by pre-padding the output while postpadding the input accordingly). this allows the network to catch more of the input before deciding on the output. they also compared this with a bidirectional long-short-term-memory (lstm) network that is able to see the entire context backward and forward. they combined the bidirectional lstm with a ctc layer, handling the longer-output-than-input issue with epsilon post-padding (we used doubling). they reported, as expected, that greater contextual information contributes to performance. the bidirectional lstm performs better than the unidirectional one even when the full context (whole word) is fed to the network before it starts its prediction. the best performance was obtained using the bidirectional lstm, combined with an n-gram (non-rnn) model. this transcription from a written language to ipa can potentially be used as a mediator to transcribe between any two languages were we to have an encoder from ipa to the source language and a decoder from ipa for the target script. another potential use for this ability of transliteration to ipa would be for improved spell checking and correction (for those who only have basic knowledge of the sounds of the letters). a model could predict the utterances from the graphemes and recognize the word by similarity in the sound domain, to arrive at the correct spelling. for example: @xmath0. ## data kuzari. to train and evaluate our rnn model, we needed a considerable amount of parallel text in judeo-arabic along with its arabic transliteration. thankfully, the kuzari (kitab al khazari), a medieval philosophical treatise composed by judah halevi in andalusia around 1140, was recast in arabic by nabih bashir @xcite . we use this arabic version and the original judeo-arabic as a parallel resource for training. the original judeo-arabic text of the kuzari is taken from the friedberg repository, and is derived from the critical edition edited by david h. baneth. it was based on several manuscripts and comprises 47,348 word tokens (15,532 unique types), about 11% of which are hebrew and aramaic insertions. it is important to mention that bashir's book provides translations for those insertions, and that there is no distinction in his edition between the transliterated and translated words. as mentioned before, we postpone handling the hebrew and aramaic citation and borrowings in our work, and we use the annotations provided by the friedberg's experts in the judeo-arabic text, to mask out all words that are not originated in arabic. after cleaning the data from translation comments, section numbering and other elements irrelevant for transliteration, we needed to align the arabic translation with the judeo-arabic source, on the word level. aligning the texts is necessary because bashir's translation does not perfectly match the judeo-arabic text, there being some word insertions, deletions and edits. to do that, we begin by breaking the two texts into words, and then iterating the two sequences of words. at each iteration we choose between skipping a word from the source (judeo-arabic) text, skipping a word from the target (arabic) text, or adding both as an aligned source/target pair. the decision is the choice that minimizes the total edit distance of the alignment. to calculate edit distance between strings of different scripts, we applied the simple map to the arabic source to get a basic transliteration into hebrew letters. (the details of the algorithm are given in (terner, 2020, appendix).) after this alignment process we only keep pairs of words in the alignment that are close enough edit-distance wise. we set the threshold for edit-distance similarity heuristically, after normalizing the edit distance result by the maximum length of the two words that were compared, to the value of 0.5. this left us with a total of 47,083 word pairs, of which 20% were chosen randomly as test data, and the rest left for training. we are aware of the generalization problem of training and testing on data from the same source. words that do not align well according to the predefined threshold, such as hebrew insertions, are removed and replaced by a fixed symbol (h) both in the source and target texts. we do the same for hebrew insertions that are annotated as such by friedberg's annotators. we do this to preserve the sentence structure since we suspect that this information is of value to the model for making better predictions. we see punctuation as an important feature of the language and we want to keep this information for the model to train with. where there are disagreements (e.g. comma vs. period) in the source and target languages, we favor the source language. additionaly, we removed from the arabic text the harakat marks (short vowel marks): fathah, kasrah, dammah and sukun. they appear rarely in written arabic (except for text intended for children or in religious texts) and are usually considered "noise" for language-related algorithms due to their scarcity (habash, 2010). in the judeo-arabic text, the equivalent to these diacritics is the hebrew niqqud signs. they appear only for hebrew insertions; hence they are removed incidentally by the replacement of hebrew insertions. sometimes we even make use of those marks to identify hebrew insertions missed by friedberg's experts. note that the tanwīn symbols when combined with alif were standardised to the modern style, so the diacritic symbol appears on top of the alif and not on the letter preceding it. ## baseline algorithm the evaluation metric that we use is simply the average edit distance over all examples in the test dataset. the edit distance (ed) that we use is the levenshtein distance, which is calculated between the predicted characters and the ground truth. it is then normalized by the length of the ground truth. the formula for label error rate (ler) is 1 |s| (x,z)∈s ed(h(x), z)/|z| for model h on test data s ⊆ x × z, where x are the inputs, z is ground truth and |z| is the length of z. this is a natural measure for a model where the aim is to produce a correct label sequence @xcite . to evaluate results, we start by creating a baseline transliteration on the test data that translate each hebrew letter to the most common letter according to the predefined mapping mentioned earlier ( ## discussion we designed a model for automatic transliteration of judeo-arabic texts into arabic scripts. endeavoring to overcome the problem of ambiguous mappings in the transliteration, we trained an rnn model using the ctc loss that has enabled us to cope with unequal length input/output sequences due to the addition of diacritics in the target (arabic) side of the data. as mentioned, we wanted to create a network that will have memory, along with some language capability in the output side that will enable it to distinguish between different word senses and overcome small variations in the transliteration. the results demonstrate a substantial decrease in error compared to the baseline, from 9.5% to 2% ler, showing that the network is capable of attaching correct diacritics to the arabic. on unseen text, the network incurs a 3.24% error rate. this is important since judeo-arabic texts vary according to the writer, time period, and region. examples and some remaining problems are included in figure 5 . we experimented with several methods to enhance training. first, we exercised pre-training of the network with cross-entropy loss to teach the network the simple mapping used in the baseline transliteration. this can enlarge and deepen the network and still guarantee convergence. without this pretraining stage, the network failed to discover this mapping by itself. since we only train on parallel text from a single source, and we are interested in making the model generalize better to unseen texts, we augment the training dataset by generating parallel texts out of available medieval arabic texts. this had a slight adverse affect on accuracy and might depend on the choice of arab texts that were used for the synthetic data. dropout was considered and implemented on the synthetic data with the desired affect. while accuracy on the first test data, taken from the same source as the original training data decreased as expected, accuracy for the additional test data improved. we suspect that the network assimilates more language skills due to the dropout on the training data. examining the top 5 results of the ctc-decode beam search, instead of only the top one, reveals that sometimes the correct transliteration, at least the one that was chosen by the human transliterator, is among those five. therefore, running a pure arabic language model should enhance model accuracy. one question that arises is how much of the language the network catches. to examine this, we perform a forward pass on the shuffled test data. shuffling is done at word level. this generates a parallel dataset of words that lack context, sharing the same word distribution as the real test data. indeed as expected, on the shuffled test data the error increases by ∼0.5% on average.