A S anskrit grammar-based approach to identify and address gaps in G oogle T ranslate’s S anskrit- E nglish zero-shot NMT (2023)

Abstract: In this work, we test the working of Google Translate’s recently introduced Sanskrit-English translation system using a relatively small set of probe test cases designed to focus on those areas that we expect, based on a knowledge of Sanskrit and English grammar, to pose a challenge for translation between Sanskrit and English. We summarize the findings that point to significant gaps in the current Zero-Shot Neural Multilingual Translation (Zero-Shot NMT) approach to Sanskrit-English translation. We then suggest an approach based on Sanskrit grammar to create a differential parallel corpus as a corrective training data to address such gaps. This approach should also generalize to other pairs of languages that have low availability of learning resources, but a good grammar theory.

## introduction and motivation translation between sanskrit and english presents significant challenges, even for expert human translation, due to the unique features of sanskrit and the large linguistic gap between sanskrit and english. also, sanskrit has a unique role, specially in the indian subcontinent as it was formerly a common language of communication across all fields. it was displaced by colonization, but is seeing a resurgence of late, specially for accessing indian knowledge systems in the original. this has important implications on the expectations from an automatic translation, and the implications of erroneous translations. in addition to the above factors, automatic machine translation between sanskrit and english presents the additional challenge of sanskrit having relatively low availability of high-quality training resources such as tagged corpora. in may 2022, it was announced (google, 2022) that 24 new languages including sanskrit were being added to google translate, using the new zero-shot machine translation adaptation @xcite of google's neural multilingual translation @xcite . in essence, zero-shot nmt leverages deep learning in a single model trained on multiple language pairs, to translate even between directions and language pairs it has not been explicitly trained on. google's zero-shot nmt is a variation of zero-resource machine translation @xcite , which requires an additional fine-tuning step using "pseudo-parallel" data of the new language pair. the need for this step is avoided in the design of zero-shot nmt. it is interesting that google has used the zero-shot nmt approach for sanskrit-english. this is a data-driven approach to mt and nlp that avoids the need for explicit encoding of knowledge. the other possible approach to mt and nlp is grammar-based or model-driven, which needs explicit encoding of knowledge. the linguistic theory and grammar behind sanskrit is very stable and sound. using this theory base, efforts have been made to create grammar-based sanskrit nlp systems, for example @xcite . the data-driven approach is normally attractive when training resource availability is good, and a good grammar model is absent or too complex. the grammar-based approach is normally attractive when there is a good grammar model, and the training resource availability is poor. with sanskrit, there is a relatively good grammar model and relatively poor training resource availability. and yet, google has chosen the data-driven approach for sanskrit-english translation through its choice of zero-shot nmt, in order to maintain a uniform approach across all languages. it is pertinent, therefore, to test the effectiveness of google's sanskrit-english translation system in actual use. we now describe the considerations we used to design this test. ## test design considerations for sanskrit-english google translate we now describe the considerations that came up when designing a test for sanskrit-english google translate, and how we dealt with them: a. directionality -should we test sanskrit to english, or english to sanskrit or both? for the sanskrit-english pair, we expect three main use cases: a. sanskrit-english for sanskrit access -people fluent in english and interested in sanskrit literature trying to translate a traditional vedic or classical sanskrit mantra, shloka, poem or text from sanskrit to english. b. english-sanskrit for sanskrit learning or communication -people fluent in english, interested in learning sanskrit (either for conversation or to access the literature) and trying to translate english to sanskrit. c. sanskrit-english for communication or learning -people fluent in sanskrit but not english, translating their original sanskrit text into english for communicating with others or to learn english. on account of the nature of the language pair and their current status, of the above three, we expect case a to overwhelmingly dominate, case b to be next, and case c to be relatively insignificant. thus, the sanskrit-english direction is the highest priority, and also has higher demands on accuracy, since it is more likely to be dealing with classical texts whose incorrect interpretation could have undesirable cultural consequences. we therefore put more emphasis on testing sanskrit-english than english-sanskrit. most of our discussion will also be about sanskrit to english translation, unless otherwise mentioned or obvious from the context. b. purpose -our purpose is to check whether google's zero-shot nmt automatic translation is robust enough for the sanskrit-english language pair, rather than an exhaustive performance analysis of the translation. therefore, rather than a test suite aimed at complete coverage of the language pair, we will create a probe test suite of a few carefully hand-crafted cases, leveraging grammar and language theory, mainly focused on areas where we expect challenges for translation. c. automation -since the test set was small and for one-off use, and to be hand-crafted leveraging human expertise, it was simpler to do it iteratively and manually for now rather than invest effort in automating it. ## test process and results based on the considerations discussed above, a small "probe" test suite of approximately 120 test cases was hand-crafted and applied iteratively. the test cases are not all independent, many are part of a group of inputs iteratively designed to test different variants of a specific area being tested. for example, correct translation of single/dual/plural number involves inputs containing various combinations of these. it is difficult to enumerate the groups, since there are sometimes overlaps where a single test case is logically part of multiple groups. hence, the above table lists the number of individual test cases and not groups. each test case was manually translated by one of the authors, who is fluent in both sanskrit and english, to create the expected reference output. the test case was then input to google translate sanskrit to english translation english to sanskrit translation 98 test cases 31 test cases and the output recorded against it. the output was manually evaluated using a 3-way rating system defined by us as follows. all the test cases, the expected and actual outputs and ratings are available in the sheet attached as appendix b -all test cases. the ratings are annotated with explanations in a remarks column where needed. the results of the test are summarized below. as seen from the above table, if we take the stricter criterion of only ok-rated outputs as correct, the accuracy of google translate for our probe test is 37.76 per cent for sanskrit to english, and 32.26 per cent for english to sanskrit. if we take the more relaxed criterion of only *-rated outputs as incorrect, the accuracy is 72.45 per cent for sanskrit to english, and 64.52 per cent for english to sanskrit. since this is only a probe test, and not an exhaustive coverage test, we cannot claim these as the actual accuracy figures, but the test probe does reveal that there are significant gaps in the performance of google translate for both the directions that need to be fixed before the translation can be considered robust. the detailed remarks about each output can be found in appendix b -all test cases. in the following section, we summarize the key observations and their implications. it must be noted here that google translate by design is a learning product and is therefore being continuously updated. the test results are valid as of the time they were conducted, namely, in the third week of december 2022. ## key observations and implications looking at the test case outputs, we find that given the multiple inherent challenges of sanskrit-english translation, the system performs surprisingly well for a zero-shot nmt that has possibly not been trained on a single input specific to the sanskrit-english pair. of the specific challenge areas tested by the probe test, it does cover quite a wide spectrum of phenomena satisfactorily, in at least a few cases, including sandhi, samāsa, taddhita, dual number, three grammatical genders, and the phenomenon of satisaptamī (absolutive locative clause). having said that, the output is often inconsistent across variations of a language feature, and dubious or incorrect in several cases. ## characterizing the gaps the current gaps in google translate's english-sanskrit translation, summarized in the previous section, can be classified into two categories: 1. learning gaps -these are gaps that can be addressed by better training of the zeroshot nmt, by feeding more training data, or tuning the learning parameters. for example, if a specific english idiom is not currently learnt as an idiom, it could be learnt by feeding in examples of its usage in the english-hindi/urdu translation corpus. we can expect such gaps to gradually reduce over time as the system is fed with more training data, without any change to the basic zero-shot nmt approach. however, with the approach being suggested in this work, this gap reduction could be speeded up. 2. systemic gaps due to pure zero-shot nmt -these are gaps that arise due to not feeding translation data specific to the target language pair (in our case english-sanskrit) in training the system, but only leveraging translation data of other language pairs that between them cover the target language pair, for example, in our case, english-hindi/urdu and sanskrit-hindi/urdu. such gaps are inherent to the pure zero-shot nmt approach and will not reduce over time. addressing such gaps needs a different approach that we shall touch upon shortly. let us try to formally characterize these two types of gaps in order to understand them better. in order to do that, we must first characterize different types of machine translation systems and see where zero-shot nmt fits in. essentially, a deep-learning based machine translation system is a transformer that takes a text s in the source language s and transforms it into a text t in the target language t. in order to do this, it uses a pre-trained language model. let d(l1, l2) represent training data consisting of parallel translations from the language l1 to the language l2. google translate's neural multilingual translation (nmt) approach @xcite ) uses a single large language model (llm) that is trained on all the languages available. this allows learning to be leveraged across different data sets and languages. thus, for example, the learning from d(l1, l2), d(l2, l3) and d(l4, l2) is merged into a single model. then, in translating (l1, l2), the learning from not just d(l1, l2), but also d(l2, l3) and d(l4, l2) gets leveraged, leading to more robust output than from just d(l1, l2) alone. moreover, this allows the system to translate even between language pairs even though that specific pair was not part of the learning data, say, due to low availability of training data for that pair. in this case, for example, the system could give a translation for (l1, l4), though this pair was not part of the training data. this would obviously not be as robust as having d(l1, l4) in the training mix, but may be better than giving no translation at all. this is what google means by zero-shot nmt @xcite , and that is what is reportedly used in english-sanskrit translation. the implication is that the system is currently not trained on d(english, sanskrit) data at all. it leverages, for example, d(english, hindi/urdu) and d(sanskrit, hindi/urdu), and possibly data in other language pairs, to attempt (english, sanskrit) translation. let f(s,t) be the set of features that would have been learnt by the nmt if it had been trained with the ideal training data set d(s,t) to correctly perform an arbitrary (s,t) translation request. now, in zero-shot nmt, there is no d(s,t). instead, the nmt is fed an n-member set dn of d(li, lj) where 1 <= @xmath0, 1 <= @xmath1, and (li, lj) is not in dn. the assumption here is that f(s,t) will be compositionally learnt by the nmt via some combination of d(li, lj) training inputs. we can now characterize the two categories of gaps we mentioned above in these terms. a learning gap is one where f(s,t) is not currently achieved, but can be learnt by either adding more data to dn, or by optimizing the parameters of the nmt, or both. for example, let us say a specific form of a verb in sanskrit is not being correctly translated into english. this could be addressed by adding data that contains that form in the d(sanskrit, hindi/urdu) set. a systemic gap is one where no combination of d(li, lj) comprising dn for any value of n can cause learning of f(s,t), because there exists a set of features fo(s,t) that are not compositional, but can only be learnt by training on d(s,t). let us look at an example each of both these gaps. first, an example of a learning gap. the word मां [māṃ] occurs both in sanskrit and hindi/urdu. in sanskrit, it is the sandhi form of माम् [mām] and means "me". in hindi/urdu, it is the simplified form of माँ [mām ̐ ] and means "mother". currently, google translate sometimes confuses these two cases, as seen from some of the test outputs. this distinction can be trained into the system by having more examples distinguishing the two cases in the d(sanskrit, hindi/urdu) and d(hindi/urdu, english) training data. hence, we can call this a learning gap. now, an example of systemic gap. the unambiguous sanskrit sentence "अ%े उपिवशन् िपता पु 1ं प3यित"[aśve upaviśan pitā putraṃ paśyati] is ideally translated into english as "the father seated on a horse sees his son". similarly the unambiguous sanskrit sentence "िपता अ%े उपिवश7तं पु 1ं प3यित"[pitā aśve upaviśantaṃ putraṃ paśyati] is ideally translated into english as "the father sees his son who is seated on a horse". in sanskrit, the distinction between the two is clear and marked by inflection on the appropriate noun. in english, the distinction is clear when marked with a relative clause marked by "who is". however, a possible hindi translation of both these would be "िपता अपने बे टे को घोड़े पर बै ठे हb ए दे खता है "[pitā apane beṭe ko ghoḍe para baiṭhe hue dekhatā hai]. this literally translates to "the father sees his son seated on a horse", which is ambiguous due to the prepositional phrase attachment ambiguity, and can convey both the meanings. the issue here is that the unambiguous case markings of sanskrit on the phrase "seated on the horse" get mapped in both the hindi translations to a single oblique case marking (बै ठे हb ए[baiṭhe hue]), which has the effect of saying "seated on a horse" without adding the relative clause marker "who is". this feature mismatch (divergence pattern) between the languages (namely inflection in sanskrit vs oblique case in hindi vs relative clause in english) causes a noncompositionality in translation. therefore there is an information loss in zero-shot nmt with transfer learning involving only d(sanskrit, hindi/urdu) and d(hindi/urdu, english). this gap cannot be addressed by adding any amount of d(sanskrit, hindi/urdu) and d(hindi/urdu, english) data or tweaking the parameters of the zero-shot nmt. it can only be addressed by adding examples of both sanskrit sentences translated correctly directly to english, that is, by adding some d(sanskrit, english) data. thus, we can call this a systemic gap. [note: this example is slightly simplified for ease of understanding. the actual sanskrit-hindi and sanskrit-english translations by google translate are marginally different, but close enough for the example and the argument to hold. the actual details are discussed in appendix c -"the systemic gap example"]. systemic gaps are an outcome of "language divergence" in translation, which was formally described in @xcite . a partial set of language divergence patterns between english and sanskrit was described in the context of a prototype rulebased machine translation for english to sanskrit in @xcite . the conclusion we can draw is that due to the presence of systemic gaps, the purely data-driven approach of zero-shot nmt, without encoding grammar knowledge, and without training on the specific language pair, does not work well enough for the sanskrit-english language pair. the zero-resource approach of @xcite ) using "pseudo-parallel" sanskrit-english data will not work in the presence of systemic gaps. google's zero-shot nmt paper @xcite discusses (in section 4.6, "zero-shot translation" and section 4.7, "effect of direct parallel data") possible approaches to go beyond zero-shot nmt by adding real parallel data in the missing language pair and direction (e.g. sanskrit-english in our case). if a lot of high-quality real-life parallel data is available, this is found to be ideal. however, we know that high-quality sanskrit-english real-life data is not readily available. for example, samanantar, a parallel corpora collection between english and 11 indic languages, does not include sanskrit @xcite . a more recent effort, indictrans2, includes parallel corpora with english and all 22 scheduled indian languages including sanskrit @xcite . this is a good start, however, the size of the corpus is only of the order of 2.5k real-life sentences, which may be insufficient for good quality learning. also, the direction is english-sanskrit, and reversing it for sanskrit-english training would not be ideal. to address this need for direct parallel sanskrit-english data to plug the gaps in zero-shot nmt, and the absence of such large high-quality corpora, we suggest the following approach -construct a "differential corpus" as a corrective data suite of sanskrit-english language-pair-specific training data, by leveraging such linguistic knowledge. after all, we leveraged linguistic knowledge to create an effective probe test without requiring an exhaustive test suite. in the same way, we propose that this linguistic knowledge could be leveraged to create the right training data to address these areas and plug the gaps to create a more robust zero-shot nmt system. to our knowledge, there is no readily available framework that can be directly leveraged to create such a differential corpus. the work on language divergence in general @xcite and english-sanskrit @xcite in particular cited earlier is a good directional starting point, but we will need to focus it on the sanskrit-english direction and cover a more comprehensive set of features specific to the systemic gap issue than addressed in those works. there has been work done in identifying the formal structure of sanskrit text @xcite that could provide a set of features for us to take into account, but they may need to be filtered to keep only the ones relevant from the viewpoint of divergence. there is also a translation of the exercises from apte's classical text on sanskrit syntax (apte, 1885) into sanskrit @xcite . this again may need to be filtered for divergence specific cases. a generic grammar framework like the grammatical framework (gf, 1998) is potentially interesting for its ability to deal with multilingual grammars, but it currently has support only for english and hindi, not sanskrit. sanskrit-specific tools and toolkits such as inria's sanskrit heritage site @xcite , the university of hyderabad department of sanskrit's samsaadhani @xcite and ashtadhyayi.com @xcite are targeted at understanding sanskrit rather than translating it to english, so they may be useful as reference tools for the human experts generating the differential corpus. large language models (llms) like chatgpt (yiheng liu et al, 2023) get their capabilities through pre-training, an expensive and long drawnout process. reinforcement learning from human feedback (rlhf) (ziegler et al, 2019) can be used to affect their behaviour (or to "fine-tune" it, such as eliciting right responses or suppressing objectionable output), which is not as expensive, and hence can be undertaken multiple times. rlhf can potentially be used with the differential corpus suggested, but this needs further study. in-context learning (sang et al, 2022) is seen as a major shift in transfer learning in the context of llms, with intimations of an "emergent" behaviour. in contrast to the classic pretraining-then-finetuning procedure for downstream prediction tasks in llms, there is only a need to provide a few "in-context" examples, without affecting existing model parameters. the differential corpus suggested might well function as in-context examples that can be taken up as future work. the proposed differential corpus would consist of d(sanskrit, english) and d(english, sanskrit) data that would be designed, based on knowledge of sanskrit and english grammar theory, to focus on addressing the systemic gap, that is, exercising the language features fo(sanskrit, english) and fo(english, sanskrit) that are not compositionally learnable from d(s,t) data sets not containing the above two language pairs. in addition, since we are going to cover all combinations of a given feature, in the process, it may incidentally cover some learning gap data as well, because, intuitively, if a feature can be compositionally learnt from d(s, l1) and d(l2,t), then it can also be directly learnt from d(s,t). thus, the suggested approach would also speed up reduction of the learning gap. the key idea behind the proposed grammar-based approach is to leverage the rich linguistic model of sanskrit from the traditional indian śāstras including vyākaraṇa (the aṣṭādhyāyī of pāṇini and its related works), the vākyapadīyam of bhartṛhari, mīmāṃsā, nyāya and vaiśeṣika, mapping approximately to linguistics, grammar, discourse analysis, logic and ontology respectively, to create a "differential corpus" of translation test cases that can be used as training data to fill the current gaps for sanskrit-english zero-shot nmt. the proposed approach is summarized as follows: a. identify the prominent divergence areas of the sanskrit-english language pair, that is, the set of language features that are present in sanskrit and either absent or rudimentary in english, fo(sanskrit, english) or vice versa, fo(english, sanskrit). for reasons stated earlier, we focus here on the first case, fo(sanskrit, english). b. for each feature, iteratively create a group of test cases to test the translation of that feature. a group consists of a set of individual test cases. collectively, the group should cover the range of variations of that feature. for example, if the divergence area is -presence of three grammatical numbers (singular/dual/plural) in sanskrit, versus only two grammatical numbers (singular/plural) in english -the feature is "grammatical number", and we have to create as test input a group of sentences containing all combinations of singular, dual and plural nouns. of particular interest are cases of ambiguity, where two or more features map to the same form (e.g. a tiṅanta and subanta, or a kṛdanta and subanta, map to the same form as seen in the भवित example). the test inputs should check whether the translation deals with the ambiguity and provides the correct translation. such a differential corrective parallel corpus can be fed to the existing zero-shot nmt in addition to the training data it has already seen, without the need for any significant modification to the architecture of the system. we now identify the key linguistic features of sanskrit that are part of the proposed approach as outlined above, and highlight the potential challenge areas to be tested in each. a. lexical features 1. sandhi -correct identification of all :वर[svara](vowel), >य@जन[vyañjana](consonant) and िवसगc [visarga](aspirant) sandhis. particularly where sandhi leads to ambiguous forms. for example: 89ाि:म[brahmāsmi] can be 89 अि:म[brahma asmi] or 89ा अि:म[brahmā asmi]. 2. special signssuch as the अवgह-िचi[avagraha-cihna]ऽ. for example: अनु गृ िहतोऽिnम[anugṛhito'smi] / अनु गृ िहतोिnम[anugṛhitosmi] / अनु गृ िहतः अिnम[anugṛhitaḥ asmi] are equivalent. b. morpho-syntactic features 3. सु बeत[subanta](noun) forms -correct handling of ambiguous forms such as ते . 4. basic ितङeत[tiṅanta](verb) forms -correct handling of same धातु [dhātu] in multiple गणs[gaṇas](groups)having same forms with different meanings, or having same form as subantas (e.g. भवित[bhavati]). 5. derived tiṅanta(verb) forms -correct handling of verbs derived from िणच् [ṇic], सन् [san] and similar ijययs[pratyayas] (suffixes). 6. compound sentences -correct handling of यद ् /तद ् [yad/tad] and similar conjoint sentences. 7. complex sentences -correct handling of clauses involving कृ दeत[kṛdanta] (participials). c. semantic features 8. word order and topicality -word order does not change the gross meaning in sanskrit, but may alter the focus and topicality. also, in some cases the order does matter, for example, placement of अिप[api] at the beginning vs middle. 9. तिmत[taddhita] (noun-noun morphology)for example, अण् [aṇ] patronymic pratyaya 10. समास[samāsa] (compound nouns) -correct translation of all the main samāsa types -तjपु oष[tatpuruṣa](including कमc धारय[karmadhāraya], िqगु [dvigu], उपपद[upapada], and नञ् [nañ]), बहt uीिह[bahuvrīhi], अ>ययीभाव[avyayibhava] and qeq[dvandva]. for example -the same samāsa (e.g. पीताvबर) can be interpreted as तjपु oष[tatpuruṣa] or बहt uीिह[bahuvrīhi] a parallel corpus based on the above feature set could be created by leveraging the related work discussed earlier, as well as taking example sentences given for different grammar features of sanskrit from a good sanskrit grammar book, for example, @xcite , extrapolating them for complete coverage of all variations, and providing english translations. in some cases, the sentences may have to be hand-crafted as we have done here. since we are looking at only a differential corpus, we estimate the number of test groups to be of the order of approximately a thousand in number, which is feasible to do manually in a reasonable time frame. we believe a "differential" corrective translation data suite based on this model will allow most of the gaps in google translate's zero-shot nmt for sanskrit-english to be addressed, leading to a more robust and usable system. ## contributions and scope for future work this work has made the following contributions: