Neural Machine Translation for M alayalam Paraphrase Generation (2024)

Abstract: This study explores four methods of generating paraphrases in Malayalam, utilizing resources available for English paraphrasing and pre-trained Neural Machine Translation (NMT) models. We evaluate the resulting paraphrases using both automated metrics, such as BLEU, METEOR, and cosine similarity, as well as human annotation. Our findings suggest that automated evaluation measures may not be fully appropriate for Malayalam, as they do not consistently align with human judgment. This discrepancy underscores the need for more nuanced paraphrase evaluation approaches especially for highly agglutinative languages.

## introduction paraphrase generation is the task of rephrasing a given text while retaining its original meaning. paraphrase generation has attracted considerable attention in natural language processing (nlp) and computational linguistics. alternatively, paraphrasing can be defined as rewriting a sentence in a different form without losing its semantic information. thus automated paraphrasing is an essential component of any successful nlp system. for example, paraphrasing is essential for an nlp system to pass the turing test. there are several notable ideas to generate paraphrases that are relevant in the context of this paper. first, the idea to use machine translation-inspired solutions for paraphrasing dates back to @xcite who developed monolingual machine translation for paraphrase generation. second, the idea of context-aware statistical paraphrase, that, to our knowledge, was first introduced in @xcite . recently, @xcite presented an approach that integrated deep reinforcement learning to ob-tain automated paraphrases. @xcite showed how deep generative networks could be used for paraphrase generation. whereas @xcite used transformers for paraphrase generation. @xcite provide a more detailed view of paraphrase generation. in the context of linguistic diversity, the focus on paraphrasing extends beyond widely spoken languages to also include regional languages with rich linguistic nuances. @xcite addressed the challenge of dialectal to standard arabic paraphrasing to enhance arabic-english statistical machine translation. their work signifies a critical effort to improve translation accuracy and fluency across different arabic linguistic variants. additionally, @xcite made a substantial contribution by creating a free, generaldomain paraphrase database for the japanese language. furthermore, @xcite explores the enhancement of english-to-chinese neural machine translation through paraphrase-based data augmentation. this work addresses paraphrase generation in malayalam. malayalam is a dravidian language spoken predominantly in kerala. it is also spoken in mahe and lakshadweep of india, altogether resulting in a population of about 34 million. malayalam is known for its complicated grammatical structures, complex verb conjugations, and extensive vocabulary. there are several research projects addressing paraphrases identification malayalam language and recognizing sentence similarities, see @xcite and @xcite . recently, (k. nambiar et al., 2023) provided a malayalam model for machine translation, text summarization, and question-answering. as an extension of the above-mentioned stud-ies, this paper aims to address the complex area of malayalam paraphrase generation. our investigation focuses on developing a specialized dataset tailored to malayalam paraphrases, leveraging insights from established paraphrase generation models. the main motivation behind this research is to address the lack of resources for non-english languages and to improve the capabilities of nlp systems in the context of languages with special linguistic features. ## related works a seminal work by @xcite emphasizes the importance of developing effective models for paraphrase generation, considering the varying syntactic and semantic expressions across different languages. these challenges become more pronounced in highly agglutinative languages, where words can be formed by stringing together multiple morphemes, adding an additional layer of complexity to the generation process. extending paraphrase generation to malayalam, a language with a complex linguistic structure, demands special attention. scientific papers on multilingual nlp, such as the work by @xcite , emphasize the need for language-specific adaptations in paraphrase generation models. the authors discuss the impact of linguistic diversity on the performance of nlp models, underscoring the importance of addressing language-specific challenges. @xcite propose four similarity measures to predict the similarity between two sentences in malayalam. those are cosine similarity, jaccard similarity, overlap coefficient, and containment measure. a shared task on detecting paraphrases in indian languages, namely, hindi, tamil, malayalam, and punjabi are proposed by anand @xcite . it consisted of two subtasks: subtask 1 is to determine whether a sentence pair is a paraphrase or not, and subtask 2 is to determine whether a sentence pair is a semi-paraphrase or a paraphrase or not. different members use different features such as stop words, lemmatization, pos tagging, synonyms, overlap, cosine similarity, jaccard similarity, etc. due to the complexity of sentences, the f1 score and accuracy of task 1 are comparatively high compared to the accuracy of task 2. they concluded that the agglutinative character of malayalam and tamil makes paraphrasing more challenging. ## data this paper uses the gyafc dataset @xcite in english as a start for the paraphrasing pipeline. this dataset consists of informal and formal sentence pairs which are built using the yahoo answers l6 corpus. the sentences in this dataset are obtained from various domains including entertainment, music, family, relationships, etc. around 1000 english sentence pairs are available in this dataset. though we explore the possibility of adopting english datasets for malayalam paraphrasing, we also provide a sample of 800 malayalam paraphrase pairs evaluated by crowd workers with overlap of five foot_0 . the details on datalabelling are provided in section 5. ## methods we try to explore four approaches that could potentially leverage the knowledge that we have for english and transfer it into malayalam paraphrase. the first approach simply uses google translate on a random sample of 200 gyafc paraphrases. we evaluate all four approaches on random 200 gyafc sentence pairs. the first model combines the output of google translate with multiindic paraphrase generation, a pre-trained model for paraphrase generation @xcite . a prior study by @xcite served as the foundation for multiindic paraphrase generation, which extracts paraphrases from a parallel corpus. the model is developed using the samanantar corpus @xcite , which contains parallel corpora between english and all 11 indic languages. 200 pairs of english phrases from the gyafc dataset are translated into malayalam using google translate. these translated malayalam sentences are then fed into the multiindic paraphrase generation to obtain desired malayalam paraphrase pairs. an illustrative example pertaining to this model can be found in figure 1 . in the second approach, we use a set of english synonym word pairs foot_1 to generate paraphrases in english with a simple synonym replacement heuristic approach to paraphrase. the generated paraphrases are then translated into malayalam using google translate to obtain the malayalam paraphrase set. in the third approach, we use the bart-large-cnn model @xcite . figure 3 contains an exemplar related to this model, offering additional clarity. finally, in the fourth model a pre-existing language translation model named, opus(open parallel corpus) tiedemann (2012). opus models are a collection of pre-trained multilingual machine translation models developed by the helsinki nlp group. opus models are designed to handle translation tasks in several languages. they are trained to support translation between different language pairs, making them versatile for multilingual applications. once again 200 pairs of sentences from the gyafc dataset are passed to this model and malayalam sentence pairs are generated. these translated sentences are then paraphrased by adjusting the beam-search parameters. figure 4 includes 3 the self-reported evaluation metric. an example associated with this model, providing supplementary clarity. the num_beams parameter controls the number of beams to use in beam search. beam search is a decoding algorithm that explores multiple possible sequences and selects the most likely ones. a larger num_beams value can increase diversity in generating phrases. additionally, the num_return_sequences parameter determines how many different sequences to return. a higher value will result in more diverse paraphrases. moreover, early_stopping is used to speed up the paraphrase generation process. these parameters collectively influence the diversity, quality, and speed of paraphrase generation. finally, we compare these paraphrase methods based on nmt with the paraphrase proposed for malayalam in anand @xcite . ## evaluation yamshchikov et al. ( 2021 ) have explored various metrics for the evaluation of paraphrases. they found bertscore @xcite to be the most adequate metric for english paraphrases. however, there is no direct analogy of bertscore for malayalam and the most commonly used metrics do correlate with human judgment @xcite on par with bertscore though not perfectly. thus, in this work, we calculate the bleu score @xcite and meteor score @xcite for evaluating the phrases generated for a reference sentence. we also use cosine similarity used for paraphrase evaluation by anand @xcite to put our results in perspective, despite cosine similarity was found to have a lower correlation with the human evaluation of paraphrases. finally, we have labelled 200 paraphrase pairs generated by each of the models with human labellers via crowd-sourcing platform. each sentence was labelled with three or more native speakers of malayalam. we measured a percentage of sentence pairs that were labelled as correct model bleu meteor cosine human similarity labels multiindic paraphrase @xcite 0.04 0.25 0.70 0.37 synonym replacement 0.05 0.28 0.60 0.42 bart @xcite 0.20 0.31 0.96 0.31 opus @xcite 0 paraphrases with high confidence. we publish the resulting human-labelled dataset of 800 sentence pairs to facilitate further research of paraphrasing in malayalam. one can see that the opus model outperforms other models in terms of automated evaluation metrics. in the meantime, the paraphrases generated with multiindic paraphrase generation, specifically designed for indian languages, show lower results on automated evaluation. comparison of the proposed methods with the best malayalam paraphrasing model described in anand @xcite also shows that on automated paraphrase evaluation metrics, direct application of machine translation methods, namely, bart or opus, leads to results that score higher in terms of bleu, meteor, and cosine similarity. however, this does not necessarily point at the weakness of the models but rather highlights the inadequacy of those popular evaluation metrics for malayalam paraphrasing as well as the opportunity to leverage nmt to significantly expand the capabilities of malayalam nlp. once we include human evaluation into the picture we see two crucial results. first, the most successful paraphrases, according to human judgement, as simply achieved by heuristic synonym replacement. this is not surprising. what is important is that humans also evaluate multiindic paraphrase higher that bart or opus, despite those models higher scores on automated metrics. ## discussion in this study, we check if one could use machine translation methods for paraphrasing in malayalam. we test several methods of generating paraphrases in english, followed by their translation into malayalam. this methodology was compared with the performance of malayalam-specific paraphrase models. our findings reveal that using english for initial paraphrase generation and then translating to malayalam can yield results that are on par with those from malayalam-specific models. this has several important implications: at the same time, one has to highlight certain limitations: ## conclusion this study evaluates how effective is the idea to apply the existing neural machine translation methods to paraphrase generation in malayalam. the core finding of this paper is that the models specifically designed for agglutinative languages like malayalam are showing performance on par with nmt machine translation pipelines that leverage available english resources. the study also highlights the demand for specific paraphrase evaluation metrics more suitable for dravidian languages. finally, we publish human-labelled dataset of paraphrases to facilitate further research on the topic.