S iamese Networks for Inference in M alayalam Language Texts (2021)

Abstract: Natural language inference is a method of finding inferences in language texts. Understanding the meaning of a sentence and its inference is essential in many language processing applications. In this context, we consider the inference problem for a Dravidian language, Malayalam. Siamese networks train the text hypothesis pairs with word embeddings and language agnostic embeddings, and the results are evaluated against classification metrics for binary classification into entailment and contradiction classes. XLM-R embeddings based Siamese architecture using gated recurrent units and bidirectional long short term memory networks provide promising results for this classification problem.

## introduction textual entailment is a uni-directional relationship between two text pairs. it is the method of identifying the meaning from two sentences. sentence pairs are labeled as entailment pairs if a sentence is inferred from the context of the other sentence in the pair. it is defined in different ways as classical definition, applied definition, and mathematical definition @xcite . in the classical definition, a text entails a hypothesis if the hypothesis is valid in all circumstances where the text is true. in applied definition, text entails hypothesis if the hypothesis is mostly true when a human reads it. the mathematical definition @xcite is a text that entails a hypothesis if the probability of the hypothesis is true given the text is greater than the likelihood of the hypothesis being true. the sentence from which we derive the information is called text, and the sentence which identifies its information as derived from text is called a hypothesis. sentence pairs are named as entailed if the hypothesis has its meaning derived from the text. contradictory pairs indicate that the information from the hypothesis is contradictory concerning the information conveyed in the text. nowadays, textual entailment has become part of the primary natural language processing tasks. it is also called natural language inference. the terms text and premise are used interchangeably. the performance of the latest transformer-based approaches is evaluated in nlp tasks like entailment recognition, semantic textual similarity, and paraphrase detection. hence, textual entailment recognition has also become an evaluation criterion for many nlp tasks. it is also a necessary sub-task in applications like multi-document summarization, information retrieval, information extraction, and question answering systems. many text entailment or natural language inference related works in english use different sized datasets but not in the malayalam language. malayalam is a dravidian language used in the southern part of india. it is a language that has various dialects and has many inflections. malayalam language computing is developing, with few resources, namely stemmer, pos tagger, sandhi splitter, and few datasets for paraphrasing, text classification, and sentiment analysis. it has agglutinated language structure, and new words are created through word compounding and inflection, and hence there can be many inferential compound words for a word. the main challenge of textual entailment recognition in the malayalam language is the absence of a dataset. dataset creation is a tedious task that involves high costs and time. we used an in-house dataset created by machines and humans in the loop translation of the stanford natural language inference dataset. it is a very cost-effective method of dataset creation that can be adapted to any low-resource language. this work attempts to use siamese networks with bidirectional long short term memory and gated recurrent units to understand the similarities and differences of texthypothesis pairs for classification. the remaining sections are organized as follows. section 2 briefs the works related to textual entailment recognition. section 3 details the dataset. section 4 discusses the design of the system for classification. the experimental settings, results, and discusses sentence similarity are in section 5. section 6 concludes the work. ## related works textual entailment recognition started with a recognizing textual entailment (rte)challenge in 2005 @xcite . this challenge continued for years and used different-sized datasets. as the dataset size increases, feature-based methods such as word overlap, n-gram match, set-based similarities, and syntactic similarities were replaced by machine learning and deep learning methods using different word and sentence representations. there are numerous entailment recognition systems in english and other languages like french, german, italian, spanish, and arabic. various textual entailment frameworks have also been developed in these languages, namely excite-ment open platform (eop) @xcite . eop uses edit distance and classification as its algorithms. lexical level features, syntactic and surface-level features, graph-based approaches were used to recognize entailments. the increase in datasize has helped to use machine learning and deep learning strategies for this classification. deep learning methods are widely used for textual entailment in english using different datasets. snli (stanford natural language inference) dataset is a collection of 570k sentence pairs mainly collected through amazon mechanical turk, referencing the flickr30k corpus. this dataset helped in using deep learning techniques to text entailment recognition. sentence models with the sum of words, recurrent neural networks, and long shortterm memory networks were discussed @xcite . mnli (multi-genre natural language inference) dataset is a collection of 433k sentence pairs from different written and spoken english genres. some of the genres are face-to-face, government, telephone, letters, fiction, and travel. it is an improvement from snli with a more diverse col-lection of sentence pairs, and hence its baseline performance is low compared to snli @xcite . xnli @xcite (cross-lingual natural language inference) corpus is a collection of data from mnli, derived for 15 languages, including some low resource languages like urdu and swahili. it uses a translation-based approach with multilingual sentence encoders and then aligning sentence embeddings for inference identification. except for english, entailment recognition systems exist for french, spanish, german, greek, bulgarian, russian, turkish, arabic, vietnamese, thai, chinese, hindi, swahili, and urdu. in languages like swahili, thai and urdu, transfer learning based approaches are used, which is helpful for smallsized datasets. for the malayalam language, there are works related to paraphrasing, sentiment analysis, summarization, whereas text entailment recognition is seemingly a new area for malayalam language processing. the performance of different embeddingbased approaches is one work in this language for entailment recognition @xcite . in this work, laser-based embedding representation showed improved performance results for entailment recognition for the malayalam language compared with bert and other models. bidirectional lstm based dependent reading represents text and hypothesis in encoding and inference stage @xcite . siamese network-based architecture with sentence embeddings from bert experiments in the english language @xcite . a neural model based on lstm using the word by word attention is another deep learning-based method for recognizing entailments @xcite . child-sum-tree-based inference of texts generalizes well for snli and other entailment datasets @xcite . text alignment based approaches along with machine learning are used for entailment recognition in arabic language @xcite ). another method used asymmetric word embeddings to produce similarity based word-word interactions for textual entailment @xcite . entailment recognition has been part of competition on legal information extraction / entailment, where sentence encoding and decomposable attention models perform entailment recognition in the context of legal texts @xcite . automatic translation-based approaches are used in the italian dataset, where the dataset is translated into the english language for entailment recognition @xcite . ## dataset the subset of the malayalam language inference (manli) dataset is used for binary entailment recognition. it consists of 7989 text hypothesis pairs which are labeled as entailment and contradiction. this dataset is created from stanford natural language inference (snli) dataset. the sentence pairs from the snli dataset are translated to the malayalam language with linguistic corrections from the department of linguistics, malayalam university, kerala. there are 4026 entailment pairs and 3963 contradiction pairs in this dataset. the reason for the creation of this dataset is the unavailability of entailment datasets in malayalam. languages like malayalam have many inherent linguistic properties like inflections, agglutinative nature, dialectrelated differences, and no specific word order. a sample from the dataset is shown in figure 1 . the english translation of the sample dataset is provided in premise hypothesis label two men on bicycles. people are riding bikes. ## proposed architecture the design of this system consists of neural networks of identical architecture consisting of an embedding layer and bidirectional long shortterm memory/gated recurrent unit networks. the siamese network for sentence representation takes each sentence from text-hypothesis pair to an embedding layer. this layer uses different types of embeddings, namely word2vec and laser (language agnostic sentence representations), and xlm-r. the different layers in the system are: ## conclusion in this work, we focused on the application of siamese network architecture to recognize entailment in malayalam language. the results shows adequate performance. the use of newer embedding models leads to better accuracy but the embedding dimension is a limiting factor with the network configuration. as the embedding dimension increases, the time and space complexity increases in siamese model architecture, where text and hypothesis are processed as sequences parallely. through this work, we aim to depict the performance of siamese network based entailment recognition with respect to malayalam language, which is a low resource dravidian language.