P ersian Natural Language Inference: A Meta-learning Approach (2022)

Abstract: Incorporating information from other languages can improve the results of tasks in low-resource languages. A powerful method of building functional natural language processing systems for low-resource languages is to combine multilingual pre-trained representations with cross-lingual transfer learning. In general, however, shared representations are learned separately, either across tasks or across languages. This paper proposes a meta-learning approach for inferring natural language in Persian. Alternately, meta-learning uses different task information (such as QA in Persian) or other language information (such as natural language inference in English). Also, we investigate the role of task augmentation strategy for forming additional high-quality tasks. We evaluate the proposed method using four languages and an auxiliary task. Compared to the baseline approach, the proposed model consistently outperforms it, improving accuracy by roughly six percent. We also examine the effect of finding appropriate initial parameters using zero-shot evaluation and CCA similarity.

## introduction in natural language processing (nlp), the goal is to improve models for the processing and production of human languages. as part of nlp, several tasks are defined, each covering a different level of natural language understanding. meanwhile, natural language inference (nli) is considered an appropriate and rigorous measure of language comprehension. this task requires recognizing the consequences of natural language sentences, which indicates how well it understands the language @xcite . nli aims to determine the inferential relationship between a premise p and a hypothesis h. the * equal contribution problem involves a three-class classification in which every pair (p, h) falls into one of three categories: entailment, contradiction, and neutral. if the hypothesis can be inferred from the premise, pair (p, h) will be assigned to the entailment class. for a hypothesis that contradicts the premise, pair (p, h) will be assigned to the contradiction and neutral otherwise @xcite . the persian language lacks sufficient linguistic resources when it comes to natural language understanding. the lack of data can be addressed by collecting annotated data, but this process is both time-consuming and expensive @xcite . farstail @xcite is currently available for persian, which is created using the same method as scitail @xcite . it contains 10,367 samples. also, parsinlu @xcite is created for high-level tasks in persian, and for nli, it consists of 2700 samples. as it turns out, this amount of data is too small compared with resourcerich languages (such as english, which has only 550,000 samples in the snli @xcite . researchers have tried to solve the data scarcity problem by using cross-language methods. recent work on cross-lingual learning has mainly focused on transfer between languages already covered by pre-trained representations @xcite . nonetheless, these techniques do not readily transfer to low-resource languages in which (1) large monolingual corpora are unavailable for pretraining, and (2) sufficient labeled data is lacking for fine-tuning downstream tasks @xcite . the results of experimental studies for persian using different embedding methods including word2vec @xcite , fasttext @xcite , elmo @xcite , and bert @xcite and various models, such as, decompatt @xcite , esim @xcite , hbmp @xcite , and ulmfit @xcite is reported in farstail @xcite . although this cross-lingual information sharing has enabled success in various natural language processing tasks, it raises the question of how we can achieve more effective collaborative learning between languages or even between different tasks. recently, meta-learning has shown to be effective for a variety of machine learning tasks, including nlp @xcite @xcite . this paper uses a metalearning-based method for learning parameters in the joint space of tasks and languages. auxiliary languages include english, spanish, french, and german, while qa is the auxiliary task. alternatively, an essential prerequisite for the successful application of meta-learning is a task distribution from which a large number of tasks can be sampled to train the meta-learner. however, in nlp, datasets are usually considered as tasks @xcite . there are two main problems with treating entire datasets as tasks. the first problem is overfitting, in which a meta-learner is overfitted to a small number of training tasks since there is only a small number of supervised datasets for each nlp problem. a second concern is that the heterogeneity of nlp datasets may result in learning episodes that lead to memorization overfitting, where a meta-learner ignores the support set and fails to adapt @xcite . to improve the quality and quantity of tasks, we use the dreca @xcite approach as our data augmentation strategy. in this paper, we employ meta-learning algorithms to enhance the persian nli task. our models are evaluated on the farstail dataset. experimental results show that we push persian nli accuracy forward by more than 6% and zero-shot accuracy by about 4%, setting a new state-of-the-art result for this task. in summary, the main contributions of our research are: the rest of the paper is arranged as follows: section 2 briefly describes related work. section 3 introduces our method, and in section 4, we explain the details of the experimental setup. section 5 presents practical results. the results analysis and some justification are described in section 6. we conclude the paper and summarize future directions in section 7. ## related work in this section, we briefly outline related work in three areas. the first area is models based on crosslingual algorithms. in the second area, we highlight methods based on meta-learning. finally, we summarize existing data augmentation strategies. ## the proposed methodology in our setting, firstly, we prepare a set of tasklanguage pairs to provide meta-learning tasks. afterward, we sample some tasks in each episode and feed them to the meta-learner. in the rest of this section, we describe the proposed task sampling strategy, the proposed meta-learning algorithm, and the proposed task augmentation strategy. ## results the meta-learning model is tested on different combinations and configurations of the auxiliary tasks. the accuracy results of the reptile algorithm are presented in additionally, we conducted zero-shot evalua-tions of both algorithms. zero-shot results are presented in the first accuracy column of tables 2 and 3 . the confusion matrices of the best-performing models for both reptile and prototypical algorithms are also depicted in appendix a.4. ## discussion and analysis as part of another experiment, we combined data from the target language with data from other auxiliary tasks for meta-training. based on the results of these experiments (last column of as indicated in the last two columns of we examined the proximity between the feature spaces of the auxiliary languages and the target language quantitatively and qualitatively. at first, we collect representations of the auxiliary and target languages from non-episodic, reptile, and prototypical models. in fig. 1 , we present a 2component pca visualization for comparison. we also evaluated the models using a distance metric compared to the non-episodic method, we observe a drastic drop of hausdorff distance from 0.18 to 0.05 for prototypical, and also we see a minor decline of hausdorff distance from 0.18 to 0.13 for the reptile. both qualitative visualization and quantitative metrics confirm that meta-learning approaches bring the distributions of auxiliary and target language data closer together, thus increasing the accuracy on the target language. the advantage of meta-learning methods is that they obtain the appropriate initial parameters for the target language, as mentioned. the zero-shot test is used as a criterion to evaluate this point, and it shows that meta-learning-based models are more accurate than other methods. the generality of the initial parameters can also be assessed via canonical correlation analysis (cca) @xcite . using this criterion, we compare the output of each layer before and after fine-tuning, and the results are presented in fig. 2 . the meta-learning models have a higher cca similarity, which indicates the model obtained more general parameters before fine-tuning. in the next experiment, we apply the dreca strategy and train the model with the prototypical algorithm. dreca is a task augmentation strategy used when a small number of auxiliary tasks are available. the identification of clusters requires a more detailed examination of the method. however, in this paper, we are not interested in obtaining meaningful clusters but in generating more auxiliary tasks. so we used the dreca strategy for the best results obtained with the prototypical method, hoping to get more improvements. according to ## conclusion we present effective use of meta-learning to benefit from other tasks or languages. we advantageously leverage this approach to improve nli in persian as a low-resource language. we found that our meta-learning model outperformed competitive baseline models. in response to the concept of treating entire datasets as tasks, we use dreca as a general-purpose task augmenting approach. finally, zero-shot evaluations illustrate the generality of the results obtained by meta-learning. this work will be extended to other cross-lingual nlp tasks in persian in the future. furthermore, we would like to use a self-supervised approach to provide a useful starting point for parameters.