Cross-Lingual Cross-Target Stance Detection with Dual Knowledge Distillation Framework (2023)

Abstract: Stance detection aims to identify the user’s attitude toward specific targets from text, which is an important research area in text mining and benefits a variety of application domains. Existing studies on stance detection were conducted mainly in English. Due to the low-resource problem in most non-English languages, cross-lingual stance detection was proposed to transfer knowledge from high-resource (source) language to low-resource (target) language. However, previous research has ignored the practical issue of no labeled training data available in target language. Moreover, target inconsistency in cross-lingual stance detection brings about the additional issue of unseen targets in target language, which in essence requires the transfer of both language and target-oriented knowledge from source to target language. To tackle these challenging issues, in this paper, we propose the new task of cross-lingual cross-target stance detection and develop the first computational work with dual knowledge distillation. Our proposed framework designs a cross-lingual teacher and a cross-target teacher using the source language data and a dual distillation process that transfers the two types of knowledge to target language. To bridge the target discrepancy between languages, cross-target teacher mines target category information and generalizes it to the unseen targets in target language via category-oriented learning. Experimental results on multilingual stance datasets demonstrate the effectiveness of our method compared to the competitive baselines.

## introduction stance detection aims to automatically identify the user's attitude (e.g., "in favor of" or "against") towards specific targets (e.g., entities, topics or claims) @xcite , which is an important research area in text mining and social me-dia analytics. it has been applied to diverse domains, such as public attitude mining, market analysis @xcite , veracity prediction @xcite and many others. existing work on stance detection has mainly focused on monolingual setting and been conducted on english datasets @xcite @xcite @xcite . in contrast to english, most other languages lack abundant annotated data for training quality stance detection models. to address the low-resource problem and improve model performance, cross-lingual stance detection has been proposed to transfer knowledge from highresource (source) language to low-resource (target) language. @xcite propose a contrastive language adaptation method to align representations in two languages, which relies on the labeled training data in target language. @xcite transfer the knowledge from english to target languages through pre-training and prompttuning for few-shot (and zero-shot) cross-lingual stance detection @xcite with few (and no) training data in target languages. none of these works has addressed the practical issue when there are only unlabeled training data available in target language, which is a prevalent phenomenon in many low-resource languages. consequently, the typical method based on supervised contrastive learning @xcite is not workable due to the lack of supervised signals for target language, and pre-training & prompttuning method @xcite cannot fully utilize the unlabeled training data in target languages as well. since labeled data are extremely scarce in low-resource languages, these unlabeled data are the valuable resource of target domain that can be utilized to facilitate various tasks in general, and in particular, we utilize them to help bridge cross-lingual gap in our work. in addition, cross-lingual stance detection usu-ally brings about target inconsistency problem, that is, the occurrences and distributions of the concerned targets may vary considerably across languages due to the differences in socio-cultural backgrounds and linguistic expressions. without the labeled data, target inconsistency causes the additional issue of unseen targets in target language, which in essence requires the transfer of both language-related and target-oriented knowledge from source to target language. to address the above issues, we consider crosslingual distillation @xcite as a good fit to train a teacher model on the source language data so as to acquire the pseudo supervised signals for a student model trained on the unlabeled target language data. since the language and targetoriented knowledge transferring from source to target language is of different types, dual teacher models are required for our purpose. previously, multiteacher distillation in language processing @xcite was typically used to enhance the transfer of the same type of knowledge. in contrast, our aim is to transfer different types of cross-language and cross-target knowledge, which needs to consider not only the design of individual teacher models but also their combination scheme. in particular, cross-target teacher model is built on the basis of cross-lingual teacher model, and relies on the finegrained target knowledge, in which semantically correlated targets can be aggregated to indicate category information and utilized to generalize for the unseen targets. in this paper, we propose the new task of crosslingual cross-target stance detection, which is fundamentally different from both cross-target stance detection task in monolingual setting and existing cross-lingual stance detection task. we develop a cross-lingual cross-target stance detection (ccsd) method via designing a dual knowledge distillation framework for this task, which trains a cross-lingual teacher and a crosstarget teacher on the source language data and distills the two types of knowledge to the student model. to bridge target inconsistency between languages, cross-target teacher first learns correlated target representations to mine the category information and fuses the aggregated semantic representations for refinement. it then devises categoryoriented contrastive learning to enhance the generalization ability of the cross-target model. crosslingual teacher is a multilingual language model which is prompt-tuned with cross-lingual templates. to reduce the impact of language differences on cross-target knowledge distillation, cross-lingual teacher model functions as the initialized encoder for cross-target teacher. finally, in the dual distillation process, cross-lingual teacher and cross-target teacher jointly produce the pseudo-labels for the student model, and distill the two types of knowledge to the unlabeled target language data with varying degrees of target inconsistency control. the contributions of our work are as follows: ## related work monolingual stance detection stance detection has been well studied on english datasets @xcite @xcite @xcite . the mainstream research is mainly as follows: (1) stance detection for pre-defined targets, which trains a classifier for several predefined targets. previous methods mainly learn target-specific representations with attention @xcite @xcite @xcite . there is also other work that conducts multidataset learning @xcite for cross-domain stance detection. ## proposed method figure 1 shows the overall structure of the proposed cross-lingual cross-target stance detection method ccsd. we first train a cross-lingual teacher and a cross-target teacher on the source language data, and then distill the learned knowledge to the student model trained with unlabeled data in target language. cross-lingual teacher is a multilingual pre-trained language model which is prompt-tuned with cross-lingual templates and consistency constraints to enhance cross-lingual ability solely with the source language data. cross-target teacher learns target representations to mine the target category with highly-correlated targets, and then generalizes the target category to the unseen targets in target language through contrastive learning. ## conclusion to address target inconsistency in low-resource cross-lingual stance detection, we propose the new task of cross-lingual cross-target stance detection and develop a dual knowledge distillation framework ccsd to tackle the challenging issues of unlabeled data and unseen targets in target language. in our dual framework, a cross-lingual teacher and a cross-target teacher are trained on the source language data and distill respective knowledge to the student model trained with the unlabeled target language data. to bridge the target gap between source and target languages, cross-target teacher further mines the semantically correlated target categories, and generalizes this information to the unseen targets in target language. experimental results on four cross-lingual datasets under varying target settings demonstrate the effectiveness of our method for cross-lingual cross-target stance detection.