Cross-Cultural Similarity Features for Cross-Lingual Transfer Learning of Pragmatically Motivated Tasks (2021)

Abstract: Much work in cross-lingual transfer learning explored how to select better transfer languages for multilingual tasks, primarily focusing on typological and genealogical similarities between languages. We hypothesize that these measures of linguistic proximity are not enough when working with pragmatically-motivated tasks, such as sentiment analysis. As an alternative, we introduce three linguistic features that capture cross-cultural similarities that manifest in linguistic patterns and quantify distinct aspects of language pragmatics: language context-level, figurative language, and the lexification of emotion concepts. Our analyses show that the proposed pragmatic features do capture cross-cultural similarities and align well with existing work in sociolinguistics and linguistic anthropology. We further corroborate the effectiveness of pragmatically-driven transfer in the downstream task of choosing transfer languages for cross-lingual sentiment analysis.

## pragmatically-motivated features we propose three language-level features that quantify the cultural similarities across languages. language context-level ratio a language's context-level reflects the extent to which the language leaves the identity of entities and predicates to context. for example, an english sentence did you eat lunch? explicitly indicates the pronoun you, whereas the equivalent korean sentence ᄌ ᅥ ᆷᄉ ᅵ ᆷ ᄆ ᅥ ᆨᄋ ᅥ ᆻᄂ ᅵ? (= did eat lunch?) omits the pronoun. context-level is considered one of the distinctive attributes of a language's pragmatics in linguistics and communication studies, and if two languages have similar levels of context, their speakers are more likely to be from similar cultures @xcite . the language context-level ratio (lcr) feature approximates this linguistic quality. we compute the pronoun-and verb-token ratio, ptr(l k ) and vtr(l k ) for each language l k , using part-of-speech tagging results. we first run language-specific postaggers over a large mono-lingual corpus for each language. next, we compute ptr as the ratio of count of pronouns in the corpus to the count of all tokens. vtr is obtained likewise with verb tokens. low ptr, vtr values may indicate that a language leaves the identity of entities and predicates, respectively, to context. we then compare these values between the target language l tg and transfer language l tf , which leads to the following definition of lcr: literal translation quality similar cultures tend to share similar figurative expressions, including idiomatic multiword expressions (mwes) and metaphors @xcite . for example, like father like son in english can be translated word-by-word into a similar idiom tel père tel fils in french. however, in japanese, a similar idiom 蛙の子は蛙 (kaeru no ko wa kaeru) "a frog's child is a frog." cannot be literally translated. literal translation quality (ltq) feature quantifies how well a given language pair's mwes are preserved in literal (word-by-word) translation, using a bilingual dictionary. a well-curated list of mwes is not available for the majority of languages. we thus follow an automatic extraction approach of mwes @xcite . first, a variant of pointwise mutual information, pmi foot_2 @xcite is used to extract noisy lists of top-scoring n-grams from two large monolingual corpora from different domains, and intersecting the lists filters out domain-specific n-grams and retains the language-specific top-k mwes. then, a bilingual dictionary between l tf and l tg and a parallel corpus between the pair are used. 3 for each n-gram in l tg 's mwes, we search for its literal translations extracted using the dictionary in parallel sentences containing the n-gram. for any word in the n-gram, if there is a translation in the parallel sentence, we consider this as hit, otherwise as miss. and we calculate hit ratio as hit (hit+miss) for each n-gram found in the parallel corpus. finally, we average the hit ratios of all n-grams and z-normalize over the transfer languages to obtain ltq(l tf , l tg ). emotion semantics distance emotion semantic distance (esd) measures how similarly emotions are lexicalized across languages. this is inspired by @xcite who used colexification patterns (i.e., when different concepts are expressed using the same lexical item) to capture the semantic similarity of languages. however, colexification patterns require human annotation, and existing annotations may not be comprehensive. we extend @xcite 's method by using cross-lingual word embeddings. we define esd as the average distance of emotion word vectors in transfer and target languages, after aligning word embeddings into the same space. more specifically, we use 24 emotion concepts defined in @xcite and use bilingual dictionaries to expand each concept into every other language (e.g., love and proud to liebe and stolz in german). we then remove the emotion word pairs from the bilingual dictionaries, and use the remaining pairs to align word embeddings of source into the space of target languages. we hypothesize that if words correspond to the same emotion concept in different languages (e.g., proud and stolz) have similar meaning, they should be aligned to the same point despite the lack of supervision. however, because each language possesses different emotion semantics, emotions are scattered into different positions. we thus define esd as the average cosine distance between languages: ## feature analysis in this section, we evaluate the proposed pragmatically-motivated features intrinsically. throughout the analyses, we use 16 languages listed in figure 4 which are later used for extrinsic evaluation ( §5). ## extrinsic evaluation: ranking transfer languages to demonstrate the utility of our features, we apply them to a transfer language ranking task for cross-lingual transfer learning. we first present the overall task setting, including the datasets and models used for the two cross-lingual tasks. next, we describe the transfer language ranking model and its evaluation metrics. ## related work quantifying cross-cultural similarity a few recent work in psycholinguistics and nlp have aimed to measure cultural differences, mainly from word-level semantics. @xcite suggested a cross-lingual word alignment method that preserves the cultural, social context of words. they derive cross-cultural similarity from the embeddings of a bilingual lexicon in the shared representation space. @xcite computed sim-ilarity by comparing the nearest neighborhood of words in different languages, showing that words in some domains (e.g., time, quantity) exhibit higher cross-lingual alignment than other domains (e.g., politics, food, emotions). jackson et al. (2019) represented each language as a network of emotion concepts derived from their colexification patterns and measured the similarity between networks. ## future directions typology of cross-cultural pragmatics the features proposed here provide three dimensions in a provisional quantitative cross-linguistic typology of pragmatics in language. having been validated, both intrinsically and extrinsically, they can be used in studies as a stand-in for cross-cultural similarity. they also open a new avenue of research, raising questions about what other quantitative features of language are correlates of cultural and pragmatic difference. model probing fine-tuning pretrained models to downstream tasks has become the de facto standard in various nlp tasks, and their success has promoted the development of their multilingual extensions @xcite . while the performance gains from these models are undeniable, their learning dynamics remain obscure. this issue has prompted various probing methods designed to test what kind of linguistic information the models retain, including syntactic and semantic knowledge @xcite @xcite . similarly, our features can be employed as a touchstone to evaluate a model's knowledge in cross-cultural pragmatics. investigating how different pretraining tasks affect the learning of pragmatic knowledge will also be an interesting direction of research. ## conclusion in this work, we propose three pragmaticallyinspired features that capture cross-cultural similarities that arise as linguistic patterns: language context-level ratio, literal translation quality, and emotion semantic distance. through feature analyses, we examine whether our features can operate as valid proxies of cross-cultural similarity. from a practical standpoint, the experimental results show that our features can help select the best transfer language for cross-lingual transfer in pragmaticallydriven tasks, such as sentiment analysis.