Zero-shot Cross-lingual POS Tagging for F ilipino (2024)

Abstract: Supervised learning approaches in NLP, exemplified by POS tagging, rely heavily on the presence of large amounts of annotated data. However, acquiring such data often requires significant amount of resources and incurs high costs. In this work, we explore zero-shot cross-lingual transfer learning to address data scarcity issues in Filipino POS tagging, particularly focusing on optimizing source language selection. Our zero-shot approach demonstrates superior performance compared to previous studies, with top-performing fine-tuned PLMs achieving F1 scores as high as 79.10%. The analysis reveals moderate correlations between cross-lingual transfer performance and specific linguistic distances–featural, inventory, and syntactic–suggesting that source languages with these features closer to Filipino provide better results. We identify tokenizer optimization as a key challenge, as PLM tokenization sometimes fails to align with meaningful representations, thus hindering POS tagging performance.

## introduction the rise of pretrained language models (plms) has revolutionized the landscape of natural language processing (nlp). while these models demonstrably address data scarcity in under-resource languages by learning universal language representations @xcite , many languages, including filipino, a widely spoken under-resource language in the philippines @xcite , continue to face significant challenges. building robust nlp pipelines for filipino remains difficult despite the abundance of textual resources like literary works, linguistic references, and social media data. filipino lacks dedicated resources for a range of language processing tasks @xcite @xcite . robust and reliable part-of-speech (pos) taggers could significantly improve the performance of such tasks by accurately classifying words into their grammatical categories. this disambiguation is essential because many words can have multiple meanings based on context. for example, the filipino word "buhay" can be a "pangngalan" (noun) meaning "life" or a "pang-uri" (adjective) meaning "lively" or "vibrant." by clearing up word confusion, pos tagging helps in performing higher-level nlp tasks such as machine translation, information extraction, text-to-speech conversion, speech recognition, etc. however, annotating datasets for pos tagging is complex and resource-intensive. one potential solution is cross-lingual transfer learning, which involves using the knowledge gained from training a model in one language to address tasks in another language @xcite . in this paradigm, a language model acquires representations from a source language and then undergoes fine-tuning to execute tasks in a target language with limited labeled data. furthermore, zero-shot learning, a specific form of cross-lingual transfer learning, presents a solution in scenarios with a complete absence of annotated data @xcite . one crucial factor in enhancing zero-shot crosslingual transfer learning is the selection of the source language. this selection process involves identifying and analyzing language similarity metrics that can improve the success of cross-lingual transfer learning @xcite . these metrics quantify and compare linguistic and structural correspondences between languages. linguists often use intuitive notions of structure to compare languages @xcite , and source language selection tends to follow similar intuitive approaches. however, quantified language similarity metrics provide a more objective basis for these comparisons, suggesting that higher similarity between a source-target language pair generally results in improved cross-lingual transfer learning performance. the challenge, however, lies in selecting the most appropriate similarity metric, given the wide array of available options. identifying which metrics are most indicative of successful cross-lingual transfer learning could streamline the source language selection process, thereby enhancing adaptability for under-resource languages such as filipino. prior studies have explored the impact of several linguistic features on cross-lingual transfer performance. one study emphasized the correlation between linguistic similarity and transfer performance, advocating for selecting source languages based on rigorous linguistic assessments rather than defaulting to english @xcite . in contrast, another study proposed exploring syntactic and morphological similarities across languages to improve model transfer capabilities @xcite . additionally, another study emphasized the importance of including linguistically similar languages in pre-training for improved transfer learning outcomes @xcite . our paper extends this line of research by examining linguistic similarity distances between filipino and source languages and within the context of zero-shot learning for pos tagging. more specifically, we examined how measures of linguistic distances across multiple dimensions contributed to the effectiveness of pos tagging. while a study @xcite investigated this aspect for the natural language inference (nli) task across all 15 languages in the xnli dataset @xcite individually, our focus is on pos tagging and filipino as the target language. futhermore, we investigated how the choice of plm influenced the outcome and effectiveness of source language selection. we also explored which source language and combination of source languages yielded the highest f1 scores for filipino pos tagging. ## language similarity lang2vec @xcite is a versatile tool for linguistic analysis that provides readily available pre-computed distances between languages represented as vectors of featural, syntactic, geographic, inventory, genetic, and phonological dimensions from multiple databases including the world atlas of language structures (wals) @xcite , syntactic structures of world languages (sswl) @xcite , phoible @xcite , glottolog @xcite tree of language fam-ilies, and ethnologue @xcite . these dimensions enable comparisons of various linguistic features across different languages. understanding cross-lingual transfer performance in filipino pos tagging will benefit an investigation of language similarity metrics. ## methods we used a selection of plms, including xlm-r @xcite , a multilingual variant of the roberta model, and roberta-tagalog (cruz and cheng, 2021), a roberta model pretrained using a filipino-language pretraining corpus. in this study, both models were finetuned and assessed in a zero-shot cross-lingual scenario, tasked with performing pos tagging for filipino texts using their base configurations. xlm-r was selected for its well-established performance in multilingual contexts and its robustness in handling large-scale text datasets across various sequence-labeling tasks @xcite . roberta-tagalog, on the other hand, was chosen because it is an improvement over the previous tagalog pretrained transformer models (cruz and cheng, 2021). ## results the results of the top-performing finetuned plms outperform all previously presented zero-shot learning methods listed in in contrast, despite these similarities, descriptive observa-tions alone are insufficient to suggest a meaningful structural connection between filipino and the source languages. the similarities are also not easily generalizable with the other top-performing source languages. therefore, an examination of quantitative linguistic distances is crucial for optimal source language selection. ## conclusion this study implements zero-shot fine-tuning using plms for filipino pos tagging, exploring the role of linguistic distances in source language selection. correlation analysis between linguistic similarity distances and plm performance suggests that featural, inventory, and syntactic distances between source languages and filipino, impact cross-lingual transfer learning outcomes. the study also explored the role of plm selection in influencing cross-lingual transfer learning performance. while roberta-tagalog is specifically designed for tagalog, the multilingual language model xlm-r outperformed it. furthermore, the exploration of a multilingual source language approach shows good results, though slightly lower than monolingual fine-tuning, suggesting potential benefits of using multiple languages simultaneously for cross-lingual transfer learning tasks. despite promising results, challenges in tokenization were observed, particularly in accurately tokenizing filipino text. errors in tokenization underscore the need for improved tokenization processes for plms, especially for under-resourced languages like filipino. future research should address these challenges by creating new treebanks and expanding existing ones to further enhance model performance. using top-performing models from this study to annotate unannotated datasets can serve as a foundation for future researches. these annotations, once manually refined, can produce gold-standard annotations for improved training and evaluation of nlp models.