Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource P hilippine Languages (2025)

Abstract: Named-entity recognition (NER) in low-resource languages is usually tackled by finetuning very large multilingual LMs, an option that is often infeasible in memory- or latency-constrained settings. We ask whether small decoder LMs can be pretrained so that they adapt quickly and transfer zero-shot to languages unseen during pretraining. To this end we replace part of the autoregressive objective with first-order model-agnostic meta-learning (MAML). Tagalog and Cebuano are typologically similar yet structurally different in their actor/non-actor voice systems, and hence serve as a challenging test-bed. Across four model sizes (11 M – 570 M) MAML lifts zero-shot micro-F1 by 2–6 pp under head-only tuning and 1–3 pp after full tuning, while cutting convergence time by up to 8%. Gains are largest for single-token person entities that co-occur with Tagalog case particles si/ni, highlighting the importance of surface anchors.

## introduction named-entity recognition (ner) locates and categorises persons (per), organisations (org) and locations (loc) in unstructured text @xcite . it is used in a variety of important domains such as healthcare @xcite @xcite and law @xcite @xcite , yet progress remains concentrated in a handful of well-resourced languages. cross-lingual named-entity recognition is therefore important to better serve underserved communities, yet recent advancements remain unevenly distributed since * corresponding author: david.demitri.africa@gmail.com ner performance in many languages remains poor due to limited training resources. a key challenge is that entity boundaries and categories are not universal: languages differ in their morphosyntactic cues, word order, and orthographic conventions. models trained primarily on indo-european data thus fail to generalize reliably to underrepresented settings. in this paper, we address this problem through meta-pretraining: shaping language model initializations to adapt rapidly to new linguistic conditions. unlike standard pretraining, which minimizes average loss over a static corpus, episodic meta-pretraining (e.g. via maml; @xcite explicitly optimizes for fast transfer. for low-resource ner, this offers two potential benefits: (i) rapid adaptation to languages with typologically distinct cues (e.g. case particles, voice systems, code-switching), and (ii) stronger zero-shot prototypes for common entity types, even without in-language exposure. while meta-learning has been explored for classification tasks in english or cross-lingually at bert scale @xcite @xcite , its efficacy for small decoder lms and morphologically rich languages is underexplored. as a case study, we focus on ner in tagalog and cebuano, the two most widely spoken philippine languages @xcite . typologically, both languages combine austronesian features such as voice alternations, case particles, and reduplication with pervasive borrowing and code-switching (figure 8 ; rq2 what transfers? which entity classes, morphological cues, and lexical patterns (especially those tied to tagalog/cebuano typology) explain the observed gains or failures? we answer these questions by systematically comparing first-order maml and vanilla pretraining on llama-style pico decoders across scales, analyzing both downstream performance and representation dynamics @xcite . this allows us to investigate: rq3 how does the effect of meta-pretraining vary with model size? are benefits stronger at small scales, or do they persist as capacity increases? we provide the following contributions: ## zero-shot transfer results zero-shot evaluation. unless stated otherwise, all scores are obtained without seeing any tagalog/cebuano data during finetune, relying solely on the uner test sets ( § 2.4). figure 1 shows that pico-maml improves cebuano/tagalog micro-f 1 at every parameter budget. the relative lift is largest for moderate sizes and tapers with scale (+6% at 570m). these results indicate that adding a single outer-loop meta-update per batch yields a cross-lingual prior not captured by vanilla pretraining under our setup. comparison of head-only tuning and full tuning. decomposing by finetuning regime (fig. 2 ), maml yields 1-2 pp gains when only the crf head is trained, implying that the frozen weights already embeds better entity cues. full tuning narrows the gap to 0.5-1.3 pp, indicating that the lift persists even when the optimiser is free to overwrite the initialisation. further, results indicate that the benefit provided by the meta-objective is scale-dependent. for the 11 m (tiny) model, maml moves the overall score @xmath0. from 65 m parameters upward the benefit becomes clearer with larger head-only lifts, suggesting a threshold at which meta-gradients can provide reusable entity features without crowding out the lm signal. sensitivity to finetuning language. slovak (+18 %) and croatian (+13 %). gain in slovak might be due to fixed case endings that consistently bracket entity names, providing a clear surface boundary signal for the model (similar in function to tagalog's case particles but realised morphologically rather than syntactically.) the sole regression (-2 pp on simplified chinese) is most likely due to a known issue in poor crossscript transfer to chinese, but it may also be due to subword sparsity in the shared vocabulary rather than a failure of the meta-objective. @xcite . overall, maml appears to teach the model to exploit shallow lexical anchors (particles, affixes) that generalise well across indo-european languages while still transferring to more typologically distant austronesian targets. to better understand the mechanisms underlying these gains, we conduct a focused qualitative analysis on a representative configuration. ## analysis of maml pretrained models in order to analyze the learning process, rather than just the last checkpoint, we focus our qualitative study on a medium-sized model (181 m parameters) finetuned in a head-only regime on slovak (sk_snk), finetuning on all 61 checkpoints from step 0 of pretraining to step 6000. we restrict our analysis to this slice because while finetuning 9760 (2 pretraining regimes x 2 finetuning regimes x 4 model sizes x 10 finetuning languages x 61 checkpoints) models would be prohibitively expensive, this configuration at least offers a reasonable signalto-cost trade-off. this is for a few reasons: (i) the medium tier is the smallest model that still exhibits a clear 2-3 pp head-only lift (figure 1 ) yet is three- ## finetuning speed of meta-pretraining finally, we assess finetuning speed using convergence time (measuring time to achieve 90% of final loss t 90 ), normalized area under the loss curve (measuring aggregate convergence behavior over the curve), and initial slope (measuring the initial speed of learning in the first few steps), as seen in ## conclusion this paper shows that maml-based metapretraining, even when applied to small decoderonly language models, can meaningfully improve zero-shot transfer to low-resource languages, as demonstrated on tagalog and cebuano ner. the gains are most pronounced for person entities and head-only finetuning, and scale best with larger model capacities. our qualitative and word-level analyses reveal that the mechanism of improvement centers on the sharpening of lexical prototypes and better anchoring to surface cues like tagalog case particles. hence, we do not expect these improvements to fully generalize to multi-token or highly contextual entity types. these findings suggest that meta-learning can provide a principled route to more adaptable small models, but also highlight key limitations: the benefits are capacity-and task-dependent, and the current approach struggles with richer entity structures. future work should explore alternative metalearning objectives, extend to more diverse tasks and languages, and investigate the dynamics of prototype formation in even lower-resource settings.