Improving Low-Resource Cross-lingual Parsing with Expected Statistic Regularization (2023)

Abstract: We present Expected Statistic Regulariza tion (ESR), a novel regularization technique that utilizes low-order multi-task structural statistics to shape model distributions for semi- supervised learning on low-resource datasets. We study ESR in the context of cross-lingual transfer for syntactic analysis (POS tagging and labeled dependency parsing) and present several classes of low-order statistic functions that bear on model behavior. Experimentally, we evaluate the proposed statistics with ESR for unsupervised transfer on 5 diverse target languages and show that all statistics, when estimated accurately, yield improvements to both POS and LAS, with the best statistic improving POS by +7.0 and LAS by +8.5 on average. We also present semi-supervised transfer and learning curve experiments that show ESR provides significant gains over strong cross-lingual-transfer-plus-fine-tuning baselines for modest amounts of label data. These results indicate that ESR is a promising and complementary approach to model-transfer approaches for cross-lingual parsing.1

## introduction in recent years, great strides have been made on linguistic analysis for low-resource languages. these gains are largely attributable to transfer approaches from (1) massive pretrained multilingual language model (plm) encoders @xcite ; (2) multi-task training across related syntactic analysis tasks @xcite ; and (3) multilingual training on diverse high-resource languages @xcite @xcite . combined, these approaches have been shown to be particularly effective for cross-lingual syntactic analysis, as shown by udify @xcite . however, even with the improvements brought about by these techniques, transferred models still make syntactically implausible predictions on low-resource languages, and these error rates increase dramatically as the target languages become more distant from the source languages @xcite . in particular, transferred models often fail to match many low-order statistics concerning the typology of the task structures. we hypothesize that enforcing regularity with respect to estimates of these structural statistics-effectively using them as weak supervision-is complementary to current transfer approaches for low-resource crosslingual parsing. to this end, we introduce expected statistic regularization (esr), a novel differentiable loss that regularizes models on unlabeled target datasets by minimizing deviation of descriptive statistics of model behavior from target values. the class of descriptive statistics usable by esr are expressive and powerful. for example, they may describe cross-task interactions, encouraging the model to obey structural patterns that are not explicitly tractable in the model factorization. additionally, the statistics may be derived from constraints dictated by the task formalism itself (such as ruling out invalid substructures) or by numerical parameters that are specific to the target dataset distribution (such as relative substructure frequencies). in the latter case, we also contribute a method for selecting those parameters using small amounts of labeled data, based on the bootstrap @xcite . although esr is applicable to a variety of problems, we study it using modern cross-lingual syntactic analysis on the universal dependencies data, building off of the strong model-transfer framework of udify @xcite . we show that esr is complementary to transfer-based approaches for building parsers on low-resource languages. we present several interesting classes of statistics for the tasks and perform extensive experiments in both oracle unsupervised and realistic semi-supervised cross-lingual multi-task parsing scenarios, with particularly encouraging results that significantly outperform state-of-the-art approaches for semi-supervised scenarios. we also present ablations that justify key design choices. ## expected statistic regularization we consider structured prediction in an abstract setting where we have inputs @xmath0, output structures @xmath1, and a conditional model p θ (y|x) ∈ p with parameters θ ∈ θ, where p is the distribution space and θ is the parameter space. in this section we assume that the setting is semi-supervised, with a small labeled dataset d l and a large unlabeled dataset d u ; let d @xmath2, y i )} m @xmath3. our approach centers around a vectorized statistic function f that maps unlabeled samples and models to real vectors of dimension d f : ## choosing the targets and margins there are several possible approaches to choosing the targets t and margins σ, and in general they can differ based on the individual statistics. for some statistics it may be possible to specify the targets and margins using prior knowledge or formal constraints from the task. in other cases, estimating the targets and margins may be more difficult. depending on the problem context, one may be able to estimate them from related tasks or domains (such as neighboring languages for cross-lingual parsing). here, we propose a general method that estimates the statistics using labeled data, and is applicable to semi-supervised scenarios where at least a small amount of labeled data is available. the ideal targets are the expected statistics under the ''true'' model p * are: in addition to estimating the target statistics for small batch sizes, the bootstrap gives us a way to estimate the natural variation of the statistics for small sample sizes. to this end, we propose to utilize the standard deviations from the bootstrap samples as our margins of uncertainty σ: this allows our loss function ℓ to adapt to more or less certain statistics. if some statistics are naturally too variable to serve as effective supervision, they will automatically have weak contribution to ℓ and little impact on the model training. ## application to cross-lingual parsing now that we have described our general approach, in this section we lay out a proposal for applying it to cross-lingual joint pos tagging and dependency parsing. we choose to apply our method to this problem because it is an ideal testbed for controlled experiments in semi-supervised structured prediction. by their nature, the parsing tasks admit many types of interesting statistics that capture cross-task, universal, and language-specific facts about the target test distributions. we evaluate in two different transfer settings: oracle unsupervised and realistic semi-supervised. in the oracle unsupervised settings, there is no supervised training data available for the target languages (and the l term is dropped from eq. 3), but we use target values and margins calculated from the held-out supervised data. this setting allows us to understand the impact of our regularizer in isolation without the confounding effects of direct supervision or inaccuracte targets. in the semi-supervised experiments, we vary the amounts of supervised data, and calculate the targets from the small supervised data samples. this is a realistic application of our approach that may be applied to low-resource learning scenarios. ## realistic semi-supervised experiments the previous experiments considered an unsupervised transfer scenario without labeled data. in these next experiments we turn to a realistic semi-supervised application of our approach where we have access to limited labeled data for the target treebank. ## related work related work generally falls into two categories: weak supervision and cross-lingual transfer. weak supervision: supervising models with signals weaker than fully labeled data has and continues to be a popular topic of interest. current trends in weak supervision focus on generating instance-level supervision, using weak information such as: relations between multiple tasks @xcite @xcite ; labeled features @xcite @xcite ; coarse-grained labels @xcite ; dictionaries and distant supervision @xcite @xcite ; or some combination thereof @xcite . in contrast, our work is more closely related to older work on population-level supervision. these techniques include constraint-driven learning (codl) (cha), posterior regularization (pr) @xcite , the measurements framework of @xcite , and the generalized expectation criteria (gec) @xcite @xcite . our work can be seen as an extension of gec to more expressive expectations and to modern mini-batch sgd training. there are a two more recent works that touch on these ideas, but both have significant downsides compared to our ap-proach. @xcite use a pr approach inspired by @xcite for crosslingual parsing, but must use very simple constraints and require a slow inference procedure that can only be used at test time. @xcite utilize gec with minibatch training, but focus on using related tasks for computing simpler constraints and do not adapt their targets to small batch sizes. cross-lingual transfer: earlier trends in cross-lingual transfer for parsing used delexicalization @xcite @xcite and then aligned multilingual word vector-based approaches @xcite @xcite . with the rapid rise of language-model pretraining @xcite @xcite , recent research has focused on multilingual plms and multi-task fine-tuning to achieve generalization in transfer. @xcite showed that a multilingual plm afforded surprisingly effective cross-lingual transfer using only english as the fine-tuning language. @xcite extended this approach by fine-tuning a plm on the concatenation of all treebanks. @xcite , however, show that transfer to distant languages benefit less. other recent successes have been found with linguistic side-information @xcite , careful methodology for source-treebank selection @xcite @xcite , self-training @xcite , and paired bilingual text for annotation projection @xcite @xcite . ## conclusion we have presented expected statistic regularization, a general approach to weak supervision for structured prediction, and studied it in the context of modern cross-lingual multi-task syntactic parsing. we evaluated a wide range of expressive structural statistics in idealized and realistic transfer scenarios and have shown that the proposed approach is effective and complementary to the state-of-the-art model-transfer approaches.