Generating from AMR s into High and Low-Resource Languages using Phylogenetic Knowledge and Hierarchical QL o RA Training ( HQL ) (2024)

Abstract: Multilingual generation from Abstract Meaning Representations (AMRs) verbalises AMRs into multiple languages. Previous work has focused on high- and medium-resource languages relying on large amounts of training data. In this work, we consider both high- and low-resource languages capping training data size at the lower bound set by our low-resource languages i.e. 31K. We propose a straightforward technique to enhance results on low-resource while preserving performance on high-resource languages. We iteratively refine a multilingua model to a set of monolingual models using Low-Rank Adaptation with a training curriculum based on a tree structure; this permits investigating how the languages used at each iteration impact generation performance on high and low-resource languages. We show an improvement over both mono and multilingual approaches. Comparing different ways of grouping languages at each iteration step we find two working configurations: grouping related languages which promotes transfer, or grouping distant languages which facilitates regularisation

## introduction abstract meaning representation (amr) @xcite is a representation language used to encode the meaning of sentences. figure 1 shows an example amr graph and some of its possible verbalisations in 4 different languages. amr-to-text generation is the task of verbalizing the meaning encoded by an amr graph. while there has been constant progress on this task for the english language @xcite @xcite and some other high-resource (hr) and medium-resource possible-01 help-01 police victim eng: the police could help the victim. deu: die polizei konnte dem opfer helfen. spa: la policía podría ayudar a la víctima. ita: la polizia potrebbe aiutare la vittima. ## a brief overview of lora and qlora lora is a parameter-efficient fine-tuning approach where, during training, the weights of the original base model (w 0 ) are frozen and two lowrank, trainable, decomposition matrices (a and b) are added to selected layers of the model, reworking the output hidden state of the layers (h) to the addition of the original weights and the product of the low-rank matrices (ab) as shown in equation 1. ab happens to be a good approximation of a full fine-tuning weight update while requiring fewer parameters to be trained. notably, after having trained a and b on some task or language, we can compute their final product (ab) and merge this product into the original weights (w 0 ) via simple matrix addition thereby creating a new model specialised for the target task or language. thus the same model can be iteratively fine-tuned on multiple tasks or languages. in our approach, we start from a pre-trained multilingual model and iteratively derive 12 monolingual models from this initial model in 4 steps, starting by fine-tuning this model tuned on 12 languages (step 0) and iteratively fine-tuning models for 6, 2 and 1 languages (steps 1, 2 and 3). by merging the weights of the original model with the parameters learned in the lora matrices, the final models have no inference overhead, which distinguishes lora from other peft approaches. furthermore, since lora matrices are smaller than the base model, loras for multiple tasks or languages can be trained and switched faster and without requiring as much storage space as other approaches. another advantage of lora adaptation is that it lowers the memory requirements for fine-tuning very large models compared with full fine-tuning. to further reduce memory requirements during training, @xcite proposed qlora, where unquantized lora modules are applied to a quantized model. while training quantized weights is unstable @xcite , only training the few unquantized weights of the lora module makes this approach stable. ## task we aim to verbalise amr graphs into both highand low-resource languages. to factor out the impact of training data size, we keep this size constant across languages restricting the number of distinct training instances per language to 31k, the lower bound set by the language with fewer resources. in this way, differences between languages can be traced back to differences between models and training strategies rather than to the size of the available data for each language. for our experiments, we select a combination of 6 low-and 6 high-resource languages (as classified by the nllb team et al. ( 2022) ). we select these languages so that they can be grouped in a balanced phylogenetic tree (see figure 2b ). ## hierarchical qlora (hql) to mitigate the effects of data scarcity (over-fitting) and multilingual training (noise), we propose a variation of curriculum learning that leverages both phylogenetic knowledge and the modularity and memory efficiency of loras to iteratively refine a base multilingual model into a set of monolingual models. base model. our base model is mt5 large @xcite foot_1 , a multilingual encoder-decoder model which we extend with lora modules to support modular parameter-efficient fine-tuning and 4-bit quantization to reduce memory footpring during training. refining models. we learn 12 monolingual models by iteratively fine-tuning a model trained in 12 languages in four steps as follows. in the first step (level 0), the base model (mt5 large ) is finetuned on 12 languages using lora fine-tuning. the resulting model -which is created by merging mt5 large 's weights with the a and b matrices as explained above -is then fine-tuned on two sets of 6 languages yielding two 6-language models, each trained with a separate lora module (level 1). we repeat this process twice: first, fine-tuning the two 6-language models into 6 bilingual models (level 2) and second, fine-tuning each of the bilingual models into 12 monolingual models (level 3). algorithm 1 in appendix a specifies our training strategy in more detail. ## results we report results obtained when generating from both silver and gold amr comparing our approach with previous works and baselines and examining results on both high-and low-resource languages. hql outperforms or is on par with mono and multilingual baselines (silver and gold amrs). on silver amrs, hql models are consistently better than both the mono and the multilingual baselines, except for tok pisin (figure 3 , hql outperforms the gen&trans baseline on all lr languages. while the gen&trans baseline outperforms our models on most hr languages (except english), our approach outperforms the gen&trans models on all lr languages (figure 3 ). this shows the benefits of hql for lr languages where mt yield low quality texts while our stacked lora approach seems to enhance transfer. similar results are seen on other metrics (appendix c) where hql comes ahead in most lr languages. we also see that two languages previously unseen by the base model (tok pisisn and asturian) show a transfer effect as they perform on par with lr languages present in the base model's training data. for limburgish and sicilian, we conjecture that the low scores result from the low-quality of the machine translation as evidenced by the poor performance of the gen&trans baseline on these languages. hql optimizes faster than the three baseline models and on average, outperforms them all. figure 4 plots the average bleu, chrf++, and bleurt-20 score for all 12 languages against the number of instances seen during training. we see that already at level l2, our hql models outperform all three baselines (monolingual, multilingual, gen&trans ) on two of the metrics despite seeing fewer total training instances. the graph also shows that each new level of the hierarchy improves performance. hql performs well compared to previous works despite being trained on fewer data. in previous work, f&g, ribeiro and xu trained on 400k to 8.9m synthetic training pairs per language while the martinez model is trained for up to 30 epochs on close to 55k monolingual instances. in contrast, our models are trained on 4 epochs and less than 31k instances per language. despite this, our models come close to and in some cases, outperform those previous approaches, while also enabling support for lr languages. ## conclusion we proposed a novel approach for multilingual amr-to-text generation and showed that it significantly outperforms fully monolingual and fully multilingual approaches. we demonstrated that, on lr languages, it can outperform a gen&trans approach, despite most training data being machinetranslated. we compared different techniques for selecting a training hierarchy and found that, while the phylogenetic approach usually achieves better results than the distant languages approach, differences were not significant.