Incorporating Hypernym Features for Improving Low-resource Neural Machine Translation (2024)

Abstract: Parallel data is difficult to obtain for low-resource languages in machine translation tasks, making it crucial to leverage monolingual linguistic features as auxiliary information. This article introduces a novel integration of hypernym features into the model by combining learnable hypernym embeddings with word embeddings, providing semantic information. Experimental results based on bilingual and multilingual models showed that: (1) incorporating hypernyms improves translation quality in low-resource settings, yielding +1.7 BLEU scores for bilingual models, (2) the hypernym feature demonstrates efficacy both in isolation and in conjunction with syntactic features, and (3) the performance is influenced by the choice of feature combination operators and hypernym-path hyperparameters.

## introduction low-resource neural machine translation (nmt) is an open challenge to nlp researchers because of a number of bottlenecks, such as a lack of parallel data and efficient linguistic tools, out-ofdomain data, and morphological complexity of the languages. the majority of the research in this field either exploits monolingual and multilingual data in different ways (including backtranslation @xcite , transfer learning @xcite , and multilingual training @xcite ) or come up with model-centric techniques for better modeling, training and inference @xcite . other than these two approaches, the use of linguistic knowledge is an effective strategy to improve translation quality under resource-scarce situations, however, relatively under-explored. linguistic analysis can be utilized for nmt both implicitly and explicitly. implicit integration refers to methods that, instead of directly applying morphology into the model, use it as a part of pre-processing (subword segmentation of words based on legitimate units @xcite or make a better contextual representation of the source sentence with the help of its syntactic/dependency structure @xcite @xcite . in case of explicit use, either morphological information is included in the data to provide richer information about the source and the target languages @xcite or the model is trained with a multi-task objective to predict words along with their linguistic properties as secondary output @xcite in order to obtain better internal word-form representation. while morphological attributes are directly used in the source side as additional input features to words, it is hard to decide which input feature(s) are optimum to feed to the model for learning source-to-target mapping. since the features are embedded in continuous space and can be combined easily, existing studies @xcite @xcite are found to use the following attributes together as supplementary components of a word -(1) part-of-speech (pos): tells syntactic behavior of individual words, (2) lemma: denotes base form and help to disambiguate inflectional variants, and (3) dependency parsing label: hypernym embedding word embedding pizza on steel table dish.n.02 alloy.n.0 furniture.n.01 transformer encoder self-rel. provides the relationship with other words within a sentence. although these three attributes are empirically established to help low-resource translation, however, they cannot impart distributional semantics which is crucial when there is not a sufficient amount of data available to learn linguistic regularities. in this work, we try to address the above issue by incorporating hypernym information as a component of source words to meet the lack of distributional semantics in low-resource scenarios. as presented in figure 1 , hypernym provides superclass information, hence it can relate two distinct words with semantic similarity to some extent (e.g., table and chair in furniture sense) despite without any syntactic relation between them. one can argue that hypernym is an expensive knowledge typically obtained from wordnet @xcite thus hardly available for low-resource languages. nevertheless, building a primitive wordnet with hypernym relation is relatively easy and we aim to explore the potency of superclass information in nmt. in a nutshell, our contributions are as follows: (1) we incorporate hypernyms as a semantic component of word embeddings in lowresource mt, (2) experimental results show bleu score improvements from english to eight diverse low-resource asian languages for both bilingual (+1.73 on avg.) and multilingual models (+0.24 on avg.), (3) we provide comparative analysis between syntactic vs. semantic feature combinations and hypernym-path hyperparameters variants. ## methodology at first we provide the basics of two important concepts -how linguistic input features are used as additional components of a word in nmt models, and measuring the relevance of a feature embedding. next, we describe the procedure of data annotation with hypernym information. linguistic input features into nmt: sennrich and haddow (2016) introduced a simple but effective way to incorporate linguistic input features into a word by concatenating word embedding and feature embeddings together. this approach supports an arbitrary number of features and enables the translation model to directly incorporate linguistic knowledge. in subword-based nmt, features corresponding to a word are replicated across its subwords. given a source sentence s, if each of its token is represented with k features, then the i th token s i can be denoted as s @xmath0, . . . , s ik ). here, s i1 is the word or sub-word embedding, while s i2 , . . . , s ik represent various linguistic features. for any feature type indexed by @xmath1, ..., k}, let v k , e k , and d k be the vocabulary, embedding matrix and dimension of the embedded vector, respectively, with the embedding of token s i , denoted by e i , can be computed as e @xmath2, where e ik is the embedding of s ik . the final embedding for s i is obtained as e @xmath3, where ∥ signifies the concatenation operation. attenuating feature embeddings by relevance: the above method by @xcite combines word and feature embeddings blindly and lacks to evaluate the functionality of a feature in terms of translation goal. @xcite claimed that providing extra morphological information may lead to noise when a word has only one sense. hence, they came up with two strategies to attenuate feature embedding. the first one, named as self-relevance, measures the importance of a feature embedding w.r.t the embedding itself, and the second one, named as word-based relevance, considers both word and feature embeddings together to weight the feature embedding. out of these two relevance mechanisms, chakrabarty et al., (2020) empirically found self-relevance to be better. hence, we use it throughout our experimentation and detail it as follows. self-relevance: for the k th feature component s ik , its embedded vector e ik is transformed by a learnable weight matrix w @xmath4. it generates a mask vec- word table chair desk entity.n.01 physical-entity.n.01 object.n.01 furniture.n.01 table.n.02 seat.n.03 table.n.02 chair.n.01 desk.n.01 …… depth-1 depth-2 depth-3 depth-8 depth-9 depth-10 hypernym hierarchy . . . figure 3: depth-wise sense similarity of the synsets table.n.02, chair.n.01, and desk.n.01. tor mask ik that contains the weight of e ik as mask @xmath5. next, e ik is elementwise multiplied by mask ik to modulate the relevance. the attenuated feature embedding is thus e ′ @xmath6. eventually all modified embeddings e ′ i1 , . . . , e ′ ik are concatenated to form the source embedding e ′ @xmath7. the process is depicted in figure 2 . ## experimental settings datasets: we chose asian language treebank (alt) @xcite for our mt experiments, which is a multi-parallel mt dataset. the data is initially in english and translated into 12 asian languages. following the experimental settings of @xcite , we fix english (en) as the source and eight asian languages -bengali (bg), filipino (fi), hindi (hi), indonesian (id), khmer (khm), malay (ms), myanmar (my) and vietnamese (vi) as the targets. the size of the train/dev/test split for each language pair is 18, 088/1, 000/1, 018. in the bilingual setup, we trained eight separate nmt models for each direction, whereas in multilingual experiments, we trained a one-to-many nmt model from english to eight asian languages. we use english as the source due to the availability of hypernyms and other morphological attributes. preprocessing: we apply byte-pair encoding (bpe) @xcite with 32k merge operations for subword segmentation. multilingual setup identifies each target language by a special token appended at the source side. for english data, stanford corenlp toolkit @xcite is used to get pos, lemma, and dependency ## results hypernym hyperparameters: to determine the optimal d/f combination, we analyze the results where only hypernyms are used (ids 2, 3, 4 and 11, 12, 13 for bilingual and multilingual setups, respectively). for bilingual models, d/@xmath8. bleu as well the best scores for @xmath9, @xmath10, @xmath11, @xmath12, @xmath13, and @xmath14. for multilingual models, all three combinations (ids 11, 12, 13) performed equally well. therefore, we further analyze where all features are used (ids 15, 16, 17) and find that 3/50k is the optimal combination for both settings, showing that shallow semantic annotation is better. ## conclusion this study investigates the role of hypernyms used as a word embedding component to exploit distributional semantics in low-resource settings. experiments over eight language pairs reveal its usefulness strongly in bilingual scenarios. we also conducted one-to-many multilingual experiments finding the superiority of semantic feature over lexical and syntactic features. we analyze training plots to show that perplexity drop is not always a good measure to evaluate model training. the future extension of this work will include -(1) finding the most appropriate hypernym-path of a contextual word, and (2) determining the optimum combination of semantic and syntactic features to leverage linguistic knowledge for lowresource translation.