Cross-Lingual Wolastoqey- E nglish Definition Modelling (2021)

Abstract: Definition modelling is the task of automatically generating a dictionary-style definition given a target word. In this paper, we consider cross-lingual definition generation. Specifically, we generate English definitions for Wolastoqey (Malecite-Passamaquoddy) words. Wolastoqey is an endangered, low-resource polysynthetic language. We hypothesize that sub-word representations based on byte pair encoding (Sennrich et al., 2016) can be leveraged to represent morphologically-complex Wolastoqey words and overcome the challenge of not having large corpora available for training. Our experimental results demonstrate that this approach outperforms baseline methods in terms of BLEU score. ﻿

## introduction definition modelling, introduced by @xcite , is the task of automatically generating a dictionary-style definition for a given target word. definition modelling can provide more-transparent, human-interpretable representations of the information in embeddings. definition modelling could also potentially be applied to automate, or semiautomate, the constructing or updating of dictionaries, for example, by generating draft definitions for newly-emerged words that are not yet listed. although there has been a range of work on definition modelling (e.g., @xcite @xcite ) the focus has been on monolingual definition modelling, with the target word and generated definition being in the same language. malecite-passamaquoddy (also maliseet-passamaquoddy, passamaquoddy-maliseet) is an eastern algonquian language spoken in regions of what is now new brunswick and quebec, canada, and maine, united states. malecite and passamaquoddy are dialects of this language. however, malecite is a mi'kmaq exonym, with wolastoqey being the term this speech community uses to refer to their language. we therefore use the term wolastoqey throughout this paper. wolastoqey is an endangered language, with roughly 300 remaining first language speakers in canada @xcite . moreover, children are typically not learning the language proficiently. wolastoqey is also a low-resource language, with no large corpora or annotated datasets available for training natural language processing (nlp) systems. however, the passamaquoddy-maliseet dictionary @xcite is available online through the passamaquoddy-maliseet language portal. 1 this dictionary includes roughly 19k entries with wolastoqey headwords and english definitions. many entries also include parallel wolastoqey-english example sentences. there has been very little prior computational work on wolastoqey, with @xcite presenting a preliminary finite-state model of nouns. wolastoqey, like other algonquian languages, is polysynthetic. verbs in particular have rich morphological structure, and often include several roots @xcite . consider the example gloss below for paskoloqessu: all examples are taken from the passamaquoddy-maliseet language portal. in this paper we propose a model for crosslingual wolastoqey-english definition modelling. we hypothesize that sub-word representations of wolastoqey words based on byte pair encoding (bpe) tokenization @xcite can be leveraged to generate english definitions. we propose a sequence-to-sequence model @xcite in which the encoder operates over wolastoqey words segmented via bpe, and the decoder generates english definitions. we show that our proposed model is able to outperform baseline systems in terms of bleu score. wolastoqey speakers regularly create new words by creatively combining roots @xcite . as such, not all words can be expected to be included in a dictionary. cross-lingual wolastoqey-english definition modelling could therefore be helpful for wolastoqey learners. ## related work the task of definition modelling is to learn to generate a dictionary-style definition for a given input word. this task was initially described by @xcite , who focused on generating english definitions for english words. noraset et al. proposed a word-to-sequence neural language model, composed of a two-layer lstm and a parallel cnn, to generate a definition given an initial input word and its embedding. this language modelbased approach generates definitions by iteratively predicting the next occurring word given some prior history. in this model, different inputs were given to each of the sub-components. the lstm component was initially given the embedding for the word being defined, but also considered the embeddings of its previous output in the form of context at a given timestep. the cnn sub-network, on the other hand, was used to extract characterlevel information about the word and, as such, was given the characters of the word being defined. the cnn was included because the word-level lstm has no knowledge of sub-word information. manual analysis of the proposed system's performance considered seven types of errors that were observed to occur in the generated definitions. these include redundancy, self-reference, wrong part-of-speech, under-specification, opposite definition, close semantic errors, and incorrect definition generation. out of all these errors, incorrect definition gener-ation was observed to be the most common. this paper further found that high-quality word embeddings were crucial for definition modelling to be successful. one challenge for our work is that we do not have a large corpus available from which to learn high-quality wolastoqey word embeddings. we propose to use bpe segmentation to overcome this. specifically, we hypothesize that sub-word representations based on bpe can be leveraged to represent morphologically-complex wolastoqey words without requiring a large corpus to be available for training word-level embeddings. the approach of @xcite is contextagnostic; i.e., the model generates a definition for a target word without any specific context of usage for the target. other context-agnostic approaches to definition modelling include yang et al. (2020) who incorporate knowledge of chinese sememes (minimum semantic units) for chinese definition modelling, and balachandran et al. (2018) who propose a domain-specific definition generation model for the software domain. in line with these previous studies, we also propose a context-agnostic approach. contrasting with context-agnostic approaches, context-aware approaches to definition modelling have also been considered (e.g., @xcite @xcite . in these approaches a definition is generated for a target word used in a specific context. some context-aware methods have used a sequence-tosequence model @xcite as does our proposed approach. ni and wang propose a sequence-to-sequence model to generate definitions for non-standard english words. their encoder uses a character-level lstm to represent the target word and a word-level lstm to represent the context. an lstm is also used for decoding. our proposed approach is similar to that of ni and wang, but we do not use an lstm to encode context, and our lstm which encodes the target word operates over bpe tokenization as opposed to characters. an alternative line of research considers definition extraction (e.g., @xcite in which sentences containing terms and their corresponding definitions are automatically identified in corpora. we focus on definition modelling, as opposed to extraction, because there are very few corpora containing english definitions of wolastoqey words on which to apply a definition extraction method, and because wolastoqey is a polysynthetic language and as such many possible words would not be expected to be found in corpora. ## model cross-lingual definition modelling can be seen as a machine translation task which involves translating a word in a source language to a definition in some target language. we therefore consider using a network architecture proposed for the task of translation rather than a word-to-sequence model proposed in previous work on monolingual definition modelling (e.g., @xcite . specifically, we consider a sequence-to-sequence model that makes use of an attention decoder @xcite . we base our model on a sample sequenceto-sequence translation model. foot_1 ## experimental setup in this section, we describe the dataset constructed for these experiments, the evaluation methodology used, and implementation details for our proposed model. ## results in this section we first consider tuning the vocabulary size for the wolastoqey bpe tokenizer in experiments on development data (section 5.1). we then present our main results on test data, including an analysis of the impact of pretraining embeddings (section 5.2) and results for the model variations presented in section 3.3 (section 5.3). we then present a qualitative evaluation of the model (section 5.4). ## conclusions in this paper, we considered cross-lingual wolastoqey-english definition modelling, in which we automatically generate english definitions for wolastoqey words. our work is in contrast to most prior work on definition modelling which has been monolingual, i.e., the word being defined and its definition are in the same language. in further contrast to most prior work on definition modelling, where wolastoqey is a low-resource language, we do not have access to a large wolastoqey background corpus for training. we showed that a sequence-to-sequence model that represents morphologically-complex wolastoqey words at the sub-word level using bpe segmentation outperforms baseline approaches. we further demonstrated that the proposed approach can be improved by pretraining on small wolastoqey and english monolingual corpora built from dictionary example sentences and by using a gru instead of lstm. qualitative analysis revealed that the generated definitions are often syntactically well-formed and topically related to the gold-standard reference definitions. in future work we plan to investigate alternative strategies for representing wolastoqey words in the encoder, including character-level approaches and segmentations based on unsupervised approaches to learning morphology @xcite . we also plan to explore alternative model architectures, including transformer-based models @xcite and models that incorporate large pre-trained english language models (e.g., @xcite .