A Graph Auto-encoder Model of Derivational Morphology (2020)

Abstract: There has been little work on modeling the morphological well-formedness (MWF) of derivatives, a problem judged to be complex and difficult in linguistics. We present a graph auto-encoder that learns embeddings capturing information about the compatibility of affixes and stems in derivation. The auto-encoder models MWF in English surprisingly well by combining syntactic and semantic information with associative information from the mental lexicon.

## introduction a central goal of morphology is, as famously put by @xcite , "to tell us what sort of new words a speaker can form." this definition is tightly intertwined with the notion of morphological well-formedness (mwf). while nonexisting morphologically well-formed words such as pro@xmath0ism conform to the morphological patterns of a language and could be formed, non-existing morphologically ill-formed words such as pro@xmath1ism violate the patterns and are deemed impossible @xcite . more recent research has shown that mwf is a gradient rather than binary property: non-existing words that conform to the morphological patterns of a language differ in how likely they are to be actually created by speakers @xcite . this is particularly true in the case of derivational morphology, which is not obligatory and often serves communicative needs @xcite . as a result, the degree of mwf of a non-existing derivative is influenced by a multitude of factors and judged to be hard to predict @xcite . in nlp, the lack of reliable ways to estimate the mwf of derivatives poses a bottleneck for generative models, particularly in languages exhibiting a rich derivational morphology; e.g., while inflected forms can be translated by generating morphologically corresponding forms in the target language @xcite , generating derivatives is still a major challenge for machine translation systems @xcite . similar problems exist in the area of automatic language generation @xcite . this study takes a first step towards computationally modeling the mwf of english derivatives. we present a derivational graph auto-encoder (dga) that combines semantic and syntactic information with associative information from the mental lexicon, achieving very good results on mwf prediction and performing on par with a character-based lstm at a fraction of the number of trainable parameters. the model produces embeddings that capture information about the compatibility of affixes and stems in derivation and can be used as pretrained input to other nlp applications. foot_0 2 derivational morphology ## models let w be a bernoulli random variable denoting the property of being morphologically well-formed. we want to model p (w |d, c r ) = p (w |s, a, c r ), i.e., the probability that a derivative d consisting of stem s and affix group a is morphologically wellformed according to sr corpus c r . given the established properties of derivational morphology (see section 2), a good model of p (w |d, c r ) should include both semantics and formal structure, figure 3 : dga model architecture. the dga takes as input an adjacency matrix b and additional feature vectors x s and x a and learns embeddings z s and z a . ## related work much recent computational research on derivational morphology in nlp has focused on two related problems: predicting the meaning of a derivative given its form, and predicting the form of a derivative given its meaning. the first group of studies models the meaning of derivatives as a function of their morphological structure by training embeddings directly on text segmented into morphemes @xcite or by inferring morpheme embeddings from whole-word vector spaces, e.g., using the vector offset method @xcite . formally, given a derived form f d , this line of research tries to find the meaning m d that maximizes p (m d |f d ). the second group of studies models the form of derivatives as a function of their meaning. the meaning is represented by the base word and a semantic tag @xcite or the sentential context @xcite . formally, given a meaning m d , these studies try to find the derived form f d of a word that maximizes p (f d |m d ). our study differs from these two approaches in that we model p (w |f d , m d ), i.e., we predict the overall likelihood of a derivative to exist. for future research, it would be interesting to apply derivational embeddings in studies of the second type by using them as pretrained input. neural link prediction is the task of inferring the existence of unknown connections between nodes in a graph. advances in deep learning have prompted various neural models for link prediction that learn distributed node representations @xcite . @xcite proposed a convolutional graph auto-encoder that allows to include feature vectors for each node. the model was adapted to bipartite graphs by @xcite . previous studies on neural link prediction for bipartite graphs have shown that the embeddings of the two node sets should ideally form separated clusters @xcite . our work demonstrates that relations transcending the two-mode graph structure can lead to a trade-off between clustering and dispersion in embedding space. ## conclusion we have introduced a derivational graph autoencoder (dga) that combines syntactic and semantic information with associative information from the mental lexicon to predict morphological well-formedness (mwf), a task that has not been addressed before. the model achieves good results and performs on par with a character-based lstm at a fraction of the number of trainable parameters (less than 10%). furthermore, the model learns embeddings capturing information about the compatibility of affixes and stems in derivation.