A Mixture-of-Experts Model for Antonym-Synonym Discrimination (2021)

Abstract: Discrimination between antonyms and synonyms is an important and challenging NLP task. Antonyms and synonyms often share the same or similar contexts and thus are hard to make a distinction. This paper proposes two underlying hypotheses and employs the mixture-of-experts framework as a solution. It works on the basis of a divide-and-conquer strategy, where a number of localized experts focus on their own domains (or subspaces) to learn their specialties, and a gating mechanism determines the space partitioning and the expert mixture. Experimental results have shown that our method achieves the state-of-the-art performance on the task.

## introduction antonymy-synonymy discrimination (asd) is a crucial problem in lexical semantics and plays a vital role in many nlp applications such as sentiment analysis, textual entailment and machine translation. synonymy refers to semanticallysimilar words (having similar meanings), while antonymy indicates the oppositeness or contrastiveness of words (having opposite meanings). although telling apart antonyms and synonyms looks simple on the surface, it actually poses a hard problem because of their interchangeable substitution. a few research efforts have been devoted to computational solutions of asd task, which comprises two mainstreams: pattern-based and distributional approaches. the underlying idea of pattern-based methods exists in that antonymous word pairs cooccur with each other in some antonymy-indicating lexico-syntactic patterns within a sentence @xcite . in spite of their high precision, pattern-based methods suffer from limited recall owing to the sparsity of lexico-syntactic patterns and the lexical variations. distributional methods work on the basis of distributional hypothesis stating that "the words similar in meaning tend to occur in similar contexts" @xcite . traditional distributional methods are based on discrete context vectors. @xcite verified that using only the contexts of certain classes can help discriminate antonyms and synonyms. @xcite thought that synonyms are expected to have broader and more salient intersection of their top-k salient contexts than antonyms, and proposed an average-precision-based unsupervised measure. with the advent of word embeddings as the continuous representations @xcite @xcite , several neural methods have been proposed to elicit asd-specific information from pretrained word embeddings in a supervised manner. @xcite used a siamese network to ensure the symmetric, reflexive and transitive properties of synonymy and a parasiamese network to model the antitransitivity of antonymy. @xcite projected word embeddings into the synonym and antonym subspaces respectively, and then trained a classifier on the features from these distilled subspaces, where the trans-transitivity of antonymy was taken into consideration. this paper follows the distributional approach and studies the asd problem on the basis of pretrained word embeddings. two hypotheses underlie our method: (a) antonymous words tend to be similar on most semantic dimensions but be different on only a few salient dimensions; (b) the salient dimensions may vary significantly for different antonymies throughout the whole distributional semantic space. with respect to the hypothesis (b), we find that a tailored model of mixture-of-experts (moe) @xcite fits it well. the semantic space is divided into a number of subspaces, and each subspace has one specialized expert to elicit the salient dimensions and learn a discriminator for this subspace. as to the hypothesis (a), a similar opinion was also expressed by cruse (1986) that antonymous words tend to have many common properties, but differ saliently along one dimension of meaning. in addition, our experimental results have shown that each expert requires only four salient dimensions to achieve the best performance. finally, we would like to point out the main difference of our method from the existing ones. firstly, our moe-asd model adopts a divide-andconquer strategy, where each subspace is in the charge of one relatively-simple localized expert that focuses on only a few salient dimensions; while existing methods rely on a global model which must grasp all the salient dimensions across all the subspaces. secondly, our method simply enforces the symmetric property of synonymy and antonymy, but ignores the other algebraic properties such as the transitivity of synonymy and transtransitivity of antonymy, because these algebraic properties do not always hold on the word level for the polysemy characteristic of words. ## method this paper proposes a novel asd method based on the mixture-of-experts framework (called moe-asd) 1 . its architecture is illustrated in figure 1 . it solves the problem in a divide-and-conquer manner by dividing the problem space into a number of subspaces and each subspace is in the charge of a specialized expert. the expert focuses on the salient dimensions of the subspace and makes the decision for word pairs. a gating module is trained jointly with these experts. the details are as follows. ## conclusions this paper first presents two hypotheses for asd task (i.e., antonymous words tend to be different on only a few salient dimensions that may vary significantly for different antonymies) and then motivates an asd method based on mixture-of-experts. finally, experimental results have manifested its effectiveness and validated the two underlying hypotheses. it is worth noting that our method is distinct from the other state-of-the-arts in two main aspects: (1) it works in a divide-and-conquer strategy by dividing the whole space into multiple subspaces and having one expert specialized for each subspace;