Antonym vs Synonym Distinction using I nterla C ed Encoder NET works ( ICE - NET ) (2024)

Abstract: Antonyms vs synonyms distinction is a core challenge in lexico-semantic analysis and automated lexical resource construction. These pairs share a similar distributional context which makes it harder to distinguish them. Leading research in this regard attempts to capture the properties of the relation pairs, i.e., symmetry, transitivity, and trans-transitivity. However, the inability of existing research to appropriately model the relation-specific properties limits their end performance. In this paper, we propose InterlaCed Encoder NETworks (i.e., ICE-NET) for antonym vs synonym distinction, that aim to capture and model the relation-specific properties of the antonyms and synonyms pairs in order to perform the classification task in a performance-enhanced manner. Experimental evaluation using the benchmark datasets shows that ICE-NET outperforms the existing research by a relative score of upto 1.8% in F1-measure.

## introduction antonyms vs synonyms distinction is a core challenge in natural language processing applications, including but not limited to: sentiment analysis, machine translation, named entity typing etc. synonyms are defined as semantically related words, whereas antonyms are defined as semantically opposite words. for example "disperse" and "scatter" are synonyms, while "disperse" and "garner" are antonyms @xcite . existing research on the antonym-synonym distinction is primarily categorized into pattern-based and embedding-based approaches. pattern-based approaches attempt to curate distinguishing lexicosyntactic patterns for the word pairs @xcite . a major limitation of the pattern-based approaches is the sparsity of the feature space. despite using massive data sets, the generalization attempts result in highly overlapping and noisy features, which further deteriorate the model's performance. embedding based methods rely on the distributional hypothesis, i.e., "words that occur in the same contexts tend to have similar meanings" @xcite . these methods use widely available embedding resources to capture/compute the semantic relatedness of synonym and antonym pairs @xcite . @xcite proposed distiller that uses non-linear projections to project the embedding vectors in task-specific dense sub-spaces. the key challenge faced by existing embeddingbased approaches is their inability to correctly model the inherent relation-specific properties among different relation pairs. these models mix different lexico-semantic relations and perform poorly when applied to a specific task @xcite . existing approaches, moreover, model each relation pair independently, which is not adequate for antonym and synonym relation pairs as these relation pairs exhibit unique properties that may be exploited by modeling the relation pair in correlation with other instances (discussed in detail in section 4). keeping in view the above-mentioned challenges, in this paper, we propose interlaced encoder networks (ice-net) for antonym vs synonym distinction. ice-net uses multiple different encoders to capture relation-specific properties of antonym and synonym pairs from pre-trained embeddings in order to augment the end-performance of the antonyms vs synonyms distinction task. specifically, it uses: (i) an encoder (enc-1) to capture the symmetry of synonyms; (ii) an encoder (enc-2) to model the symmetry for antonyms; and (iii) an encoder (enc-3) to preserve the transitivity of the synonyms and trans-transitivity of antonym and synonym relation pairs by employing attentive graph convolutions. these relation-specific properties of antonym and synonym relation pairs are illustrated in figure 1 (a) and explained in section 3.1. we are the first to make an attempt to use attentive graph convolutions for modeling the underlying characteristics of antonym and synonym relation pairs. note, this work is different from existing works using graph convolutional networks for relational data, e.g., @xcite , as antonyms and synonyms possess unique properties which makes them different from relation pairs in the knowledge graphs (kg), e.g., fb15k, wn18 @xcite . ice-net is a shift from the existing instancebased modeling approaches to graph-based framing which allows effective information sharing across multiple instances at a time to perform the end classification in a performance-enhanced fashion. ice-net can be used with any available pre-trained embedding resources, which makes it more flexible than the existing approaches relying on huge text corpora. we summarize the major contributions of this paper as follows: ## related work earlier research on antonym synonym distinction attempts at capturing lexico-syntactic patterns between the word pairs co-occurring within the same sentence. @xcite considered pharasal patterns: "from x to y", and "either x or y" to identify synonyms amongst distributionally similar words. baroni and bisi (2004) used co-occurrence statistics to discover synonyms and distinguish them from unrelated terms. @xcite used word alignment measures using parallel corpora from multiple different languages to capture synonyms. @xcite used a set of seed pairs to capture patterns in the data and later used these patterns to extract new antonym pairs from text corpora. @xcite proposed discourse markers as features alternate to the lexico-syntactic patterns. @xcite proposed automated routines to acquire a set of symmetric patterns for word similarity prediction. @xcite proposed antsynnet that uses a set of lexico-syntactic patterns between the word pairs within the same sentence captured over huge text corpora. in the recent past, embedding models have received considerable research attention for antonyms vs synonyms distinction. these models are based on distributional hypotheses, i.e., words with similar meanings co-occur in a similar context @xcite @xcite . a major advantage offered by the embedding-based approaches is the freedom to curate and train embedding vectors for features extracted from text corpora. @xcite used skip-gram modeling to train embedding vectors using coreference chains. @xcite used lexical contrast information in the skip-gram model for antonym and synonym distinction. @xcite uses dictionaries along with distributional information to detect probable antonyms. @xcite used a set of encoder functions to project the word embeddings in constrained subspaces in order to capture the relation-specific properties of the data. @xcite employed a mixture-of-experts framework based on a divideand-conquer strategy. they used a number of localized experts focused on different subspaces and a gating mechanism to formulate the expert mixture. we observe some of the limitations of the existing work as follows. the pattern-based approaches are limited owing to the noisy and overlapping nature of the patterns. the embedding models are limited by the challenges posed by the distributional nature of the word embeddings, e.g., in glove embeddings top similar words for the word "small" yields a combination of synonyms, antonyms, and irrelevant words @xcite . these properties are depicted in figure 1 (a ). for ease of interpretation, we use (h, r, t) to represent a relation tuple, where h corresponds to the "head" and t is the "tail" of relation r. for word pair (h, t) and relation r, symmetry implies (h, r, t) iff (t, r, h). the transitivity between the relation implies: if (h, r, t) and (t, r, t ′ ) hold then (h, r, t ′ ) also holds, as shown by the words "nasty" and "horrible" in figure 1(a) . trans-transitivity implies: if (h, r a , t) and (t, r s , t ′ ) hold then (h, r a , t ′ ) also holds, also illustrated between the words "nasty" and "pleasing". @xcite pointed out a key limitation of the translational embedding methods (commonly used for kg embeddings) in modeling symmetric relations. for instance, for a symmetric relation r, it is not possible for translational embeddings to preserve both vector operations: h + @xmath0. this is also illustrated in figure 1(b) , where we show t ′ ̸ = t. for details refer to the original article by @xcite . likewise, some of the key difference of our work from existing work, i.e., r-gcn by @xcite are explained in appendix a.2. h r t h + @xmath1 ## proposed approach given that existing kg embeddings are not able to model the relation-specific properties of the antonym and synonym pairs, we propose ice-net that takes pre-trained word embeddings as inputs and projects them to low-dimensional space. in order to ensure that low-dimensional space captures the relation-specific properties of the data to the best possible extent ice-net uses three different encoder networks. we call overall architecture as interlaced structure, because these networks are interconnected, i.e., (a) loss function of enc-2 also depends upon enc-1, (b) output of encoders (enc-1, and enc-2) is used as input to the enc-3. details about each encoder are as follows: ## analyses in this section, we perform a detailed analyses of the ice-net under different settings, namely: (i) dlce embeddings @xcite , (ii) lexical split, (iii) ablation analysis, and (iv) error analyses. ## conclusion & future work in this work we propose ice-net, which uses a set of interlaced encoder networks to capture the relation-specific properties of antonym and synonym pairs, i.e., symmetry, transitivity, and transtransitivity, in order to perform antonyms vs synonyms distinction task. results show that ice-net outperforms the existing research by a relative score of up to 1.8% for f1-measure. some promising future directions include: (i) using domainspecific text corpora along with training seeds, (ii) strategy to cut down the attention weights for the erroneous edges.