Combining W ord N et and Word Embeddings in Data Augmentation for Legal Texts (2022)

Abstract: Creating balanced labeled textual corpora for complex tasks, like legal analysis, is a challenging and expensive process that often requires the collaboration of domain experts. To address this problem, we propose a data augmentation method based on the combination of GloVe word embeddings and the WordNet ontology. We present an example of application in the legal domain, specifically on decisions of the Court of Justice of the European Union.Our evaluation with human experts confirms that our method is more robust than the alternatives.

## introduction many of the state-of-the-art natural language processing (nlp) techniques are based on deep learning methods with millions of parameters @xcite , and therefore they usually require vast amounts of data to be trained. even if a lot of progress has been made in the development of unsupervised or semisupervised methods, many high-level tasks are still addressed in a supervised fashion, especially when they concern complex tasks or very specific domains, such as predictions on legal documents @xcite @xcite . at the same time, creating corpora for such applications is particularly challenging and expensive since this process requires the collaboration of domain experts for the labeling process. one possible way to address this problem is data augmentation @xcite , which exploits existing data to generate new synthetic ones. these synthetic samples must be different enough from the original ones to provide a valuable contribution to the training. still, at the same time, their semantic content must remain similar enough not to invalidate their labels. in nlp, one possibility is to replace some words or sentences of the original samples with other ones that hold the same semantic meaning. this can be done by exploiting similarities between sub-symbolic representations of text, such as word and sentence embeddings, or exploiting relationships in symbolic representations, such as wordnet @xcite . inspired by works regarding semantic relatedness @xcite , we propose to merge graph-structured and embedding-based augmentation by combining the use of wordnet and similarity between word embeddings. in particular, we create new synthetic samples by replacing some terms with words with similar semantic meaning. we exploit wordnet to compute a set of candidate words and then choose the most similar one according to its glove word embedding @xcite . we present an example of the application of such a method in the legal domain. our context is a task of sentence classification, where we want to automatically predict whether a sentence extracted from a judgment is representative of a principle of law. since the distribution between the negative and positive classes is heavily unbalanced, we need to rely on data augmentation. we compare different techniques and ask a team of legal experts to evaluate the new synthetic data. their evaluation confirms that the quality of the synthetic data generated through our method is superior to data generated exploiting only wordnet or glove embeddings. our contribution is three-fold: we make our code, data, and evaluation publicly available. foot_0 2 related works data augmentation is a frequently used strategy in nlp to introduce diversity in the datasets that will help models overcome phenomena such as overfitting @xcite . in particular, paraphrasing-based data augmentation techniques @xcite aim to create new synthetic data preserving the meaning of the original source. one popular family of augmentation methods relies on knowledge graphs, thesauruses, and lexical database such as wordnet. wordnet (fellbaum, 2010) is a lexical database where words are grouped into sets of cognitive synonyms called "synsets". serving as a relational network, it is widely used as a source of synonyms and for the measurement of similarity between terms. for example, @xcite use wordnet to retrieve a list of synonyms of a word, and replace it with one chosen randomly. @xcite expand such approach by constraining candidates according to part of speech (pos) tags by selecting them based on a similarity measure, and test their approach on various text classification tasks. wang and @xcite follow a different approach and instead they rely on semantic embeddings, embedding words with word2vec and replacing candidate words with their nearest neighbour. our approach stems from xiang et al.'s and follows the intuition of wang and yang. we rely on wordnet to select a pool of candidate words, but we choose the replacement by measuring the similarity between their glove word embeddings @xcite . however, we provide a simpler definition of the candidate list considering the synsets collected from the wordnet opening room for syntactic differences while preserving the semantic integrity of the sentences. moreover, we address the challenging domain of legal documents, in which retaining domain-specific validity while introducing textual diversity is a critical factor. finally, we provide an evaluation of synthetic samples involving human experts. ## method our augmentation method augwn+gv combines the use of the lexical database wordnet (wn) with the properties of the vector space defined by glove pre-trained word embeddings (gv). given a sample sentence, composed of a list of words {w 1 , ..., w n }, we randomly choose one word to be replaced among those that are adjectives, nouns, or adverbs. we do so by computing the pos tags of each word p os w i through the nltk library and considering only the words for which p os w @xmath0, n n s, n n p, n n p s, jj, jjr, jjs, rb, rbr, rbs, rp }. 2 then, given a word w j to replace, we proceed as follows: we compared our method against four baselines: ## evaluation to perform a preliminary evaluation of our method, we generated a small set of synthetic samples and then asked domain experts to judge them. we also measure the difference between the augmented sentences and the original ones in terms of similarity between their embeddings. we generated the synthetic starting from a given textual sentence, randomly selecting one suitable candidate word in it, and applying one augmentation method to it. the original sample and the synthetic one thus obtained would therefore differ only for one term. this process was then applied multiple times to the synthetic sample, replacing other words and generating new samples. we repeated this process until we replaced about 60% of the candidate terms of the original sentence. ## conclusion we presented a data augmentation method that leverages both the symbolic information available in knowledge graphs and the sub-symbolic information provided by word embeddings. we have applied this technique to the challenging domain of legal documents and asked a team of experts to evaluate each replacement. the results confirm the quality of our method with respect to alternative approaches, yet they emphasize that more work is needed to obtain satisfactory results. we relied on glove since is a popular and widely adopted representation with a low computational footprint. nonetheless, our proposal can be adapted to other embeddings. in future work, we plan to further test this technique in a task-based setting where we train a machine learning model to recognize the sentences that contain a principle of law. moreover, we will apply it to other legal tasks where data is difficult to produce or where some classes are greatly underrepresented. examples of these tasks are argument mining @xcite @xcite and identification of unfair clauses in contracts @xcite @xcite .