Eliminating Fuzzy Duplicates in Crowdsourced Lexical Resources (2016)

Abstract: Collaboratively created lexical resources is a trending approach to creating high quality thesauri in a short time span at a remarkably low price. The key idea is to invite non-expert participants to express and share their knowledge with the aim of constructing a resource. However, this approach tends to be noisy and error-prone, thus making data cleansing a highly topical task to perform. In this paper, we study different techniques for synset deduplication including machine- and crowd-based ones. Eventually, we put forward an approach that can solve the deduplication problem fully automatically, with the quality comparable to the expert-based approach.

## introduction a wordnet-like thesaurus is a dictionary of a special type that represents different semantic relations between synsets-sets of quasi-synonyms @xcite . it is a crucial resource for addressing such problems as word sense disambiguation, search query extension and many other problems in the fields of natural language processing (nlp) and artificial intelligence (ai). typical semantic relations represented by thesauri are synonymy, antonomy (primarily for nouns and adjectives), troponymy (for verbs), hypo-/hypernymic relations, and meronymy. a good linguistic resource should not contain duplicated lexical senses, because duplicates violate the data integrity and complicate addition of semantic relations to the resource. therefore, removing duplicated synsets from thesauri is an important problem to be addressed, especially in collaboratively created lexical resources like wiktionary, which is known to suffer this problem @xcite . however, deduplication is rather problematic because thesauri may contain fuzzy duplicated synsets composed of different words. the work, as described in this paper, makes the following contributions: (1) it proposes an automatic approach to synset deduplication, (2) presents a synonymic dictionary-based technique for assessing synset quality, and (3) compares the proposed approach with the crowdsourcing-based one. the rest of this paper is organized as follows. section 2 reviews the related work. section 3 defines the problem of synset duplicates existing in thesauri. section 4 presents a novel approach to synset deduplication. section 5 describes the experimental setup. section 6 shows the obtained results. section 7 discusses the interesting findings. section 8 concludes the paper and defines directions for future work. ## related work one of the most straightforward ways to clear a thesaurus of sense duplicates is to align its entries with another resource of proven quality, e.g. using the ontoclean methodology proposed by @xcite . consequently, synsets that will be linked with one synset from another resource represent the same concepts, and should be merged. however, such alignment can be performed only manually. it is also a time-consuming process that requires careful examination of every synset by an expert. therefore, it is crucial to fo-cus on methods that are either automatic or involve lesser amount of human intervention. many studies nowadays aim to evaluate the feasibility of crowdsourcing for various nlp problems. for instance, @xcite showed that non-expert annotators can produce the data whose quality may compete with the expert annotation in such tasks as word sense disambiguation and word similarity estimation (they conducted their study using amazon mechanical turk foot_0 (amt), a popular online labor marketplace). @xcite assumed that semantically related words tend to co-occur in texts. given such an assumption, they managed to find and eliminate the words that had been added to synsets by mistake. this approach can be used to find sense duplicates, but it requires a large amount of semantic relations to be present in a resource. it should be noted that some resources that contain synsets may not contain any links between them. for instance, wiktionary represents certain words and relations between them, but it does not explicitly link its synsets. @xcite presented a method for semi-automatic enrichment of the wiktionaryderived synsets. first, they analyzed the contents of wiktionary and produced new synonymy relations that had not been previously included in the resource. after that, they invited collaborators to manually process the data using a custom firefox plugin to add missing synonyms to the data. a similar approach was used by @xcite to bootstrap yarn (yet another russ-net) project, which aims at creating a large open wordnet-like machine-readable thesaurus for the russian language by means of crowdsourcing. in this project, a dedicated collaborative synset editing tool was used by the annotators to construct synsets by adding and removing words. the most recognized crowdsourcing workflow is the find-fix-verify pattern proposed by bernstein et al. and used in soylent, a microsoft word plugin that submits human intelligence tasks to amt for rephrasing and improving the original text @xcite . as the name implies, the workflow includes the three stages: (1) in the find stage crowd workers find the text area that can be shortened without changing the meaning, (2) in the fix stage the workers propose improvements for these text areas, and (3) in the verify stage the workers select the worst proposed fixes. inspired by this pattern, @xcite presented the add-remove-confirm workflow for improving synset quality. similarly, it contains three stages: (1) in the add stage workers choose the words to be added to a synset from a given list of candidates, (2) in the remove stage the workers choose the words that should be removed from a synset, (3) in the confirm stage the workers choose which synset is better-the initial one or the fixed one. ## problem in our study, we focus on the synsets represented in a wordnet-like thesaurus. hence, we regard a thesaurus as a set of synsets s, where every synset @xmath0. in lexical resources created by expert lexicographers, synsets usually correspond to different meanings, so synset duplicates never arise. unfortunately, it is not true for the resources created by non-expert users, e.g. through the use of crowdsourcing. one approach to synset creation would be to combine manually constructed synsets with synsets that are imported from open resources. obviously, it is going to lead to the situation where there is a plenty of synsets representing identical concepts. the crowdsourcing approach to synset creation is also prone to this drawback, as the crowd is likely to create duplicate synsets. the following example from the russian wiktionary foot_1 shows that it contains synsets with identical meanings. for example, the synset {стоматолог (stomatologist), дантист (dentist), зубной врач ("tooth doctor")} and the synset {дантист (dentist), стоматолог (stomatologist)} definitely describe the same concept "a person qualified to treat the diseases and conditions that affect the teeth". hence, such synsets should be combined, yet they both are present in the russian wiktionary. note that in this example the second synset is a full subset of the first one; however, it is possible that two synsets may intersect only partly while sharing the same meaning. for a native speaker, it is relatively easy to detect whether two synsets share the same meanings. so, the detection may be done by non-experts via crowdsourcing. however, the key problem here is how to retrieve the pairs of synsets that presumably represent identical concepts. in the next section, we propose a simple, yet effective approach. ## approach suppose the word w has several meanings. according to @xcite , it is usually enough to provide one synonym for every meaning of w to a native speaker of a language to be able to distinguish the meanings from each other (provided that the speaker is familiar with the corresponding concepts). this phenomenon is widely exploited by explanatory dictionaries. it is also utilized in some thesauri which assume that a synset itself is enough to deduce its meaning, therefore definitions of synsets may be omitted. hence, we formulate the meaning deduplication problem as follows. given a pair of different synsets s 1 ∈ s and s 2 ∈ s, we treat them as duplicates if they share at least two words: obviously, this is a strong criterion that may be violated, so we propose the following two-stage workflow for synset deduplication. filtering. in this stage, the possible duplicates are retrieved using the above described criterion resulting in the set of synset pairs (s 1 , s 2 ) for further validation. voting. in this stage, the obtained synset pairs are subject to manual verification. the pairs voted as equivalent are combined. the assessment required in the voting stage may be provided by expert lexicographers; in crowdsourced resources, the contributors may be invited not only to add the new data, but also to increase the quality of the created data and to deduplicate it. ## experiments since task submission to amazon mechanical turk requires a u.s. billing address, this solution is not accessible to users from other countries. although there are many other crowdsourcing platforms, e.g. crowdflower, microworkers, prolific academic, etc., yet the proportion of russian speakers on such platforms is still low @xcite . given the fact that our workers are native russian speakers, we decided to use the open source crowdsourcing engine mechanical tsar foot_2 , which is designed for rapid deployment of mechanized labor workflows @xcite . inspired by the similar annotation study conducted by @xcite , we used the default configuration, i.e. the majority voting strategy for answer aggregation, the fixed answer number per task strategy for task allocation, and the no worker ranking. the workers were invited from vk, facebook and twitter via a short-term open call for participation posted by us. ## discussion the f 1 -measure shows no change after applying the voting stage, yet the precision increases by it is also of interest that yarn contains 24.8 thousand synsets that presumably have a duplicate (58% of the synsets with two or more words), while the russian wiktionary has 13.2 thousand (40%), and ruthes-lite has only 6.3 thousand (28%). we may therefore conclude that the proposed approach should mainly be applied to resources that a priori are known to contain duplicate synsets rather than to improve the quality of expert-built resources. ## conclusion in this study, we presented an automated approach to synset deduplication. the results were obtained from expert labels and annotations provided by crowd work. at least three different annotations per every synset pair from two different resources (yarn and ruthes-lite) were used. the approach allows to significantly increase the synset quality in crowdsourcing lexical resources. participation of people does not notably affect the average synset quality, though the precision slightly increases when people are involved. the results showed that two synonyms are not sufficient for defining a meaning, but three words usually give a satisfactory result. so, it is three words that should be used as a threshold value for merging duplicate synsets when using the proposed deduplication approach in a fully automatic mode. our results, including the crowd answers and the produced gold standard, are available foot_5 under the terms of creative commons attribution-sharealike 3.0 license. as a possible future direction, we may suggest using more sophisticated similarity measures to select a threshold for fully automatic merging of synsets. another possible way to improve the approach is to detect not just pairs, but clusters of synsets. this is hardly possible in resources that are manually crafted by a team of experts, but it is definitely worth exploring for crowdsourcing resources.