Experiences of Lexicographers and Computer Scientists in Validating E stonian W ordnet with Test Patterns (2016)

Abstract: New concepts and semantic relations are constantly added to Estonian Wordnet (EstWN) to increase its size. In addition to this, with the use of test patterns, the validation of EstWN hierarchies is also performed. This parallel work was carried out over the past four years (2011-2014) with 10 different EstWN versions (60-70). This has been a collaboration between the creators of test patterns and the lexicographers currently working on EstWN. This paper describes the usage of test patterns from the points of views of information scientists (the creators of test patterns) as well as the users (lexicographers). Using EstWN as an example, we illustrate how the continuous use of test patterns has led to significant improvement of the semantic hierarchies in EstWN.

## motivation there are many reasons for why test patterns should be chosen as a way to validate multiple inheritance in the wordnet hierarchical structure (formed by its semantics). to begin with, due to the nature of multiple inheritance, it requires checking. more precisely, multiple inheritance is prone to semantic errors: 1) inappropriate use of multiple inheritance @xcite . there are many cases where multiple inheritance is not used as a conjunction of two properties @xcite . 2) sometimes an is-a relation is used instead of other semantic relations @xcite . multiple inheritance makes it possible to compare relations that connect the parents of a synset. 3) in many cases, multiple inheritance causes topological rings @xcite , @xcite . according to @xcite , one synset cannot inherit properties from both parents. 4) multiple inheritance may refer to a short cut problem @xcite , @xcite , @xcite . one synset has a two-fold connection to another one, both directly and indirectly. the direct link is illegal. 5) multiple inheritance may refer to dangling uplinks in the hierarchical structure @xcite . secondly, the use of test patterns has many advantages: ## http://wordnetweb.princeton.edu/perl/webwn an example of a heart-shaped substructure in figure 2 originates from @xcite . the question arises why {homoeopathy_1} is not a subcase of {naturopathy_1}. secondly, are "mud cure, mud treatment" and {homoeopathy_1} subcases of {alternative medicine_1} or of {cura-tive_1, cure_1, …}? on the basis of the definitions of these concepts, the lexicographers decided that both are subcases of {curative_1, cure_1, …} and that {alternative medicine_1} is connected to them via a holonymy relation. there is still no thorough analysis of the heartshaped substructure. therefore, there is no such instance in the latest version of estwn. in addition, as discovered in @xcite , most of the cases of heart-shaped substructures in princeton wordnet pointed to situations where instead of a hypernymy relation there should have been a role or type relation. ## iterative evolution of estwn applying the test patterns to estwn has taken place gradually. as mentioned earlier, we began validating estwn with the closed subset test pattern. at that time, we studied approximately 20 instances of estwn and princeton wordnet. some of the results are reflected in two joint papers with kadri vare @xcite and @xcite . later, we started to use short cut as well as other patterns. in the iterative evolution of estwn, test pattern instances were separated with help of our programs and subsequently delivered to lexicographers who validated all instances and corrected wordnet semantic hierarchies where necessary. the bold font in the largest changes in the number of multiple inheritance cases appear when dense components are taken into use in version 66. this is due to the fact that dense component contains at least two or more multiple inheritance cases in one instance. in the paper of @xcite , it was discovered that only 12% (14) of 121 dense component instances do not need any correction. nevertheless, the next version (67) revealed 8 new instances. the decrease in the number of multiple inheritance cases continues even after version 67 when two more patterns are applied (heart-shaped substructure and "compound" pattern). in the last version, there are only 3 dense component instances and 0 heart-shaped substructure instances. comparing the numbers of multiple inheritance cases in versions 66 and 70, it is noted that the last number (51) is approximately 32 times smaller, i.e. multiple inheritance cases have been shrunk by approximately 97%. ## conclusion and future works the main collaboration between computer scientists and lexicographers in order to validate estwn (version 60) began with the closed subset test pattern. the closed subset was successful in finding possible errors in semantic relations. later, nine other test patterns dealing with multiple inheritance were developed (see more @xcite . two patterns, namely short cut and ring patterns are inspired from different authors and one pattern can in certain cases include a dangling uplink. in this paper, six test patterns were described but the examples covered four test patterns. typically, the work for using test patterns was organized as follows: the first author of this paper generated the instances of test patterns, then based on that document, the lexicographer made corrections using the estwn editing tool. the experience of validating estonian wordnet assured that the continuous usage of test patterns can significantly improve the semantic hierarchy. multiple inheritance decreased 32 times or 97% in the last five versions of estwn. in the future, we plan to apply these test patterns to other types of semantic relations, for in-stance to near synonymy, fuzzynymy and holonymy. moreover, as there are about 70 wordnets in the world, we believe that applying these test patterns to them may "automatically characterize their modelling decisions (i.e. potential modelling errors)" foot_4 .