Automatic Identification of Basic-Level Categories (2018)

Abstract: Basic-level categories have been shown to be both psychologically significant and useful in a wide range of practical applications. We build a rule-based system to identify basic-level categories in WordNet, achieving 77% accuracy on a test set derived from prior psychological experiments. With additional annotations we found our system also has low precision, in part due to the existence of many categories that do not fit into the three classes (superordinate, basic-level, and subordinate) relied on in basic-level category research.

## introduction wordnet organizes concepts into a hierarchy of hypernyms and hyponyms @xcite . while wordnet also identifies other information, such as meronymy, one interesting property that is not currently captured is which concepts represent basic-level categories. this is an important and valuable property to capture. @xcite first noted that, although there are many terms that could be used to refer to an object at different levels of abstraction, "it often happens that a hierarchy develops in both directions from a middle level of abstraction." @xcite called this the basic-level, identifying psychological advantages basic-level categories have as well as psychological tests to find these concepts in a hierarchy. examples of basic-level categories include table, car, tree, bird, guitar, shirt, fish, and apple @xcite . unfortunately, though, the process of identification does not scale well and only dozens of these concepts have been identified in the psychology literature @xcite . while there has been little work to automate the identification of basic-level categories (discussed in section 2), knowing the basic-level has been shown to be valuable. knowing the basic-level helps with word sense disambiguation @xcite , image searches @xcite , ad targeting @xcite , accurately measuring the readability of a text @xcite ), making search result entity cards more easily consumable @xcite , linking together different domain-specific information classification systems @xcite , and user-centered design of image-browsing interfaces @xcite . we also believe it could help with having a common set of words to work from in building wordnets for other languages, as well as language grounding and many other problem areas. given the wide variety of demonstrated applications of this information as well as the opportunity for application in other areas, we attempt to automate the identification of basic-level categories. we specifically look at heuristics to identify the basic-level noun categories in the princeton wordnet of english @xcite , hereinafter pwn. one author assigned this task as a project in a class he taught in 2010 and 2011. this work builds on the various techniques students used and combines them with novel rules into a rule-based system to identify basic-level categories. ## data we are aware of two major lists of basic-level categories as well as corresponding superordinates and some subordinates. the original experiments that started much of the work in this area @xcite ) include nine superordinate taxonomies for their first two experiments. for the three of these superordinates falling in the biological taxonomy, the experimental results showed the presumed superordinate level (tree, fish, bird) is actually the basiclevel. so, for these three groups the taxonomy was shifted down one level (e.g. basic to supordinate) and new superordinates (plant, animal, animal) were added to ensure the experimental results were accounted for. additionally, eight additional basic-level categories were used in their later experiments 3-4 @xcite ), so these were also added. markman and wisniewski (1997) also provide a large list of superordinates, basic-level categories, and subordinates, though there is overlap with the aforementioned list. a summary of the lists is shown in ## our approach we start with 29 student projects each independently trying to solve this problem, cataloging the types of approaches and rules considered and then combining a slightly-constrained set of these, as well as novel rules, into a combined system. while the goal is to produce one system by evaluating the collective set of rules, some boundaries are needed to constrain this. for example, one student only considered words also appearing in the 'adventure' category of the brown corpus @xcite , a small, categorized corpus of english, which restricts the project beyond the goals of this work. we therefore start with a general approach common to most solutions (section 4.1), describe the relevant rules (section 4.2), experimenting to determine which filtering rules are more and less effective (section 5.1), and then combine the more effective rules into a combined system before experimenting with a set of voting rules (section 5.2). ## experiments for the purpose of evaluation, we mapped the gold standard labels mentioned in section 3 to synsets in pwn. some categories, such as green seedless grapes and double knit pants, did not have corresponding pwn synsets and were discarded. the labels also included four superordinates under which the psychology experiments and pwn had substantial incompatibilities, and these were also discarded. for example, whereas one superordinate in psychology experiments was taken to be exercise equipment @xcite , the three basic-level categories underneath this mapped to very different hypernym trees in pwn: sports equipment, exercise device, and even athletic facility. we then divided the mapped categories into a train, development, and test set. this division was done manually at the superordinate level rather than completely randomly because there are several hypernyms with many basic-level categories labeled underneath them and having those split across sets may result in reporting better-thanreal-world results as a result of learning locationspecific patterns. instead, splits have been made manually at higher levels in the hypernym hierarchy, though the available labels leave out significant portions of the pwn hypernym hierarchy so this is still imperfect. the number of categories in each set is shown in ## evaluation our system's overall performance on the test data is listed in accuracy is measured as the percentage of categories filtered (or not filtered) correctly based on the test data. our system did well at filtering out superordinates, made a moderate number of mistakes filtering out basic-level categories, and was least successful at filtering out subordinates. just as when tuning the filtering rules on the development set, there was a substantial degradation in performance when extrapolating to the test set. results on the development set, including subsystem breakdowns, are shown in the filtering rules provide the most substantial portion of the impact as measured on the development set with 77% accuracy, while the voting rules improved accuracy by 17 points to 94%. comparing this to the results on the test data from the labels used here are sparse and non-random, collected from psychological research papers which had experimental reasons to control the ratios of subordinates to basic-level and superordinate categories. as an additional measure of system performance, we manually annotated a random set of 250 categories predicted to be at the basic-level by our system to estimate the precision of the system. unfortunately, the estimated precision was only 10.4% (26 of 250). this annotation was done by two annotators using as a standard the property from @xcite that basiclevel categories are the most inclusive level at which a concrete picture of the category as a whole can be formed (previously mentioned in section 2.1). this was chosen because it was a simple mental test to perform unlike many of the other properties and it was a pattern observed in all of the basic-level categories from that experiment @xcite . that same experiment was one of the underlying sources of our labeled data. the inter-annotator agreement is 92% and the kappa score is 59%. disputes were resolved through discussion. our system predicts 13,082 synsets are basiclevel. using our accuracy on the basic-level as a measure of recall, combined with our estimated precision, we estimate that there are around a total of 1,620 basic-level categories in pwn. this is a quantity we have not previously seen estimated. judging by the examples predicted as basiclevel in our estimate of precision, there are some systematic errors in cases where the system predicts a category is basic-level but it turns out not to be. the most interesting type of mistake accounted for just over half of the mistakes. in this case, the categories were not basic-level but also were not clearly superordinates or subordinates either, at least as described in the psychology literature. in the psychology experiments, the focus is primarily on physical objects and organisms. rather, there were many examples where the word was a noun describing an action (e.g. violence), denoting a relation (e.g. proportion), or denoting a role in a more complicated semantic frame (e.g. defalcation). it is possible we are being too restrictive in our labeling, but it appears to us that there are many nominal categories which describe things that belong to classifications other than superordinate, basic-level, and subordinate. this suggests the low precision is not just due to the prevalence of subordinates relative to basic-level categories (which is an issue). in addition, though, much of the imprecision may be due to phenomena outside our limited theoretical label space. we are making our list of predicted basic-level categories available for download at http://e22pii.com/research/files/gwc2018/predicted_basic_level_categories_synsets.txt. the labels we used, mapping labeled words in the psychology literature to pwn synsets, is also available at http://e22pii.com/research/files/gwc2018/labels.txt ## conclusion we built a rule-based system to automatically identify basic-level categories using pwn. we were effective at including most basic-level categories and excluding superordinates, but not as effective at excluding subordinates. we were 77% accurate overall at classifying our limited test data derived from psychological experiments. however, we have evidence that suggests these labels are based on a simplistic view that divides categories into 3 groups which do not appear to cover the full range of phenomena described by nouns. outside of this limited test data, we manually annotated a sample of our system's predicted basic-level categories and found low precision with the majority of the mistakes outside of these three groups. this suggests that for greater broad-coverage accuracy it may be necessary to model cross-part-of-speech relationships and other phenomena that do not fit nicely in the existing label space. in the future, we hope to refine and scale a labeling process using mechanical turk to build a larger and less-biased training set. we hope to rely on several of the tests in the psychology research, although modeling additional phenomena may require extending these tests. additionally, we hope to build a machine learning-based system to turn the many weak rules we have into features that can help improve system performance, as well as to evaluate the system on a much larger test set with this rule-based system as a baseline.