Synthetic Textual Features for the Large-Scale Detection of Basic-level Categories in E nglish and M andarin (2021)

Abstract: Basic-level categories (BLC) are an important psycholinguistic concept introduced by Rosch et al. (1976); they are defined as the most inclusive categories for which a concrete mental image of the category as a whole can be formed, and also as those categories which are acquired early in life. Rosch’s original algorithm for detecting BLC (called cue-validity) is based on the availability of semantic features such as “has tail” for “cat”, and has remained untested at large. An at-scale algorithm for the automatic determination of BLC exists, but it operates without Rosch-style semantic features, and is thus unable to verify Rosch’s hypothesis. We present the first method for the detection of BLC at scale that makes use of Rosch-style semantic features. For both English and Mandarin, we test three methods of generating such features for any synset within Wordnet (WN): extraction of textual features from Wikipedia pages, Distributional Memory (DM) and BART. The best of our methods outperforms the current SoA in BLC detection, with an accuracy of English BLC detection of 75.0%, and of Mandarin BLC detection 80.7% on a test set. When applied to all of WordNet, our model predicts that 1,118 synsets in English Wordnet (1.4%) are BLC, far fewer than existing methods, and with a precision improvement of over 200% over these. As well as confirming the usefulness of Rosch’s cue validity algorithm, we also developed and evaluated our own new indicator for BLC, which models the fact that BLC features tend to be BLC themselves.

## related work cue validity @xcite ) (cv) is defined as the conditional probability of a category given a feature. the cue validity of a specific category is the sum of all cue validity scores for each attribute of this category. @xcite hypothesized that blc have higher cue validity scores compared to other categories. there is some disagreement in the literature as to the borderline between blc and other categories @xcite and the order of acquisition of blc and superordinate categories. rosch claimed that children learn blc concepts earlier than other concepts, but mandler produced evidence that only partially supports this statement. while the best performance in children across all ages was indeed for blc, mandler also found that some children were responsive to the superordinate and contextual categories (e.g. kitchen) very early on, even at only 12 months old. our own study of the aoa list by @xcite confirms that some superordinate categories such as "plant" and "furniture" are learnt before the age of 5. other factors may play a role too: @xcite claimed that objects that lend themselves best to becoming blc are those that are complex and possess structured information. there is also disagreement in the literature about the blc-like superordinates in natural kinds such as fish, tree and bird, which rosch counts as superordinates, but several researchers including ourselves consider blc @xcite . we will in this work rely on existing semantic feature norms derived experimentally. @xcite provided norms for 541 concepts (living and non-living objects) which they consider blc. the norms were collected using 725 participants. @xcite al. (2019 ) extended these feature norms into 4,436 concepts with combination of terms from vinson and vigliocco (2008), who collected norms of verbs as well; however, buchanan et al's list no longer attempts to include only blc, but additionally included both superordinates and subordinates. bulat et al. (2017) used 2,526 features from mcrae et al. (2005)'s semantic norms to build semantic representations for metaphor identification. motivated by the lack of large-scale feature norms, @xcite tested the performance of distributional semantic vector space @xcite for the task of generating properties for concepts in a visual world experiment. they measured eye fixations when participants looked at target concepts and established distracting contexts designed for priming participants. to create the contexts, they used three features generated by a distributional model. they observed that participants fixate on the target concept when listening to both the target and competitor word, and were thereby able to prove that the features automatically generated successfully biased the participants and therefore expressed the desired contexts sufficiently well. @xcite introduced a rule-based system for the detection of blc via wordnet @xcite and external knowledge such as frequency of words in brown and gutenberg corpora. when measured on a small test set of obvious cases from the literature, their model achieved an accuracy of 77%. however, when run on the entire wn, it predicted that 13,082 synsets out of the total 82,115 synsets (16%) are blc. this number seems far too high, and indeed, when they estimated the accuracy of their blc detection on a subset of those 13,082 synsets, it was found that only 10.4% are true positives. their prediction was therefore that the entire wn contains only around 1,620 synsets. we can see from this experiment that the true difficulty in the blc detection task may lie in achieving high ## approach we aim to identify blc in english and mandarin by more closely following rosch's theoretical approach; our target is to directly test whether using the properties suggested by her helps in large-scale blc detection. we recast the problem as a supervised ml task, in contrast to the rule-based system by @xcite . we also want to test whether other theoretical hypotheses about blc can be tested if suitable semantic features are available. we use the indicators listed in tab. 1, six of which have been used in earlier work or are obvious. the remaining two indicators are somewhat more involved in independently worthy of study: one indicator simulates rosch's cue validity concept, and the remaining one called blc-pr is due to us. it follows the notion that if blc are central, early-learnt concepts, then maybe the features associated with them could also be blc themselves, which might help and reinforce the early acquisition of these concepts. we simulate this indicator using the pagerank algorithm. the indicators word length, corpus frequency, age of acquisition (aoa), concreteness rating and wn depth can be derived for english as well as for mandarin, and do not require any semantic features. word length means the number of letters and characters, for english and mandarin respectively. number of brushstrokes is only defined for mandarin and expresses the intuition that culturally older, more important concepts would exhibit fewer brush strokes. we extracted the english frequency data from the american part of the google books corpus and chinese one from the tencent ai lab embedding corpus @xcite . concreteness ratings were drawn from a list of 3 million rated word forms @xcite , where each word has been given a 10-point rating from abstract (1) to concrete (10). we expect this to help in classification as blc are by definition never abstract. we also use a 300-dimension standard word2vec representation @xcite for english and 200-dimension one for mandarin @xcite . despite not having interpretable dimensions, we hypothesise that such representations may capture context in an indirect manner. we applied chinese open wordnet (cow) @xcite , which was built on top of the synsets of english wn, to match mandarin concepts to their english counterparts. cow contains 42,312 synsets, 27,888 of which are nouns, compared to the 82,115 nouns in wn. the acquisition age indicator we use is based on test-based age of acquisition norms of 30,000 words/senses with information of the age when children learned this word/sense (kuperman et al., 2012) 1 we successfully matched 22,138 of our (bilingual) wn synsets to the aoa test words/senses, as well to their concreteness ratings. the remaining indicators depend on our method of generating semantic features. we will therefore first describe how we generate the synthetic semantic features. ## experiment we perform an experiment in learning the blc status for every noun synset in wn based on the set of indicators introduced in section 3, using different generation methods for the synthetic features. our baseline is the method by @xcite , which uses no semantic features. the working hypothesis is that the use of semantic features should help blc detection, even if the generation of the features is sub-optimal. evaluation is performed in two ways: 1. for a small test/train set, blc detection is reported in terms of f-measure 2. for all synsets in wn, we perform a separate precision study using human subjects, in the same way as @xcite . ## further analyses and discussion we next performed some ablation studies (tab. 7) to investigate which indicators performed better than others. all features on their own are significantly different from the baseline (no features). concreteness on its own performs well, but this is somewhat uninteresting as blc are never concrete (and as the list of concrete distractors was created from this indicator). we also see that blc-pr on its own performs well, while cue validity on its own does not. word2vec stands out because while quite strong on its own, it is the only feature that decreases results. because it tends to group words under similar context, subordinate categories are likely to have high similarity score with their respective blc, which actually hurts the performance of our classifier. blc-pr also plays a beneficial role in detecting blc as we expected, proving that blc concepts tend to overall have more blc features. tab. 10 (appendix) shows the top 20 performing set of synsets for blc-pr, which reveals that 7 out of these are abstract. this may be caused by the fact that abstract concepts have many abstract features. we are particularly interested in the performance of cue validity, as it has never before been tested in a large-scale experiment. on its own, cue validity performs rather disappointingly and in particular misclassifies abstract concepts as blc (cf. tab. 9 ), but it manages to increase results in tandem with the other features. rosch devised cue validity as a metric in taxonomies of concrete objects; she never considered abstract concepts. therefore, cue validity may only work well in the categorization of concrete objects, not in the distinction between abstract and blc concepts. this is a particular project with experiments in wn, as abstract concepts also occupy a large part of nouns in wn (48,033 abstract synsets according to our classifier). given that blc definition is debated in the literature, our inclusion of the likely age of acquisition into the definition given to the judges deserves some justification. @xcite used only the first part of @xcite 's original definition of blc, concerning the concrete image of the category and inclusivity, but left out the second part of her definition concerning early age of acquistion. we included the second part of her definition because we wanted a definition that is easier to convey to judges. in a separate experiment, we verified the link between age of acquisition and blc in order to empirically estimate the age we would use in our instructions. using the list provided by @xcite we were able to verify that most of the concrete concepts a 4 year old child learns are blc, if we take the list of 86 blc defined by rosch as our gold standard: 88% of these rosch-blc are known to children of that age, although only 26% recall is achieved. higher ages of acquisition are listed in tab. 8. we are therefore fairly confident that the definition of blc we employed in our experiment is easily understandable and realistic. ## conclusion how to determine what is a blc and what not, from first principles, is a long-standing problem in computational psychology. we have presented here the beginning of a methodology for studying which aspects of blc might contribute to their blc-ness: the fact that they are universally recognised, learned early in language development, and have certain information-theoretic properties about categorisation as a language-wide phenomenon. in particular, we studied the generation of synthetic semantic features resembling human semantic norms from corpora, and which relationship these features have to blc. we showed that even imperfect features as available with current nlp techniques could serve to improve the detection of blc. we find that bart, a methodology borrowed from neural machine translation, is capable of generating indicators that are able to improve the detection of blc. this also contributes a potential solution to the general problem of providing feature norms automatically at scale, although that would also necessitate additional experiments about the stand-alone quality of the features. what we were able to empirically validate, however, is that they are good enough to substantially improve the soa in blc detection. we were also able to substantiate some longstanding claims in psycholinguistics about the validity of cue validity, first proposed by @xcite . our own hypothesis, that blc concepts tend to be described by semantic features that themselves are blc, was also validated to a certain degree. we hope that the current work opens the door for much-needed further research into synthetic semantic features. in future work, we plan to include visual features into our blc detection system. @xcite found that in early child vocabulary development, the shape of objects is more important than other features such as size, colors and texture. we also plan to pay more attention to methods that can distinguish subordinate terms from blc.