R a TE : a Reproducible automatic Taxonomy Evaluation by Filling the Gap (2023)

Abstract: Taxonomies are an essential knowledge representation, yet most studies on automatic taxonomy construction (ATC) resort to manual evaluation to score proposed algorithms. We argue that automatic taxonomy evaluation (ATE) is just as important as taxonomy construction. We propose RaTE, an automatic label-free taxonomy scoring procedure, which relies on a large pre-trained language model. We apply our evaluation procedure to three state-of-the-art ATC algorithms with which we built seven taxonomies from the Yelp domain, and show that 1) RaTE correlates well with human judgments and 2) artificially degrading a taxonomy leads to decreasing RaTE score.

## introduction a domain taxonomy is a tree-like structure that not only aids in knowledge organization but also serves an integral part of many knowledge-rich applications including web search, recommendation systems and decision making processes. taxonomies are also inevitably used as business and product catalogs and for managing online sales. notable taxonomy products in this domain include amazon category taxonomy, 2 google product taxonomy, 3 yelp business category foot_3 and google content categories. 5 recent years have witnessed interest in new automatic taxonomy construction (atc) systems, but there are no systematic methods for objectively evaluating their figure of merit. for instance, taxo-gen @xcite ) -see section 3 -was evaluated by asking at least three human evaluators if a taxonomy concept pair contains a hypernymy relationship, which can lead to bias and low reproducibility. it is not only difficult to compare or rank different algorithms, but changing the hyperparameters or settings of a parameterized atc system can also result in drastically different outputs, and make optimization unfeasible. because ontologies and taxonomies in particular are typically created in contexts to address specific problems or achieve specific goals, e.g. classification, their evaluation is evidently contextdependent, and many researchers actually believe that a task-independent automatic evaluation remains elusive @xcite . still, researchers have argued that objective evaluation metrics must be well available for significant progress in the development and deployment of taxonomies and ontologies @xcite . in this work, we propose rate, a reproducible procedure for automatic taxonomy evaluation. rate does not require external knowledge but instead depends on masked language modelling (mlm) to query a large language model for subsumption relations. we show that with some care, mlm is a valuable proxy to human judgments. we apply rate to the yelp corpus (a corpus of restaurant reviews) ranking seven taxonomies we extracted using three state-of-the-art atc systems. we observe it correlates well with our manual evaluation of those taxonomies, and also show that artificially degrading a taxonomy leads to a decrease of score proportional to the level of noise injected. in the remainder, we discuss related work in section 2. in section 3, we describe the atc systems we used for building up our taxonomies, and their evaluation procedures. we then present rate in section 4 including refinements that we found necessary for our approach to work. we report in section 5 the experiments we conducted to demonstrate the relevance of rate, and conclude in section 6. gold standard evaluation focusses on comparing and measuring the similarity of the target taxonomy with an existing ground truth such as word-net @xcite , @xcite . semantic similarity metrics have been proposed, including @xcite , @xcite and lin @xcite . we include in this category specific measures such as topic coherence @xcite which scores the quality of a word cluster which rely on similarity measures. there are several issues with such a process: mapping concepts from the output system to the ground truth is not trivial and gold standards do not necessarily cover well the domains of interest. application-based evaluation is an attractive alternative to gold-standard evaluation. @xcite for instance proposed several possible applications for evaluation including conceptpair relation classification. @xcite underlines however that it is in fact hard to correlate ontology quality with the application performance. data-driven evaluation intends to select the ontology o with the best structural fit to a target corpus c, which boils down into estimating p (c|o) as in @xcite . practically however, it remains unclear how to approximate such conditional probability. ## automatic taxonomy extractors in this work, we replicated results of three stateof-the-art atc systems that are publicly available and that are producing quality results on selected datasets and domains. in this section, we describe those systems and discuss their corresponding evaluation methods. ## rate a critical part of taxonomy/ontology evaluation is knowledge about subsumptions, e.g. "is fluorescence spectroscopy a type of fluorescence technology?" or "is crj200 a bombardier?". thus, rate measures the accuracy of the hypernym relations present in a taxonomy we seek to evaluate. the main difference between our work and earlier ones is that we do not rely on human judgments to determine the quality of a parentchild pair, nor do we consider an external reference (that often is not available or simply too shallow). instead, we rely on a large language model tasked to check subsumption relations. ultimately, an optimized language model should be able to generate an accurate list of the most canonical hypernyms for a given domain, similar to domain experts. but because we are mainly interested in domain-specific taxonomies, there is a high risk that specific terms of the domain are not well recognized by the model, and therefore, we investigate three methods for increasing the hit rate of hypernymy prediction of taxonomy subjects and reducing false negatives by (1) creating various prompts, (2) fine-tuning mlms with different masking procedures, and (3) extending the model's vocabulary with concept names. ## experiments we conducted our experiments on the yelp corpus which contains around 1.08m restaurant reviews such as the one in figure 2 (top box). we used the very same corpus prepared by huang et al. (2020). 8 5.1 taxonomies we created seven taxonomies for evaluation using the atc systems mentioned in section 3. here our goal was to obtain meaningful taxonomies that best cover the yelp domain using each taxonomy extractor. we did so by experimenting with different extractor settings and input. for taxogen, we only had to specify some parameters. 9 for corel and hiexpan however, we had to provide a seed taxonomy. hence we produced 5 such taxonomies using corel and hiexpan by providing frequentlyappearing parent-child pairs in the seeds. 10 ## discussion we presented rate, a procedure aimed at automatically evaluating a domain taxonomy without gold standard references or human evaluations. it relies on a large language model and an unmasking procedure for producing annotations. we tested rate on the yelp corpus which gathers restaurant reviews, and found that it correlated well with human judgments, and (artificially) degrading a taxonomy led to a score degradation proportional to the amount of noise injected. still, we observed that the quality of the language model predictions varies according to the strategies used to fine-tune them. there remains a number of avenues to investigate. first, we have already identified a number of decisions that could be revisited. in particular, we must test rate on other domains, possibly controlling variables such as the size of the fine-tuning material or the frequency of terms. second, rate is an accuracy measure, and depending on the evaluation scenario, it should eventually be coupled with a measure of recall. last, an interesting avenue is to investigate whether rate can be used to optimize the hyper-parameters of an atc system.