Top-Rank-Focused Adaptive Vote Collection for the Evaluation of Domain-Specific Semantic Models (2020)

Abstract: The growth of domain-specific applications of semantic models, boosted by the recent achievements of unsupervised embedding learning algorithms, demands domain-specific evaluation datasets. In many cases, content-based recommenders being a prime example, these models are required to rank words or texts according to their semantic relatedness to a given concept, with particular focus on top ranks. In this work, we give a threefold contribution to address these requirements: (i) we define a protocol for the construction, based on adaptive pairwise comparisons, of a relatedness-based evaluation dataset tailored on the available resources and optimized to be particularly accurate in top-rank evaluation; (ii) we define appropriate metrics, extensions of well-known ranking correlation coefficients, to evaluate a semantic model via the aforementioned dataset by taking into account the greater significance of top ranks. Finally, (iii) we define a stochastic transitivity model to simulate semantic-driven pairwise comparisons, which confirms the effectiveness of the proposed dataset construction protocol.

## introduction in recent years, we have been witnessing a growth of natural language processing (nlp) applications in a wide range of specific domains, such as recruiting (inda; @xcite , law @xcite , oil and gas @xcite , social media analysis (alrashdi and o'keefe, 2019), online education @xcite , and biomedical @xcite . embedding-based models have been playing a crucial role in this specialization, as they allow the application of the same learning algorithm to a variety of different corpora of unlabeled texts, obtaining domain-specific models @xcite @xcite @xcite @xcite . the evaluation and validation of a domainspecialized model requires manually-annotated domain-specific datasets @xcite @xcite . however, the construction of such datasets is a very resourceconsuming process, and particular care is needed to ensure their ability to evaluate the desired features @xcite @xcite . in particular, it is fundamental to carefully consider the so-called downstream task (i.e., the final purpose of the model), because the appropriate evaluation metric depends on this task @xcite @xcite @xcite . semantic similarity and relatedness are related but distinct notions in linguistics, the first being associated with concepts which share taxonomic properties and being maximized by synonyms; on the other hand, semantically related concepts can share any kind of semantic relation, including antonym @xcite . these notions underlie the downstream tasks of countless nlp applications, including information retrieval @xcite @xcite @xcite @xcite , content-based recommendation @xcite @xcite , semantic matching @xcite @xcite , ontology learning and knowledge management @xcite @xcite , and word sense disambiguation @xcite . in view of the widespread of these applications, we propose a methodology to construct appropriate domain-specific datasets and metrics to assess the accuracy of relatedness and similarity estimations. in particular, due to its suitability for non-expert human annotation, we mainly focus on semantic relatedness; however, the proposed protocol can be easily extended to semantic similarity. a standard approach to evaluate a relatednessbased model is the comparison of the semantic ranking it produces with the corresponding ranking determined from human annotations. however, the relevance of rank mismatches may depend on the involved positions; in particular, top ranks are considered more important in many contexts, two prominent examples being content-based recommenders @xcite @xcite and semantic matching @xcite @xcite . the greater significance of top ranks compared with low ranks is actually a pretty common phenomenon, as it can be argued from the attempts to overweight the former in the context of ranking correlation @xcite @xcite @xcite @xcite . our contribution is framed within the requirement to create domain-specific datasets to evaluate semantic relatedness measure with particular focus on top ranks and is threefold. (i) in section 2, we define a protocol for the construction, based on adaptive pairwise comparisons, of a relatednessbased evaluation dataset tailored on the available resources and optimized to be particularly accurate in top-rank evaluation. (ii) in section 3, we define appropriate metrics to evaluate a semantic model via the aforementioned dataset by taking into account the greater significance of top ranks; the proposed metrics are extensions of well-known ranking correlation measures and they can be used to compare rankings, independently from their origin, whenever top ranks are particularly important. finally, (iii) in section 4.1, we define a stochastic model to simulate semantic-driven pairwise comparisons, whose predictions (described in section 4.2) confirm the effectiveness of the proposed dataset construction protocol; more in detail, we adapt a stochastic transitivity model, originally defined in the context of comparative judgment, in order to make it suitable for either similarity-driven or relatedness-driven comparisons. ## dataset construction in this section, we describe and justify a methodology to construct a dataset for the evaluation of a domain-specific relatedness-based model. relatedness-based evaluation -known as intrinsic evaluation in the context of embedding-based models -requires the construction of a dataset of human annotations, which may be collected via two different approaches. the former relies on a small group of linguistic experts to create a gold standard dataset, which is reliable but very expensive and, due to the subjectivity of relatedness and to the limited number of annotations, highly susceptible to bias and lack of statistical significance @xcite . the latter relies on a large group of non-experts, typically associated with a crowdsourcing service (e.g., amazon mturk, prolificacademic, socialsci, crowd-flower, clickworker, crowdsource), it is typically more affordable, and it has been proven to be repeatable and reliable @xcite . in the next sections we describe and justify a protocol to construct a dataset based on semantic relatedness between pairs of tokens foot_0 collected via a crowdsourcing approach. to simplify the reading of the paper, figure 1 shows the main steps for the practical construction of a dataset within the proposed approach, while ## evaluation metrics although the approximations of the borda scores described in sections 2.4 and 2.5 can be thought as estimates of the semantic relatedness, we rely on rankings rather than scores to avoid inconsistency issues that frequently emerge in score comparisons @xcite . @xcite proposed the quite general form for a ranking correlation coefficient in order to take into account the larger importance of top ranks in our context, we define weighted versions of ρ and τ , with increasing weight at the increasing of the rank position. namely, we define (i) different weighting schemes have been proposed in the literature @xcite ; here we adopt the additive scheme with w a @xmath0, where f (n) is a monotonically decreasing function, in view of its ability in discriminating different rankings even when they only differ by the exchange of a top rank and a low rank @xcite . a common choice is f (n) = 1/n @xcite ; however, in the large n items limit, it causes the divergence of the denominator in equation 9 and makes thus any w i negligible. this phenomenon is responsible for the decreased sensitivity on top ranks, observed by @xcite , in case of long rankings. for this reason, we prefer to use f (n; n 0 ) = 1/(n + n 0 ) 2 , where the offset n 0 has been introduced to control the weight fraction associated to the first rank in the large n items limit, i.e., r(n 0 ) = f (1; n 0 )/ ∞ @xmath1; n 0 ), which can be expressed as r(n 0 ) = 1/[(n 0 + 1) 2 ψ (1) (n 0 + 1)], where ψ (1) (x) is the first derivative of the digamma function. with this choice, both ρ w and τ w defined in equation 8 represent a family of correlation coefficients, depending on the value of n 0 , whose choice depends on the particular task (namely, on the relative importance of the first rank). the value n 0 = 0 causes an extremely high sensitivity on the first rank (r(0) ∼ 0.61), which may be excessive; hereafter, we focus therefore on the value n 0 = 2, which appears to be a reasonable trade-off (r(2) ∼ 0.28) that allows focusing on the first rank while avoiding neglecting other ranks. the metrics ρ w and τ w are suitable to compare rankings, whenever top ranks are particularly important; in particular, they can be used to evaluate a semantic model using a dataset produced as described in this paper. ## evaluation of the data-collection framework the collection of human annotations to construct a domain-specific dataset is a resource-consuming process, even within the proposed optimized data collection approach, whose person-hours cost can be estimated as ## conclusion & future work in this paper, we provided a protocol for the construction -based on adaptive pairwise comparisons and tailored on the available resources -of a dataset, which can be used to test or validate any relatedness-based domain-specific semantic model and which is optimized to be particularly accurate in top-rank evaluation. moreover, we defined the metrics ρ w and τ w , extensions of well-known ranking correlation coefficients, to evaluate a semantic model via the aforementioned dataset by taking into account the greater significance of top ranks. finally, we defined a stochastic transitivity model to simulate semantic-driven pairwise comparisons, which allows tuning the parameters of the data collection approach and which confirmed a significant increase in the performance metrics ρ w and τ w of the proposed adaptive approach compared with the uniform approach (see as future work, we plan to collect human annotations (i) to test the proposed data collection approach on real data and (ii) to assess the validity and estimate the parameters of the proposed stochastic transitivity model. additional future investigations may include a deeper analysis of the mathematical and statistical properties of the weighted coefficients ρ w , τ w , as well as a rigorous derivation of the optimal values for the parameters of the data collection approach.