QSTS : A Question-Sensitive Text Similarity Measure for Question Generation (2022)

Abstract: While question generation (QG) has received significant focus in conversation modeling and text generation research, the problems of comparing questions and evaluation of QG models have remained inadequately addressed. Indeed, QG models continue to be evaluated using traditional measures such as BLEU, METEOR, and ROUGE scores which were designed for other text generation problems. We propose QSTS, a novel Question-Sensitive Text Similarity measure for comparing two questions by characterizing their target intent based on question class, named-entity, and semantic similarity information from the two questions. We show that QSTS addresses several shortcomings of existing measures that depend on n -gram overlap scores and obtains superior results compared to traditional measures on publicly-available QG datasets. We also collect a novel dataset SimQG, for enabling question similarity research in QG contexts. SimQG contains questions generated by state-of-the-art QG models along with human judgements on their relevance with respect to the passage context they were generated for as well as when compared to the given reference question. Using SimQG, we showcase the key aspect of QSTS that differentiates it from all existing measures. QSTS is not only able to characterize similarity between two questions, but is also able to score questions with respect to passage contexts. Thus QSTS is, to our knowledge, the first metric that enables the measurement of QG performance in a reference-free manner.

## introduction automatic question generation (qg), the task of generating natural language questions for a given input text passage continues to garner significant research focus in the nlp community @xcite due its potential application in education @xcite , tutoring @xcite and interactive dialog systems @xcite . in current research, in lieu of human evaluation, the standard practice for evaluating the perfor-mance of qg models involves the use of question answering (qa) datasets containing pairs of (reference question, passage context) elements. for evaluating qg, the machine-generated question for a given passage context is compared with the given reference question by applying metrics such as bleu @xcite , @xcite , and rouge @xcite . the above widely-used measures were originally developed for evaluating tasks such as summarization and translation and are based on overlap of n-grams between a given reference text and the model-generated text. though studies have indicated that these measures do not correlate well with human judgements of fluency, relevance, and coherence @xcite @xcite these measures are easy to compute and continue to be used for various natural language generation (nlg) tasks. recently though, research studies are addressing metrics learning for nlg using transformers and these learnt metrics were shown to obtain stateof-the-art performance in evaluation @xcite . we posit that the existing metrics for measuring text generation tasks are inadequate for comparing questions due to their inability to incorporate various features that characterize questions. the first among these features is the question class (alternatively referred to as answer type) which places constraints on the answer to a given question. 1 for example, for the question, "who was lincoln?", in context of a passage on the former us president, a correct answer is most likely looking for a description referring to his job/role/occupation whereas the answer to the question "what is humidity?" is a definition. we argue that question class as well as named entities, when present in a question, directly affect the intent of the question and need to be the system-level scores of the traditional metrics (bleu, meteor, rouge), are shown in this table along with qbleu values (the extension of bleu scores for questions proposed by @xcite ), as well as bleurt scores that measure semantic similarity between two texts using bert @xcite . in we demonstrate reference-free evaluation for qg using qsts on a novel dataset, simqg. simqg contains human judgements for machine-generated questions from latest qg models for a selection of about 500 (reference question, passage) pairs from squad @xcite . simqg and an implementation of qsts in python have been made available for academic research. foot_1 organization: we present the details of comput-ing qsts in section 2. our novel dataset simqg is described in section 3. experiments and results are described in section 4 while closely-related work is summarized in section 5. finally, we conclude the paper with a summary and remarks on future directions in section 6. ## question-sensitive text similarity a necessary aspect to capture while comparing two questions is a measure of whether the target intent behind the two questions is the same. as highlighted in the examples from we use question class information in qsts to characterize if the two questions under consideration are seeking the same answer type. with an accurate question-class classifier, both questions in the first row of named-entities when present in a question constrain the question with reference to the mentioned entity. for instance, if "columbus" in the question "when was columbus born?" is replaced by another name, it will become a completely different question. therefore, similar to question classes, named entities require a hard measurement. to account for multi-word names and partial matches, we isolate the tokens in a given reference question referring to named entities and look for their presence in the generated question. 5 the named-entity similarity (nesim) is measured as the fraction of named-entity tokens in the reference question that are present in the generated question. that is, for a given reference question, "who was abraham lincoln?" and the question "who was lincoln?", the named-entity similarity score is computed as 1 2 . semantic similarity forms the third component of qsts. we use the dependency parse of ques-tions to compute the semantic similarity between them. dependency trees of sentences capture syntactic dependencies among the words in a sentence such as subject-object, modifier, and clausal links. dependency-tree based kernels are widely-used in measuring sentence similarity @xcite . let @xmath1, rel, t) represent a directed edge in the dependency tree of a given question where the typed relation rel exists between two tokens, h and t. given the dependency edges of two questions (reference and generated), we match the dependency edges of the reference question (e(q r )) with those of the generated question (e(q g )) and pick the best matching or the most similar edge ∀@xmath2. the edge similarity esim(e mr , e ng ) is computed as in the above formulation, δ(rel mr , rel ng ) refers to the kronecker δ function that assigns a value of 1 if the two relation types are the same and zero otherwise and sim(a, b) is computed using cosine similarity of the word embeddings if a, b are nonname tokens. however, if either the head or tail of the edge is a name token, we use exact match on that side of the edge. that is, for the two edges e @xmath3, rel mr , t mr ) and e @xmath4, rel ng , t ng ), if h mr is a name token, esim(e mr , e ng ) = the same principle applies if t mg is a name token. the above formulation ensures that name tokens are not treated like regular tokens and a hard match is enforced while at the same time the edge similarity values stay in the range [0, 1]. the final semantic similarity (semsim) of the two questions is the average similarity of the edges in the reference question that match best with the edges in the generated question. since named entity tokens and question cue words are handled separately, only edges involving content words are considered in this computation. additionally, we ignore edges representing less informative dependency relations such as "punctuation", "possessive modifier" and seven others in line with previous works @xcite . qsts : note that each of the similarity functions, qcsim,nesim,semsim assign normalized scores between [0, 1] for an independent aspect of matching the two questions. these three scores can be summarized using the geometric mean @xcite to obtain a single score between [0, 1] for question-sensitive text similarity as ## the simqg dataset current models for qg are evaluated using qa datasets containing (passage, reference-question) pairs. model-generated questions are compared against these reference questions using traditional metrics. it is our contention that given a passage context and questions generated by qg models against that context, several valid questions may be possible and it may not be representative to only compare generated questions against a specific given reference. to demonstrate this claim, we collected a novel dataset containing human judgements of machine-generated questions against their associated passage contexts and the reference questions available for these contexts. our novel dataset is based on squad @xcite , a widely-used dataset in both qa and qg studies. about 500 (question, passage) pairs were randomly sampled from the test portion of the squad dataset (used in @xcite ). recent qg models from prophetnet @xcite , t5 @xcite , and one of the early neural models based on gated self-attention (gsa) networks @xcite were used for obtaining machine-generated questions. by choosing machine-generated questions from models with qg performance ranging from high (prophetnet) to significantly low (gsa), we seek to include questions in our dataset with varying degrees of answerability, fluency and relevance @xcite . our annotation task on the crowdsourcing platform amazon mechanical turk (amt) was set up along the lines of previous qg works @xcite . each passage along with the machine generated question was examined by three independent crowdworkers to characterize if the question is (1) fluent: is the question grammatically correct, natural sounding, and semantically valid for the given passage context (@xmath5.0, ac-@xmath6.5, @xmath7.0)?; (2) answerable: is the answer to the generated question present in the passage (@xmath8.0, @xmath9.0)?; (3) relevant: is the question relevant to the passage and only based on the content in the passage (@xmath10.0, @xmath11.0)? the workers were also asked to compare the machine-generated question with the reference question provided in squad and to identify whether the question is similar to the reference question (@xmath12.0), similar but has less/more information compared to the reference question (@xmath13.5), different but has the same answer as the reference question (@xmath14, or related but different (@xmath15. by averaging worker scores for each question, we obtain relevance/fluency/answerability scores for each (machine-generated question, passage) pair as well as a similarity score for pairs of (machine-generated, reference) question pairs all in the range [0, 1] and by suitably thresholding at 0.5, we can obtain pairs of similar and dissimilar questions for our study as well as questions that are not fluent, answerable, or relevant. we refer to the dataset collected above as the simqg dataset. a summary of simqg is provided in additional notes on data collection: on the amt platform, we required the crowdworkers to have greater than 95% hit approval rate, a minimum of 10,000 hits, and be located in the united states and clear a qualification test to be able to work on our task. each worker was paid $0.30 per hit. we met the ethics, quality, and reliability considerations for our collected dataset as follows: as part of the amt data collection process, the anonymity and privacy of the crowdworkers is already ensured. furthermore, the settings for the hit approval rates, and location of the worker, described previously are set similar to previous qa/qg data collection efforts to ensure the english language skills of the data annotators and thus the quality of the collected dataset. a total of 7 workers helped in creating our dataset. about 47% of the workers who attempted the qualification test were able to obtain a score of 80% or more and gained the eligibility to work on our task. their annotations can, therefore, be considered reasonably reliable on average. ## experiments baseline measures: we demonstrate the performance of our proposed qsts measure by comparing with several existing measures. the first set of measures are traditionally employed in various text generation tasks including qg and comprise of bleu, meteor, and rouge scores @xcite @xcite . all these measures are based on n-gram overlap between the generated text and the reference text (of the same "type", for example, two summaries, or two sentences). the qbleu metric was designed specifically for qg systems and includes the notion of answerability, that is, does the question include enough information to enable answer retrieval for the given question @xcite ? to this end, various weights are estimated and incorporated for question cue words, content words, named entities and combined linearly to assign answerability score for a question. answerability is further combined with the traditional bleu score to obtain a qbleu score. 7 a recent research direction involves the use of transformers for learning metrics for text generation tasks @xcite . based on its state-of-the-art performance on various nlg tasks compared to other variants such as bertscore, we include bleurt as one of our baselines for comparing questions. to the best of our knowledge, bleurt has not been specifically evaluated for matching questions and we seek to bridge this gap as part of our experiments. 8 datasets: we used two existing qa/qg datasets with paraphrase information for evaluation. the first is the comqa dataset that includes about 3.3k paraphrase pairs @xcite while the second is the recently-compiled firs dataset, containing approximately 5k question pairs @xcite . the firs dataset includes rewrites of a given question which also include an extra fact from a knowledge base. that is, the rewritten question has the same intent as the original question but includes additional facts of relevant named entities. we randomly selected one of questions from each paraphrase clusters in comqa as the reference question whereas in firs, the original question forms the reference question. in addition, we evaluate on questions from the quora question pairs (qqp) dataset. foot_7 qqp is a large dataset of about 400k question pairs obtained from quora and includes duplicate and nonduplicate labels indicative of whether the intent of the two questions is the same. note that this dataset is not used for qg since passage contexts and answers are unavailable. moreover, the labels are known to be noisy in this dataset, and the questions on community forums are stylistically different from standard qg @xcite . despite these differences, we study a randomly selected 5% sample of the qqp dataset separated into duplicate pairs (qqp-dup) and non-duplicate pairs (qqp-nd). finally, we provide evaluation on simqg, the dataset specifically collected by us to model qg contexts (section 3). ## conclusions and future work we discussed existing metrics for question generation evaluation and highlighted cases where a deeper understanding of question semantics need to be modeled by metrics for a more representative evaluation. as an alternative, we designed the question-sensitive text similarity metric (qsts ) that comprises of interpretable scoring components. we also underscored the need for reference-free evaluation in qg systems. the potential of qsts in serving this purpose was demonstrated on a novel dataset simqg, compiled from human judgements on (machine-generated question, passage) pairs. qsts provides, to our knowledge, the first approach to characterizing qg system performance in practical deployments where reference questions may not always be available. as can be seen in experiments, there is still a large room for improvement for question similarity computation as well as qg evaluation. in future, we hope to pursue these directions further as well as study metric learning approaches for a referencefree evaluation.