Varanalysis@ SV -Ident 2022: Variable Detection and Disambiguation Based on Semantic Similarity (2022)

Abstract: This paper describes an approach to the SV-Ident Shared Task which requires the detection and disambiguation of survey variables in sentences taken from social science publications. It deals with both subtasks as problems of semantic textual similarity (STS) and relies on the use of sentence transformers. Sentences and variables are examined for semantic similarity for both detecting sentences containing variables and disambiguating the respective variables. The focus is placed on analyzing the effects of including different parts of the variables and observing the differences between English and German instances. Additionally, for the variable detection task a bag of words model is used to filter out sentences which are likely to contain a variable mention as a preselection of sentences to perform the semantic similarity comparison on.

## introduction one important way of improving reproducibility and reusability of research is to make its results accessible and comparable. besides the interlinking of scientific papers, researchers of different disciplines can also benefit from the interlinking of publications and primary data @xcite . social scientists often refer to the same survey datasets. unfortunately, these are seldom properly linked in the publications and if they are, the different surveys and studies often contain a large amount of single questions, called variables which need to be found in the respective corpus @xcite . it would be really helpful to have an automized way of detecting and disambiguating survey variables in scientific papers. for this reason, the very first shared task for survey variable identification is organized as sv-ident as an organizer of sv-ident, this author contributed to the model discussion and the preparation of the system description under the terms and conditions of this task. @xcite at the 3rd scholarly document processing (sdp) workshop at coling 2022. we mainly approached the task as a problem of semantic textual similarity (sts) and used language-dependent sentence embedding models to detect and disambiguate variables. for variable detection we additionally used a bag of words (bow) model. although the results did not exceed the baselines, our approach gives insights into which parts of the variables provide the most useful information for semantic similarity based disambiguation. this paper is structured as follows. in section 2, we present the task and the related data. in section 3, we describe our approach to both tasks. this section starts with an introduction on how the semantic similarity comparison is done, which is the same for both subtasks. section 4 contains a presentation of experiments performed to find the best parameters for our system. in section 5 and 6 we present our results and we discuss lessons learned, respectively. the paper is concluded in section 7. ## task description and data the shared task consists of two subtasks: variable detection (task 1) and variable disambiguation (task 2). both subtasks relate to the same dataset consisting of examples of sentences taken from social science publications foot_0 . the provided training set consists of 3,823 sentences with labels that indicate whether they contain a variable or not. each sentence has a document id referring to its source document and an unique id. if the sentence contains one or more variables, ids of these variables are also given, together with a research id which refers to the specific corpus or corpora the variables were taken from. moreover, the sentences have a language label (see ## approach and system description the task dataset features several difficulties. one of them is that it is multi-lingual containing both attribute value research id za3950 variable id exploredata-za3950_varv31 study title international social survey programme: citizenship -issp 2004 variable label q7b rights in democr: gov respect minorities variable name v31 question text there are different opinions about people's rights in a democracy. on a scale of 1 to 7, where 1 is not at all important and 7 is very important, how important is it: question text en there are different opinions about people's rights in a democracy. on a scale of 1 to 7, where 1 is not at all important and 7 is very important, how important is it: sub question q.7b -... that government authorities respect and protect the rights of minorities item category ... that all citizens have an adequate standard of living;... that government authorities treat everybody equally regardless of their position in society;... that politicians take into account the views of citizens before making decisions;... that people be given more opportunities to participate in public decision-making answer category not at all important;2;3;4;5;6;very important;can't choose, don't know;no answer, refused topic ['soziales verhalten und soziale einstellungen', 'internationale politik und institutionen', 'politische verhaltensweisen und einstellungen/meinungen', 'regierung, politische systeme, parteien und verbände'] topic en ['social behaviour and attitudes', 'international politics and organisation', 'mass political behaviour, attitudes/opinion', 'government, political systems and organisation'] german and english sentences. the fact that the variables consist of different parts with different semantic structures which are not available for all of the variables is another one. our approach focuses on the analysis of how the diverse information available can be beneficial for semantic similarity comparison. ## experiments most of the settings described above were chosen because they proved to be successful in experiments on the validation data. this section provides the results of different experimental setups for bow and sts models, respectively. ## results while the official evaluation metric for task 1 is f1macro, or averaged f1 (averaged harmonic mean of precision and recall), it is map@10 (mean average precision of the ten top ranked items) for task 2. we achieved the best results using the sts model for task 1. it scored 0.6016 on the test data (compared to 0.58 for the bow model) which is still beneath the baseline foot_2 of 0.6609, but the best result provided by participants. in task 2, our model achieved a result of 0.1359, which is also beneath the baseline foot_3 of 0.1893 and it was the only submission made by participants. ## lessons learned the task proved to be challenging. this can partly be explained by the challenging nature of the data in general. variable mentions in social science publications typically vary a lot on the linguistic level @xcite . additionally, dealing with a very large corpus of variables might explain why the results on the test data were so much worse than the results on the trial data.. since the pre-processing and evaluation of taking into account different variable parts were the main factors improving the results, it would be beneficial to further concentrate on these approaches for future work. one step into this direction could be the use of data augmentation. this already showed to be successful implicitly, since for the english data better results were achieved including the question text and question text en, which are the same sentences (see the sentence embedding models used in our approach have the advantage of being suitable for general sts tasks and perform competitively for a variety of such tasks without further finetuning @xcite . nevertheless, the baseline models of which one is fine-tuned on social science literature and the other is multilingual achieved better results for this task. for future work, it therefore would be interesting to experiment with models fine-tuned on data similar to the data at hand and multilingual models. ## conclusion we presented a solution to the sv-ident shared task relying on semantic similarity and basically treating the subtasks of variable detection and variable disambiguation as the same problem. we encoded the input sentences and parts of the variables using sentence transformer models and treating english and german sentences separately. for task 1, we used a bow model with random undersampling in order to create a preselection of likely candidates to contain a variable and then looked for sufficiently similar variables to decide whether a sentence contains a variable or not. for task 2, we ranked the most similar variables to every input sentence. throughout, we experimented with different preprocessing methods and different variable parts which proved to be beneficial. it showed that a promising approach for future work could be the consideration of data augmentation techniques.