Overview of the SV -Ident 2022 Shared Task on Survey Variable Identification in Social Science Publications (2022)

Abstract: In this paper, we provide an overview of the SV-Ident shared task as part of the 3rd Workshop on Scholarly Document Processing (SDP) at COLING 2022. In the shared task, participants were provided with a sentence and a vocabulary of variables, and asked to identify which variables, if any, are mentioned in individual sentences from scholarly documents in full text. Two teams made a total of 9 submissions to the shared task leaderboard. While none of the teams improve on the baseline systems, we still draw insights from their submissions. Furthermore, we provide a detailed evaluation. Data and baselines for our shared task are freely available at https://github.com/vadis-project/sv-ident .

## introduction social science publications often use and reference survey datasets, containing hundreds or thousands of questions, using so-called survey variables. 1 while publications may focus only on a specific subset of these variables, explicit references are usually missing: the lack of explicit links between survey variables and publications, in turn, limits access to research along the fair principles @xcite . to address this issue, we propose a task where variable mentions in unstructured documents are linked to items from a catalog of survey research datasets using natural language processing (nlp) methods. automatically identifying which variable is mentioned in a given text is challenging due to the diverse linguistic realizations of variables @xcite . a short example text is shown in figure 1 . all three qd3_1 eu citizenship: feel to be eu citizen qd2_3 attachment to: european union in the subsequent section, we focus on two theoretical concepts -mere exposure and hostile media perceptions -both of which have not been employed within the context of european identity. the dependent variable (identity) was operationalized by two indicators: citizens were asked to answer the following question: "please tell me how attached you feel to the european union" by using a scale from 1 (not attached at all) to 4 (very attached). furthermore, the item "you feel you are a citizen of the eu" was used in index building. the survey variable identification 3 (henceforth, sv-ident) shared task aims at promoting the developing of systems that can identify variables within the text of scholarly publications from the social sciences in different languages (initially, we focus here on english and german). the shared task is divided into two sub-tasks: a) variable detection and b) variable disambiguation. the former deals with identifying sentences that contain variable mentions, while the latter focuses on linking the correct variables mentioned in a sentence. variable mentions are often implicit (e.g., sentences 1 and 3 in figure 1 ), and understanding when a variable is mentioned may require contextual information as well as knowledge from external sources (e.g., a variable vocabulary). since annotating scientific texts requires domain knowledge, training data is costly to create and thus scarce. to overcome these limitations, nlp systems, e.g., models using pre-trained language models (plms) and transfer learning are promising technologies to use. in this paper, we report the results on the first edition of the sv-ident shared task. two teams made a total of 9 submissions to the leaderboard. one of the teams developed systems for both sub-tasks and submitted a system description paper. while none of the teams improve on the baselines, we use the submissions provided by the teams to collect a few initial findings on the difficulties and challenges of the sv-ident task. crucially, we find that there is a difference between the performance on two types of variable mentions: explicit and implicit. implicitly mentioned variables (sentence 1 in figure 1 ) are significantly more difficult to detect and disambiguate, as they require contextual knowledge. this opens up new research questions for future work, such as, for instance: can implicit mentions of survey variables be further categorized into finergrained classes or can co-reference resolution be used to link variable mentions across different parts of a document? in order to foster future research on this task, we release all of our code to reproduce the analysis results and the annotation guidelines for creating the dataset at https://github.com/ vadis-project/sv-ident . the remainder of this paper is organized as follows: we provide an overview of the dataset used in §2. in §3, we describe the task definition and evaluation metrics. we present the submitted systems in §4 and provide a detailed analysis of the results in §5. we briefly discuss related work in §6 and frame the shared task into a broader context in §7. finally, we summarize the shared task and propose future work in §8. ## data the sv-ident 2022 shared task has been conceived in the context of the vadis project foot_3 and organized as part of the third workshop on scholarly document processing (sdp) @xcite , co-located at the 2022 international conference on computational linguistics. in the following, we describe the data collection process and the dataset used for the shared task. ## experimental setup the task of sv-ident deals with identifying variable mentions in a text. for simplicity, the task is formulated as a sentence-level task, but can also be solved using document-level information (inline with the data annotation process). the shared task is decomposed into two sub-tasks: variable detection and variable disambiguation, where the former task can be used to help filter candidate sentences for the latter. ## participating systems two teams participated in our challenge on co-dalab, and one of the teams submitted a system description, which is included in the proceedings. we summarize the report here. the participant (hövelmeyer and kartal, 2022) treated both tasks, at least partly, as a problem of semantic textual similarity @xcite . for task 1, sentences were first preprocessed by randomly undersampling in order to balance the data, removing stopwords, lemmatizing the data, and using only a subset of the fields from the vocabulary metadata based on preliminary experiments. then, test sentences and vocabulary data were converted into dense sentence representations using sentence-t5 @xcite for english and sahajtomar/german-semantic foot_10 (henceforth, gs) for german. similarity scores were computed for those test sentence and vocabulary item pairs. pairs with a score greater than a predetermined threshold were classified as sentences containing variables. for task 2, the same sentence representations were used, but for all test sentences. the variables were then ranked based on their scores, with a higher score implying a greater similarity. while other methods were also implemented, such a logistic regression and multinominal naive bayes classifiers, the best performing systems used sentence-t5. ## evaluation this section first describes the baseline systems for each task and later provides the results of the shared task. ## related work identifying mentions of survey variables in text was first introduced by @xcite in the openminted project (om). 15 15 http://openminted.eu/ om sv-ident documents 64 44 research datasets 1 76 total # variables 406 27,365 (vocabulary size) # annotated 414 8,721 variables (tokens) # uniquely used 243 851 variables (types) instances annotate 1,217 5,972 (# annotated sentences) as the predecessor of our task, they created the first dataset for the problem of sv-ident. given that identifying variables requires semantic relations, other nlp tasks deal with a fundamentally similar perspective, such as entity linking (el), recognizing textual entailment (rte), semantic textual similarity (sts), plagiarism detection, and detecting previously fact-checked news. el can be conceptualized as linking mentions to variables in a knowledge base @xcite . since there are many similar survey variables in research datasets, disambiguating the right variable for a sentence is similar to determining the identity of an entity from a knowledge base. the rte task is to identify whether a sentence entails a given candidate hypothesis or not @xcite . a question answering adaptation of rte @xcite is similar to sv-ident, as the question and each answer form a hypothesis, which then re-quires the system to determine whether a sentence entails a given candidate hypothesis. sts is yet another similar task, which aims to find the similarity level between given texts @xcite . sts was organized as a shared international workshop on semantic evaluation between 2012 and 2017, and sts models have been developed for various domains @xcite @xcite . in the task of plagiarism detection of pan, 16 a system should extract all plagiarized passages from a given set of candidate documents with (external) or without (intrinsic) comparing them to potential source documents @xcite . lastly, detecting previously fact-checked claims, a shared task by the checkthat! lab @xcite , aims to match the most similar claimstext fragments from social media or political debate scripts -to a corpus of verified claims. the corpus is used to find the most similar claims, which does not require direct linking, as is done in sv-ident task 2, because implicit links are inferred. ## why sv-ident? today's search engines are the core elements of information access for social scientists. while search engines have seen many improvements in terms of keyword search and text understanding, they suffer from a limited capability of retrieving information from interconnected data sources, such as academic literature and research datasets. nonetheless, they show outstanding performance on retrieving such documents individually. current interlinking infrastructures typically only link research datasets to publications on the citation-level. such systems do not yet consider fine-grained linking of publications to individual survey variables from research datasets. as demonstrated in the sv-ident shared task, survey variables may be mentioned implicitly, which makes their manual or automatic identification non-trivial. currently, social scientists have to manually identify such variables, which is timeconsuming. in addition to these limitations, search engines do not yet support queries specific to social science topics, concepts, or relations. yet, keyword search, which is widely used, has many known problems (e.g., vocabulary mismatch or complex queries). as a result, social scientists are unable to access interlinked publications and research data. thus, the re-use and reproducibility of research is limited. sv-ident, and more generally the vadis project, plays an important role in filling the gap in the lack of infrastructure for social scientists @xcite . sv-ident aims to build automatic models for identifying survey variables in social science publications. this directly enables a more finegrained interlinking of publications and research datasets. more specifically, variables can be linked on the sentence-level, which allows new features to be developed. within the vadis project, we aim to develop variable-based automatic summarization, which will allow scientists to quickly get an overview of a publication with respect to the variables used. furthermore, we plan to incorporate variable recommendation algorithms into the gesis search platform to enable scientists to find relevant variables outside the scope of variables they are already familiar with. ## conclusion this overview reports on the results of the sv-ident 2022 shared task. we introduce two subtasks relevant for sv-ident, namely, variable detection and variable disambiguation. we report on data, which is currently the largest of its kind, that was collected, annotated, and made publicly available for this challenge. baseline as well as participants' systems are described and evaluated. we find that nearly all systems perform better on explicit variable mentions, opening up new directions of research. finally, we contextualize the shared task into related work and highlight its importance within a broader context. future work will further analyze the distinction between different variable mention types. in addition, multi-task learning could solve both tasks jointly or in combination with adjacent tasks. co-reference resolution could be used to help disambiguate implicit variable mentions. finally, evaluating systems on more diverse metrics, such as fairness or robustness, is critical for applied research. in the subsequent section, we focus on two theoretical concepts -mere exposure and hostile media perceptions -both of which have not been employed within the context of european identity. the dependent variable (identity) was operationalized by two indicators: citizens were asked to answer the following question: "please tell me how attached you feel to the european union" by using a scale from 1 (not attached at all) to 4 (very attached). furthermore, the item "you feel you are a citizen of the eu" was used in index building. the independent variable was operationalized by a combination of several items: the participants were first asked the following question: "could you tell me to what extent you…a) watch television on a tv set or via the internet, b) listen to the radio, c) read the written press d) use the internet?" later they were questioned: "where do you get most of your news on european political matters? firstly? and then?" possible answers were respectively: television, the press, radio, and the internet. the respondents were asked, "do you think that the [national] television present(s) the eu too positively, objectively, or too negatively?" the same question was repeated regarding the radio and the press. even the unintended, casual contact with news on the eu provided by the media fostered european identity.