Exploring Social Sciences Archives with Explainable Document Linkage through Question Generation (2023)

Abstract: This paper proposes a new approach for exploring digitized humanities and social sciences collections based on explainable links built from questions. Our experiments show the quality of our automatically generated questions and their relevance in a local context as well as the originality of the links produced by embeddings based on these questions. Analyses have also been performed to understand the types of questions generated on our corpus, and the related uses that can enrich the exploration. The relationships between the co-references and the questions generated, and the answers extracted from the text were also discussed and open a path for future improvements for our system in their resolution.

## introduction from an information and communication science perspective, two steps are essential to bring together computer technology and human and social science objectives. the first essential step is the availability of annotated data in order to train and evaluate with objective metrics the model deployed to ensure their relevance and scientific interest. the second essential step is the creation of an interface adapted to the objectives of the device and respectful of the user by putting forward the explainability of the results provided. this methodology was followed in the context of the archival project 1 : firstly existing annotated data have been used to train and evaluate deep neural network question generation models; secondly we applied these models to a corpus of social science archives for evaluating their relevance in a real archive exploration application. this paper describes this second step, which study if recent advances in natural language processing thanks to deep learning models translate into novel mediation interfaces for social science researchers. this paper is structured as follows: section 2 presents our methodology for generating explainable links among documents based on questiongeneration models; section 3 presents the archive corpus used in this study; section 4 presents our question generation and filtering method; section 5 describes how generated questions can be used to 1 https://anr.fr/projet-anr-19-ce38-0011 create explainable links within documents; finally sections 6, 7 and 8 presents an experimental study on our archive corpus with quantitative, descriptive and qualitative evaluations of the method proposed. ## exploration through questions generation when exploring a thematic archive collection, links can be made between documents or parts of documents according to various criteria such as cooccurence of entities (person, location, organisation, date, . . . ), keywords related to a knowledge base or a thesaurus @xcite , or directly by a statistical similarity measure between documents or parts of documents such as sentences @xcite or paragraphs @xcite . furthermore, some methods produce links directly between the embedding of the whole documents @xcite , @xcite , the sum of the word embeddings of the document @xcite or by representing them with word-graphs and using shortest-path algorithms for linking @xcite . the graph structure obtained can then be used to design navigation interfaces such as maps representing linked documents or directly by inserting hypertext links @xcite . the weaknesses of keywork/entity links are on the one hand the amount of links generated that can be very big if large sets of keywords or entities are considered and on the other hand the fact that the simple occurrence of relevant terms does not mean that their contexts of occurrence are relevant or interesting to users. on the contrary, similarity-based links take words in context into consideration, but the use of statistical similarity metrics make the links often difficult to interpret. recently, advances in question answering (qa) models from text have enabled the use of asking direct natural language questions in order to access to electronic documents. impressive results have been obtained with current deep learning language models on benchmark corpora such as squad @xcite , however it has been shown that the kind of questions that these models handle best are simple literal questions for which a factual answer can be found in the text and that performance drops when dealing with more abstract questions or questions needing a larger context than a sentence in order to be addressed. moreover, most of these studies have been applied only on wikipedia text. a recent study @xcite have shown that realistic questions, like those that can be asked by a professional reader analyzing social science archives are quite far from the simple benchmark questions used to evaluate qa systems, leading to poor performance. however, even if current qa models might be too simplistic for use in a real archive exploration setting, we believe that question generation models can still be useful in order to characterize documents. such models can be trained on the same corpora as qa models: while qa models are trained to generate a response given a question and a text document, question generation models are trained to predict a question given an answer and a text document. by selecting potential answers on text segments and generating questions from these answers and their context of occurrence, we obtain an abstraction of a text segment which contains the set of questions that can be asked on it. by estimating similarities between questions and answers belonging to different documents, we can predict links between them that can be explained by the two qa pairs, adding an explainability layer to the process. we believe that this is an efficient way of presenting links to a user: by looking only at the linked qa pairs, readers can decide if it is worth or not to follow this link, saving time compared to the standard solution consisting of following every link to decide if the similarity between two text segments is interesting or not. in this study, we developed a question generation and filtering process which is used to obtain links between documents of a collection of social science archive corpus. this study presents the first quantitative and qualitative evaluation done of this method on this archive corpus. ## the self-management archive corpus in order to assess the previously described "exploration through questions generation" paradigm, we have chosen to focus on a particular type of social science archive source: a full collection of the autogestion ("self-management") journal published for 20 years between 1966 and 1986. the originality of our work is hence to propose a way to access this rich source through explainable links. the "self-management" notion falls within the large spectrum of social sciences. it concerns daily social environment, economic life, as well as political life, education, ecology, culture, architecture . . . nowadays, "self-management" supports in an underlying way the concepts of radical democracy, confederalism, social and solidarity economy and sustainable development. as a source of social innovation, self-management has variations all over the world and questions societal and economic models of development. it is a particularly transversal and interdisciplinary notion which can feed research in sociology, political science, economy, law, political anthropology and social history. the autogestion journal foot_0 is distributed in its digitized form by the french persée organization. it is part of a larger pluridisciplinary multilingual mixed collection (archives and documents) that has been gathered since the 1960's by the fmsh foot_1 foundation's library. the full collection has been granted the collex label (collection d'excellence or excellency collection) from the collex-persée foot_2 network under the supervision of higher education and research for the preservation of corpus of digitized or natively digital documents. @xcite describes the journal as an observatory of liberation movements and states that it « accompaniedpreceeding and following -the liberation movement which called for workers' self-management. through analysis of its precursors, contemporary practices and historical precedents, the journal was a conceptual tool capable of inspiring action. its disappearance coincided with the abandonment of the reference to workers' self-management in socio-political movements, although the aspiration it represented continues to exist.» we are using an ocrized version of the corpus. the structure of the journal is rather standard (mono-column, few figures) and the quality of the ocr provided by tesseract is sufficient to be exploited as is without manual corrections. studying the impact of ocr errors is outside the scope of this study and should be investigated in further research work. the resulting corpus is composed of 46 issues of the journal, ranging over 20 years, for an overall amount of 6298 pages and 1.98m tokens. ## question generation the question-generation (qg) task is a classical nlp task that has been revisited thanks to the development of efficient deep learning sequence-tosequence models @xcite @xcite and large language models @xcite . it can be modeled as a neural generation task, where a sequenceto-sequence model is trained to translate a sequence of words representing a text segment (the context) containing an answer (a sub-sequence of words belonging to the context) into another sequence of words representing a question on the input. the task is then to generate a question given a (context, answer) pair. the availability of large databases of question/answer/context triplets such as squad can be used to directly fine-tune sequence-to-sequence generation models such as bart. one of the key decision that has to be made before generating a question is the choice of the answers on which questions will be generated. choosing all noun phrases as answers can lead to an over generation of questions, most of them being not very relevant if the contexts of occurrence of answers is not informative. that is why we chose to use semantic annotations in order to select answer candidates in order to generate more informative questions. as proposed in @xcite and @xcite , we use a semantic role labelling (srl) model following the propbank formalism @xcite in order to select answers candidates among the semantic roles detected. in our study, we train the question generation model by fine-tuning the barthez (kamal eddine et al., 2021) language model on a french corpus of question-answer-context triplets called @xcite . this is a three steps process: at inference time, generating questions on a given sentence involves performing semantic analysis on the sentence, generating an input sequence for each detected semantic role, and using the fine-tuned seq-to-seq model to generate a question for each input sequence. the following translated example is from the fquad training set with an s being the answer, lu the lexical unit that triggers the semantic relation, and ct x the context: source : [ans:arg2] héra (hera)[lu] appelé (called) the application of the question generation model on a sentence from the self-management corpus processed by an srl parser is illustrated in the following example: source : [ans:arg0] une bonne partie du c.n.r.s. (a good part of the c.n.r.s.)[lu] évolue (is evolving) we apply a series of filters to enhance the quality and reduce the quantity of generated examples. the first step (f1) is to restrict the srl analysis to only include frames with a strictly verbal trigger (rejecting auxiliary verbs), as these are deemed to be of higher quality due to their ease of detection. to further improve the quality of the generated examples, we apply a filter (f2) on the queries to remove those with non-informative answers or contexts. this includes answers that are less than 5 characters, or belonging to the nltk @xcite stopwords list in order to eliminate answers containing only pronominal coreferences. queries with a context of fewer than 5 words are also filtered out. the generated questions are also subjected to a filter (f3) based on the "roundtrip consistency" methodology proposed by @xcite . this filter involves retaining only the synthetic examples where a qa model 5 is able to retrieve a portion of the target answer from the generated question. we consider that the model has successfully retrieved the answer if there is a minimum overlap of 30% between the predicted answer and the answer of the query. finally, we apply a final filter (f4) to eliminate duplicate questions, which are a frequent phenomenon due to slight variations in some queries, often resulting in very similar or identical questions. ## generating explainable links the main originality of our approach is the use of our synthetic questions/answers to establish links between documents in our corpus. while traditional methods involve computing similarity via document embeddings at a chosen level of granularity (sentence, paragraph or textblock, page), our approach involves computing a similarity measure between question+answer (q+a) embeddings. we consider a source and a target q+a embeddings obtained by the concatenation of questions and answers produced by our method described in section 4. for example, this is the "<@xmath0: quel organisme évolue vers une structure autogérée ? (which organization is moving towards a self-managed structure?) | une bonne partie du c.n.r.s. (a good part of the c.n.r.s.) 5 in our case, a camembert-large @xcite ) model trained on fquad our embedding projection for each q+a pair use the sentencetransformer (reimers and gurevych, 2019) library foot_3 . a cosine similarity measure is then employed between all pairwise combinations of these embeddings, resulting in the computation of a similarity matrix. for each q+a pair in our corpus, we extract the 49 most similar pairs across three granularities: ## quantitative and qualitative description of the generated questions in this section, we analyze the application of our generation and filtering method to our selfmanagement corpus. we provide first a quantitative study of the set of generated questions followed by a more in-depth descriptive study according to three criteria: question types, question themes and coreference chains. ## qualitative evaluation of the generated questions in addition to the quantitative and descriptive evaluation of the generated questions on the selfmanagement corpus, we performed a first qualitative evaluation on a subset of the collection. in order to evaluate the quality and relevance of the generated questions, we annotate the generated questions according to two dimensions. the first dimension focuses on the quality of the question form, with questions being categorized as "valid", "incoherent question", or "ungrammatical question". in the second dimension, we assess the relevance of the question after it has been validated in the previous dimension. this evaluation involves three 5-point likert scales: for the relevance annotations, the results are also promising. in terms of answer adequacy (figure 3(a) ), the majority of questions (67%) received a score that indicates a high level of adequacy foot_6 . the two likert scales measuring question relevance are more subjective, but a large proportion of questions (over 68%) were rated as relevant in the local context (figure 3(b) ). in the global context (figure 3(c )) of the reading, the percentage of questions rated as relevant drops, with just over half of the questions meeting the same score criterion. to check inter-annotator agreement, a subset of 129 questions were annotated by two annotators. on the first dimension (surface form) we noticed only 11 disagreements between the two annotators. on the second dimension, concerning the first likert with a simplified 3 category evaluation by grouping the 1 and 2 choices and the 4 and 5 ones, we measured 25 disagreements out of 115 annotations. with the same grouping, we obtain 43 disagreements out of 115 annotations for likert 2 and 65 out of 115 annotations for likert 3. these higher numbers of disagreement were expected, as this last evaluation is highly subjective. ## document linking evaluation we quantitatively evaluate the difference in the links generated by the question embeddings compared to two conventional embedding similarity methods. we create a link between two sentences or two paragraphs (textblock) if the similarity between their embeddings is below a threshold (or the top n links are kept). the comparison results in three "similarity sets": ## conclusion this paper proposes a new approach for exploring digitized humanities and social sciences collections based on explainable links built from questions. our experiments show the quality of our automatically generated questions and their relevance in a local context as well as the originality of the links produced by embeddings based on these questions. analyses have also been performed to understand the types of questions generated on our corpus, and the related uses that can enrich the exploration. additionally, we discussed the relationships between co-references, generated questions, and extracted answers from the text, which opens a path for future improvements for our system in their resolution. experiments are still to be conducted to study more qualitatively the generated links, as well as to enrich and filter in a finer way the large quantity of questions on the corpus.