Increasing the Difficulty of Automatically Generated Questions via Reinforcement Learning with Synthetic Preference for Cost-Effective Cultural Heritage Dataset Generation (2024)

Abstract: As the cultural heritage sector increasingly adopts technologies like Retrieval-Augmented Generation (RAG) to provide more personalised search experiences and enable conversations with collections data, the demand for specialised evaluation datasets has grown. While end-to-end system testing is essential, it’s equally important to assess individual components. We target the final, answering task, which is well-suited to Machine Reading Comprehension (MRC). Although existing MRC datasets address general domains, they lack the specificity needed for cultural heritage information. Unfortunately, the manual creation of such datasets is prohibitively expensive for most heritage institutions. This paper presents a cost-effective approach for generating domain-specific MRC datasets with increased difficulty using Reinforcement Learning from Human Feedback (RLHF) from synthetic preference data. Our method leverages the performance of existing question-answering models on a subset of SQuAD to create a difficulty metric, assuming that more challenging questions are answered correctly less frequently. This research contributes: (1) A methodology for increasing question difficulty using PPO and synthetic data; (2) Empirical evidence of the method’s effectiveness, including human evaluation; (3) An in-depth error analysis and study of emergent phenomena; and (4) An open-source codebase and set of three llama-2-chat adapters for reproducibility and adaptation.

## introduction the cultural heritage sector is increasingly leveraging advanced technologies like large language models (llms) @xcite ) and ai assistants @xcite to increase and improve access to collections and their associated data. these technologies provide new opportunities for more dynamic and intuitive interactions with heritage content. one particularly promising technology is retrieval-augmented generation (rag) @xcite , which retrieves relevant information from a database of vectorized content to generate accurate, fact-based responses to user queries. we believe that rag, and iterations on the approach, will play a significant role in improving the search capabilities of heritage institutions in the coming years. heritage search systems are used by the public and academics alike; however, the latter tend to submit more complex and specific queries @xcite . rag has the capability to fulfil these needs but still requires robust evaluation. this includes not only end-to-end system testing but also the evaluation of individual components. as the response is generally required to be written based only on the retrieved documents to mitigate language model hallucinations, we argue that the task is one of machine reading comprehension (mrc). while mrc datasets are well-established in the general domain, they are notably lacking in cultural heritage and the cost of their construction is prohibitive for most institutions. we estimate that the popular squad dataset cost about @xmath09 @xcite ; the actual cost is likely much higher. to address these challenges, we propose using automatic question generation (aqg) systems to generate mrc datasets. however, we argue that many automatically generated questions, particularly those from zero-or few-shot approaches, do not provide an adequate challenge for modern language models. manipulating difficulty is challenging through traditional training approaches given its abstract and subjective nature, and prompt based solutions are intractable when considering the infinite permutations and interactions between different aspects of difficulty @xcite @xcite @xcite . to control difficulty, we adapt the reinforcement learning from human feedback protocol used in ai assistant steering @xcite . in this regime, samples are ranked based on specific criteria and paired into chosen and rejected samples for training a reward model. this reward model learns to distinguish good samples from bad and outputs a signal which steers a policy model. rather than relying on costly human annotations, we generate synthetic preference data by evaluating question-answering model performance on a subset of squad, assuming that questions answered correctly less frequently are inherently more difficult. this approach leverages the language model's innate feature extraction capabilities, eliminating the need to explicitly define difficulty components. figure 1 demonstrates this feature extraction ability by comparing questions generated with and without reinforcement fine-tuning. we selected squad over an in-domain qa dataset for two main reasons. first, it is a wellstudied, large, and diverse dataset. second, comparable qa datasets at squad's scale are either visual question-answering focused @xcite or have data reliability concerns such as ocr text @xcite . this approach enables cultural heritage practitioners to generate challenging evaluation datasets more efficiently and cost-effectively than manual curation. the primary expense is compute resources, which can be accessed in the cloud for only a few dollars per hour. 1 we summarise this paper's contributions as follows: ## related work a similar question generation approach to ours is employed by @xcite who adopt a pipeline structure. however, their primary objective is to generate suitable questions rather than specifically focusing on difficulty. an important distinction lies in their extensive pre-processing applied to identify candidate answers before feeding them to the question generation model. we argue that pre-identifying answers may limit diversity and prevent the inclusion of potentially complex answer types. ## method to challenge the high cost of manual annotation while maintaining quality and increasing difficulty, we design and implement a robust system capable of generating contextually relevant, coherent, and challenging question-answer pairs from textual input. the process follows the core methodology of rlhf, deviating only in the use of synthetic preference data to train a reward model. rather than explicitly defining the characteristics of difficulty and risking failure to capture certain aspects, we exploit the ability of leading question-answer models to derive which questions are challenging, and allow a reward model to extract the component features of the task. we task three qa models with answering all questions in our validation split of squad. these questions are assigned a score based on the number of times they were answered incorrectly, which are in turn used to generate pairwise preference data. these pairwise samples enable the training of a reward model (rm) for use in fine-tuning a supervised model (sft) on the task of question generation using proximal policy optimisation (ppo) @xcite . we embed this synthetic rlhf process into a greater pipeline for generating samples, shown in figure 2 . this ensures the quality of the final dataset. the pipeline also contains a set of rulebased critics which are used to exclude samples that are malformed and those with non-unique answers in the source text. samples are then deduplicated using exact string matching. the remainder of this section discusses each of the relevant components of the pipeline and the rlhf process. ## results and discussion model accuracy. to measure performance, we observe the difference in prediction accuracy for qa models on each dataset. in general we find the reference-free metrics to show limited correlation with model prediction accuracy and an ability to differentiate human written samples from model generations. we believe this is evidence for the continued need for more reliable, reference-free evaluation tools for question generation. human evaluation. to evaluate question quality, we conduct a human evaluation on a subset of 50 passages from the test split. each input passage and question is filtered through the format critic then provided to two annotators who select either the correct answer span or indicate that the question cannot be answered. in the case of annotator disagreement or the annotated answers differing from the model generated answer, the annotator responses and the model answer are provided to two new annotators who both select which responses are appropriate. we allow annotators to select multiple responses as correct but only include those that were selected unanimously by both annotators as valid. we observe an agreement of κ = 0.7975 between annotators. the results of this evaluation, shown in the results demonstrate that reinforcement learning can effectively manipulate question difficulty, while highlighting important avenues for future work. while squad's synchronic nature served our experimental needs, cultural heritage datasets typically present diachronic challenges that add complexity to question generation. although specialised diachronic models exist @xcite , they lack the extensive training of generaldomain llms. however, these larger models' exposure to historical corpora, combined with their advanced instruction-following capabilities, sug- ## conclusion in this paper, we introduce a low-cost methodology for generating challenging mrc datasets to meet the growing need for evaluation datasets in the cultural heritage sector. by using high-performing question-answering models to identify the most difficult questions, we were able to create synthetic pairwise data for training a reward model. rather than manually defining question difficulty, our approach allows the model to learn and extract these features autonomously, leading to a significant improvement in the difficulty of questions generated for evaluation. with this said, we trained on a general domain dataset in order to single out the training behaviour, in doing so losing many of the characteristic features of heritage datasets. in future work we will examine how the training paradigm fares under the unique challenges presented by such a varied industry. although this work was produced to meet the evaluation demands of our ongoing work in rag at our institution, we also highlight that the approach can work in any domain and that with some modification, it could be used to augment other dataset formats. we believe this approach can be extended further, allowing for the manipulation of multiple abstract properties simultaneously through multireward model setups @xcite .