Finnish SQuAD : A Simple Approach to Machine Translation of Span Annotations (2025)

Abstract: We apply a simple method to machine translate datasets with span-level annotation using the DeepL MT service and its ability to translate formatted documents. Using this method, we produce a Finnish version of the SQuAD2.0 question answering dataset and train QA retriever models on this new dataset. We evaluate the quality of the dataset and more generally the MT method through direct evaluation, indirect comparison to other similar datasets, a backtranslation experiment, as well as through the performance of downstream trained QA models. In all these evaluations, we find that the method of transfer is not only simple to use but produces consistently better translated data. Given its good performance on the SQuAD dataset, it is likely the method can be used to translate other similar span-annotated datasets for other tasks and languages as well. All code and data is available under an open license: data at HuggingFace TurkuNLP/squad_v2_fi, code on GitHub TurkuNLP/squad2-fi, and model at HuggingFace TurkuNLP/bert-base-finnish-cased-squad2.

## introduction question answering (qa) is an important practical information retrieval task as well as a common benchmark of computational models of human language. extractive qa models are typically built as a two step retriever-reader pipeline, first retrieving the documents relevant to the query (retriever) and then using an encoder model to extract the correct answer span from those documents (reader). generative qa models replace the reader component with a generative large language model (llm), in an approach commonly referred to as retrieval-augmented generation @xcite . no matter which of the qa paradigms is applied, large-scale question answering datasets such as the squad dataset play a key role. both in terms of benchmarking model performance, and model training. whereas for extractive qa these datasets are used directly, in generative llm development, qa datasets are commonly used as a source of examples for instruction fine-tuning. unfortunately, these large-scale qa datasets are mostly available only for english and a small number of well-resourced languages, making the direct development of retriever-reader qa models for languages without such a dataset almost impossible, as well as negatively impacting benchmarking of llm-based qa. with the improvements to machine translation (mt) output quality seen in the recent years, machine translating datasets is becoming a frequent choice to obtain a dataset in a new language in cases where native annotation is not possible due to lack of resources. while such an approach is technically very simple to implement for datasets consisting of unannotated text, it becomes considerably more complex for datasets with dense text span annotations, such as the qa datasets. numerous approaches have been introduced aiming to transfer the span annotations during translation. in this paper we contribute to this overall line of research by demonstrating a simple, yet effective approach to translate english question answering datasets to finnish (or other languages) using a little-known feature of the deepl machine translation service. the primary contribution of this paper is a finnish version of the publicly available sections of the squad 2.0 dataset. this dataset can serve both for the development of extractive qa systems on top of finnish encoder models, as well as provide a source of finnish data for instruction tuning and benchmarking of finnish llms. our other contribution is a finnish extractive qa model trained on this dataset. the paper is organized as follows: in section 2 we review prior work on machine translating qadatasets. in section 3 we explain our process of machine translating these datasets. in section 4 we evaluate the new resource and compare it to other similar resources. finally, section 5 concludes the work. ## related work there are numerous open-domain question answering datasets for english. among the most commonly used is the stanford question answering dataset (squad). squad1.1 @xcite consists of 100,000 questions posed by crowdsourced workers on a set of text passages (paragraphs of 536 wikipedia articles). the questions are produced by the workers, while the answers constitute spans present in the text passages. squad2.0 @xcite is a superset of squad1.1 with an additional 50,000 crowdsourced unanswerable distractor questions that only make the impression of being answered in the given passage. 1 native human generated question answering datasets for other languages include chinese @xcite , korean @xcite and @xcite , but a large number of languages lack a large qa dataset. the squad dataset has been machine translated to several languages. arabic @xcite squad1.1 version starts by machine translating the passages, questions and answers separately. subsequently, all the paragraphs and answers are transliterated to arabic and the span of text of length at most 15 words with the least edit-distance with respect to the answer is identified. only 231 articles containing 48,344 question-answer pairs are translated, and a full 25,490 question-answer pairs are not recovered by the initial translation and the transliteration heuristic step is applied. a reported small-scale evaluation shows that approximately 64% of these are correctly recovered. persian @xcite squad2.0 version starts by machine translating the passages, questions and answers separately. then an alignment is established by finding the position of the sentence that the answer appears in the english dataset. if the translated answer does not appear in the equivalent translated sentence, the questionanswer pair is removed from the final dataset. the final dataset salvages 70,560 question-answer pairs. the tar-method (translate-align-retrieve) used to create the spanish translation of squad1.1 @xcite also starts by machine translating the passages, questions and answers separately. if the translated answer can be found in the translated passage, it is retrieved as is. in the opposite case, a word alignment between the source and translated passage is established using the eflomal word alignment method @xcite and this alignment is then used to locate the translated answer. the final dataset salvages almost all of the question-answer pairs, but a manual error analysis showed that 50% of the answer spans were either misaligned (7%) or under-/over-extended (43%). for finnish, which is our target language of interest, there exists an earlier machine translated version of the squad2.0 dataset @xcite . the passages, questions and answers are translated separately and their spans in the translations are identified using a number of normalization steps designed to improve the chance of successful matching. the dataset preserves 66,000 question-answer pairs from the original approx. 92,000. the unpublished swedish translation of squad2.0 foot_1 deviates from the common approach, and translates one question-answer pair at a time, marking the answer span with a recognizable token (e.g. "[0]"), and retrieves the span after translation, relying on the mt system preserving the special tokens. this process is reported to preserve 90% of the original question-answer pairs. in a more recent approach, a separate alignment model is first trained for the target language @xcite . then each context, question, and answer are translated together as a single unit using the google translate service. if the answer is not found with exact matching from the translation, the alignment model is used. finally, if the first two steps fail, the context is segmented into subsets of words with a total word count that approximates the word count of the answer. then the embeddings of the answer and all the context segments are calculated using a pre-trained multilingual bert model from which the closest segment to the answer is searched using cosine similarity with a threshold on the similarity score to prevent weak alignments. this method is reported to preserve 93.4% of the original question-answer pairs. in another recent approach, an annotated clinical corpus is translated from english to dutch @xcite . in the dataset the annotation and the context are stored separately. in the paper the annotations are first integrated directly into the clinical text by enclosing the text span and the cui (concept unique identifier) in square brackets ' @xcite ]'. then the text with embedded annotations is machine translated, keeping the annotations intact. finally the annotations are extracted from the translated text using regular expressions to separate the annotations and the context again to the original format. the google translate service and gpt 4 turbo are compared. the google translate service lost up to 1.7% of annotations and gpt4 turbo lost up to 5.9%. most of lost annotations for google were formatting errors, but for gpt, the lost annotations were mostly entirely omitted. in summary, the clearly most common approach to machine translating datasets with span level annotations relies on translating the elements in isolation, and subsequently identifying through a varied set of heuristics their positions in the translated passages. this is naturally an error-prone process due to the fact that the answers when translated in isolation are not guaranteed to match their in-context translation within the passage, preventing reliable alignment. this is demonstrated by the substantial proportions of "lost" examples reported for most of these machine translated datasets. and while metadata-tagging approaches like that of @xcite preserve most of the annotations, they are not able to preserve overlapping annotations without multiple rounds of translations. in the following, we apply an approach which uses the functionality of a commercial mt engine to avoid the tedious alignment of answer segments with the original passages. ## evaluation one of the main challenges with machine translated datasets is the absence of a large-enough, manually annotated, representative test set. such a test set is in many cases difficult to create, as it entails replicating the entire annotation task and procedure, which is a major undertaking for tasks with complex annotations, such as qa. therefore, in addition to reporting model performance on the machine translated test set, we also carry out several other evaluations: a backtranslation experiment, a manual evaluation of the translated examples, and a comparison of our method respective to two other machine translated squad datasets, one for spanish and one for finnish. these comparisons allow us in particular to establish the relative merits of our approach to other methods of figure 1 : example of the colored answer spans from an actual squad passage: the original english passage (top), its finnish translation (middle), and its backtranslation from finnish into english (bottom). this example is shown as-is without any manual corrections (other than adjusting colors for better readability). note the two overlapping answers documents obtained by wikileaks and wikileaks at the very beginning of the passage. ## discussion and conclusions in this paper, we have demonstrated a practical method for annotation transfer through an affordable, high-quality machine translation service, relying on its ability to translate formatted text documents. we have applied this method to create a finnish qa dataset with very little effort and negligible cost, resulting in a finnish squad2.0 translation with higher coverage and better overall model performance than what was previously available for finnish. as a side product of our evaluation, we have also created an alternate spanish squad dataset of seemingly better quality than that previously available. we have shown, through comparison to other machine translated qa datasets, and more directly also through an english-finnish-english backtranslation experiment, that the dataset is unlikely to result in substantially worse models than a (hypothetical) finnish dataset created manually. the backtranslation experiment suggest the penalty for mt is about 5pp in terms of em and 2.5pp in terms of f1. we argue that the value of the approach is in allowing for a substantial expansion in the availability of numerous nlp tasks in a number of languages that currently lack the relevant native datasets. while it is clear that a high-quality dataset manually annotated in the target language is the best resource for training nlp models, it is clear that for many task-language pairs such a dataset will not be created for many years to come, if ever. in these cases, we argue that the method gives a practical, viable alternative which, thanks to its simplicity can be implemented with ease and applied quite broadly to produce datasets for many tasks in many languages. the applicability of the method will naturally depend on the task, and likely to a degree the language at hand. the method is naturally limited to the language pairs supported by the translation service used and may not be practical for very large datasets in the billion word range. it also relies on the availability of a suitable translation service with terms and conditions not restricting such application (as is the case at present). while such dependence is not ideal, it is nevertheless becoming somewhat the norm in nlp, where large, high-quality models and systems are increasingly exposed through a service, rather than distributed openly, which is understandable given their development and deployment costs. our code is available under an open source license, and can be used to generate qa datasets for other languages supported by the translation service. the finnish dataset is on the hugging-face dataset repository as turkunlp/squad v2 fi, the code is on github as turkunlp/squad2-fi, and the finnish model is on the huggingface model repository as turkunlp/bert-base-finnish-cased-squad2. one limitation of this work is in relying on a particular property of an existing mt system, which also limits the applicability only to the languages supported by it. this is alleviated by the fact that deepl supports 33 languages, allowing for a potentially very large number of datasets to be translated in the simple manner we outline. further, since professional mt systems are primarily targeting translators and need to support formatting transfer to remain competitive, it is conceivable that a suitable mt system can be found also for other languages. another limitation is in relying on a closed, commercial system, which naturally negatively affects e.g. replicability. however, the system only needs to be used once, when creating the new dataset, and after that the dataset is available openly and can be evaluated in a transparent manner. the closed nature of the mt system thus does not fully transfer onto the dataset. we note that our use of a closed mt system is fully comparable to the current wide-spread practice in which nlp datasets are created using closed, commercial llms such as openai's chatgpt.