F innish Paraphrase Corpus (2021)

Abstract: In this paper, we introduce the first fully manually annotated paraphrase corpus for Finnish containing 53,572 paraphrase pairs harvested from alternative subtitles and news headings. Out of all paraphrase pairs in our corpus 98% are manually classified to be paraphrases at least in their given context, if not in all contexts. Additionally, we establish a manual candidate selection method and demonstrate its feasibility in high quality paraphrase selection in terms of both cost and quality.

## introduction the powerful language models that have recently become available in nlp have also resulted in a distinct shift towards more meaning-oriented tasks for model fine-tuning and evaluation. the most typical example is entailment detection, with the paraphrase task raising in interest recently. paraphrases, texts that express the same meaning with differing words @xcite , arealready by their very definition -a suitable target to induce and evaluate models' ability to represent meaning. paraphrase detection and generation has numerous direct applications in nlp @xcite , among others in question answering @xcite , plagiarism detection (altheneyan and menai, 2019), and machine translation @xcite . research in paraphrase naturally depends on the availability of datasets for the task. we will review these in more detail in section 2, nevertheless, barring few exceptions, paraphrase corpora are typically large and gathered automatically using one of several possible heuristics. typically a comparatively small section of the corpus is manually classified to serve as a test set for method development. the heuristics used to gather and filter the corpora naturally introduce a bias to the corpora which, as we will show later in this paper, demonstrates itself as a tendency towards short examples with a relatively high lexical overlap. addressing this bias to the extent possible, and providing a corpus with longer, lexically more diverse paraphrases is one of the motivations for our work. the other motivation is to cater for the needs of finnish nlp, and improve the availability of high-quality, manually annotated paraphrase data specifically for the finnish language. in this paper, we therefore aim for the following contributions: firstly, we establish and test a fully manual procedure for paraphrase candidate selection with the aim of avoiding a selection bias towards short, lexically overlapping candidates. secondly, we release the first fully manually annotated paraphrase corpus of finnish, sufficiently large for model training. the number of manually annotated examples makes the released dataset one of the largest, if not the largest manually annotated paraphrase corpus for any language. and thirdly, we report the experiences, tools, and baseline results on this new dataset, hopefully allowing other language nlp communities to assess the potential of developing a similar corpus for other languages. ## related work statistics of the different paraphrase corpora most relevant to our work are summarized in ## text selection as discussed previously, we elect to rely on fully manual candidate extraction as a measure against any bias introduced through heuristic candidate selection methods. in order to obtain sufficiently many paraphrases for the person-months spent, the text sources need to be paraphrase-rich, i.e. have a high probability for naturally occurring paraphrases. such text sources include for example news headings and articles reporting on the same news, alternative translations of the same source material, different student essays and exam answers for the same assignment, and related questions with their replies in discussion fora, where one can assume different writers using distinct words to state similar meaning. for this first version of the corpus, we use two different text sources: alternative finnish subtitles for the same movies or tv episodes, and headings from news articles discussing the same event in two different finnish news sites. ## paraphrase annotation the paraphrase annotation is comprised of multiple annotation steps, including candidate selection as described above, manual classification of candidates based on an annotation scheme, as well as the possibility of rewriting partial paraphrases into full paraphrases. next, we will discuss the different paraphrase types represented in our annotation scheme, and afterwards the annotation workflow is discussed in a more detailed fashion. ## corpus statistics and evaluation the released corpus includes 45,663 naturally occurring paraphrases with additional 7,909 rewrites, resulting in the total size of 53,572 paraphrase pairs. the data is randomly divided into training, development and test sections using 80/10/10 split, however, with a restriction of all paraphrases from the same movie or tv episode being in the same section. basic data statistics are summarized in in figure 2 we measure the density of different label combinations in the training set conditioned on cosine similarity of paraphrase pairs based on tf-idf weighted character n-grams of lengths 2-4. up to cosine similarity of 0.5 the most common labels are evenly represented, while the prevalence of label 4 increases throughout the range and dominates the sparsely populated range of similarities over 0.8. ## paraphrase classification baseline in order to establish a baseline classification performance on the new dataset, we train a classifier based on the finbert model @xcite . each paraphrase pair of statements a and b is encoded as the sequence [cls] a [sep] b [sep], where [cls] and [sep] are the special marker tokens of the bert model. subsequently, the output embeddings of the three special tokens are concatenated together with the averaged embeddings of the tokens in a and b. these five concatenated embeddings are then propagated into four decision layers: one for the base label 2/3/4, one for the subsumption @xmath0, and one for each the binary flag s and i. since the flags only apply to base label 4, no gradients are applied to these layers for examples with base labels 2 and 3. we have explored also other bert-based architectures, such as basing the classification on the [cls] embedding only as is customary, and having a single classification layer comprising all possible base label and flag combinations. these resulted in a consistent drop in prediction accuracy, and we did not pursue them any further. the baseline results are listed in ## discussion and future work in this work, we set out to build a paraphrase corpus for finnish that would be (a) in the size category allowing deep model fine-tuning and (b) manually gathered maximizing the chance of finding more non-trivial, longer paraphrases than would be possible with the traditional automatic candidate extraction. the annotation so far took 14 person-months and resulted in little over 50,000 manually classified paraphrases. we have demonstrated that, indeed, the corpus has longer, more lexically dissimilar paraphrases. building such a corpus is therefore shown feasible and presently it is likely the largest manually annotated paraphrase dataset for any language, naturally at the inevitably higher data collection cost. the manual selection is only feasible for texts rich in paraphrase, and the domains and genres covered by the corpus is necessarily restricted by this condition. in our future work, we intend to extend the manually annotated corpus, ideally roughly double its present size. we expect the pursued data size will allow us to build sufficiently accurate models, both in terms of embedding and pair classification, to gather further candidates automatically at a level of accuracy sufficient to support down-stream applications. we are also investigating further text sources, especially parallel translations outside of the present subtitle domain. the additional flags in our annotation scheme, as well as the nearly 10,000 rewrites allow for interesting further investigations in their own right. while in the current study we concentrated on training a classifier for categorizing the paraphrases into fine-grained sub-categories, where only 2% of the paraphrases in the current release belonged to related but not a paraphrase category (label 2), which can be seen as a negative class in the more traditional paraphrase or not a paraphrase classification task. in order to better account for this traditional classification task, in future work, in addition to extending the number of positive examples, we will also look into methods for expanding the training section with negative examples. while extending the data with unrelated paraphrase candidates (label 1) can be considered a trivial task, as more or less any random sentence pair can be considered unrelated, the task of expanding the data with interesting related but not a paraphrase candidates (label 2) is an intriguing question. one option to consider in future work is active learning, where the confidence scores provided by the initial classifier could be used to collect difficult negatives. ## conclusions in this paper we presented the first entirely manually annotated paraphrase corpus for finnish including 45,663 naturally occurring paraphrases gathered from alternative movie or tv episode subtitles and news headings. further 7,909 hand-made rewrites are provided, turning contextdependent paraphrases into perfect paraphrases whenever possible. the total size of the released corpus is 53,572 paraphrase pairs of which 98% are manually classified to be at least paraphrases in their given context if not in all contexts. additionally, we evaluated the advantages and costs of manual paraphrase candidate selection from two 'parallel' but monolingual documents. we demonstrated the approach on alternative subtitles showing the technique being feasible for high quality candidate selection yielding sufficient amount of paraphrase candidates for the given annotation effort. we have shown the candidates to be notably longer and less lexically overlapping than what automated candidate selection permits. the corpus is available at github.com/ turkunlp/turku-paraphrase-corpus under the cc-by-sa license.