Cross-lingual Annotation Projection in Legal Texts (2020)

Abstract: We study annotation projection in text classification problems where source documents are published in multiple languages and may not be an exact translation of one another. In particular, we focus on the detection of unfair clauses in privacy policies and terms of service. We present the first English-German parallel asymmetric corpus for the task at hand. We study and compare several language-agnostic sentence-level projection methods. Our results indicate that a combination of word embeddings and dynamic time warping performs best.

## introduction the european union considers cultural and linguistic diversity, and in particular multilinguality, as some of its fundamental principles. so much so, that all the regulations and laws given by the parliament are published in all of eu's twenty-four official languages. these could be an immense resource towards transparency, egalitarianism, accountability and democracy, giving the eu citizen access to legislative and policy proposals in their own and also in other languages @xcite . however, to transform such resources into assets for the citizen, linguistic tools are needed, that can automatically analyze textual sources and yield actionable information @xcite . unfortunately, producing the annotated corpora required to train such linguistic tools is, even for a single language, notoriously expensive. moreover, the vast majority of linguistic resources and tools focus on english. likewise, the workforce of professionals needed for annotating legal documents may not be readily available in each language. we thus investigate methods for automatically transferring the annotations made on legal documents in a language with significant linguistic resources and domain experts, such as english, onto the corresponding versions of the same documents in a target language, where such resources and expertise may be lacking. our ultimate goal is to use automatically generated annotations for training linguistic tools for the target language without resorting to expert annotators in that language.this would leverage the creation of classifiers that can leverage the linguistic resources available in the target language, to analyze documents in that language. we chose to focus on the detection of unfair clauses in online terms of service (tos) and privacy policies (pp), which are usually published in multiple languages. however, the domain of interest of our study spans across many other types of legal texts. other eu official documents published in multiple languages include, for instance, eu parliament laws and regulations, documents of the eu court of justice, policy documents, documents for public consultations, and so on. therefore, the potential import of effective methods for facilitating cross-lingual legal text analysis is considerable. reasons for focusing on tos and pp are the interest for such documents from a consumer protection perspective, especially since the recent adoption of the european general data protection regulation (gdpr), as well as the availability of tools for the analysis of such documents. one such tool is claudette @xcite , a web server for the automatic detection of potentially unfair clauses in tos. at the time of writing, claudette is only available for the english language. an effective method for cross-lingual annotation projection could extend its scope to a variety of languages other than english, without having to resort to domain expertise in any of these languages. to carry out the present study, we built the first english-german parallel corpus for the task at hand. it turned out that the corpus is asymmetric, meaning a one-to-one sentence-by-sentence correspondence between source and target document is not always guaranteed. this aspect makes the problem particularly challenging, since we need to ensure a correct behaviour when a sentence does not appear in one of the documents or it has been integrated in another sentence: a property of sentence alignment called robustness @xcite . for example, we found source-document sentences corresponding to multiple target-document sentences, as well as significant rephrasing, resulting in the same sentence being annotated differently in the two documents. the solution we propose makes use of an automated translation of the source document into the target document, obtained via a third-party tool, of a sentence-wise dissimilarity metric, and of a method for finding an alignment between warped time series. we evaluate performance with several combinations of dissimilarity scores, with or without dynamic time warping. our results indicate that the best performance is achieved by a combination of word embedding-based dissimilarity and dynamic time warping. the main novelty of our study lies in the confluence of three aspects: the projection is performed at sentence-level; the multi-lingual corpus is asymmetric; and the proposed methods are language-agnostic. sections 2 and 3 formalize the problem and describe the architecture of our solution; 4 and 5 present corpus and experiments; 6 discusses related work in the areas of sentence alignment and annotation projection; 7 concludes. ## problem definition our problem is transferring the knowledge provided by some experts and encoded in the form of document annotations in a source language, into a target language. annotations consist of tags attached to sentences in a collection of documents. our approach is based on the projection of annotations @xcite between the two languages. henceforth we will use english (e) and german (g) as source and target, respectively. the input is defined by three resources: the annotated english version of the source document; d g ) the corresponding german version of the same document, which is the target to be annotated; the goal is to find a correspondence between the sentences e 1 , . . . , e n in d e and the sentences g 1 , . . . , g m in d g via the automatically translated sentences g t 1 , . . . , g t n . in this way, the original annotations (i.e., the labels) ℓ 1 , . . . , ℓ n associated with the sentences in d e can be transferred (projected) from the english document into a sequence of corresponding labels ℓ 1 , . . . , ℓ m in the target. all the correspondences are thus evaluated among pairs of english sentences. we shall remark that our setting is asymmetrical, therefore owing to the different way the content of documents may be rendered in different languages, n may differ from m. although our experimental results are limited to the english/german language pair, the approach itself is language-agnostic. in particular, it does not rely on any language-specific features. for the same reason, we consider automatic translation as a service, and not as a variable in the experimental study. ## projection architecture our methodology for annotation projection is two-fold. the first step is the computation of a set of matches between each sentence of the translated target document, d t g , and one or more sentences of the source document d e . each sentence of d t g is thus labeled using the labels of the corresponding sentences of d e , if any. the second step amounts to transferring the labels from d t g to the target 1 the choice to translate into english is arbitrary. one could equally choose to rely on translating de into german. the simplest procedure, which we take as a baseline, p t , is illustrated in figure 1 . it consists in computing the dissimilarity between any two sentences from the two documents, by simply considering them as strings of symbols, and then matching each non-annotated sentence of one document with the least dissimilar sentence in the other one. a similar procedure, p e , requires creating a neural embedding for each sentence and then computing the dissimilarities between embeddings. these procedures create a relation between source and target sentences: each target sentence is associated with one source sentence, whereas each source sentence can be associated with any number of target sentences (possibly none). in the case of online tos and pp written in different languages, it is not surprising to observe a high degree of parallelism between the source and the target documents. therefore, it is fair to assume that in most cases the sentences will appear in the source document in roughly the same order as their matching counterparts appear in in the target document. 2 to exploit this property, we design two additional procedures relying on dynamic time warping @xcite to compute the match: p t+dt w for textual input data, and p e+dt w for embedded input sentences. ## dataset the english-german corpus created for this task includes tos and pp: two types of documents employing very different languages and drafting techniques. ## experimental results we performed projection as a fully unsupervised process. to translate the documents from german to english we used apache joshua, a standalone tool that poses no usage restrictions. to compute the dissimilarity measures described in section 3.1 between the d e and d t g we used the python textdistance library. to create the embeddings of the source sentences and of the translated target sentences, we use the pre-trained elmo model available on tensorflowhub, which follows @xcite , with an embedding size of 1024, and create sentence embeddings by average of word embeddings. the corpus used for pre-training is 1b word benchmark @xcite , which includes a variety of sources in english such as news commentaries and parliamentary debates. to compute the distance between embedding vectors we used the scipy library @xcite . after obtaining the projected labels for d g , we evaluated the performance by comparing such projected labels with the original golden annotations. in order to measure the performance, we considered the task as a multi-label classification problem on the sentences of the target documents. tables 2 and 3 report f1 scores, precision and recall associated with each approach. these scores are computed with no distinction between different documents. the f1-micro score is computed considering the total number of true positives, false positives, and false negatives, without making any distinction between classes. the weighted score is computed as a weighted average between the f1 scores obtained for each class, where the weight is given by the number of times the label appears in the ground truth. the macro score is obtained with a similar procedure, without using different weights, but considering all the classes alike. we shall remark that, in principle, the task does not guarantee that an f1 equal to 1 is always achievable, due to possible discrepancies between the two sets of labels in the original english and german corpora. with reference to example 4.1, even if the english sentence had been correctly associated with both german sentences, each of the latter would inherit both tags of the source language, thus resulting in a false positive for what concerns the evaluation metrics. in finally, figure 2 summarizes the f1-weighted scores obtained for each different category in the setting without dtw, in tos and pp, respectively. the plot shows that the embedding-based approaches consistently outperform the other dissimilarity scores. it is noteworthy that the categories of tos in which we perform best and worst are the ones with less data points. a meaningful qualitative analysis should not focus on those two classes but rather sample on the whole dataset. we will further investigate these differences in our future work. one issue worth consideration is the impact of the translation service on the overall performance. we have conducted preliminary experimentation using professional tools such as google translate. we did notice a performance improvement overall. however, the relations between the performance of p t , p e , p t+dt w and p e+dt w are unchanged. 6 related works our work is closely related to two research strands, namely sentence alignment and annotation projection. the sentence alignment task is addressed in the work of @xcite by defining a "corridor of alignment" based on global information. basically, a candidate matching between sentences takes into account the position of the sentences inside each document. then, corresponding words are matched, and this local information is further exploited to align full sentences following an ordered one-to-one relationship. this is somehow similar to our approach based on dtw, although we directly operate at the sentence level. to address the task of aligning documents in which sentences do not appear in the same order, @xcite present an approach based on integer linear programming, while @xcite propose instead an approach based on the length of the sentences to create one-to-one matches. other examples of alignement between parallel corpora can be found in @xcite . with respect to these works, our task is different, since we do not aim for the perfect alignment of words, or paragraphs, but we are interested only in a faithful projection of labels. the idea of projecting annotations between two parallel corpora in different languages, to enable the construction of a machine learning model in both languages, is firstly framed in @xcite . the authors there address various word-level nlp tasks. interestingly, they address the problem of noisy projection by using the uncertainty level given by the alignment model. the same problem is addressed by @xcite by exploiting multiple sources for the same target document. projection has been used also for argumentation mining @xcite , to create training data for machine learning models for low-resource languages. in particular, @xcite argue against the necessity of human-translated parallel corpora as a resource, since they obtain comparable results using machine translated parallel documents. in contrast with these approaches, which use projected labels for supervised training, in the work by @xcite the projected labels are used as features for unsupervised training. projection has also been used by @xcite to create a parallel version of an existing corpus. the projection of structural information between parallel documents is tackled by @xcite , where alignment is performed firstly sentence-wise (1-1) and then word-wise. none of the reviewed approaches is directly applicable to our setting. the works of @xcite , @xcite , @xcite are all based on symmetric corpora, while imposing a 1-1 alignment is not possible in our domain. @xcite , @xcite , @xcite , @xcite , and @xcite all address word-level alignment instead of sentence level-alignment. these methods cannot be extended to address sentence-level alignment without introducing an element of subjectivity, for instance, in resolving conflicts of words matched across different sentences. finally, @xcite rely on the language-dependent concept of "cognate" words, which is applicable only to selected language pairs, while our approach is language-agnostic. besides projection, other approaches can be used to apply machine-learning methods for low-resources languages. direct transfer @xcite consists of exploiting features which are shared across the languages, such as multilingual word embeddings, and then training a model on the labelled corpus, so as to use it directly on the test data, without any need of parallel corpora or projection. while this approach may seem less noisy than projection, in @xcite direct transfer approaches have achieved worse performance. other works @xcite have made use of weak supervision, a mixture of supervised and unsupervised learning, which requires a setting where a few labelled documents of the target language are available. finally, another approach consists in the cross-lingual transfer/alignment of word embedding spaces @xcite , where a mapping between the word embeddings of the source and the target language is first learned; then, word embeddings in the target language are mapped into the ones in the source language word, so that they can be finally fed into any model trained on the source language documents. for what concerns other methods to asses the distance between two sentences, gomaa and fahmy (2013) present an extensive survey from which we have selected the most common and significant algorithms. @xcite and @xcite train neural model to assess the distance between two sentence, but this approach is applicable only to languages where a dataset of sentence distances is available. finally, @xcite and @xcite involve the training of advanced neural models and require a large database for training, thus do not suit our setting, since ours is not a learning-based approach. ## conclusion we devised and compared several methods for annotation projection. the motivation behind this study stemmed from our long-term goal of extending the scope of existing classification systems of legal texts, currently available only in english, so as to have them work with a variety of languages, without having to resort to legal experts in such languages. we studied the performance of a number of alternative methods for text encoding, measuring sentence dissimilarity, and integrating sequence information in the alignment problem. we tested several possibilities with an english-german corpus created by legal experts in the two languages, and interpreted the results as an indication that a combination of word embeddings and dynamic time warping seems most effective. this is fortunate since the methods involved in such a combination also yield a much smaller computational footprint compared to other methods we have investigated. these results pave the way to several directions for future research. one of them is architectural. as we remarked when we defined the problem, our approach could be implemented by relying on automated translations of source documents to the target language or vice versa. in our study, we arbitrarily opted for the second alternative. we thus plan to implement the option we have not yet considered, and measure its performance in comparison to the results we obtained so far. moreover, we plan to make an experimental comparison between the results obtained with a tool for legal text analysis trained to work on a target language using the approach presented here, and the results obtained by translating the query document and performing the analysis with the original tool. we speculate that answers to the above questions may depend partly on the task, partly on the availability and quality of language resources and tools in different languages. finally, another direction worth of investigation is the use of state-of-the-art sentence embeddings, such as sentence-bert @xcite and universal sentence encodings @xcite . in conclusion, we shall remark that although our analysis has focussed on a specific set of legal documents, cross-lingual annotation projection in legal texts has, in fact, a much wider import. in the longer run we would like to apply this method to other areas of cross-lingual legal text analysis, with the ultimate purpose of contributing positively to the relations between citizens and institutions across nations characterized by linguistic and cultural diversity and united under a common institutional framework.