Creating and Evaluating a Multilingual Corpus of UN General Assembly Debates (2024)

Abstract: This paper presents a multilingual aligned corpus of political debates from the United Nations (UN) General Assembly sessions between 1978 and 2021, which covers five of the six official UN languages: Arabic, Chinese, English, French, Russian, and Spanish. We explain the preprocessing steps we applied to the corpus. We align the sentences by using word vectors to numerically represent the meaning of each sentence and then calculating the Euclidean distance between them. To validate our alignment methods, we conducted an evaluation study with crowd-sourced human annotators using Scale AI, an online platform for data labelling. The final dataset consists of around 300,000 aligned sentences for En-Es, En-Fr, En-Zh and En-Ru. It is publicly available for download.

## introduction multilingual corpora are valuable resources for natural language processing (nlp) research and applications, as they enable the development and evaluation of cross-lingual and low-resource models and systems. however, creating and maintaining large-scale and high-quality multilingual corpora is a challenging task, as it involves collecting, processing, and aligning texts from multiple languages and domains, while ensuring their accuracy, consistency, and relevance. in this paper, we align and evaluate a multilingual corpus that is based on the plenary sessions of the united nations (un) general assembly, which is the main organ of the un where all member states have equal representation and voice. the plenary sessions are held every year and are translated and transcribed in the six official languages of the un: arabic, chinese, english, french, russian, and spanish. these sessions cover a wide range of global issues, such as peace and security, human rights, development, climate change, and health, and reflect the views and positions of different countries and regions on these issues. therefore, our corpus provides a rich source of multilingual texts within the political domain that can be used for various nlp tasks, such as machine translation, in-domain text classification, question-answering, and multilingual argument mining. we describe the methods we used to collect, clean, segment, and align the plenary sessions across languages. we then cover our approach to bilingual sentence alignment using embeddings and euclidean distance, and the special considerations and difficulties we encountered for the different languages. for example, we faced some challenges in aligning arabic with the other languages, due to technical issues in converting the arabic documents into a suitable format for alignment. we also noticed some differences in the order and structure of sentences across languages, which made the alignment more difficult. we discuss how we addressed these challenges and how we validated the quality of our alignment. ## related work previous multilingual parallel corpora have been based on united nations data, including the mul-tiun @xcite ) and the united nations parallel corpus @xcite . in the multiun corpus, data was retrieved from the united nations official document sys-tem (ods), a web-based repository of official documents of the united nations. the data was filtered by publication symbols, which are unique identifiers that indicate the issuing body, the type of document and the year of publication. the paper selected documents with publication symbols that correspond to official records and other parliamentary documents of the un. the multilingual sentence alignment starts with pairwise alignments based primarily on sentence lengths and then on a dictionary. pairwise alignments are available, and later merged into multilingual alignments across all six languages. the updated version of mul-tiun, v2, contains documents up to and including 2011 @xcite . the united nations parallel corpus (unpc) is composed of official records and other parliamentary documents of the united nations that are in the public domain. it contains sentence-level alignments for content between 1990 and 2014. the corpus contains 799,276 documents in six languages and contains 86,307 documents that have translations across all six languages. the sentencelevel alignments were generated using giza++ @xcite to align sentences based on word co-occurrences. the authors validate their dataset through a number of machine translation baselines, with bleu scores results varying between 29 and 61, depending on the language pair. while robust, neither of these corpora offer a full evaluation of the accuracy and precision of their alignment, nor are they recent enough to include the later documents. furthermore, sentence alignment methods have come a long way since their collection. more modern methods of multilingual sentence alignments are based on multilingual pretrained language models, such as mbert, that can learn cross-lingual representations of sentences. these methods have been shown to outperform giza++ @xcite . @xcite use a sequence-tosequence architecture to train a multilingual sentence encoder on an initial parallel corpus. the encoder maps sentences from different languages into a shared embedding space, where similar sentences are close to each other. the authors then use a margin-based scoring method to measure the similarity between sentence embeddings. the authors evaluate their their method on three tasks, the bucc mining task, the un reconstruction task, and the paracrawl filtering task, and show that the proposed method outperforms existing methods on all three tasks by a large margin. ## corpus collection the united nations (un) plenary meetings are meticulously recorded in each of the six official un languages, making them an ideal source for a multilingual corpus. the records are then made available on the official website foot_0 , in the form of pdf files, separated by language and session. all the documents are public domain. we downloaded the documents in all 6 languages and converted them using an ocr-based tool in a pdf editor, discarding pictures, tables and style markers. in total, we processed 2113 documents between 1978 and 2021. however, due to the age of the documents and the limitations of the ocr-based tool, we were unable to convert enough arabic-language documents for use in alignment. furthermore, the documents we did manage to convert were of poor quality. as a result, we were unable to align the arabic sentences and eventually discarded the arabic language documents until such a time that we can properly convert them. we then processed these documents using the natural language toolkit (nltk) package @xcite . we used the toolkit to separate the documents into individual sentences, as the documents only provide paragraph boundaries. we then tokenised the sentences and removed stop words to made them ready for alignment. ## sentence alignment the documents described in section 3 are not translated at the sentence level, but rather at the level of individual speeches taken as a whole. this means that each speech in one language has a corresponding speech in another language, but not necessarily each sentence. therefore, in order to create a parallel corpus at the sentence level, we need to match each sentence in one language with the equivalent sentence in another language. this is a challenging task, as the sentences may not have the same order, structure, or length across languages. furthermore, translations are not always a one to one mapping. sometimes a sentence can be represented by multiple sentences in the other language, or multiple sentences can be condensed into a single sentence in another. therefore, a simple probabilistic model based on sentence length would fail across languages with different scripts and language families. to solve this problem, we use a semantic similarity approach that aligns the sentences based on their meaning and content, rather than their form or position. we do this by using word vectors to represent the meaning of each sentence as a numerical vector. then, we calculate the euclidean distance between the vectors of each language pair. the sentence pair with the smallest distance is the correct match. for word vectorisation, we used language-agnostic sentence representations (laser) foot_1 , an open-source nlp toolkit developed by facebook and trained on the tatoeba corpus foot_2 . laser performs sequence-to-sequence processing with an encoder-decoder approach. the encoder network, which is used to generate the embeddings we need, is a five layered bi-directional long-short-term memory (bilstm) network whose input is a string and output is a fixed-size vector in a 1024 dimensional space. crucially, this space is shared by all languages, meaning that sentences with similar meaning in two different languages would be mapped to very near points in the space, regardless of how different the languages are. the vectors are normalised and stored in a matrix, where each row represents a sentence and each column represents a language. we calculate the euclidean distance between each sentence in one language and a window of 25 sentences in another language for each language pair. we select the pair of sentences with the smallest distance as the match. the window size is implemented to decrease time complexity as well as improve accuracy by not considering sentences too far away to have been the intended translation. this is done to prevent long, vague sentences that may be close to several other sentences from being matched numerous times, while also allowing for genuine cases where a sentence in one language has legitimately been represented by multiple sentences in the other. we also perform anchoring, where we identify special entities such as dates and numbers, and include only sentences in the target language that that contain the same terms to be considered for matching. ## evaluation we evaluated the quality of our final validated dataset using crowd-sourced human annotations. to obtain reliable and consistent evaluations, we used scale ai foot_3 , an online platform whose purpose is to generate labelled datasets for training ai models. scale ai allows for the labelling of data such as images, videos, texts and 3d models. we uploaded our parallel documents to scale ai and requested the annotators to mark the sentences that are translations of each other in each language pair. we also provided them with clear instructions and examples of how to perform the task. we received the annotations from scale ai in a json format, which we converted into a tabseparated format for further analysis. scale also selects a "training set" of 20 sentence pairs, which it chooses from the corpus, for its crowd-sourced users, and discards results from users that perform below a threshold of 70% on the training set. we designed the task to present annotators with two sentences, the "original" sentence in english and the "target" sentence in one of the target languages: french, russian, spanish or chinese. the annotators were asked to read the sentences carefully and decide whether or not the two sentences are a match, a partial match, no match, or if they were unsure. in the instructions, the annotators were given detailed guidelines about what constitutes a match. if two sentences are direct translations of each other, or if all the information in the target sentence is present in the original sentence, they are considered to be a match. furthermore, if the target sentence conveys the full meaning of the original sentence, annotators are to consider them a match. partial matches occur when some information in the target sentence is not present in the original sentence, or vice versa. if the sentences are neither a full match nor a partial match, then annotators were to choose "no match". we also included an "unsure" option, and discarded any responses that included it. figures 1 and shows an example of the instructions for the english-french evaluation, the way they appear next to each sentence pair in the task. in addition to these instructions, further workflow directions were made available. the task required 3 reviews per sentence, meaning three different annotators had to agree on a label for it to be accepted. annotators were required to have a basic proficiency in the source language and native proficiency in the target language. however, due to crowd-sourcing, there was no way to verify their actual proficiency. the agreement between reviewers was pretty high, with cohen's kappa at 0.87 across all four language pairs. furthermore, the evaluators found that over 80% of the presented sentences were a match, and less 5% were completely unaligned. the number of evaluated sentences varied across languages, as it depended on the number of available annotators that scale was able to train for each task. as a result, while we only had 1000 sentences evaluated for english-french, we managed to evaluate upwards of 6000 sentences for english-chinese. ## limitations we acknowledge some limitations of our dataset that we aim to overcome in future iterations. one of the main limitations is that we could not include arabic as one of the languages in our corpus, due to technical difficulties in converting the arabic documents into a suitable format for alignment. this means that our dataset does not cover all six official languages of the united nations, and thus misses an important and widely spoken language in the world. we hope to solve this problem by finding a more reliable way to process the arabic documents and align them with the other languages. another limitation of our dataset is that we relied on crowd-sourcing for evaluating the quality of our alignment. while crowd-sourcing is a convenient and cost-effective way to obtain human judgments, it also comes with some drawbacks, such as inconsistency and bias among the annotators. we tried to mitigate this issue by auditing the results and filtering out the outliers, but we could only review a small fraction of the evaluations. therefore, our evaluation may not reflect the true quality of our dataset, and may be influenced by the subjective opinions of the annotators. we plan to address this issue by conducting a more rigorous and systematic evaluation of our dataset, using multiple sources of human feedback and objective metrics. ## conclusion in this paper, we introduced a novel parallel corpus that consists of texts from the plenary sessions of the united nations general assembly. our corpus covers five languages: english, french, spanish, russian, and chinese. we described the process of extracting and preprocessing the sentences from the original documents, and aligning them based on semantic similarity using a state-of-theart cross-lingual sentence encoder. we evaluated the quality of our dataset using two methods: a simple validation that uses a regression model to predict sentence length based on the source language and the target language, and a crowd-source human evaluation that measures the accuracy and precision of our alignment. the resulting aligned dataset has a high degree of accuracy across languages, and can be used for various natural language processing tasks, such as machine translation, cross-lingual information retrieval, and multilingual text summarisation. our work contributes to the field of multilingual natural language processing by providing a largescale and high-quality parallel corpus that covers multiple languages in the field of political discourse and debate. we believe that our corpus can facilitate the development and evaluation of crosslingual models and applications. in the future, we plan to solve the problem of arabic-language documents that prevented us from completing our dataset for all six official languages of the united nations. we also intend to extend our corpus to include more languages and more sources of multilingual texts. the current version of our dataset is available for download foot_4 .