Extension M ulti30 K : Multimodal Dataset for Integrated Vision and Language Research in U krainian (2023)

Abstract: We share the results of the project within the well-known Multi30k dataset dedicated to improving machine translation of text from English into Ukrainian. The main task was to manually prepare the dataset and improve the translation of texts. The importance of collecting such datasets for low-resource languages for improving the quality of machine translation has been discussed. We also studied the features of translations of words and sentences with ambiguous meanings. The collection of multimodal datasets is essential for natural language processing tasks because it allows the development of more complex and comprehensive machine learning models that can understand and analyze different types of data. These models can learn from a variety of data types, including images, text, and audio, for more accurate and meaningful results.

## introduction creating and processing high-quality datasets for such low-resource languages as ukrainian is incredibly important for solving machine learning tasks. the task of machine translation, like other tasks, requires large amounts of data to effectively learn and understand the nuances and complexities of language. however, for low-resource languages, the available data may be limited, which can make it difficult to develop accurate and efficient models. datasets directly affect the performance of machine learning models. this can lead to higher accuracy and better generalization for tasks such as language translation, speech recognition, and natural language processing. a multimodal dataset refers to a dataset that consists of various data types, each representing a different modality. these modalities can include images, text, audio, and other types of data, each with its own unique meaning. by including multiple domains, a dataset can collect a wider range of information, allowing the development of more complex machine learning models. for our task we decided to choose multi30k @xcite dataset which consists of two modalities: images and their descriptions. multi30k is a modification of the flickr30k dataset @xcite with 31,014 german translations of english annotations and more than 150,000 collected annotations in german. this dataset was edited by professional translators, and one picture corresponds to one annotation in english and german, which is the reason for choosing this dataset for further adaptation into ukrainian. the problem under consideration consists of three parts: ## datasets and tasks the main dataset used for the tasks described above is the multi30k dataset, which includes 31,000 images originally described in english. the presented dataset also includes translations created by professional german translators. in the next iterations, this dataset was also translated into french @xcite , czech @xcite and turkish @xcite . dataset overview is presented on figure 1 . the dataset can be used to boost performance of some existing multilingual multimodal models for various machine learning tasks, such as multimodal machine translation, image captioning, image se-mantic search, cross-lingual transfer learning, and multilingual text summarization etc. as a result, we managed to process 31,014 sentences for ukrainian and english, and the number of words that are in this dataset was also counted. we prepared a ukrainian version of multi30k dataset with the following features. comparison of the number of tokens for the languages from the original article (english and german) and ukrainian can be seen in this number for the english language exceeds the given number for the ukrainian language, due to its linguistic features, for example, there are no articles in ukrainian. 3 dataset preparation process ## cosine similarity the data obtained after translation were analyzed. we decided to calculate the cosine similarity using a multilingual model distiluse-base-multilingualcased-v2 @xcite for the original translation using google cloud translator and for the translation obtained after manual correction. we got a high value of cosine similarity for all sentences for both languages. as a result, 4997 sentences have a high value, and 3 have a low value. here, values above 0.4 are taken into account, which is considered sufficient for a general understanding of the meaning of a sentence or phrase. the columns "initial text" and "manually translated text" indicate the number of sentences that exceed the corresponding cosine similarity value. thus, as a result of our translation adjustment, an additional 30 values went out of range of 0.9, 63 values went out of range 0.8, and so on. this reflects the effectiveness of the work done on manual verification of the selected data. ## conclusion in conclusion, the importance of collecting high quality datasets for low-resource languages such as ukrainian cannot be overestimated for machine learning tasks. an example of this was our project to improve the machine translation of text from english to ukrainian by manually preparing the multi30k dataset and examining translations of ambiguous words and sentences. collecting multimodal datasets that include different types of data such as images, text, and audio is especially important as they provide richer and more complex data for developing accurate and meaningful machine learning models. the results from our project demonstrate the impact of such datasets in improving the performance of machine learning models for tasks such as machine translation. as a result, the creation and processing of such data sets will lead to a significant improvement in the solution of the problem of machine translation and many other tasks. the project involved loading and validating a selected data set to determine the data type, image format, and word count. the data set was translated from english into ukrainian using google translate, which served as the basis for manual verification and correction by a team of 8 people as a further development we want to research siamese language models and cosine similarity of their embeddings even more to finally either prove or disprove that they can be used as benchmarks for machine translation. also, we want to check how our gathered dataset will affect the performance of existing multimodal multilingual semantic search models by finetuning them using ukrainian multi30k. another area for further research is to combine token metrics and text embeddings from a multilingual semantic search network to capture figurative meaning and some professional or just domain specific words and phrases.