C roatian Film Review Dataset (Cro- F i R e D a): A Sentiment Annotated Dataset of Film Reviews (2023)

Abstract: This paper introduces Cro-FiReDa, a sentiment-annotated dataset for Croatian in the domain of movie reviews. The dataset, which contains over 10,000 sentences, has been annotated at the sentence level. In addition to presentingthe overall annotation process, we also present benchmark results based on the transformer-based fine-tuning approach.

## introduction the goal of sentiment analysis is to classify the polarity of text (e.g., positive, negative, neutral, or mixed). in this paper, we describe the process of annotating a sentiment analysis dataset in croatian. as shown in the example below, the label indicates the sentiment polarity of the text. hr "i bio sam zadivljen i tijekom finalne borbene scene ." en "and i was also amazed during the final battle scene." the dataset presented here is the first sentiment movie review dataset. the texts for the annotation campaign are taken from the croatian movie review website and cover multiple genres, namely adventure, series (serija), and sci-fi. in addition to the other metadata described below, the website includes a summary of the entire text of the author's review. the dataset, annotation guidelines, trained models, and associated code will be made available to the public. in this work, we describe our entire workflow for creating the resource. we also present the experimental scores for the sentiment analysis task using pre-trained transformer models. the rest of the paper is structured as follows: in section 2, we review the related work on the dataset with regard to its annotation as well as modelling. in section 3, we describe the annotation process in detail. in section 4, we present the statistics of the annotated dataset before presenting the baseline scores in section 5. we complete the paper with the conclusion, discussion, and future work in section 6. ## related work in this section, we will highlight the related work on resources and models for sentiment analysis. sentiment analysis is a well-researched field, and there are a number of resources for various languages, such as english @xcite @xcite , german @xcite @xcite , french @xcite , and italian @xcite . there are few resources available for croatian sentiment analysis. the stance (and sentiment) annotated dataset @xcite ) contains comments submitted by users for online news articles. @xcite created a dataset for sentiment analysis of croatian news articles and performed zero-shot classification using slovene resources. @xcite performed multiple levels of sentiment analysis on multilingual wikipedia articles using machine translation. @xcite compiled a parallel dataset for sentiment and emotion analysis based on movie subtitles. the dataset was created by manually annotating 25k finnish and 30k english sentences, which were then projected onto 30 other languages, including croatian. @xcite presented rule-based annotated croatian news articles in the finance domain that captured the general sentiment of the text. @xcite compiled a dataset of gaming review text spans in croatian that were tagged with positive and negative labels. there are also a few croatian sentiment lexicons, such as those developed by @xcite ; @xcite . the parlasent-bcs @xcite dataset is another resource that has croatian sentences in parliamentary debates tagged with sentiment polarity. @xcite provide an overview of existing resources for east european languages, including croatian. ## text and annotations in this section, we describe our annotation procedure in greater depth. first, we describe the backgrounds of the annotators. second, the guidelines and methodology for annotation are explained. third, the statistical aspects of the dataset are discussed. the task is defined as a sentence-level sentiment task in which each sentence in the training set is annotated with a single label. the dataset consists of professional reviews from the croatian movie review website 1 . the adventure, tv series, and science fiction (sci-fi) genres were chosen as subcategories. each review instance is accompanied by the following data fields: ## results the scores for the sentiment task are reported using the f1, and accuracy (macro) metrics. in the case of fine-tuning setup, each experiment was performed five times with different random seeds, and the mean of all the scores is reported in ## conclusion and future work with this paper, we have presented the sentiment annotated movie review dataset for croatian. we performed experiments using curated datasets for the sentiment analysis task for the croatian language. out of 21 unique categories of film reviews, to name a few, we have processed only three categories (adventure, series (serija), and sci-fi).