Advancing Sentiment Analysis in S erbian Literature: A Zero and Few–Shot Learning Approach Using the Mistral Model (2024)

Abstract: This study presents the Sentiment Analysis of the Serbian old novels from the 1840-1920 period, employing the Mistral Large Language Model (LLM) to pioneer zero and few-shot learning techniques. The main approach innovates by devising research prompts that include guidance text for zero-shot classification and examples for few-shot learning, enabling the LLM to classify sentiments into positive, negative, or objective categories. This methodology aims to streamline sentiment analysis by limiting responses, thereby enhancing classification precision. Python, along with the Hugging Face Transformers and LangChain libraries, serves as our technological backbone, facilitating the creation and refinement of research prompts tailored for sentence-level sentiment analysis. The results of sentiment analysis in both scenarios, zero-shot and few-shot, have indicated that the zero-shot approach outperforms, achieving an accuracy of 68.2%.

## introduction over the years, the need for sentiment analysis as one of the pivotal fields of natural language processing (nlp) has significantly grown across various domains of interest, including but not limited to medicine @xcite , finance @xcite , education @xcite , digital humanities @xcite , politics and social media @xcite . previous research has mainly focused on a small number of languages that had a larger amount of training data available. interest in languages with low resources such as arabic @xcite , bangla @xcite , african @xcite , and serbian @xcite has grown over the years. considering the insufficient resources for the serbian language, the possibility of training large language models (llm) without a large amount of training data represents an important step in sentiment analysis. so far, various approaches have been employed for sentiment analysis over the serbian language. the sentiment analysis on the serbian movie review dataset using by using unigram, bigram, and trigram features in a combination of naïve bayes (nb) and support vector machines (svm) @xcite showed the best accuracy of 85.5% for 2 classes and 62.2% for 3 classes. the sentiment analysis framework for a morphologically rich language (safos) @xcite had the best accuracy of 78.3% for movie reviews and 79.2% for newspapers using a combination of unigram and bigram features reduced by sentiment feature mapping. within the same research, the sentiment lexicon and serbian wordnet (swn) synsets were integrated using sentiment polarity scores for feature selection and the lexicon derived from swn was augmented by incorporating morphological variants of phrases and emotional terms from serbian morphological electronic dictionaries @xcite . the lexiconbased approach using three existing lexicons: nrc, affin and bing with additional extensive corrections, using multinomial naïve bayes (mnb) with bag-of-words approach combined with the features of the sentiment lexicon. this approach gave the accuracy of sa on the evaluation dataset of 82% for two classes, and 72% for 3 classes @xcite . the main motivation for this study lies in the fact that, to the best of our knowledge, sentiment analysis in serbian literature utilizing the zero-shot and few-shot learning approach using the mistral model has not been jet explored. machine learning has been highly successful in data-intensive applications but is often hampered when the data set is small, and this study offers a new approach to sentiment analysis in cases of smaller data sets. the sentiment analysis was applied to the selected, annotated, and balanced sentences from the serbian part of the eltec 1 multilingual corpus of novels. novels written in the period 1840-1920 are built to test various distant reading methods among them sentiment analysis, presented in section 2. four human annotators performed careful checks of sentiment in sentences, yielding 1089 balanced sentences with three classes: positive, negative, and neutral. techniques used for automated classification were zero-shot and few-shot. zero-shot learning techniques, where the llm is prompted without prior specific training on the task, rely solely on the general capabilities of the model @xcite @xcite conversely, few-shot learning involves providing the llm with a small number of examples before requesting it to perform the task. this method aims to prime the model with relevant context, enhancing its performance on specific sentiment classification tasks @xcite . the mistral 7b-instruct (jiang et al., 2023) variant, specifically utilized in this work, has been finetuned to follow instructions with remarkable precision, thus providing an advantage in generating contextually relevant and accurate sentiment analysis. it achieves this by leveraging the base model's architectural efficiencies without sacrificing performance on complex text inputs. this version of mistral 7b outperforms comparative models in human and automated benchmarks, showcasing its utility in nuanced language tasks such as sentiment classification. furthermore, further elaboration will be provided in section 3.1. in section 3 the methodological approach is depicted through various prompts, while a detailed evaluation of the model on prepared sentences (with the findings and a thorough discussion) is given in section 4. finally, conclusions and plans for future work can be found in section 5. one of the main goals is to ascertain whether llms can provide a consistent, efficient, and potentially less biased means of sentiment annotation, 1 eltec: european literary text collection thereby overcoming some of the limitations associated with human annotators. through this comparison, the feasibility and advantages of integrating llms into the sentiment analysis process are aimed to be illuminated, potentially revolutionizing how sentiment data is processed and interpreted in various applications. this aspect is particularly significant for languages with limited linguistic resources, such as serbian. these languages often lack comprehensive corpora with annotated sentiment, presenting a substantial challenge for traditional sentiment analysis techniques that rely heavily on such datasets. the scarcity of annotated corpora in these languages not only hinders the development of effective sentiment analysis models but also limits the applicability of these models in real-world scenarios. ## dataset serbian part of eltec corpus @xcite , dubbed srpeltec, comprises 100 novels in the main collection and 20 in the extended collection. these novels are digitized and freely accessible, thus presenting no constraints on their usage. however, challenges arise concerning the analysis and extraction of information from such text collection, which consists of 5.886,528 tokens and 4.769,262 words. novels are automatically annotated with part of speech, lemma, and named entity information, thereby paving the way for the application of advanced text analysis methods, in line with the distant reading paradigm. for sentiment analysis, a subset of this text collection is used in previous research. for evaluation, we will rely on a previously manually annotated dataset with 1089 sentences @xcite . figure 1 presents the distribution of sentence length, quantified by the number of words, which corresponds uniquely to each sentiment label. to evaluate the models and demonstrate the capabilities offered by zero-shot and few-shot methodologies compared to previous research, the same dataset was employed for evaluation purposes. the process of annotating sentences occurred in several phases: 1) extraction of 30k sentences from srpeltec; 2) manual evaluation by four annotators, where the annotation is conducted on a scale from -5 to -2 for negative sentiment gradation; -1, 0, 1 for neutral (objective) sentiment; and 2 to 5 for positive sentiment and 3) calculating inter-proceedings of clib 2024 annotator agreement was calculated using recal2 tool (deen freelon, 2011) that showed: percent agreement 82.5%, scott's pi 0.737, cohen's kappa 0.739, krippendorff's alpha (nominal) 0.737. the human annotator's task in this context relied heavily on their intuition as a native speaker of the language. however, this approach had limitations, particularly when dealing with sentences that are sarcastic or express victory of one side over another, for example in sentence "kad su ga drugi dan iz crkve sa krštenja doneli, dodje i kršteni kum deteta, sava srbin, dobra duša ti je on bio, al ' sav beše suzama poliven kad je u sobu ušao." (when they brought him back from the church the next day, the baptized godfather of the child, sava the serb, arrived, he was a good soul, but he was entirely bathed in tears when he entered the room.). in such cases, determining the polarity of the sentence became challenging without clear specifications on what constitutes positive, negative, and neutral sentiment. to address these challenges, annotators may require additional context or guidelines to determine the intended sentiment accurately. providing specifications on what constitutes positive, negative, and neutral sentiment can help standardize the annotation process and minimize subjective interpretation. moreover, leveraging advanced natural language processing techniques, such as sentiment analysis algorithms, can complement human annotation efforts by identifying sentiment patterns and detecting nuances in language that may be challenging for human annotators to discern alone. in conclusion, while annotating sentences for sentiment analysis, relying on the annotator's intuition as a native speaker is essential. however, to ensure accuracy and consistency, it is crucial to provide clear guidelines and consider contextual factors, especially when dealing with ambiguous or nuanced expressions like sarcasm or conflicting sentiments. manual annotation not only requires significant time investment but also heavily relies on the human annotators' comprehension of the instructions and their proficiency in the native language of the text being analyzed. this dependency introduces a potential for variability and subjectivity in the annotations, which can influence the reliability of sentiment analysis outcomes. ## methodology the research on sentiment analysis of eltec texts was performed using llms mistral 7b model which will be briefly introduced in section 3.1. the proceedings of clib 2024 methodology for this research employed a "prompt and response" technique @xcite , utilizing llms to analyze sentiment within a corpus. prompts were generated from prompt templates. prompt templates are crafted so that the prompts generated from them contain sentences or examples from the corpus, designed to elicit llm responses that reflect a range of sentiments. prompt templates were prepared for both zeroshot and few-shot learning scenarios, with the former requiring no examples for the llm to generate responses, and the latter incorporating specific examples to guide the model's output. four templates were devised for the zero-shot learning approach, aiming to evaluate the model's innate understanding and response generation capabilities without prior context. conversely, two templates were established for the few-shot approach, each including examples intended to orient the model toward the desired output, as will be detailed in section 3.2. the llms responses to the prompts are parsed and classified into the same categories as those used for manual annotation within the corpus: positive, negative, and neutral, where the parsing process is crucial, given the llm's potential to generate subtly nuanced responses. finally, the llm-generated sentiment classifications are compared to the manual annotations using accuracy and confusion matrices presented in section 4. the approach taken emphasizes minimizing extraneous elements in the llm's responses. this was achieved by limiting the responses to specific instructions or grammatical structures, thereby simplifying the subsequent text-parsing process. to prepare the responses for classification, the following steps were systematically implemented in three steps: 1) extraneous characters, including spaces, new lines, and punctuation, were removed from the llm's responses. additionally, all text was converted to lowercase to maintain consistency and eliminate any discrepancies caused by case sensitivity; 2) the cleaned text was then parsed to identify keywords that indicate sentiment. specifically, the presence of words corresponding to "positive", "negative", "neutral", or variants thereof, such as "objective", was checked. 3) based on the keywords identified, each response was classified into categories: this method of response processing ensures that the textual responses from the mistral model are efficiently classified, allowing for clear and quantifiable analysis of sentiment trends based on the llm's outputs. figure 2 outlines the systematic workflow for preparing the mistral model for sentiment analysis. in addition to quantitative analysis, this study also employed qualitative analysis to examine instances where llms may surpass human annotators in sentiment analysis accuracy. this qualitative examination focused on identifying specific cases within the corpus where the llm's sentiment classification demonstrated a higher level of precision, nuanced understanding, or consistency compared to manual annotations. this facet of analysis involved a detailed review of the llm responses. scenarios in which llms provided superior sentiment analysis were highlighted to uncover the potential advantages of integrating llms in areas requiring high levels of accuracy and objectivity in sentiment classification. ## results/discussion the results given in this section represent sentiment analysis on the serbian novels dataset by using responses generated by the mistral model in both scenarios, i.e. zero-shot and few-shot learning. the accuracy values (acc.) depicted in the evaluation of the zero-shot templates reveals a varied range of accuracy, where the first template exhibited the highest result, suggesting that straightforward and direct prompts are most effective in eliciting accurate sentiment analysis from this model for serbian sentiment. figure 3 presents the confusion matrix for the first zero-shot template. in appendix a is presented a confusion matrix for the rest zero-shot templates. the first and fourth templates were most effective in identifying positive and negative sentiments. however, they struggled with objective sentences, showing a high rate of mislabeling. interestingly, the first template, despite its higher accuracy in sentiment classification, also exhibited a higher number of errors where the llm responses could not be classified into any of the categories. the fourth template utilized all-caps to emphasize key instructions, and also performed well, indicating that clarity in instruction plays a crucial role. the second zero-shot template, which attempted a more complex chain-of-thought analysis, resulted in the lowest accuracy, highlighting the limitations of the model's processing capacity in its current configuration. the third zero-shot template achieved the best accuracy in classifying objective sentiments. nevertheless, it performed poorly with negative sentiments and was not very effective for positive sentiments. it is designed with a role-play scenario involving old literary serbian, and showed moderate success, reflecting the added difficulty of interpreting historical and stylistic language nuances. the first zero-shot template recorded the highest accuracy but also the most unclassified responses, marked as errors at 47. in it is important to note that templates 2 and 3 tended to detect sarcasm where it was not recognized by ## conclusion and future work in this study, the simplification necessitated by using a quantified model with a limited context window appeared to strip away many of the benefits typically associated with the mistral model. despite its notable speed, the diminished performance suggests that such an approach may not be viable, particularly for less commonly studied languages like serbian. it is important to mention that zero-shot prompts were not run on the quantified model in our study. therefore, it remains unclear whether the quantification itself degrades performance for less commonly trained parts of the model (such as serbian language processing), or if the limitations imposed by the reduced context window, especially when combined with the addition of examples, render the model unsuitable for this type of text. one potential method to further investigate these findings would be to run zero-shot prompts on quantified models. however, the value of such research remains uncertain. the findings of this study demonstrate that in literary texts of old serbian novels, the zero-shot approach exhibits superior performance, particularly in the case of the simplest prompt, thereby leaving room for further exploration in this direction. using all caps to highlight the part of instructions has proven useful in the elimination of unusable responses, but instructing llm to detect sarcasm resulted in overdetection. additionally, comparing this approach with fine-tuned xlm-r models will represent one of the future objectives. proceedings of clib 2024 1. pročitajte i razumite dati tekst. 2. identifikujte ključne reči ili fraze u tekstu koje ukazuju na sentiment. posebnu pažnju obratite na pridjeve, priloge i bilo koje specifične glagole koji obično nose emotivnu težinu. 3. razmotrite ukupni kontekst poruke. ponekad, sentiment nije u vezi sa prisustvom specifičnih reči, već kako su te reči upotrebljene zajedno u rečenicama. 4. odredite da li tekst primarno izražava pozitivna osećanja (kao što su sreća, zadovoljstvo ili nada), negativna osećanja (kao što su tuga, ljutnja ili frustracija), ili je primarno činjeničan ili neutralan, bez ikakvog emotivnog sadržaja. tekst: {text} sentiment teksta je english translation of zero-shot second (using chain of though) template in serbian: as an expert in sentiment analysis, analyze the following text in serbian and determine its sentiment. the sentiment should be strictly classified as "positive", "negative", or "objective". no other response will be accepted. 1. read and understand the given text.