Scaling Sustainable Development Goal Predictions across Languages: From E nglish to F innish (2024)

Abstract: In this paper, we leverage an exclusive English dataset to train diverse multilingual classifiers, investigating their efficacy in adapting to Finnish data. We employ an exclusively English classification dataset of UN Sustainable Development Goals (SDG) in an education context, to train various multilingual classifiers and examine how well these models can adapt to recognizing the same classes within Finnish university course descriptions. It’s worth noting that Finnish, with a mere 5 million native speakers, presents a significantly less-resourced linguistic context compared to English. The best performing model in our experiments was mBART with an F1-score of 0.843.

## introduction the list of 17 sustainable development goals (sdgs) established by the united nations (un) has gained significance in assessing the societal, humanitarian, and environmental impact of companies in eu. this is particularly relevant for large companies compelled to include robust sustainability reporting in their annual reporting to authorities 1 . as a response to this growing importance, numerous universities and educational institutions have incorporated the un sdgs into their academic curricula. this development prompts a crucial inquiry into how educational institutions, at a higher administrative level, can ascertain which specific sdgs are being integrated into different degree programs (see @xcite . to address this concern, adopting smaller local models -specifically distil-mbert, mbert, mbart, and xlm-roberta-tailored to the content of course descriptions proves beneficial for 1 https://finance.ec.europa.eu/capital-marketsunion-and-financial-markets/company-reporting-andauditing/company-reporting/corporate-sustainability-reporting_en universities. such an approach facilitates compliance with the general data protection regulation (gdpr) foot_0 , ensuring the preservation of data privacy more than reliance on commercial large language models. the selection of the models stems from a need to accommodate linguistic diversity and the intricacies of academic content, ensuring accurate sdg classification while respecting data confidentiality. each model brings distinct advantages: distil-mbert and mbert for their efficiency and language coverage, and mbart and xlm-roberta for their superior cross-lingual and contextual understanding capabilities, making them well-suited for analyzing finnish and english course descriptions in the context of sdgs. in our work, we experiment with the viability of producing sdg classification data automatically using a large language model (llm) in english. we use this english only data to train several multilingual classifiers and study the scalability of these models to finnish data. finnish, with only 5 million native speakers, is considerably less resourced than english. this is not to say that finnish would be particularly under-studied in the context of nlp, given the vast amount of different nlp applications available for the language (see hämäläinen and alnajjar 2021). ## related work the examination of sustainable development within the realm of nlp has been approached from various perspectives, including investigations into fairness in nlp @xcite , exploration of poverty and societal sustainability through interviews @xcite , analysis of argumentation mining @xcite , and community profiling @xcite , among other aspects. our approach distinguishes itself by striving to encompass all un sustainable devel-opment goals and implementing them within an educational framework. the effect of multilingual models has been studied before in several contexts such as sentiment analysis @xcite and persuasion detection @xcite using parallel data in several languages. the findings suggest that translation strategies have a huge impact on the performance of the models. regarding the educational aspect of our study, there exists ample prior research on integrating sustainable development goals (sdgs) into teaching methods @xcite @xcite . however, this previous research is non-computational, and as far as we are aware, there is no existing computational research on this subject from the standpoint of natural language processing (nlp). ## sdg data our dataset, containing 5988 entries from metropolia university, was accessed via their api foot_1 , a resource available to staff and researchers upon formal request and approval by the university's it department. in acquiring and handling this data, we observed strict ethical standards, including anonymization of identifiable information and adherence to the university's data use policies, ensuring the preservation of data confidentiality. this dataset, spanning from 2010 to 2023, encompasses a diverse range of courses across various departments and majors, offered in both finnish and english. the time frame was chosen to provide a comprehensive collection of course materials, aiding in the robustness of our model training. after the initial data collection, we utilized the vertex ai api foot_2 to conduct batch processing, a pivotal step in annotating each course description with the corresponding sustainable development goals (sdgs). the api facilitated the automation of this task by allowing us to process large volumes of text data and generate labels that indicate the relevance of specific sdgs to the course content. our study focuses on the following sdgs: 3 (good health and well-being), 7 (affordable and clean energy), 8 (decent work and economic growth), 9 (industry, innovation, and infrastructure), and 10 (reduced inequalities), selected for their significant relevance to the university curriculum, as other goals were less represented, leading to potential data imbalance. the data is formatted as json objects for multilabel classification with multilingual models; one typical entry would look as follows: "input": "heat distribution systems in buildings, the student learns: the building's heating power demand and its calculation. different heat distribution methods and devices. ways of adjusting the heating system. water-circulating radiator and floor heating system. dimensioning of pipework and selection of radiator, circulation pump, expansion vessel and safety devices. the student can calculate the heating power demand of the building, can dimension the pipework, radiators, circulation pump and expansion vessel and safety devices.", "labels": [0, 1, 0, 0, 0], where the input text is related to goal 7 affordable and clean energy. therefore, the entries are structured as "input"a detailed course description and "labels" -binary encoding indicating the course's relevance to the selected sdgs. figure 2 displays the distribution of the sustainable development goal (sdg) mentioned in the training dataset. ## cross-lingual models for sdg prediction in this study, we utilized four advanced multilingual models: distil-mbert @xcite , efficient with significant language understanding nuance understanding may be limited mbert comprehensive language comprehension finnish-specific tuning may be less mbart strong in deep contextual understanding high computational requirements xlm-roberta excellent in cross-lingual tasks possible compromise in finnish depth our dataset, split into 70% training, 15% validation, and 15% testing, primarily consisted of english for training and validation, with finnish reserved for testing. this approach was intended to test the models' transfer learning capabilities from english to finnish. each model underwent fine-tuning for multilabel sdg classification using a pytorch-based @xcite framework. key steps in the training process included tokenization, encoding, and employing a bcewithlogitsloss function, as shown in equation ( 1 ). this study not only demonstrates the effectiveness of these models in a multilingual context but also sheds light on the scaling behavior of llms, particularly in adapting from high-resource to low-resource languages. the findings provide valuable insights into the adaptability of multilingual models, with a special focus on finnish, illustrating the broader applicability of these models in diverse linguistic settings. the exploration of these multilingual models in predicting sdgs in finnish highlights significant insights into the scaling behavior and adaptability of llms. our strategic data split, along with the tailored training and architecture of each model, demonstrates our approach in tackling the challenges of language representation within nlp. this methodology is particularly pertinent in understanding how llms perform across languages with varying resource levels. ## results and evaluation in addition, figure 3 depicts the f1 scores, which represent the harmonic mean of precision and recall for each sdg. our in-depth analysis of the classifiers' performance on a per-label (per sdg) basis provides subtle insights into their prediction capability and limitations. these per-label findings highlight the importance of model architecture and training corpus diversity in addressing the unique linguistic issues given by each sdg. the variations in performance across objectives indicate that, while certain sdgs are well-represented and easier to forecast with current nlp models, others require additional research and targeted data enrichment to improve model performance. ## conclusions this study takes a novel approach to assessing the integration of the united nations sustainable development goals (sdgs) into university curricula. we investigated the effectiveness of multilingual classifiers in adapting to finnish data, a language with significantly fewer resources in the field of natural language processing (nlp). our study sought to ascertain the scalability of these models in recognizing sdg-related content within finnish university course descriptions. the findings show that multilingual models can bridge the language gap effectively, with notable success in identifying sdg-related content across multiple languages. the performance varied across sdgs, with some models excelling in some areas while struggling in others. this variation emphasizes the significance of model selection based on the target language's specific characteristics and linguistic nuances. our work, which aligns with workshop talks on scaling behavior across linguistic settings, advances the nlp community's understanding of multilingual model applicability in resolving language resource inequities by utilizing finnish as a case study. the study opens the door to further investigation into the use of multilingual models for other low-resource languages. in conclusion, the study successfully demonstrates the feasibility of using english-trained multilingual models to process and analyze data in finnish, a language with limited resources. this method not only provides a useful tool for educational institutions' sustainability reporting, but it also contributes to a better understanding of multilingual nlp applications.