WHOS e H eritage: C lassification of UNESCO W orld H eritage Statements of " O utstanding U niversal V alue” with Soft Labels (2021)

Abstract: The UNESCO World Heritage List (WHL) includes the exceptionally valuable cultural and natural heritage to be preserved for mankind. Evaluating and justifying the Outstanding Universal Value (OUV) is essential for each site inscribed in the WHL, and yet a complex task, even for experts, since the selection criteria of OUV are not mutually exclusive. Furthermore, manual annotation of heritage values and attributes from multi-source textual data, which is currently dominant in heritage studies, is knowledge-demanding and time-consuming, impeding systematic analysis of such authoritative documents in terms of their implications on heritage management. This study applies state-of-the-art NLP models to build a classifier on a new dataset containing Statements of OUV, seeking an explainable and scalable automation tool to facilitate the nomination, evaluation, research, and monitoring processes of World Heritage sites. Label smoothing is innovatively adapted to improve the model performance by adding prior inter-class relationship knowledge to generate soft labels. The study shows that the best models fine-tuned from BERT and ULMFiT can reach 94.3% top-3 accuracy. A human study with expert evaluation on the model prediction shows that the models are sufficiently generalizable. The study is promising to be further developed and applied in heritage research and practice.

## introduction since the world heritage convention was adopted in 1972, 1121 sites has been inscribed worldwide in the world heritage list (whl) up to 2019, aiming at a collective protection of the cultural and natural heritage of outstanding universal value (ouv) for mankind as a whole @xcite @xcite . first proposed in 1976, ouv, meaning the "cultural and/or natural significance which is so exceptional as to transcend national boundaries and to be of common importance for present and future generations of all humanity", has been operationalized and formalized into an administrative requirement for new inscriptions on the whl since 2005. @xcite @xcite . all nominations must meet one or more of the ten selection criteria (6 for culture and 4 for nature), focusing on different cultural and natural values. since 2007, complete statements of ouv (souv) need to be submitted and approved for new world heritage (wh) nominations, which should include, among others, a section of "justification for criteria", giving a short paragraph to explain why a site (also known as property) satisfies each of the criteria it is inscribed under. these statements are to be drafted by the state parties after scientific research for any tentative nominations, further reviewed and revised by the advisory bodies from icomos and/or iucn, and eventually approved and adopted by the world heritage committee for inscription. similarly, retrospective souv have been required for sites inscribed before 2006 to revise or refill the section justification of criteria @xcite . however, the evaluation of souv can be ambiguous in the sense that: 1) the selection criteria are not mutually exclusive and contain common information about historical and aesthetic/artistic values as an integral part @xcite ; 2) the key stakeholders to evaluate the souv for a nomination occasionally disagree with each other at early stages, leading to recursive reviews and revisions, though all are considered to be domain experts @xcite @xcite . a tool to check the accuracy, objectivity, consistency, and coherence of such statements can significantly benefit the inscription process involving thousands of experts worldwide each year. not only for new nominations, the souv are also essential reference points for monitoring and interpreting inscribed heritage sites @xcite . researchers and practitioners actively and regularly check if the justified criteria are still relevant for the sites, as to decide on further planning and managerial actions. moreover, these same statements are also used in support of legal court cases, should wh sites be endangered by human development @xcite . under the support of the recommendation of historic urban landscape and the recent our world heritage campaign, multiple data sources (e.g., news articles, policy documents, social media posts) are encouraged in such analyses of identifying and mapping ouv @xcite @xcite . the traditional method of manually annotating heritage values and attributes by experts can be time-consuming and knowledge-demanding for analysing massive social media posts by people in cities with wh sites to find ouv-related statements, albeit dominantly applied in practice @xcite @xcite . to approximate both ultimate goals of this study: 1) aiding the inscription process by checking the coherence and consistency of souv, and 2) identifying heritage values from multiple data sources (e.g., social media posts), a computational solution rooted on souv is desired. by training nlp models with the officially written and approved souv, a machine replica of the collective authoritarian view could be obtained. this machine replica will not be employed at this stage to justify ouv for new nominations from scratch. rather, it will assess the written souv of wh sites (either existing or new) and classify ouv-related texts with the learned collective authoritarian view. furthermore, it can investigate the existing souv from bottom up and capture the subtle intrinsic associations within the statements and among the corresponding selection criteria @xcite . this yields a new perspective on interpreting the whl, which would give insights for furthering amending the concept of ouv and selection criteria to be better discernible. therefore, this study aims at training an explainable and scalable classifier that can reveal the intrinsic associations of world heritage ouv selection criteria, which can be feasible to apply in realworld analyses by researchers and practitioners. as outcome, this paper presents the classifier of un-esco world heritage statements of ouv with soft labels (whose heritage). the contributions of this paper can be summarized as follows: 1) a novel text classification dataset is presented, concerning a domain-specific task about outstanding universal value for un-esco world heritage sites; 2) innovative variants of label smoothing are applied to introduce the prior knowledge of label association into training as soft labels, which turned out effective to improve performance in most investigated popular models as baselines in this task; 3) several classifiers are trained and compared on the statements of ouv classification task as initial benchmarks, supplemented with explorations on their explainability and generalizability using expert evaluation. ## related work text classification in the past decades, numerous models have been proposed from shallow to deep learning models for text classification tasks. in shallow learning models, the raw input text is pre-processed to extract features of the text, which are then fed into machine learning classifiers, e.g., naive bayes @xcite and support vector machine @xcite for prediction. in deep learning models, deep neural networks are leveraged to extract information from the input data, such as convolutional neural networks (cnn) @xcite , recurrent neural networks (rnn) @xcite , attention networks @xcite and transformers @xcite . multi-class and multi-label tasks are two extensions of the simplest binary classification, where every sample can belong to one or more classes within a class list @xcite , where the labels may also be correlated @xcite . this work explores the combined application of some popular shallow and deep learning models for a multi-class classification task. label smoothing label smoothing (ls) is originally proposed as a regularization technique to alleviate overfitting in training deep neural networks @xcite . it assigns a noise distribution on all the labels to prevent the model from predicting too confidently on 'ground-truth' labels. it is widely used in computer vision @xcite , speech @xcite and natural language processing @xcite tasks. originally the distribution is uniform across the labels, which is data independent. recently, other variants of ls are also proposed that are able to incorporate the interrelation information from the data into the distribution @xcite @xcite . in this work, the technique is applied to generate soft labels with a distribution derived from domain knowledge since the classes in this task are clearly interrelated with each other. transfer learning in nlp in many real-world applications, labelled data are limited and expensive to collect. training models with limited data from scratch affects the performance. transfer learning @xcite is widely used to solve this by using word embeddings that are pretrained on massive corpus and fine-tuning them on target task. earlier works @xcite provide static word embeddings that ignore the contextual information in the sentences. more recent works, e.g., ulm-fit @xcite and bert @xcite , take the context into account and generate dynamic contextualized word vectors, showing excellent performance, which also prove to be sufficiently generalizable across many tasks. this task, with a relatively small data size, employs the idea of transfer learning and applies both embedding methods. ## discussion and conclusions this paper presents a new text classification benchmark from a real-world problem about unesco world heritage statements of outstanding universal value (ouv). the problem is essentially a multi-class single label classification task, while the classes are not necessarily mutually exclusive. the prior knowledge of the class association is added to the training process as soft labels through novel variants of label smoothing (ls). the study shows that introducing ls improved the performance on most baselines, reaching a top-3 accuracy of 94.3%. the models also performed reasonably well in an independent test dataset and received positive outcomes in a human study with domain experts, suggesting that the classifiers have the potential to be further developed and applied in the world heritage research and practice. ls was not tuned together with other hyperparameters during the training. yet, it still showed an improvement in most baselines. however, the complex effect of ls on different baselines needs more investigation. the top-1 accuracy is limited even on the best models, which is not uncommon in the literature for non-binary multi-class classification when the labels are not sufficiently distinct @xcite . applying data augmentation and training supplemental binary classifiers may improve the performance on difficult classes. the choice of replacing all numbers @xmath0, which invites more investigations. moreover, more studies on the generalizability and reliability of the models on data from different distributions (e.g., from policy documents or news article) are needed before further application. this work would support a series of follow-up studies respectively exploring the intrinsic associations of ouv based on the models' behaviour @xcite , application of the proposed methods in social media mining in venice @xcite , and generalizability in case studies worldwide. this work is intended to aid, but not replace the workload of human stakeholders: for state parties to identify ouv-related statements through documentation, for advisory bodies and whc to review and revise the yearly nomination proposals, for researchers to investigate massive official discourse and user-generated content, and for the public to visually understand the values of their world heritage around them. therefore, this work whose heritage can be another milestone for the digital transformation of world heritage studies, aiming at a more socially inclusive future practice.