AI 4 C ulture: Towards Multilingual Access for Cultural Heritage Data (2024)

Abstract: The AI4Culture project (2023-2025), funded by the European Commission, and involving a 12-partner consortium led by the National Technical University of Athens, develops a platform serving as an online capacity building hub for AI technologies in the cultural heritage (CH) sector, enabling multilingual access to CH data. It offers access to AI-related resources, including openly labelled datasets for model training and testing, deployable and reusable tools, and capacity building materials. The tools are aimed at optical character recognition (OCR) for printed and handwritten documents, subtitle generation and validation, machine translation (MT), and metadata enrichment via image information extraction and semantic linking. The project also customises these tools to enhance interface and component usability. We illustrate this with technology that corrects OCR output using language models and adapts it for MT.

## introduction the ai4culture project aims to develop an online capacity building hub for ai technologies in the cultural heritage (ch) sector. this innovative platform seeks to make ch data more accessible and understandable in today's multilingual digital era, by facilitating data sharing, promoting cultural content reuse and linking the vast european data space (which ensures data availability for eco-© 2024 the authors. this article is licensed under a creative commons 4.0 licence, no derivative works, attribution, cc-by-nd. nomic, societal and research use) with ch institutions. the project is funded by the european commission (ec) and runs from april 2023 to march 2025. the consortium is led by the ails laboratory of the national technical university of athens (ntua). the other consortium partners are the ntua spin-off datoptron and organisations specialised in digital ch (i.e. europeana foundation, european fashion heritage association, the digitglam unit at university of leuven, and the company datable), natural language processing (the companies crosslang and pangeanic, the mt research unit at fondazione bruno kessler (fbk), the digital safety and security center of the austrian institute of technology), online translation services (the company translated), and media culture (institute for sound and vision). the ai4culture platform, expected to launch its first version mid-2024, will offer access to airelated resources, i.e. to openly labelled datasets for training and testing models, to deployable and reusable tools, and to capacity building materials on the use of these tools and datasets for training, testing, and evaluation. the platform targets ch students and professionals, data providers, researchers and ai model developers, amongst others. towards the end of 2024, several workshops will be organised on the technologies involved. ## technology and language coverage the tools made accessible through the platform relate to four technologies: accessibility of the above technologies will be achieved through online interfaces, application programming interfaces (apis), which allow to send requests to various services, and docker images, which can be locally deployed. the interfaces include transcribathon 1 (supporting the transcription of documents and the translation of transcriptions), subbit! 2 (supporting the editing of automatically generated subtitles), the interface of sage 3 (serving semantic annotation and generation of enrichments), and pecat 4 (supporting validation and post-editing of automatic translations). the software built during the project to achieve accessibility will be provided as open source. moreover, this software focuses on the reuse of existing open-source tools, such as pero-ocr. @xcite on the multilingual level, the transcription and subtitling services, as well the ch metadata, cover numerous languages, with a focus on eu official languages. the translation functionality uses various systems, including the europeana translate 6 and the ec's etranslation engines. ## customisation of technologies the tools made accessible on the platform are customised to increase the usability of interfaces and components. for instance, fbk's subtitling components incorporate the whisper 7 pre-trained model for improving speech recognition. another example is the post-correction component developed by crosslang, which aims at enhancing both 1 transcribathon.eu 2 subbit.eu 3 pro.europeana.eu/page/sage 4 pangeanic.com/datasets-for-ai/ ai-data-annotation-platform 5 github.com/dcgm/pero-ocr 6 pro.europeana.eu/project/ europeana-translate 7 github.com/openai/whisper ocr and mt outputs of auto-generated transcriptions. this component particularly targets documents for which no (closely) matching specialised transcription engine exists in terms of language, time period, script of writing, etc. the ocr post-correction component consist of several steps. first, it removes word-splitting hyphens at line breaks using a language model-based technique. next, it segments the transcribed lines into sentences using an advanced rule-based approach. finally, it performs word post-correction through either a basic method using lexicon and language model lookup, or through a computationally more demanding strategy that prompts a chatbot based on a large language model (llm) to make corrections to the transcriptions. figure 1 shows a part of a dutch letter from world war ii. @xcite