---
language:
- es
- fr
- it
- cs
- pt
- en
- ar
- fi
- de
- ru
- hu
- zh
license: cc-by-4.0
tags:
- multilingual
- bert
- poetry
- digital humanities
pipeline_tag: fill-mask
widget:
- text: ¿Qué es la vida? Un [MASK].
datasets:
- linhd-postdata/pulpo
metrics:
- accuracy
library_name: transformers
---

# ALBERTI

ALBERTI is a set of two BERT-based multilingual model for poetry. One for verses and another one for stanzas. This model has been further trained with the PULPO corpus for verses using [Flax](https://github.com/google/flax), including training scripts.

See https://arxiv.org/abs/2307.01387.

This is part of the
[Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organised by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google.

## PULPO

PULPO, the Prolific Unannotated Literary Poetry Corpus, is a set of multilingual corpora of verses and stanzas with over 72M words.

The following corpora has been downloaded using the [Averell](https://github.com/linhd-postdata/averell/) tool, developed by the [POSTDATA](https://postdata.linhd.uned.es/) team:

### Spanish
- [Disco v3](https://github.com/pruizf/disco)
- [Corpus of Spanish Golden-Age Sonnets](https://github.com/bncolorado/CorpusSonetosSigloDeOro)
- [Corpus general de poesía lírica castellana del Siglo de Oro](https://github.com/bncolorado/CorpusGeneralPoesiaLiricaCastellanaDelSigloDeOro)
- [Gongocorpus](https://github.com/linhd-postdata/gongocorpus) - [source](http://obvil.sorbonne-universite.site/corpus/gongora/gongora_obra-poetica)
### English
- [Eighteenth-Century Poetry Archive (ECPA)](https://github.com/alhuber1502/ECPA)
- [For better for verse](https://github.com/waynegraham/for_better_for_verse)
### French
- [Métrique en Ligne](https://crisco2.unicaen.fr/verlaine/index.php?navigation=accueil) - [source](https://github.com/linhd-postdata/metrique-en-ligne)
### Italian
- [Biblioteca italiana](https://github.com/linhd-postdata/biblioteca_italiana) - [source](http://www.bibliotecaitaliana.it/)
### Czech
- [Corpus of Czech Verse](https://github.com/versotym/corpusCzechVerse)
### Portuguese
- [Stichotheque](https://gitlab.com/stichotheque/stichotheque-pt)

Also, we obtained the following corpora from these sources:
### Spanish 
- [Poesi.as](https://github.com/linhd-postdata/poesi.as) - [source](http://www.poesi.as/)
### English
- [A Gutenberg Poetry Corpus](https://github.com/aparrish/gutenberg-poetry-corpus)
### Arabic
- [Arabic Poetry dataset](https://www.kaggle.com/ahmedabelal/arabic-poetry)
### Chinese
- [THU Chinese Classical Poetry Corpus](https://github.com/THUNLP-AIPoet/Datasets/tree/master/CCPC)
### Finnish
- [SKVR](https://github.com/sks190/SKVR)
### German
- [TextGrid Poetry Corpus](https://github.com/linhd-postdata/textgrid-poetry) - [source](https://textgrid.de/en/digitale-bibliothek)
- [German Rhyme Corpus](https://github.com/tnhaider/german-rhyme-corpus)
### Hungarian
- [verskorpusz](https://github.com/ELTE-DH/verskorpusz)
### Portuguese
- [Poems in Portuguese](https://www.kaggle.com/oliveirasp6/poems-in-portuguese)
### Russian
- [19 000 Russian poems](https://www.kaggle.com/grafstor/19-000-russian-poems)

## Team members

- Álvaro Pérez ([alvp](https://huggingface.co/alvp))
- Javier de la Rosa ([versae](https://huggingface.co/versae))
- Aitor Díaz ([aitordiaz](https://huggingface.co/aitordiaz))
- Elena González-Blanco
- Salvador Ros ([salva](https://huggingface.co/salva))

## Useful links

- [Community Week timeline](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104#summary-timeline-calendar-6)
- [Community Week README](https://github.com/huggingface/transformers/blob/master/examples/research_projects/jax-projects/README.md)
- [Community Week thread](https://discuss.huggingface.co/t/bertin-pretrain-roberta-large-from-scratch-in-spanish/7125)
- [Community Week channel](https://discord.com/channels/858019234139602994/859113060068229190)
- [Masked Language Modelling example scripts](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)
- [Model Repository](https://huggingface.co/flax-community/alberti-bert-base-multilingual-cased/)

## Acknowledgments

This project would not have been possible without the infrastructure and resources provided by HuggingFace and Google Cloud. Moreover, we want to thank POSTDATA Project (ERC-StG-679528) and the Computational Literary Studies Infrastructure (CLS INFRA No. 101004984) of the European Union's Horizon 2020 research and innovation programme for their support and time allowance.