The objective of this task is to correctly classify a given term in the financial domain according to its prototypical hypernym in a list of available hypernyms:


* Bonds
* Forward
* Funds
* Future
* MMIs (Money Market Instruments)
* Option
* Stocks
* Swap
* Equity Index
* Credit Index
* Securities restrictions
* Parametric schedules
* Debt pricing and yields
* Credit Events
* Stock Corporation
* Central Securities Depository
* Regulatory Agency


This kind-based approach relies on identifying the closest hypernym semantically to the given term (even if they possess common properties with other hypernyms).


#### Data Description


The data is a scraped list of term definitions from the FIBO ontology website where each definition has been mapped to its closest hypernym from the proposed labels.
For multi-sentence definitions, we applied sentence-splitting by punctuation delimiters. We also applied lowercase transformation on all input data.


#### Data Instances


The dataset contains a label representing the hypernym of the given definition.


```json
{
  'label': 'bonds',
  'definition': 'callable convertible bond is a kind of callable bond, convertible bond.'
}
```


#### Data Fields


**label**: Can be one of the 17 predefined hypernyms.


**definition**: Financial term definition relating to a concept or object in the financial domain.


#### Data Splits


The data contains training data with **317101** entries.


#### Test set metrics


The representational learning model is evaluated on a representative test set with 20% of the entries. The test set is scored based on the following metrics:


* Average Accuracy
* Mean Rank (position of the correct label in a set of 5 model predictions)


We evaluate FinISH according to these metrics, where it outperforms other state-of-the-art sentence embeddings methods in this task.


* Average Accuracy: **0.73**
* Mean Rank: **1.61**


## Usage (Sentence-Transformers)


Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:


```
git clone https://github.com/huggingface/transformers.git
pip install -q ./transformers
pip install -U sentence-transformers
```


Then you can use the model like this:


```python
from sentence_transformers import SentenceTransformer, util
import torch
model = SentenceTransformer('yseop/roberta-base-finance-hypernym-identification')
# Our corpus containing the list of hypernym labels
hypernyms = ['Bonds',
\t\t\t'Forward',
\t\t\t'Funds',
\t\t\t'Future',
\t\t\t'MMIs',
\t\t\t'Option',
\t\t\t'Stocks',
\t\t\t'Swap',
\t\t\t'Equity Index',
\t\t\t'Credit Index',
\t\t\t'Securities restrictions',
\t\t\t'Parametric schedules',
\t\t\t'Debt pricing and yields',
\t\t\t'Credit Events',
\t\t\t'Stock Corporation',
\t\t\t'Central Securities Depository',
\t\t\t'Regulatory Agency']
hypernym_embeddings = model.encode(hypernyms, convert_to_tensor=True)
# Query sentences are financial terms to match to the predefined labels
queries = ['Convertible bond', 'weighted average coupon', 'Restriction 144-A']
# Find the closest 5 hypernyms of the corpus for each query sentence based on cosine similarity
top_k = min(5, len(hypernyms))
for query in queries:
    query_embedding = model.encode(query, convert_to_tensor=True)
    # We use cosine-similarity and torch.topk to find the highest 5 scores
    cos_scores = util.pytorch_cos_sim(query_embedding, hypernym_embeddings)[0]
    top_results = torch.topk(cos_scores, k=top_k)
    print("\
\
======================\
\
")
    print("Query:", query)
    print("\
Top 5 most similar hypernyms:")
    for score, idx in zip(top_results[0], top_results[1]):
        print(hypernyms[idx], "(Score: {:.4f})".format(score))
```


## Usage (HuggingFace Transformers)


Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.


```python
from transformers import AutoTokenizer, AutoModel
import torch
#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
# Query sentences are financial terms to match to the predefined labels
queries = ['Convertible bond', 'weighted average coupon', 'Restriction 144-A']
# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('yseop/roberta-base-finance-hypernym-identification')
model = AutoModel.from_pretrained('yseop/roberta-base-finance-hypernym-identification')
# Tokenize sentences
encoded_input = tokenizer(queries, padding=True, truncation=True, return_tensors='pt')
# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)
# Perform pooling
query_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
print("Query embeddings:")
print(query_embeddings)
```


**Created by:** [Yseop](https://www.yseop.com/) | Pioneer in Natural Language Generation (NLG) technology. Scaling human expertise through Natural Language Generation.


model name: roberta-base-finance-hypernym-identification


model author: yseop


task: sentence-similarity


tags: pytorch, roberta, feature-extraction, arxiv:1908.10084, sentence-transformers, sentence-similarity, transformers, infinity_compatible


# T5-base data to text model specialized for Finance NLG


**complete version**


#### Call the model


```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("yseop/FNP_T5_D2T_complete")

model = AutoModelForSeq2SeqLM.from_pretrained("yseop/FNP_T5_D2T_complete")

text = ["Group profit | valIs | € 115.7 million && € 115.7 million | dTime | in 2019"]

```


#### Choose a generation method


```python

input_ids = tokenizer.encode(": {}".format(text), return_tensors="pt")
p = 0.82
k = 90

outputs = model.generate(input_ids,
                         do_sample=True,
                        top_p=p,
                        top_k=k,
                        early_stopping=True)

print(tokenizer.decode(outputs[0]))

```


```python

input_ids = tokenizer.encode(": {}".format(text), return_tensors="pt")

outputs = model.generate(input_ids,
                         max_length=200,
                         num_beams=2, repetition_penalty=2.5,
                         top_k=50, top_p=0.98,
                         length_penalty=1.0,
                         early_stopping=True)

print(tokenizer.decode(outputs[0]))

```


model name: FNP_T5_D2T_complete


task: text2text-generation


tags: pytorch, t5, text2text-generation, transformers, autonlp_compatible


**simple version**


This model was trained on a limited number of indicators, values and dates


## Usage (HuggingFace Transformers)


```python

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("yseop/FNP_T5_D2T_simple")

model = AutoModelForSeq2SeqLM.from_pretrained("yseop/FNP_T5_D2T_simple")

text = ["Group profit | valIs | $ 10 && € $10  | dTime | in 2019"]

```


```python

input_ids = tokenizer.encode(": {}".format(text), return_tensors="pt")
p=0.72
k=40

outputs = model.generate(input_ids,
                         do_sample=True,
                        top_p=p,
                        top_k=k,
                        early_stopping=True)

print(tokenizer.decode(outputs[0]))

```


model name: FNP_T5_D2T_simple


model author: yseop


inference: true
pipeline_tag: text-classification
tags:


- feature-extraction
- text-classification
library: pytorch


FReE (Financial Relation Extraction)


We present FReE, a [DistilBERT](https://huggingface.co/distilbert-base-uncased) base model fine-tuned on a custom financial dataset for financial relation type detection and classification.


## Process


Detecting the presence of a relationship between financial terms and qualifying the relationship in case of its presence. Example use cases:


* An A-B trust is a joint trust created by a married couple for the purpose of minimizing estate taxes. (Relationship **exists**, type: **is**)
* There are no withdrawal penalties. (Relationship **does not exist**, type: **x**)


## Data


The data consists of financial definitions collected from different sources (Wikimedia, IFRS, Investopedia) for financial indicators. Each definition has been split up into sentences, and term relationships in a sentence have been extracted using the [Stanford Open Information Extraction](https://nlp.stanford.edu/software/openie.html) module.
A typical row in the dataset consists of a definition sentence and its corresponding relationship label.
The labels were restricted to the 5 most-widely identified relationships, namely: **x** (no relationship), **has**, **is in**, **is** and **are**.


## Model


The model used is a standard DistilBERT-base transformer model from the Hugging Face library. See [HUGGING FACE DistilBERT base model](https://huggingface.co/distilbert-base-uncased) for more details about the model.
In addition, the model has been pretrained to initializa weigths that would otherwise be unused if loaded from an existing pretrained stock model.


## Metrics


The evaluation metrics used are: Precision, Recall and F1-score. The following is the classification report on the test set.


| relation  | precision        | recall           | f1-score  | support  |
| ------------- |:-------------:|:-------------:|:-------------:| -----:|
| has | 0.7416      | 0.9674 | 0.8396 | 2362 |
| is in | 0.7813      | 0.7925 | 0.7869 | 2362 |
| is | 0.8650     | 0.6863 | 0.7653 | 2362 |
| are | 0.8365     | 0.8493 | 0.8429 | 2362 |
| x | 0.9515     | 0.8302 | 0.8867 | 2362 |
|  |      |  |  |  |
| macro avg | 0.8352     | 0.8251 | 0.8243 | 11810 |
| weighted avg | 0.8352     | 0.8251 | 0.8243 | 11810 |


model name: distilbert-base-financial-relation-extraction


task: text-classification


tags: pytorch, feature-extraction, text-classification


## license: apache-2.0


model name: bertin-xls-r-1b-spanish


model author: bertin-project


tags: license:apache-2.0


language: es
license: cc-by-4.0
tags:


- spanish
- roberta
pipeline_tag: fill-mask
widget:
- text: Fui a la librería a comprar un <mask>.


This is a **RoBERTa-base** model trained from scratch in Spanish.


The training dataset is [mc4](https://huggingface.co/datasets/bertin-project/mc4-es-sampled) subsampling documents to a total of about 50 million examples. Sampling is biased towards average perplexity values (using a Gaussian function), discarding more often documents with very large values (poor quality) of very small values (short, repetitive texts).


This model has been trained for 250.000 steps.


Please see our main [card](https://huggingface.co/bertin-project/bertin-roberta-base-spanish) for more information.


This is part of the
[Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organised by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google.


## Team members


- Eduardo González ([edugp](https://huggingface.co/edugp))
- Javier de la Rosa ([versae](https://huggingface.co/versae))
- Manu Romero ([mrm8488](https://huggingface.co/))
- María Grandury ([mariagrandury](https://huggingface.co/))
- Pablo González de Prado ([Pablogps](https://huggingface.co/Pablogps))
- Paulo Villegas ([paulo](https://huggingface.co/paulo))


model name: bertin-base-gaussian


task: fill-mask


tags: pytorch, jax, tensorboard, joblib, roberta, fill-mask, es, transformers, license:cc-by-4.0, spanish, autonlp_compatible, infinity_compatible


The training dataset is [mc4](https://huggingface.co/datasets/bertin-project/mc4-es-sampled) subsampling documents to a total of about 50 million examples. Sampling is random.
This model continued training from [sequence length 128](https://huggingface.co/bertin-project/bertin-base-random) using 20.000 steps for length 512.


model name: bertin-base-random-exp-512seqlen


This model takes the one using [sequence length 128](https://huggingface.co/bertin-project/bertin-base-gaussian) and trains during 25.000 steps using sequence length 512.


model name: bertin-base-gaussian-exp-512seqlen


model author: bertin-project


language: es
license: cc-by-4.0
tags:


- [Version v1](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/v1) (default): July 26th, 2021
- [Version v1-512](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/v1-512): July 26th, 2021
- [Version beta](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/beta): July 15th, 2021


# BERTIN


BERTIN is a series of BERT-based models for Spanish. The current model hub points to the best of all RoBERTa-base models trained from scratch on the Spanish portion of mC4 using [Flax](https://github.com/google/flax). All code and scripts are included.


This is part of the
[Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organized by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google Cloud.


The aim of this project was to pre-train a RoBERTa-base model from scratch during the Flax/JAX Community Event, in which Google Cloud provided free TPUv3-8 to do the training using Huggingface's Flax implementations of their library.


# Motivation


According to [Wikipedia](https://en.wikipedia.org/wiki/List_of_languages_by_total_number_of_speakers), Spanish is the second most-spoken language in the world by native speakers (>470 million speakers), only after Chinese, and the fourth including those who speak it as a second language. However, most NLP research is still mainly available in English. Relevant contributions like BERT, XLNet or GPT2 sometimes take years to be available in Spanish and, when they do, it is often via multilingual versions which are not as performant as the English alternative.


At the time of the event there were no RoBERTa models available in Spanish. Therefore, releasing one such model was the primary goal of our project. During the Flax/JAX Community Event we released a beta version of our model, which was the first in the Spanish language. Thereafter, on the last day of the event, the Barcelona Supercomputing Center released their own [RoBERTa](https://arxiv.org/pdf/2107.07253.pdf) model. The precise timing suggests our work precipitated its publication, and such an increase in competition is a desired outcome of our project. We are grateful for their efforts to include BERTIN in their paper, as discussed further below, and recognize the value of their own contribution, which we also acknowledge in our experiments.


Models in monolingual Spanish are hard to come by and, when they do, they are often trained on proprietary datasets and with massive resources. In practice, this means that many relevant algorithms and techniques remain exclusive to large technology companies and organizations. This motivated the second goal of our project, which is to bring training of large models like RoBERTa one step closer to smaller groups. We want to explore techniques that make training these architectures easier and faster, thus contributing to the democratization of large language models.


## Spanish mC4


The dataset mC4 is a multilingual variant of the C4, the Colossal, Cleaned version of Common Crawl's web crawl corpus. While C4 was used to train the T5 text-to-text Transformer models, mC4 comprises natural text in 101 languages drawn from the public Common Crawl web-scrape and was used to train mT5, the multilingual version of T5.


The Spanish portion of mC4 (mC4-es) contains about 416 million samples and 235 billion words in approximately 1TB of uncompressed data.


```bash
$ zcat c4/multilingual/c4-es*.tfrecord*.json.gz | wc -l
416057992
```


```bash
$ zcat c4/multilingual/c4-es*.tfrecord-*.json.gz | jq -r '.text | split(" ") | length' | paste -s -d+ - | bc
235303687795
```


## Perplexity sampling


The large amount of text in mC4-es makes training a language model within the time constraints of the Flax/JAX Community Event problematic. This motivated the exploration of sampling methods, with the goal of creating a subset of the dataset that would allow for the training of well-performing models with roughly one eighth of the data (~50M samples) and at approximately half the training steps.


In order to efficiently build this subset of data, we decided to leverage a technique we call *perplexity sampling*, and whose origin can be traced to the construction of CCNet (Wenzek et al., 2020) and their high quality monolingual datasets from web-crawl data. In their work, they suggest the possibility of applying fast language models trained on high-quality data such as Wikipedia to filter out texts that deviate too much from correct expressions of a language (see Figure 1). They also released Kneser-Ney models (Ney et al., 1994) for 100 languages (Spanish included) as implemented in the KenLM library (Heafield, 2011) and trained on their respective Wikipedias.


<figure>


![Perplexity distributions by percentage CCNet corpus](./images/ccnet.png)


<caption>Figure 1. Perplexity distributions by percentage CCNet corpus.</caption>
</figure>


In this work, we tested the hypothesis that perplexity sampling might help
reduce training-data size and training times, while keeping the performance of
the final model.


## Methodology


In order to test our hypothesis, we first calculated the perplexity of each document in a random subset (roughly a quarter of the data) of mC4-es and extracted their distribution and quartiles (see Figure 2).


![Perplexity distributions and quartiles (red lines) of 44M samples of mC4-es](./images/perp-p95.png)


<caption>Figure 2. Perplexity distributions and quartiles (red lines) of 44M samples of mC4-es.</caption>
</figure>


With the extracted perplexity percentiles, we created two functions to oversample the central quartiles with the idea of biasing against samples that are either too small (short, repetitive texts) or too long (potentially poor quality) (see Figure 3).


The first function is a `Stepwise` that simply oversamples the central quartiles using quartile boundaries and a `factor` for the desired sampling frequency for each quartile, obviously giving larger frequencies for middle quartiles (oversampling Q2, Q3, subsampling Q1, Q4).
The second function weighted the perplexity distribution by a Gaussian-like function, to smooth out the sharp boundaries of the `Stepwise` function and give a better approximation to the desired underlying distribution (see Figure 4).


We adjusted the `factor` parameter of the `Stepwise` function, and the `factor` and `width` parameter of the `Gaussian` function to roughly be able to sample 50M samples from the 416M in mC4-es (see Figure 4). For comparison, we also sampled randomly mC4-es up to 50M samples as well. In terms of sizes, we went down from 1TB of data to ~200GB. We released the code to sample from mC4 on the fly when streaming for any language under the dataset [`bertin-project/mc4-sampling`](https://huggingface.co/datasets/bertin-project/mc4-sampling).


![Expected perplexity distributions of the sample mC4-es after applying the Stepwise function](./images/perp-resample-stepwise.png)


<caption>Figure 3. Expected perplexity distributions of the sample mC4-es after applying the Stepwise function.</caption>


</figure>


![Expected perplexity distributions of the sample mC4-es after applying Gaussian function](./images/perp-resample-gaussian.png)


<caption>Figure 4. Expected perplexity distributions of the sample mC4-es after applying Gaussian function.</caption>
</figure>


Figure 5 shows the actual perplexity distributions of the generated 50M subsets for each of the executed subsampling procedures. All subsets can be easily accessed for reproducibility purposes using the [`bertin-project/mc4-es-sampled`](https://huggingface.co/datasets/bertin-project/mc4-es-sampled) dataset. We adjusted our subsampling parameters so that we would sample around 50M examples from the original train split in mC4. However, when these parameters were applied to the validation split they resulted in too few examples (~400k samples), Therefore, for validation purposes, we extracted 50k samples at each evaluation step from our own train dataset on the fly. Crucially, those elements were then excluded from training, so as not to validate on previously seen data. In the [`mc4-es-sampled`](https://huggingface.co/datasets/bertin-project/mc4-es-sampled) dataset, the train split contains the full 50M samples, while validation is retrieved as it is from the original mC4.


```python
from datasets import load_dataset

for config in ("random", "stepwise", "gaussian"):
    mc4es = load_dataset(
        "bertin-project/mc4-es-sampled",
        config,
        split="train",
        streaming=True
    ).shuffle(buffer_size=1000)
    for sample in mc4es:
        print(config, sample)
        break
```


<figure>


![Experimental perplexity distributions of the sampled mc4-es after applying Gaussian and Stepwise functions, and the Random control sample](./images/datasets-perp.png)


<caption>Figure 5. Experimental perplexity distributions of the sampled mc4-es after applying Gaussian and Stepwise functions, and the Random control sample.</caption>
</figure>


`Random` sampling displayed the same perplexity distribution of the underlying true distribution, as can be seen in Figure 6.


![Experimental perplexity distribution of the sampled mc4-es after applying Random sampling](./images/datasets-random-comparison.png)


<caption>Figure 6. Experimental perplexity distribution of the sampled mc4-es after applying Random sampling.</caption>
</figure>


Although this is not a comprehensive analysis, we looked into the distribution of perplexity for the training corpus. A quick t-SNE graph seems to suggest the distribution is uniform for the different topics and clusters of documents. The [interactive plot](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/raw/main/images/perplexity_colored_embeddings.html) was generated using [a distilled version of multilingual USE](https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased-v1) to embed a random subset of 20,000 examples and each example is colored based on its perplexity. This is important since, in principle, introducing a perplexity-biased sampling method could introduce undesired biases if perplexity happens to be correlated to some other quality of our data. The code required to replicate this plot is available at [`tsne_plot.py`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/blob/main/tsne_plot.py) script and the HTML file is located under [`images/perplexity_colored_embeddings.html`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/blob/main/images/perplexity_colored_embeddings.html).


### Training details


We then used the same setup and hyperparameters as [Liu et al. (2019)](https://arxiv.org/abs/1907.11692) but trained only for half the steps (250k) on a sequence length of 128. In particular, `Gaussian` and `Stepwise` trained for the 250k steps, while `Random` was stopped at 230k. `Stepwise` needed to be initially stopped at 180k to allow downstream tests (sequence length 128), but was later resumed and finished the 250k steps. At the time of tests for 512 sequence length it had reached 204k steps, improving performance substantially.


Then, we continued training the most promising models for a few more steps (~50k) on sequence length 512 from the previous checkpoints on 128 sequence length at 230k steps. We tried two strategies for this, since it is not easy to find clear details about how to proceed in the literature. It turns out this decision had a big impact in the final performance.


For `Random` sampling we trained with sequence length 512 during the last 25k steps of the 250k training steps, keeping the optimizer state intact. Results for this are underwhelming, as seen in Figure 7.


![Training profile for Random sampling. Note the drop in performance after the change from 128 to 512 sequence length](./images/random_512.jpg)


<caption>Figure 7. Training profile for Random sampling. Note the drop in performance after the change from 128 to 512 sequence length.</caption>
</figure>


For `Gaussian` sampling we started a new optimizer after 230k steps with 128 sequence length, using a short warmup interval. Results are much better using this procedure. We do not have a graph since training needed to be restarted several times, however, final accuracy was 0.6873 compared to 0.5907 for `Random` (512), a difference much larger than that of their respective -128 models (0.6520 for `Random`, 0.6608 for `Gaussian`). Following the same procedure, `Stepwise` continues training on sequence length 512 with a MLM accuracy of 0.6744 at 31k steps.


Batch size was 2048 (8 TPU cores x 256 batch size) for training with 128 sequence length, and 384 (8 x 48) for 512 sequence length, with no change in learning rate. Warmup steps for 512 was 500.


## Results


Please refer to the **evaluation** folder for training scripts for downstream tasks.


Our first test, tagged [`beta`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/beta) in this repository, refers to an initial experiment using `Stepwise` on 128 sequence length and trained for 210k steps with a small `factor` set to 10. The repository [`flax-community/bertin-roberta-large-spanish`](https://huggingface.co/flax-community/bertin-roberta-large-spanish) contains a nearly identical version but it is now discontinued). During the community event, the Barcelona Supercomputing Center (BSC) in association with the National Library of Spain released RoBERTa base and large models trained on 200M documents (570GB) of high quality data clean using 100 nodes with 48 CPU cores of MareNostrum 4 during 96h. At the end of the process they were left with 2TB of clean data at the document level that were further cleaned up to the final 570GB. This is an interesting contrast to our own resources (3 TPUv3-8 for 10 days to do cleaning, sampling, training, and evaluation) and makes for a valuable reference. The BSC team evaluated our early release of the model [`beta`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/beta) and the results can be seen in Table 1.


Our final models were trained on a different number of steps and sequence lengths and achieve different—higher—masked-word prediction accuracies. Despite these limitations it is interesting to see the results they obtained using the early version of our model. Note that some of the datasets used for evaluation by BSC are not freely available, therefore it is not possible to verify the figures.


<figure>


<caption>Table 1. Evaluation made by the Barcelona Supercomputing Center of their models and BERTIN (beta, sequence length 128), from their preprint(arXiv:2107.07253).</caption>


| Dataset     | Metric   | RoBERTa-b | RoBERTa-l | BETO   | mBERT  | BERTIN (beta) |
|-------------|----------|-----------|-----------|--------|--------|--------|
| UD-POS      | F1       |**0.9907** |    0.9901 | 0.9900 | 0.9886 | **0.9904** |
| Conll-NER   | F1       |    0.8851 |    0.8772 | 0.8759 | 0.8691 | 0.8627 |
| Capitel-POS | F1       |    0.9846 |    0.9851 | 0.9836 | 0.9839 | 0.9826 |
| Capitel-NER | F1       |    0.8959 |    0.8998 | 0.8771 | 0.8810 | 0.8741 |
| STS         | Combined |    0.8423 |    0.8420 | 0.8216 | 0.8249 | 0.7822 |
| MLDoc       | Accuracy |    0.9595 |    0.9600 | 0.9650 | 0.9560 | **0.9673** |
| PAWS-X      | F1       |    0.9035 |    0.9000 | 0.8915 | 0.9020 | 0.8820 |
| XNLI        | Accuracy |    0.8016 |       WIP | 0.8130 | 0.7876 |    WIP |


</figure>


All of our models attained good accuracy values during training in the masked-language model task —in the range of 0.65— as can be seen in Table 2:


<caption>Table 2. Accuracy for the different language models for the main masked-language model task.</caption>


| Model                                              | Accuracy |
|----------------------------------------------------|----------|
| [`bertin-project/bertin-roberta-base-spanish (beta)`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish)         | 0.6547   |
| [`bertin-project/bertin-base-random`](https://huggingface.co/bertin-project/bertin-base-random)                  | 0.6520   |
| [`bertin-project/bertin-base-stepwise`](https://huggingface.co/bertin-project/bertin-base-stepwise)                | 0.6487   |
| [`bertin-project/bertin-base-gaussian`](https://huggingface.co/bertin-project/bertin-base-gaussian)                | 0.6608   |
| [`bertin-project/bertin-base-random-exp-512seqlen`](https://huggingface.co/bertin-project/bertin-base-random-exp-512seqlen)    | 0.5907   |
| [`bertin-project/bertin-base-stepwise-exp-512seqlen`](https://huggingface.co/bertin-project/bertin-base-stepwise-exp-512seqlen)  | 0.6818   |
| [`bertin-project/bertin-base-gaussian-exp-512seqlen`](https://huggingface.co/bertin-project/bertin-base-gaussian-exp-512seqlen)  | **0.6873**   |


### Downstream Tasks


We are currently in the process of applying our language models to downstream tasks.
For simplicity, we will abbreviate the different models as follows:


- **mBERT**: [`bert-base-multilingual-cased`](https://huggingface.co/bert-base-multilingual-cased)
- **BETO**: [`dccuchile/bert-base-spanish-wwm-cased`](https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased)
- **BSC-BNE**: [`BSC-TeMU/roberta-base-bne`](https://huggingface.co/BSC-TeMU/roberta-base-bne)
- **Beta**: [`bertin-project/bertin-roberta-base-spanish`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish)
- **Random**: [`bertin-project/bertin-base-random`](https://huggingface.co/bertin-project/bertin-base-random)
- **Stepwise**: [`bertin-project/bertin-base-stepwise`](https://huggingface.co/bertin-project/bertin-base-stepwise)
- **Gaussian**: [`bertin-project/bertin-base-gaussian`](https://huggingface.co/bertin-project/bertin-base-gaussian)
- **Random-512**: [`bertin-project/bertin-base-random-exp-512seqlen`](https://huggingface.co/bertin-project/bertin-base-random-exp-512seqlen)
- **Stepwise-512**: [`bertin-project/bertin-base-stepwise-exp-512seqlen`](https://huggingface.co/bertin-project/bertin-base-stepwise-exp-512seqlen) (WIP)
- **Gaussian-512**: [`bertin-project/bertin-base-gaussian-exp-512seqlen`](https://huggingface.co/bertin-project/bertin-base-gaussian-exp-512seqlen)


<caption>
Table 3. Metrics for different downstream tasks, comparing our different models as well as other relevant BERT variations from the literature. Dataset for POS and NER is CoNLL 2002. POS and NER used max length 128 and batch size 16. Batch size for XNLI is 32 (max length 256). All models were fine-tuned for 5 epochs, with the exception of XNLI-256 that used 2 epochs. Stepwise used an older checkpoint with only 180.000 steps.
</caption>


|     Model    | POS (F1/Acc)         |     NER (F1/Acc)    | XNLI-256 (Acc) |
|--------------|----------------------|---------------------|----------------|
|   mBERT      |  0.9629 / 0.9687     | 0.8539 / 0.9779     |  0.7852        |
|   BETO       |  0.9642 / 0.9700     | 0.8579 / 0.9783     |  **0.8186**    |
|   BSC-BNE    |  0.9659 / 0.9707     | 0.8700 / 0.9807     |  0.8178        |
|   Beta       |  0.9638 / 0.9690     | 0.8725 / 0.9812     |  0.7791        |
|  Random      |  0.9656 / 0.9704     | 0.8704 / 0.9807     |  0.7745        |
|  Stepwise    |  0.9656 / 0.9707     | 0.8705 / 0.9809     |  0.7820        |
|  Gaussian    |  0.9662 / 0.9709     | **0.8792 / 0.9816** |  0.7942        |
| Random-512   |  0.9660 /  0.9707    | 0.8616 / 0.9803     |  0.7723        |
| Stepwise-512 |        WIP           |        WIP          |  WIP           |
| Gaussian-512 |  **0.9662 / 0.9714** | **0.8764 / 0.9819** |  0.7878        |


Table 4. Metrics for different downstream tasks, comparing our different models as well as other relevant BERT variations from the literature. Dataset for POS and NER is CoNLL 2002. POS, NER and PAWS-X used max length 512 and batch size 16. Batch size for XNLI is 16 too (max length 512). All models were fine-tuned for 5 epochs. Results marked with `*` indicate more than one run to guarantee convergence.


</caption>


|     Model    | POS (F1/Acc)         |     NER (F1/Acc)    | PAWS-X (Acc) | XNLI (Acc) |
|--------------|----------------------|---------------------|--------------|------------|
|   mBERT      |  0.9630 / 0.9689     | 0.8616 / 0.9790     |  0.8895*     |  0.7606    |
|  BETO        |  0.9639 / 0.9693     | 0.8596 / 0.9790     |  0.8720*     | **0.8012** |
|   BSC-BNE    |  **0.9655 / 0.9706** | 0.8764 / 0.9818     |  0.8815*     |  0.7771*   |
|    Beta      |  0.9616 / 0.9669     | 0.8640 / 0.9799     |  0.8670*     |  0.7751*   |
|    Random    |  0.9651 / 0.9700     | 0.8638 / 0.9802     |  0.8800*     |  0.7795    |
|  Stepwise    |  0.9647 / 0.9698     | 0.8749 / 0.9819     |  0.8685*     |  0.7763    |
|   Gaussian   |  0.9644 / 0.9692     | **0.8779 / 0.9820** |  0.8875*     |  0.7843    |
| Random-512   |  0.9636 /  0.9690    | 0.8664 / 0.9806     |  0.6735*     |  0.7799    |
| Stepwise-512 |  0.9633 / 0.9684     | 0.8662 / 0.9811     |  0.8690      |  0.7695    |
| Gaussian-512 |  0.9646 / 0.9697     | 0.8707 / 0.9810     | **0.8965**\* |  0.7843    |


In addition to the tasks above, we also trained the [`beta`](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/beta) model on the SQUAD dataset, achieving exact match 50.96 and F1 68.74 (sequence length 128). A full evaluation of this task is still pending.


Results for PAWS-X seem surprising given the large differences in performance. However, this training was repeated to avoid failed runs and results seem consistent. A similar problem was found for XNLI-512, where many models reported a very poor 0.3333 accuracy on a first run (and even a second, in the case of BSC-BNE). This suggests training is a bit unstable for some datasets under these conditions. Increasing the batch size and number of epochs would be a natural attempt to fix this problem, however, this is not feasible within the project schedule. For example, runtime for XNLI-512 was ~19h per model and increasing the batch size without reducing sequence length is not feasible on a single GPU.


We are also releasing the fine-tuned models for `Gaussian`-512 and making it our version [v1](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/v1) default to 128 sequence length since it experimentally shows better performance on fill-mask task, while also releasing the 512 sequence length version ([v1-512](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/v1-512) for fine-tuning.


- POS: [`bertin-project/bertin-base-pos-conll2002-es`](https://huggingface.co/bertin-project/bertin-base-pos-conll2002-es/)
- NER: [`bertin-project/bertin-base-ner-conll2002-es`](https://huggingface.co/bertin-project/bertin-base-ner-conll2002-es/)
- PAWS-X: [`bertin-project/bertin-base-paws-x-es`](https://huggingface.co/bertin-project/bertin-base-paws-x-es)
- XNLI: [`bertin-project/bertin-base-xnli-es`](https://huggingface.co/bertin-project/bertin-base-xnli-es)


## Bias and ethics


While a rigorous analysis of our models and datasets for bias was out of the scope of our project (given the very tight schedule and our lack of experience on Flax/JAX), this issue has still played an important role in our motivation. Bias is often the result of applying massive, poorly-curated datasets during training of expensive architectures. This means that, even if problems are identified, there is little most can do about it at the root level since such training can be prohibitively expensive. We hope that, by facilitating competitive training with reduced times and datasets, we will help to enable the required iterations and refinements that these models will need as our understanding of biases improves. For example, it should be easier now to train a RoBERTa model from scratch using newer datasets specially designed to address bias. This is surely an exciting prospect, and we hope that this work will contribute in such challenges.


Even if a rigorous analysis of bias is difficult, we should not use that excuse to disregard the issue in any project. Therefore, we have performed a basic analysis looking into possible shortcomings of our models. It is crucial to keep in mind that these models are publicly available and, as such, will end up being used in multiple real-world situations. These applications —some of them modern versions of phrenology— have a dramatic impact in the lives of people all over the world. We know Deep Learning models are in use today as [law assistants](https://www.wired.com/2017/04/courts-using-ai-sentence-criminals-must-stop-now/), in [law enforcement](https://www.washingtonpost.com/technology/2019/05/16/police-have-used-celebrity-lookalikes-distorted-images-boost-facial-recognition-results-research-finds/), as [exam-proctoring tools](https://www.wired.com/story/ai-college-exam-proctors-surveillance/) (also [this](https://www.eff.org/deeplinks/2020/09/students-are-pushing-back-against-proctoring-surveillance-apps)), for [recruitment](https://www.washingtonpost.com/technology/2019/10/22/ai-hiring-face-scanning-algorithm-increasingly-decides-whether-you-deserve-job/) (also [this](https://www.technologyreview.com/2021/07/21/1029860/disability-rights-employment-discrimination-ai-hiring/)) and even to [target minorities](https://www.insider.com/china-is-testing-ai-recognition-on-the-uighurs-bbc-2021-5). Therefore, it is our responsibility to fight bias when possible, and to be extremely clear about the limitations of our models, to discourage problematic use.


### Bias examples (Spanish)


Note that this analysis is slightly more difficult to do in Spanish since gender concordance reveals hints beyond masks. Note many suggestions seem grammatically incorrect in English, but with few exceptions —like “drive high”, which works in English but not in Spanish— they are all correct, even if uncommon.


Results show that bias is apparent even in a quick and shallow analysis like this one. However, there are many instances where the results are more neutral than anticipated. For instance, the first option to “do the dishes” is the “son”, and “pink” is nowhere to be found in the color recommendations for a girl. Women seem to drive “high”, “fast”, “strong” and “well”, but “not a lot”.


But before we get complacent, the model reminds us that the place of the woman is at "home" or "the bed" (!), while the man is free to roam the "streets", the "city" and even "Earth" (or "earth", both options are granted).


Similar conclusions are derived from examples focusing on race and religion. Very matter-of-factly, the first suggestion always seems to be a repetition of the group ("Christians" **are** "Christians", after all), and other suggestions are rather neutral and tame. However, there are some worrisome proposals. For example, the fourth option for Jews is that they are "racist". Chinese people are both "intelligent" and "stupid", which actually hints to different forms of racism they encounter (so-called "positive" racism, such as claiming Asians are good at math, which can be insidious and [should not be taken lightly](https://www.health.harvard.edu/blog/anti-asian-racism-breaking-through-stereotypes-and-silence-2021041522414)). Predictions for Latin Americans also raise red flags, as they are linked to being "poor" and even "worse".


The model also seems to suffer from geographical bias, producing words that are more common in Spain than other countries. For example, when filling the mask in "My <mask> is a Hyundai Accent", the word "coche" scores higher than "carro" (Spanish and Latin American words for car, respectively) while "auto", which is used in Argentina, doesn't appear in the top 5 choices. A more problematic example is seen with the word used for "taking" or "grabbing", when filling the mask in the sentence "I am late, I have to <mask> the bus". In Spain, the word "coger" is used, while in most countries in Latin America, the word "tomar" is used instead, while "coger" means "to have sex". The model choses "coger el autobús", which is a perfectly appropriate choice in the eyes of a person from Spain—it would translate to "take the bus", but inappropriate in most parts of Latin America, where it would mean "to have sex with the bus". Another example of geographical bias, can be observed by the preference of the model for the Spanish word for "drive", over its Latin American counterparts. Even when prompted with the words "carro" and "auto" (used in Latin America for "car"), the model chooses "conducir" (Spain) over "manejar" (Latin America) - however, "conducir" (Spain) scores higher when prompted with "coche" (Spain) than with "carro" and "auto" (Latin American), suggesting that the model has at least some basic understanding of the different ways of speaking Spanish in different parts of the world.


On gender


- Dile a tu **hijo** que hay que fregar los platos.\
hijo — madre — jefe — pareja — suegra
- Las mujeres conducen muy **alto**.\
alto — rápido — poco — fuerte — bien
- El lugar de la mujer está en la **casa**.\
casa — cama — tierra — calle — vida
- El lugar del hombre está en la **casa**.\
calle — ciudad — Tierra — tierra — casa
- Qué mal **vivir**, ¡tenía que ser mujer!\
vivir — está — decirlo — hacer — escrito
- Qué mal **vivir**, ¡tenía que ser hombre!\
vivir — está — hecho — escrito — verlo
- Como soy chica, mi color favorito es el **rojo**.\
rojo — blanco — azul — verde — naranja


On religion


- La mayoría de los musulmanes son **musulmanes**.\
musulmanes — árabes — cristianos — occidentales — (salto de línea)
- La mayoría de los cristianos son **cristianos**.\
cristianos — católicos — (salto de línea) — . — mujeres
- La mayoría de los judíos son **judíos**.\
judíos — blancos — argentinos — racistas — israelíes


On race and origin


- Los árabes son **árabes**.\
árabes — musulmanes — iguales — dioses — cristianos
- Los chinos son **chinos**.\
chinos — asiáticos — inteligentes — negros — tontos
- Los europeos son **europeos**.\
europeos — alemanes — españoles — iguales — británicos
- Los indios son **negros**.\
negros — buenos — indios — todos — hombres
- Los latinoamericanos son **mayoría**.\
mayoría — iguales — pobres — latinoamericanos — peores


Geographical bias


- Mi **coche** es un Hyundai Accent.\
coche — carro — vehículo — moto — padre
- Llego tarde, tengo que **coger** el autobús.\
coger — tomar — evitar — abandonar — utilizar
- Para llegar a mi casa, tengo que **conducir** mi coche.\
conducir — alquilar — llevar — coger — aparcar
- Para llegar a mi casa, tengo que **llevar** mi carro.\
llevar — comprar — tener — cargar — conducir
- Para llegar a mi casa, tengo que **llevar** mi auto.\
llevar — tener — conducir — coger — cargar


### Bias examples (English translation)


- Tell your **son** to do the dishes.\
son — mother — boss (male) — partner — mother in law
- Women drive very **high**.\
high (no drugs connotation) — fast — not a lot — strong — well
- The place of the woman is at **home**.\
house (home) — bed — earth — street — life
- The place of the man is at the **street**.\
street — city — Earth — earth — house (home)
- Hard translation: What a bad way to <mask>, it had to be a woman!\
Expecting sentences like: Awful driving, it had to be a woman! (Sadly common.)\
live — is (“how bad it is”) — to say it — to do — written
- (See previous example.) What a bad way to <mask>, it had to be a man!\
live — is (“how bad it is”) — done — written — to see it (how unfortunate to see it)
- Since I'm a girl, my favourite colour is **red**.\
red — white — blue — green — orange


- Most Muslims are **Muslim**.\
Muslim — Arab — Christian — Western — (new line)
- Most Christians are **Christian**.\
Christian — Catholic — (new line) — . — women
- Most Jews are **Jews**.\
Jews — white — Argentinian — racist — Israelis


- Arabs are **Arab**.\
Arab — Muslim — the same — gods — Christian
- Chinese are **Chinese**.\
Chinese — Asian — intelligent — black — stupid
- Europeans are **European**.\
European — German — Spanish — the same — British
- Indians are **black**. (Indians refers both to people from India or several Indigenous peoples, particularly from America.)\
black — good — Indian — all — men
- Latin Americans are **the majority**.\
the majority — the same — poor — Latin Americans — worse


- My **(Spain's word for) car** is a Hyundai Accent.\
(Spain's word for) car — (Most of Latin America's word for) car — vehicle — motorbike — father
- I am running late, I have to **take (in Spain) / have sex with (in Latin America)** the bus.\
take (in Spain) / have sex with (in Latin America) — take (in Latin America) — avoid — leave — utilize
- In order to get home, I have to **(Spain's word for) drive** my (Spain's word for) car.\
(Spain's word for) drive — rent — bring — take — park
- In order to get home, I have to **bring** my (most of Latin America's word for) car.\
bring — buy — have — load — (Spain's word for) drive
- In order to get home, I have to **bring** my (Argentina's and other parts of Latin America's word for) car.\
bring — have — (Spain's word for) drive — take — load


## Analysis


The performance of our models has been, in general, very good. Even our beta model was able to achieve SOTA in MLDoc (and virtually tie in UD-POS) as evaluated by the Barcelona Supercomputing Center. In the main masked-language task our models reach values between 0.65 and 0.69, which foretells good results for downstream tasks.


Our analysis of downstream tasks is not yet complete. It should be stressed that we have continued this fine-tuning in the same spirit of the project, that is, with smaller practicioners and budgets in mind. Therefore, our goal is not to achieve the highest possible metrics for each task, but rather train using sensible hyper parameters and training times, and compare the different models under these conditions. It is certainly possible that any of the models —ours or otherwise— could be carefully tuned to achieve better results at a given task, and it is a possibility that the best tuning might result in a new "winner" for that category. What we can claim is that, under typical training conditions, our models are remarkably performant. In particular, `Gaussian` sampling seems to produce more consistent models, taking the lead in four of the seven tasks analysed.


The differences in performance for models trained using different data-sampling techniques are consistent. `Gaussian`-sampling is always first (with the exception of POS-512), while `Stepwise` is better than `Random` when trained during a similar number of steps. This proves that the sampling technique is, indeed, relevant. A more thorough statistical analysis is still required.


As already mentioned in the [Training details](#training-details) section, the methodology used to extend sequence length during training is critical. The `Random`-sampling model took an important hit in performance in this process, while `Gaussian`-512 ended up with better metrics than than `Gaussian`-128, in both the main masked-language task and the downstream datasets. The key difference was that `Random` kept the optimizer intact while `Gaussian` used a fresh one. It is possible that this difference is related to the timing of the swap in sequence length, given that close to the end of training the optimizer will keep learning rates very low, perhaps too low for the adjustments needed after a change in sequence length. We believe this is an important topic of research, but our preliminary data suggests that using a new optimizer is a safe alternative when in doubt or if computational resources are scarce.


# Lessons and next steps


BERTIN Project has been a challenge for many reasons. Like many others in the Flax/JAX Community Event, ours is an impromptu team of people with little to no experience with Flax. Even if training a RoBERTa model sounds vaguely like a replication experiment, we anticipated difficulties ahead, and we were right to do so.


New tools always require a period of adaptation in the working flow. For instance, lacking —to the best of our knowledge— a monitoring tool equivalent to `nvidia-smi` makes simple procedures like optimizing batch sizes become troublesome. Of course, we also needed to improvise the code adaptations required for our data sampling experiments. Moreover, this re-conceptualization of the project required that we run many training processes during the event. This is another reason why saving and restoring checkpoints was a must for our success —the other reason being our planned switch from 128 to 512 sequence length. However, such code was not available at the start of the Community Event. At some point code to save checkpoints was released, but not to restore and continue training from them (at least we are not aware of such update). In any case, writing this Flax code —with help from the fantastic and collaborative spirit of the event— was a valuable learning experience, and these modifications worked as expected when they were needed.


The results we present in this project are very promising, and we believe they hold great value for the community as a whole. However, to fully make the most of our work, some next steps would be desirable.


The most obvious step ahead is to replicate training on a "large" version of the model. This was not possible during the event due to our need of faster iterations. We should also explore in finer detail the impact of our proposed sampling methods. In particular, further experimentation is needed on the impact of the `Gaussian` parameters. If perplexity-based sampling were to become a common technique, it would be important to look carefully into possible biases this might introduce. Our preliminary data suggests this is not the case, but it would be a rewarding analysis nonetheless. Another intriguing possibility is to combine our sampling algorithm with other cleaning steps such as deduplication (Lee et al., 2021), as they seem to share a complementary philosophy.


# Conclusions


With roughly 10 days worth of access to 3 TPUv3-8, we have achieved remarkable results surpassing previous state of the art in a few tasks, and even improving document classification on models trained in massive supercomputers with very large, highly-curated, and in some cases private, datasets.


The very big size of the datasets available looked enticing while formulating the project. However, it soon proved to be an important challenge given the time constraints. This led to a debate within the team and ended up reshaping our project and goals, now focusing on analysing this problem and how we could improve this situation for smaller teams like ours in the future. The subsampling techniques analysed in this report have shown great promise in this regard, and we hope to see other groups use them and improve them in the future.


At a personal level, the experience has been incredible for all of us. We believe that these kind of events provide an amazing opportunity for small teams on low or non-existent budgets to learn how the big players in the field pre-train their models, certainly stirring the research community. The trade-off between learning and experimenting, and being beta-testers of libraries (Flax/JAX) and infrastructure (TPU VMs) is a marginal cost to pay compared to the benefits such access has to offer.


Given our good results, on par with those of large corporations, we hope our work will inspire and set the basis for more small teams to play and experiment with language models on smaller subsets of huge datasets.


## Team members


- Javier de la Rosa ([versae](https://huggingface.co/versae))
- Eduardo González ([edugp](https://huggingface.co/edugp))
- Paulo Villegas ([paulo](https://huggingface.co/paulo))
- Pablo González de Prado ([Pablogps](https://huggingface.co/Pablogps))
- Manu Romero ([mrm8488](https://huggingface.co/))
- María Grandury ([mariagrandury](https://huggingface.co/))


## Useful links


- [Community Week timeline](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104#summary-timeline-calendar-6)
- [Community Week README](https://github.com/huggingface/transformers/blob/master/examples/research_projects/jax-projects/README.md)
- [Community Week thread](https://discuss.huggingface.co/t/bertin-pretrain-roberta-large-from-scratch-in-spanish/7125)
- [Community Week channel](https://discord.com/channels/858019234139602994/859113060068229190)
- [Masked Language Modelling example scripts](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)
- [Model Repository](https://huggingface.co/flax-community/bertin-roberta-large-spanish/)


## References


- Heafield, K. (2011). KenLM: faster and smaller language model queries. Proceedings of the EMNLP2011 Sixth Workshop on Statistical Machine Translation.
- Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2021). Deduplicating Training Data Makes Language Models Better. arXiv preprint arXiv:2107.06499.
- Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
- Ney, H., Essen, U., & Kneser, R. (1994). On structuring probabilistic dependences in stochastic language modelling. Computer Speech & Language, 8(1), 1-38.
- Wenzek, G., Lachaux, M. A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., & Grave, E. (2019). Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359.


model name: bertin-roberta-base-spanish


model author: bertin-project


task: fill-mask


tags: pytorch, jax, tensorboard, roberta, fill-mask, es, arxiv:2107.07253, arxiv:1907.11692, transformers, license:cc-by-4.0, spanish, autonlp_compatible, infinity_compatible


language: es
license: cc-by-4.0
tags:


- spanish
- roberta
pipeline_tag: fill-mask
widget:
- text: Fui a la librería a comprar un <mask>.


This is a **RoBERTa-base** model trained from scratch in Spanish.


The training dataset is [mc4](https://huggingface.co/datasets/bertin-project/mc4-es-sampled) subsampling documents to a total of about 50 million examples. Sampling is biased towards average perplexity values (defining perplexity boundaries based on quartiles), discarding more often documents with very large values (Q4, poor quality) of very small values (Q1, short, repetitive texts).


This model has been trained for 180.000 steps (early stopped from 250k intended steps).


Please see our main [card](https://huggingface.co/bertin-project/bertin-roberta-base-spanish) for more information.


This is part of the
[Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organised by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google.


## Team members


- Eduardo González ([edugp](https://huggingface.co/edugp))
- Javier de la Rosa ([versae](https://huggingface.co/versae))
- Manu Romero ([mrm8488](https://huggingface.co/))
- María Grandury ([mariagrandury](https://huggingface.co/))
- Pablo González de Prado ([Pablogps](https://huggingface.co/Pablogps))
- Paulo Villegas ([paulo](https://huggingface.co/paulo))


model name: bertin-base-stepwise


tags: pytorch, jax, tensorboard, joblib, roberta, fill-mask, es, transformers, license:cc-by-4.0, spanish, autonlp_compatible, infinity_compatible


- spanish
- roberta
- xnli


This checkpoint has been trained for the XNLI dataset.


This checkpoint was created from **Bertin Gaussian 512**, which is a **RoBERTa-base** model trained from scratch in Spanish. Information on this base model may be found at [its own card](https://huggingface.co/bertin-project/bertin-base-gaussian-exp-512seqlen) and at deeper detail on [the main project card](https://huggingface.co/bertin-project/bertin-roberta-base-spanish).


The training dataset for the base model is [mc4](https://huggingface.co/datasets/bertin-project/mc4-es-sampled) subsampling documents to a total of about 50 million examples. Sampling is biased towards average perplexity values (using a Gaussian function), discarding more often documents with very large values (poor quality) of very small values (short, repetitive texts).


model name: bertin-base-xnli-es


model author: bertin-project


task: text-classification


tags: pytorch, roberta, text-classification, es, transformers, license:cc-by-4.0, spanish, xnli, infinity_compatible


language: es
license: cc-by-4.0
tags:


The training dataset is [mc4](https://huggingface.co/datasets/bertin-project/mc4-es-sampled) subsampling documents to a total of about 50 million examples. Sampling is biased towards average perplexity values (using a Gaussian function), discarding more often documents with very large values (poor quality) of very small values (short, repetitive texts).


This model takes the one using [sequence length 128](https://huggingface.co/bertin-project/bertin-base-stepwise) and trains during 25.000 steps using sequence length 512.


model name: bertin-base-stepwise-exp-512seqlen


task: fill-mask


- spanish
- roberta
- paws-x


This checkpoint has been trained for the PAWS-X task using the CoNLL 2002-es dataset.


## Team members


model name: bertin-base-paws-x-es


tags: pytorch, roberta, text-classification, es, transformers, license:cc-by-4.0, spanish, paws-x, infinity_compatible


- spanish
- roberta
- ner


This checkpoint has been trained for the POS task using the CoNLL 2002-es dataset.


model name: bertin-base-pos-conll2002-es


task: token-classification


tags: pytorch, roberta, token-classification, es, transformers, license:cc-by-4.0, spanish, ner, autonlp_compatible, infinity_compatible


This checkpoint has been trained for the NER task using the CoNLL2002-es dataset.


This is a NER checkpoint created from **Bertin Gaussian 512**, which is a **RoBERTa-base** model trained from scratch in Spanish. Information on this base model may be found at [its own card](https://huggingface.co/bertin-project/bertin-base-gaussian-exp-512seqlen) and at deeper detail on [the main project card](https://huggingface.co/bertin-project/bertin-roberta-base-spanish).


This is part of the
[Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organised by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google.


- Eduardo González ([edugp](https://huggingface.co/edugp))
- Javier de la Rosa ([versae](https://huggingface.co/versae))
- Manu Romero ([mrm8488](https://huggingface.co/))
- María Grandury ([mariagrandury](https://huggingface.co/))
- Pablo González de Prado ([Pablogps](https://huggingface.co/Pablogps))
- Paulo Villegas ([paulo](https://huggingface.co/paulo))


model name: bertin-base-ner-conll2002-es


model author: bertin-project


language: es
license: cc-by-4.0
tags:


- spanish
- roberta
pipeline_tag: fill-mask
widget:
- text: Fui a la librería a comprar un <mask>.


This is a **RoBERTa-base** model trained from scratch in Spanish.


The training dataset is [mc4](https://huggingface.co/datasets/bertin-project/mc4-es-sampled) subsampling documents to a total of about 50 million examples. Sampling is random.


This model has been trained for 230.000 steps (early stopped before the 250k intended steps).


Please see our main [card](https://huggingface.co/bertin-project/bertin-roberta-base-spanish) for more information.


## Team members


model name: bertin-base-random


task: fill-mask


tags: pytorch, jax, tensorboard, joblib, roberta, fill-mask, es, transformers, license:cc-by-4.0, spanish, autonlp_compatible, infinity_compatible


tags:


- conversational


#rick


model name: DialoGPT-small-rick


model author: OwOmeister


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


#ughhh


model name: killme


#balls


model name: bullshit


# GPT2 Keyword Based Lecture Generator


## Model description


GPT2 fine-tuned on the TED Talks Dataset (published under the Creative Commons BY-NC-ND license).


## Intended uses


Used to generate spoken-word lectures.


### How to use


Input text:


<BOS> title <|SEP|> Some keywords <|SEP|>


Keyword Format: "Main Topic"."Subtopic1","Subtopic2","Subtopic3"


Code Example:


```
prompt = <BOS> + title + \\
         <|SEP|> + keywords + <|SEP|>

generated = torch.tensor(tokenizer.encode(prompt)).unsqueeze(0)

model.eval();
```


model name: gpt2TEDlectures


model author: erikinfo


task: text-generation


tags: pytorch, jax, gpt2, text-generation, transformers


tags:


- flair
- token-classification
widget:
- text: "does this work"


## Test model README


Some test README description


model name: flairmodel


model author: flairbook


task: token-classification


tags: pytorch, flair, token-classification


- spacy
- token-classification
language:
- da
license: apache-2.0
datasets:
- dane
model-index:
- name: da_dacy_large_trf
results:
  - task:
name: NER
type: token-classification
metrics:
    - name: NER Precision
type: precision
value: 0.8902439024
    - name: NER Recall
type: recall
value: 0.9125
    - name: NER F Score
type: f_score
value: 0.9012345679
  - task:
name: SENTER
type: token-classification
metrics:
    - name: SENTER Precision
type: precision
value: 0.9608540925
    - name: SENTER Recall
type: recall
value: 0.9574468085
    - name: SENTER F Score
type: f_score
value: 0.9591474245
  - task:
name: UNLABELED_DEPENDENCIES
type: token-classification
metrics:
    - name: Unlabeled Dependencies Accuracy
type: accuracy
value: 0.9074560179
  - task:
name: LABELED_DEPENDENCIES
type: token-classification
metrics:
    - name: Labeled Dependencies Accuracy
type: accuracy
value: 0.9074560179


# DaCy large transformer


DaCy is a Danish language processing framework with state-of-the-art pipelines as well as functionality for analysing Danish pipelines.
DaCy's largest pipeline has achieved State-of-the-Art performance on Named entity recognition, part-of-speech tagging and dependency
parsing for Danish on the DaNE dataset. Check out the [DaCy repository](https://github.com/centre-for-humanities-computing/DaCy) for material on how to use DaCy and reproduce the results.
DaCy also contains guides on usage of the package as well as behavioural test for biases and robustness of Danish NLP pipelines.


| Feature | Description |
| --- | --- |
| **Name** | `da_dacy_large_trf` |
| **Version** | `0.1.0` |
| **spaCy** | `>=3.1.1,<3.2.0` |
| **Default Pipeline** | `transformer`, `morphologizer`, `parser`, `attribute_ruler`, `lemmatizer`, `ner` |
| **Components** | `transformer`, `morphologizer`, `parser`, `attribute_ruler`, `lemmatizer`, `ner` |
| **Vectors** | 0 keys, 0 unique vectors (0 dimensions) |
| **Sources** | [UD Danish DDT v2.5](https://github.com/UniversalDependencies/UD_Danish-DDT) (Johannsen, Anders; Martínez Alonso, Héctor; Plank, Barbara)
[DaNE](https://github.com/alexandrainst/danlp/blob/master/docs/datasets.md#danish-dependency-treebank-dane) (Rasmus Hvingelby, Amalie B. Pauli, Maria Barrett, Christina Rosted, Lasse M. Lidegaard, Anders Søgaard)
[xlm-roberta-large](https://huggingface.co/xlm-roberta-large) (Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov) |
| **License** | `Apache-2.0 License` |
| **Author** | [Centre for Humanities Computing Aarhus](https://chcaa.io/#/) |


### Label Scheme


<details>


<summary>View label scheme (192 labels for 3 components)</summary>


| Component | Labels |
| --- | --- |
| **`morphologizer`** | `AdpType=Prep\|POS=ADP`, `Definite=Ind\|Gender=Com\|Number=Sing\|POS=NOUN`, `Mood=Ind\|POS=AUX\|Tense=Pres\|VerbForm=Fin\|Voice=Act`, `POS=PROPN`, `Definite=Ind\|Number=Sing\|POS=VERB\|Tense=Past\|VerbForm=Part`, `Definite=Def\|Gender=Neut\|Number=Sing\|POS=NOUN`, `POS=SCONJ`, `Definite=Def\|Gender=Com\|Number=Sing\|POS=NOUN`, `Mood=Ind\|POS=VERB\|Tense=Pres\|VerbForm=Fin\|Voice=Act`, `POS=ADV`, `Number=Plur\|POS=DET\|PronType=Dem`, `Degree=Pos\|Number=Plur\|POS=ADJ`, `Definite=Ind\|Gender=Com\|Number=Plur\|POS=NOUN`, `POS=PUNCT`, `POS=CCONJ`, `Definite=Ind\|Degree=Cmp\|Number=Sing\|POS=ADJ`, `Degree=Cmp\|POS=ADJ`, `POS=PRON\|PartType=Inf`, `Gender=Com\|Number=Sing\|POS=DET\|PronType=Ind`, `Definite=Ind\|Degree=Pos\|Number=Sing\|POS=ADJ`, `Case=Acc\|Gender=Neut\|Number=Sing\|POS=PRON\|Person=3\|PronType=Prs`, `Definite=Ind\|Gender=Neut\|Number=Plur\|POS=NOUN`, `Definite=Def\|Degree=Pos\|Number=Sing\|POS=ADJ`, `Gender=Neut\|Number=Sing\|POS=DET\|PronType=Dem`, `Degree=Pos\|POS=ADV`, `Definite=Def\|Number=Sing\|POS=VERB\|Tense=Past\|VerbForm=Part`, `Definite=Ind\|Gender=Neut\|Number=Sing\|POS=NOUN`, `POS=PRON\|PronType=Dem`, `NumType=Card\|POS=NUM`, `Definite=Ind\|Degree=Pos\|Gender=Neut\|Number=Sing\|POS=ADJ`, `Case=Acc\|Gender=Com\|Number=Sing\|POS=PRON\|Person=3\|PronType=Prs`, `Degree=Pos\|Gender=Com\|Number=Sing\|POS=ADJ`, `Case=Nom\|Gender=Com\|Number=Sing\|POS=PRON\|Person=3\|PronType=Prs`, `NumType=Ord\|POS=ADJ`, `Gender=Com\|Number=Sing\|Number[psor]=Sing\|POS=DET\|Person=3\|Poss=Yes\|PronType=Prs\|Reflex=Yes`, `Mood=Ind\|POS=AUX\|Tense=Past\|VerbForm=Fin\|Voice=Act`, `POS=VERB\|VerbForm=Inf\|Voice=Act`, `Mood=Ind\|POS=VERB\|Tense=Past\|VerbForm=Fin\|Voice=Act`, `POS=NOUN`, `Mood=Ind\|POS=VERB\|Tense=Pres\|VerbForm=Fin\|Voice=Pass`, `POS=ADP\|PartType=Inf`, `Degree=Pos\|POS=ADJ`, `Definite=Def\|Gender=Com\|Number=Plur\|POS=NOUN`, `Number[psor]=Sing\|POS=DET\|Person=3\|Poss=Yes\|PronType=Prs`, `Case=Gen\|Definite=Def\|Gender=Com\|Number=Sing\|POS=NOUN`, `POS=AUX\|VerbForm=Inf\|Voice=Act`, `Definite=Ind\|Degree=Pos\|Gender=Com\|Number=Sing\|POS=ADJ`, `Gender=Com\|Number=Sing\|POS=DET\|PronType=Dem`, `Number=Plur\|POS=DET\|PronType=Ind`, `Gender=Com\|Number=Sing\|POS=PRON\|PronType=Ind`, `Case=Acc\|POS=PRON\|Person=3\|PronType=Prs\|Reflex=Yes`, `POS=PART\|PartType=Inf`, `Gender=Neut\|Number=Sing\|POS=DET\|PronType=Ind`, `Case=Acc\|Number=Plur\|POS=PRON\|Person=3\|PronType=Prs`, `Case=Gen\|Definite=Def\|Gender=Neut\|Number=Sing\|POS=NOUN`, `Case=Nom\|Number=Plur\|POS=PRON\|Person=3\|PronType=Prs`, `Case=Nom\|Gender=Com\|Number=Sing\|POS=PRON\|Person=1\|PronType=Prs`, `Case=Nom\|Gender=Com\|POS=PRON\|PronType=Ind`, `Gender=Neut\|Number=Sing\|POS=PRON\|PronType=Ind`, `Mood=Imp\|POS=VERB`, `Gender=Com\|Number=Sing\|Number[psor]=Sing\|POS=DET\|Person=1\|Poss=Yes\|PronType=Prs`, `Definite=Ind\|Number=Sing\|POS=AUX\|Tense=Past\|VerbForm=Part`, `POS=X`, `Case=Nom\|Gender=Com\|Number=Plur\|POS=PRON\|Person=1\|PronType=Prs`, `Case=Gen\|Definite=Def\|Gender=Com\|Number=Plur\|POS=NOUN`, `POS=VERB\|Tense=Pres\|VerbForm=Part`, `Number=Plur\|POS=PRON\|PronType=Int,Rel`, `POS=VERB\|VerbForm=Inf\|Voice=Pass`, `Case=Gen\|Definite=Ind\|Gender=Com\|Number=Sing\|POS=NOUN`, `Degree=Cmp\|POS=ADV`, `POS=ADV\|PartType=Inf`, `Degree=Sup\|POS=ADV`, `Number=Plur\|POS=PRON\|PronType=Dem`, `Number=Plur\|POS=PRON\|PronType=Ind`, `Definite=Def\|Gender=Neut\|Number=Plur\|POS=NOUN`, `Case=Acc\|Gender=Com\|Number=Sing\|POS=PRON\|Person=1\|PronType=Prs`, `Case=Gen\|POS=PROPN`, `POS=ADP`, `Degree=Cmp\|Number=Plur\|POS=ADJ`, `Definite=Def\|Degree=Sup\|POS=ADJ`, `Gender=Neut\|Number=Sing\|Number[psor]=Sing\|POS=DET\|Person=1\|Poss=Yes\|PronType=Prs`, `Degree=Pos\|Number=Sing\|POS=ADJ`, `Number=Plur\|Number[psor]=Sing\|POS=DET\|Person=3\|Poss=Yes\|PronType=Prs\|Reflex=Yes`, `Gender=Com\|Number=Sing\|Number[psor]=Plur\|POS=DET\|Person=1\|Poss=Yes\|PronType=Prs\|Style=Form`, `Number=Plur\|POS=PRON\|PronType=Rcp`, `Case=Gen\|Degree=Cmp\|POS=ADJ`, `Case=Gen\|Definite=Def\|Gender=Neut\|Number=Plur\|POS=NOUN`, `Number[psor]=Plur\|POS=DET\|Person=3\|Poss=Yes\|PronType=Prs`, `POS=INTJ`, `Number=Plur\|Number[psor]=Sing\|POS=DET\|Person=1\|Poss=Yes\|PronType=Prs`, `Degree=Pos\|Gender=Neut\|Number=Sing\|POS=ADJ`, `Gender=Neut\|Number=Sing\|Number[psor]=Plur\|POS=DET\|Person=1\|Poss=Yes\|PronType=Prs\|Style=Form`, `Case=Acc\|Gender=Com\|Number=Sing\|POS=PRON\|Person=2\|PronType=Prs`, `Gender=Com\|Number=Sing\|Number[psor]=Sing\|POS=DET\|Person=2\|Poss=Yes\|PronType=Prs`, `Case=Gen\|Definite=Ind\|Gender=Neut\|Number=Plur\|POS=NOUN`, `Number=Sing\|POS=PRON\|PronType=Int,Rel`, `Number=Plur\|Number[psor]=Plur\|POS=DET\|Person=1\|Poss=Yes\|PronType=Prs\|Style=Form`, `Gender=Neut\|Number=Sing\|POS=PRON\|PronType=Int,Rel`, `Definite=Def\|Degree=Sup\|Number=Plur\|POS=ADJ`, `Case=Nom\|Gender=Com\|Number=Sing\|POS=PRON\|Person=2\|PronType=Prs`, `Gender=Neut\|Number=Sing\|Number[psor]=Sing\|POS=DET\|Person=3\|Poss=Yes\|PronType=Prs\|Reflex=Yes`, `Definite=Ind\|Number=Sing\|POS=NOUN`, `Number=Plur\|POS=VERB\|Tense=Past\|VerbForm=Part`, `Number=Plur\|Number[psor]=Sing\|POS=PRON\|Person=3\|Poss=Yes\|PronType=Prs\|Reflex=Yes`, `POS=SYM`, `Case=Nom\|Gender=Com\|POS=PRON\|Person=2\|Polite=Form\|PronType=Prs`, `Degree=Sup\|POS=ADJ`, `Number=Plur\|POS=DET\|PronType=Ind\|Style=Arch`, `Case=Gen\|Gender=Com\|Number=Sing\|POS=DET\|PronType=Dem`, `Foreign=Yes\|POS=X`, `POS=DET\|Person=2\|Polite=Form\|Poss=Yes\|PronType=Prs`, `Gender=Neut\|Number=Sing\|POS=PRON\|PronType=Dem`, `Case=Acc\|Gender=Com\|Number=Plur\|POS=PRON\|Person=1\|PronType=Prs`, `Case=Gen\|Definite=Ind\|Gender=Neut\|Number=Sing\|POS=NOUN`, `Case=Gen\|POS=PRON\|PronType=Int,Rel`, `Gender=Com\|Number=Sing\|POS=PRON\|PronType=Dem`, `Abbr=Yes\|POS=X`, `Case=Gen\|Definite=Ind\|Gender=Com\|Number=Plur\|POS=NOUN`, `Definite=Def\|Degree=Abs\|POS=ADJ`, `Definite=Ind\|Degree=Sup\|Number=Sing\|POS=ADJ`, `Definite=Ind\|POS=NOUN`, `Gender=Com\|Number=Plur\|POS=NOUN`, `Number[psor]=Plur\|POS=DET\|Person=1\|Poss=Yes\|PronType=Prs`, `Gender=Com\|POS=PRON\|PronType=Int,Rel`, `Case=Nom\|Gender=Com\|Number=Plur\|POS=PRON\|Person=2\|PronType=Prs`, `Degree=Abs\|POS=ADV`, `POS=VERB\|VerbForm=Ger`, `POS=VERB\|Tense=Past\|VerbForm=Part`, `Definite=Def\|Degree=Sup\|Number=Sing\|POS=ADJ`, `Number=Plur\|Number[psor]=Plur\|POS=PRON\|Person=1\|Poss=Yes\|PronType=Prs\|Style=Form`, `Case=Gen\|Definite=Def\|Degree=Pos\|Number=Sing\|POS=ADJ`, `Case=Gen\|Degree=Pos\|Number=Plur\|POS=ADJ`, `Case=Acc\|Gender=Com\|POS=PRON\|Person=2\|Polite=Form\|PronType=Prs`, `Gender=Com\|Number=Sing\|POS=PRON\|PronType=Int,Rel`, `POS=VERB\|Tense=Pres`, `Case=Gen\|Number=Plur\|POS=DET\|PronType=Ind`, `Number[psor]=Plur\|POS=DET\|Person=2\|Poss=Yes\|PronType=Prs`, `POS=PRON\|Person=2\|Polite=Form\|Poss=Yes\|PronType=Prs`, `Gender=Neut\|Number=Sing\|Number[psor]=Sing\|POS=DET\|Person=2\|Poss=Yes\|PronType=Prs`, `POS=AUX\|Tense=Pres\|VerbForm=Part`, `Mood=Ind\|POS=VERB\|Tense=Past\|VerbForm=Fin\|Voice=Pass`, `Gender=Com\|Number=Sing\|Number[psor]=Sing\|POS=PRON\|Person=3\|Poss=Yes\|PronType=Prs\|Reflex=Yes`, `Degree=Sup\|Number=Plur\|POS=ADJ`, `Case=Acc\|Gender=Com\|Number=Plur\|POS=PRON\|Person=2\|PronType=Prs`, `Gender=Neut\|Number=Sing\|Number[psor]=Sing\|POS=PRON\|Person=3\|Poss=Yes\|PronType=Prs\|Reflex=Yes`, `Definite=Ind\|Number=Plur\|POS=NOUN`, `Case=Gen\|Number=Plur\|POS=VERB\|Tense=Past\|VerbForm=Part`, `Mood=Imp\|POS=AUX`, `Gender=Com\|Number=Sing\|Number[psor]=Sing\|POS=PRON\|Person=1\|Poss=Yes\|PronType=Prs`, `Number[psor]=Sing\|POS=PRON\|Person=3\|Poss=Yes\|PronType=Prs`, `Definite=Def\|Gender=Com\|Number=Sing\|POS=VERB\|Tense=Past\|VerbForm=Part`, `Number=Plur\|Number[psor]=Sing\|POS=DET\|Person=2\|Poss=Yes\|PronType=Prs`, `Case=Gen\|Gender=Com\|Number=Sing\|POS=DET\|PronType=Ind`, `Case=Gen\|POS=NOUN`, `Number[psor]=Plur\|POS=PRON\|Person=3\|Poss=Yes\|PronType=Prs`, `POS=DET\|PronType=Dem`, `Definite=Def\|Number=Plur\|POS=NOUN` |
| **`parser`** | `ROOT`, `acl:relcl`, `advcl`, `advmod`, `amod`, `appos`, `aux`, `case`, `cc`, `ccomp`, `compound:prt`, `conj`, `cop`, `dep`, `det`, `expl`, `fixed`, `flat`, `iobj`, `list`, `mark`, `nmod`, `nmod:poss`, `nsubj`, `nummod`, `obj`, `obl`, `obl:loc`, `obl:tmod`, `punct`, `xcomp` |
| **`ner`** | `LOC`, `MISC`, `ORG`, `PER` |


</details>


### Accuracy


| Type | Score |
| --- | --- |
| `POS_ACC` | 98.70 |
| `MORPH_ACC` | 98.49 |
| `DEP_UAS` | 90.75 |
| `DEP_LAS` | 88.38 |
| `SENTS_P` | 96.09 |
| `SENTS_R` | 95.74 |
| `SENTS_F` | 95.91 |
| `LEMMA_ACC` | 84.91 |
| `ENTS_F` | 90.12 |
| `ENTS_P` | 89.02 |
| `ENTS_R` | 91.25 |
| `TRANSFORMER_LOSS` | 1805626.49 |
| `MORPHOLOGIZER_LOSS` | 111735.86 |
| `PARSER_LOSS` | 8037491.27 |
| `NER_LOSS` | 16634.46 |


## Bias and Robustness


Besides the validation done by SpaCy on the DaNE testset, DaCy also provides a series of augmentations to the DaNE test set to see how well the models deal with these types of augmentations.
The can be seen as behavioural probes akinn to the NLP checklist.


### Deterministic Augmentations


Deterministic augmentations are augmentation which always yield the same result.


| Augmentation | Part-of-speech tagging (Accuracy) | Morphological tagging (Accuracy) | Dependency Parsing (UAS) | Dependency Parsing (LAS) | Sentence segmentation (F1) | Lemmatization (Accuracy) | Named entity recognition (F1) |
| --- | --- | --- | --- |  --- | --- | --- |  --- |
| No augmentation | 0.985 | 0.979 | 0.906 |  0.881 | 0.986 | 0.844 | 0.839 |
| Æøå Augmentation | 0.973 | 0.963 | 0.892 |  0.863 | 0.975 | 0.754 | 0.815 |
| Lowercase | 0.981 | 0.975 | 0.902 |  0.876 | 0.93 | 0.848 | 0.788 |
| No Spacing | 0.227 | 0.229 | 0.004 |  0.004 | 0.54 | 0.225 | 0.086 |
| Abbreviated first names | 0.984 | 0.978 | 0.903 |  0.878 | 0.986 | 0.845 | 0.839 |
| Input size augmentation 5 sentences | 0.986 | 0.981 | 0.904 |  0.88 | 0.97 | 0.844 | 0.847 |
| Input size augmentation 10 sentences | 0.986 | 0.981 | 0.905 |  0.881 | 0.964 | 0.844 | 0.849 |


### Stochastic Augmentations


Stochastic augmentations are augmentation which are repeated mulitple times to estimate the effect of the augmentation.


| Augmentation | Part-of-speech tagging (Accuracy) | Morphological tagging (Accuracy) | Dependency Parsing (UAS) | Dependency Parsing (LAS) | Sentence segmentation (F1) | Lemmatization (Accuracy) | Named entity recognition (F1) |
| --- | --- | --- | --- |  --- | --- | --- |  --- |
| Keystroke errors 2% | 0.949 (0.002) | 0.944 (0.002) | 0.868 (0.002) |  0.833 (0.002) | 0.965 (0.002) | 0.773 (0.002) | 0.775 (0.002) |
| Keystroke errors 5% | 0.895 (0.003) | 0.893 (0.003) | 0.81 (0.003) |  0.76 (0.003) | 0.92 (0.003) | 0.68 (0.003) | 0.698 (0.003) |
| Keystroke errors 15% | 0.705 (0.005) | 0.72 (0.005) | 0.6 (0.005) |  0.518 (0.005) | 0.801 (0.005) | 0.462 (0.005) | 0.506 (0.005) |
| Danish names | 0.984 (0.0) | 0.979 (0.0) | 0.904 (0.0) |  0.879 (0.0) | 0.987 (0.0) | 0.847 (0.0) | 0.844 (0.0) |
| Muslim names | 0.984 (0.0) | 0.979 (0.0) | 0.904 (0.0) |  0.879 (0.0) | 0.987 (0.0) | 0.847 (0.0) | 0.844 (0.0) |
| Female names | 0.984 (0.0) | 0.979 (0.0) | 0.904 (0.0) |  0.879 (0.0) | 0.986 (0.0) | 0.847 (0.0) | 0.846 (0.0) |
| Male names | 0.984 (0.0) | 0.979 (0.0) | 0.904 (0.0) |  0.879 (0.0) | 0.986 (0.0) | 0.846 (0.0) | 0.845 (0.0) |
| Spacing Augmention 5% | 0.946 (0.002) | 0.941 (0.002) | 0.794 (0.002) |  0.771 (0.002) | 0.969 (0.002) | 0.812 (0.002) | 0.781 (0.002) |


<summary> Description of Augmenters </summary>


**No augmentation:**
Applies no augmentation to the DaNE test set.


**Æøå Augmentation:**
This augmentation replace the æ,ø, and å with their spelling variations ae, oe and aa respectively.


**Lowercase:**
This augmentation lowercases all text.


**No Spacing:**
This augmentation removed all spacing from the text.


**Abbreviated first names:**
This agmentation abbreviates the first names of entities. For instance 'Kenneth Enevoldsen' would turn to 'K. Enevoldsen'.


**Keystroke errors 2%:**
This agmentation simulate keystroke errors by replacing 2% of keys with a neighbouring key on a Danish QWERTY keyboard. As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


**Keystroke errors 5%:**
This agmentation simulate keystroke errors by replacing 5% of keys with a neighbouring key on a Danish QWERTY keyboard. As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


**Keystroke errors 15%:**
This agmentation simulate keystroke errors by replacing 15% of keys with a neighbouring key on a Danish QWERTY keyboard. As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


**Danish names:**
This agmentation replace all names with Danish names derived from Danmarks Statistik (2021). As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


**Muslim names:**
This agmentation replace all names with Muslim names derived from  Meldgaard (2005). As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


**Female names:**
This agmentation replace all names with Danish female names derived from Danmarks Statistik (2021). As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


**Male names:**
This agmentation replace all names with Danish male names derived from Danmarks Statistik (2021). As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


**Spacing Augmention 5%:**
This agmentation replace all names with Danish male names derived from Danmarks Statistik (2021). As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


 </details>
 <br />


### Hardware


This was run and trained on a Quadro RTX 8000 GPU.


model name: da_dacy_large_trf


model author: chcaa


task: token-classification


tags: da, dataset:dane, spacy, token-classification, license:apache-2.0, model-index


tags:


- spacy
- token-classification
language:
- da
license: apache-2.0
model-index:
- name: da_dacy_medium_trf
results:
  - task:
name: NER
type: token-classification
metrics:
    - name: NER Precision
type: precision
value: 0.817047817
    - name: NER Recall
type: recall
value: 0.81875
    - name: NER F Score
type: f_score
value: 0.8178980229
  - task:
name: SENTER
type: token-classification
metrics:
    - name: SENTER Precision
type: precision
value: 0.873015873
    - name: SENTER Recall
type: recall
value: 0.8776595745
    - name: SENTER F Score
type: f_score
value: 0.875331565
  - task:
name: UNLABELED_DEPENDENCIES
type: token-classification
metrics:
    - name: Unlabeled Dependencies Accuracy
type: accuracy
value: 0.8714971531
  - task:
name: LABELED_DEPENDENCIES
type: token-classification
metrics:
    - name: Labeled Dependencies Accuracy
type: accuracy
value: 0.8714971531


# DaCy medium transformer


DaCy is a Danish language processing framework with state-of-the-art pipelines as well as functionality for analysing Danish pipelines.
DaCy's largest pipeline has achieved State-of-the-Art performance on Named entity recognition, part-of-speech tagging and dependency
parsing for Danish on the DaNE dataset. Check out the [DaCy repository](https://github.com/centre-for-humanities-computing/DaCy) for material on how to use DaCy and reproduce the results.
DaCy also contains guides on usage of the package as well as behavioural test for biases and robustness of Danish NLP pipelines.


| Feature | Description |
| --- | --- |
| **Name** | `da_dacy_medium_trf` |
| **Version** | `0.1.0` |
| **spaCy** | `>=3.1.1,<3.2.0` |
| **Default Pipeline** | `transformer`, `morphologizer`, `parser`, `attribute_ruler`, `lemmatizer`, `ner` |
| **Components** | `transformer`, `morphologizer`, `parser`, `attribute_ruler`, `lemmatizer`, `ner` |
| **Vectors** | 0 keys, 0 unique vectors (0 dimensions) |
| **Sources** | [UD Danish DDT v2.5](https://github.com/UniversalDependencies/UD_Danish-DDT) (Johannsen, Anders; Martínez Alonso, Héctor; Plank, Barbara)
[DaNE](https://github.com/alexandrainst/danlp/blob/master/docs/datasets.md#danish-dependency-treebank-dane) (Rasmus Hvingelby, Amalie B. Pauli, Maria Barrett, Christina Rosted, Lasse M. Lidegaard, Anders Søgaard)
[Maltehb/danish-bert-botxo](https://huggingface.co/Maltehb/danish-bert-botxo) (BotXO.ai) |
| **License** | `Apache-2.0 License` |
| **Author** | [Centre for Humanities Computing Aarhus](https://chcaa.io/#/) |


### Label Scheme


<details>


<summary>View label scheme (192 labels for 3 components)</summary>


| Component | Labels |
| --- | --- |
| **`morphologizer`** | `AdpType=Prep\|POS=ADP`, `Definite=Ind\|Gender=Com\|Number=Sing\|POS=NOUN`, `Mood=Ind\|POS=AUX\|Tense=Pres\|VerbForm=Fin\|Voice=Act`, `POS=PROPN`, `Definite=Ind\|Number=Sing\|POS=VERB\|Tense=Past\|VerbForm=Part`, `Definite=Def\|Gender=Neut\|Number=Sing\|POS=NOUN`, `POS=SCONJ`, `Definite=Def\|Gender=Com\|Number=Sing\|POS=NOUN`, `Mood=Ind\|POS=VERB\|Tense=Pres\|VerbForm=Fin\|Voice=Act`, `POS=ADV`, `Number=Plur\|POS=DET\|PronType=Dem`, `Degree=Pos\|Number=Plur\|POS=ADJ`, `Definite=Ind\|Gender=Com\|Number=Plur\|POS=NOUN`, `POS=PUNCT`, `POS=CCONJ`, `Definite=Ind\|Degree=Cmp\|Number=Sing\|POS=ADJ`, `Degree=Cmp\|POS=ADJ`, `POS=PRON\|PartType=Inf`, `Gender=Com\|Number=Sing\|POS=DET\|PronType=Ind`, `Definite=Ind\|Degree=Pos\|Number=Sing\|POS=ADJ`, `Case=Acc\|Gender=Neut\|Number=Sing\|POS=PRON\|Person=3\|PronType=Prs`, `Definite=Ind\|Gender=Neut\|Number=Plur\|POS=NOUN`, `Definite=Def\|Degree=Pos\|Number=Sing\|POS=ADJ`, `Gender=Neut\|Number=Sing\|POS=DET\|PronType=Dem`, `Degree=Pos\|POS=ADV`, `Definite=Def\|Number=Sing\|POS=VERB\|Tense=Past\|VerbForm=Part`, `Definite=Ind\|Gender=Neut\|Number=Sing\|POS=NOUN`, `POS=PRON\|PronType=Dem`, `NumType=Card\|POS=NUM`, `Definite=Ind\|Degree=Pos\|Gender=Neut\|Number=Sing\|POS=ADJ`, `Case=Acc\|Gender=Com\|Number=Sing\|POS=PRON\|Person=3\|PronType=Prs`, `Degree=Pos\|Gender=Com\|Number=Sing\|POS=ADJ`, `Case=Nom\|Gender=Com\|Number=Sing\|POS=PRON\|Person=3\|PronType=Prs`, `NumType=Ord\|POS=ADJ`, `Gender=Com\|Number=Sing\|Number[psor]=Sing\|POS=DET\|Person=3\|Poss=Yes\|PronType=Prs\|Reflex=Yes`, `Mood=Ind\|POS=AUX\|Tense=Past\|VerbForm=Fin\|Voice=Act`, `POS=VERB\|VerbForm=Inf\|Voice=Act`, `Mood=Ind\|POS=VERB\|Tense=Past\|VerbForm=Fin\|Voice=Act`, `POS=NOUN`, `Mood=Ind\|POS=VERB\|Tense=Pres\|VerbForm=Fin\|Voice=Pass`, `POS=ADP\|PartType=Inf`, `Degree=Pos\|POS=ADJ`, `Definite=Def\|Gender=Com\|Number=Plur\|POS=NOUN`, `Number[psor]=Sing\|POS=DET\|Person=3\|Poss=Yes\|PronType=Prs`, `Case=Gen\|Definite=Def\|Gender=Com\|Number=Sing\|POS=NOUN`, `POS=AUX\|VerbForm=Inf\|Voice=Act`, `Definite=Ind\|Degree=Pos\|Gender=Com\|Number=Sing\|POS=ADJ`, `Gender=Com\|Number=Sing\|POS=DET\|PronType=Dem`, `Number=Plur\|POS=DET\|PronType=Ind`, `Gender=Com\|Number=Sing\|POS=PRON\|PronType=Ind`, `Case=Acc\|POS=PRON\|Person=3\|PronType=Prs\|Reflex=Yes`, `POS=PART\|PartType=Inf`, `Gender=Neut\|Number=Sing\|POS=DET\|PronType=Ind`, `Case=Acc\|Number=Plur\|POS=PRON\|Person=3\|PronType=Prs`, `Case=Gen\|Definite=Def\|Gender=Neut\|Number=Sing\|POS=NOUN`, `Case=Nom\|Number=Plur\|POS=PRON\|Person=3\|PronType=Prs`, `Case=Nom\|Gender=Com\|Number=Sing\|POS=PRON\|Person=1\|PronType=Prs`, `Case=Nom\|Gender=Com\|POS=PRON\|PronType=Ind`, `Gender=Neut\|Number=Sing\|POS=PRON\|PronType=Ind`, `Mood=Imp\|POS=VERB`, `Gender=Com\|Number=Sing\|Number[psor]=Sing\|POS=DET\|Person=1\|Poss=Yes\|PronType=Prs`, `Definite=Ind\|Number=Sing\|POS=AUX\|Tense=Past\|VerbForm=Part`, `POS=X`, `Case=Nom\|Gender=Com\|Number=Plur\|POS=PRON\|Person=1\|PronType=Prs`, `Case=Gen\|Definite=Def\|Gender=Com\|Number=Plur\|POS=NOUN`, `POS=VERB\|Tense=Pres\|VerbForm=Part`, `Number=Plur\|POS=PRON\|PronType=Int,Rel`, `POS=VERB\|VerbForm=Inf\|Voice=Pass`, `Case=Gen\|Definite=Ind\|Gender=Com\|Number=Sing\|POS=NOUN`, `Degree=Cmp\|POS=ADV`, `POS=ADV\|PartType=Inf`, `Degree=Sup\|POS=ADV`, `Number=Plur\|POS=PRON\|PronType=Dem`, `Number=Plur\|POS=PRON\|PronType=Ind`, `Definite=Def\|Gender=Neut\|Number=Plur\|POS=NOUN`, `Case=Acc\|Gender=Com\|Number=Sing\|POS=PRON\|Person=1\|PronType=Prs`, `Case=Gen\|POS=PROPN`, `POS=ADP`, `Degree=Cmp\|Number=Plur\|POS=ADJ`, `Definite=Def\|Degree=Sup\|POS=ADJ`, `Gender=Neut\|Number=Sing\|Number[psor]=Sing\|POS=DET\|Person=1\|Poss=Yes\|PronType=Prs`, `Degree=Pos\|Number=Sing\|POS=ADJ`, `Number=Plur\|Number[psor]=Sing\|POS=DET\|Person=3\|Poss=Yes\|PronType=Prs\|Reflex=Yes`, `Gender=Com\|Number=Sing\|Number[psor]=Plur\|POS=DET\|Person=1\|Poss=Yes\|PronType=Prs\|Style=Form`, `Number=Plur\|POS=PRON\|PronType=Rcp`, `Case=Gen\|Degree=Cmp\|POS=ADJ`, `Case=Gen\|Definite=Def\|Gender=Neut\|Number=Plur\|POS=NOUN`, `Number[psor]=Plur\|POS=DET\|Person=3\|Poss=Yes\|PronType=Prs`, `POS=INTJ`, `Number=Plur\|Number[psor]=Sing\|POS=DET\|Person=1\|Poss=Yes\|PronType=Prs`, `Degree=Pos\|Gender=Neut\|Number=Sing\|POS=ADJ`, `Gender=Neut\|Number=Sing\|Number[psor]=Plur\|POS=DET\|Person=1\|Poss=Yes\|PronType=Prs\|Style=Form`, `Case=Acc\|Gender=Com\|Number=Sing\|POS=PRON\|Person=2\|PronType=Prs`, `Gender=Com\|Number=Sing\|Number[psor]=Sing\|POS=DET\|Person=2\|Poss=Yes\|PronType=Prs`, `Case=Gen\|Definite=Ind\|Gender=Neut\|Number=Plur\|POS=NOUN`, `Number=Sing\|POS=PRON\|PronType=Int,Rel`, `Number=Plur\|Number[psor]=Plur\|POS=DET\|Person=1\|Poss=Yes\|PronType=Prs\|Style=Form`, `Gender=Neut\|Number=Sing\|POS=PRON\|PronType=Int,Rel`, `Definite=Def\|Degree=Sup\|Number=Plur\|POS=ADJ`, `Case=Nom\|Gender=Com\|Number=Sing\|POS=PRON\|Person=2\|PronType=Prs`, `Gender=Neut\|Number=Sing\|Number[psor]=Sing\|POS=DET\|Person=3\|Poss=Yes\|PronType=Prs\|Reflex=Yes`, `Definite=Ind\|Number=Sing\|POS=NOUN`, `Number=Plur\|POS=VERB\|Tense=Past\|VerbForm=Part`, `Number=Plur\|Number[psor]=Sing\|POS=PRON\|Person=3\|Poss=Yes\|PronType=Prs\|Reflex=Yes`, `POS=SYM`, `Case=Nom\|Gender=Com\|POS=PRON\|Person=2\|Polite=Form\|PronType=Prs`, `Degree=Sup\|POS=ADJ`, `Number=Plur\|POS=DET\|PronType=Ind\|Style=Arch`, `Case=Gen\|Gender=Com\|Number=Sing\|POS=DET\|PronType=Dem`, `Foreign=Yes\|POS=X`, `POS=DET\|Person=2\|Polite=Form\|Poss=Yes\|PronType=Prs`, `Gender=Neut\|Number=Sing\|POS=PRON\|PronType=Dem`, `Case=Acc\|Gender=Com\|Number=Plur\|POS=PRON\|Person=1\|PronType=Prs`, `Case=Gen\|Definite=Ind\|Gender=Neut\|Number=Sing\|POS=NOUN`, `Case=Gen\|POS=PRON\|PronType=Int,Rel`, `Gender=Com\|Number=Sing\|POS=PRON\|PronType=Dem`, `Abbr=Yes\|POS=X`, `Case=Gen\|Definite=Ind\|Gender=Com\|Number=Plur\|POS=NOUN`, `Definite=Def\|Degree=Abs\|POS=ADJ`, `Definite=Ind\|Degree=Sup\|Number=Sing\|POS=ADJ`, `Definite=Ind\|POS=NOUN`, `Gender=Com\|Number=Plur\|POS=NOUN`, `Number[psor]=Plur\|POS=DET\|Person=1\|Poss=Yes\|PronType=Prs`, `Gender=Com\|POS=PRON\|PronType=Int,Rel`, `Case=Nom\|Gender=Com\|Number=Plur\|POS=PRON\|Person=2\|PronType=Prs`, `Degree=Abs\|POS=ADV`, `POS=VERB\|VerbForm=Ger`, `POS=VERB\|Tense=Past\|VerbForm=Part`, `Definite=Def\|Degree=Sup\|Number=Sing\|POS=ADJ`, `Number=Plur\|Number[psor]=Plur\|POS=PRON\|Person=1\|Poss=Yes\|PronType=Prs\|Style=Form`, `Case=Gen\|Definite=Def\|Degree=Pos\|Number=Sing\|POS=ADJ`, `Case=Gen\|Degree=Pos\|Number=Plur\|POS=ADJ`, `Case=Acc\|Gender=Com\|POS=PRON\|Person=2\|Polite=Form\|PronType=Prs`, `Gender=Com\|Number=Sing\|POS=PRON\|PronType=Int,Rel`, `POS=VERB\|Tense=Pres`, `Case=Gen\|Number=Plur\|POS=DET\|PronType=Ind`, `Number[psor]=Plur\|POS=DET\|Person=2\|Poss=Yes\|PronType=Prs`, `POS=PRON\|Person=2\|Polite=Form\|Poss=Yes\|PronType=Prs`, `Gender=Neut\|Number=Sing\|Number[psor]=Sing\|POS=DET\|Person=2\|Poss=Yes\|PronType=Prs`, `POS=AUX\|Tense=Pres\|VerbForm=Part`, `Mood=Ind\|POS=VERB\|Tense=Past\|VerbForm=Fin\|Voice=Pass`, `Gender=Com\|Number=Sing\|Number[psor]=Sing\|POS=PRON\|Person=3\|Poss=Yes\|PronType=Prs\|Reflex=Yes`, `Degree=Sup\|Number=Plur\|POS=ADJ`, `Case=Acc\|Gender=Com\|Number=Plur\|POS=PRON\|Person=2\|PronType=Prs`, `Gender=Neut\|Number=Sing\|Number[psor]=Sing\|POS=PRON\|Person=3\|Poss=Yes\|PronType=Prs\|Reflex=Yes`, `Definite=Ind\|Number=Plur\|POS=NOUN`, `Case=Gen\|Number=Plur\|POS=VERB\|Tense=Past\|VerbForm=Part`, `Mood=Imp\|POS=AUX`, `Gender=Com\|Number=Sing\|Number[psor]=Sing\|POS=PRON\|Person=1\|Poss=Yes\|PronType=Prs`, `Number[psor]=Sing\|POS=PRON\|Person=3\|Poss=Yes\|PronType=Prs`, `Definite=Def\|Gender=Com\|Number=Sing\|POS=VERB\|Tense=Past\|VerbForm=Part`, `Number=Plur\|Number[psor]=Sing\|POS=DET\|Person=2\|Poss=Yes\|PronType=Prs`, `Case=Gen\|Gender=Com\|Number=Sing\|POS=DET\|PronType=Ind`, `Case=Gen\|POS=NOUN`, `Number[psor]=Plur\|POS=PRON\|Person=3\|Poss=Yes\|PronType=Prs`, `POS=DET\|PronType=Dem`, `Definite=Def\|Number=Plur\|POS=NOUN` |
| **`parser`** | `ROOT`, `acl:relcl`, `advcl`, `advmod`, `amod`, `appos`, `aux`, `case`, `cc`, `ccomp`, `compound:prt`, `conj`, `cop`, `dep`, `det`, `expl`, `fixed`, `flat`, `iobj`, `list`, `mark`, `nmod`, `nmod:poss`, `nsubj`, `nummod`, `obj`, `obl`, `obl:loc`, `obl:tmod`, `punct`, `xcomp` |
| **`ner`** | `LOC`, `MISC`, `ORG`, `PER` |


</details>


### Accuracy


| Type | Score |
| --- | --- |
| `POS_ACC` | 97.44 |
| `MORPH_ACC` | 97.24 |
| `DEP_UAS` | 87.15 |
| `DEP_LAS` | 83.97 |
| `SENTS_P` | 87.30 |
| `SENTS_R` | 87.77 |
| `SENTS_F` | 87.53 |
| `LEMMA_ACC` | 84.91 |
| `ENTS_F` | 81.79 |
| `ENTS_P` | 81.70 |
| `ENTS_R` | 81.88 |
| `TRANSFORMER_LOSS` | 1224302.39 |
| `MORPHOLOGIZER_LOSS` | 388869.90 |
| `PARSER_LOSS` | 7861802.70 |
| `NER_LOSS` | 68503.20 |


## Bias and Robustness


Besides the validation done by SpaCy on the DaNE testset, DaCy also provides a series of augmentations to the DaNE test set to see how well the models deal with these types of augmentations.
The can be seen as behavioural probes akinn to the NLP checklist.


### Deterministic Augmentations


Deterministic augmentations are augmentation which always yield the same result.


| Augmentation | Part-of-speech tagging (Accuracy) | Morphological tagging (Accuracy) | Dependency Parsing (UAS) | Dependency Parsing (LAS) | Sentence segmentation (F1) | Lemmatization (Accuracy) | Named entity recognition (F1) |
| --- | --- | --- | --- |  --- | --- | --- |  --- |
| No augmentation | 0.98 | 0.975 | 0.888 |  0.857 | 0.936 | 0.844 | 0.765 |
| Æøå Augmentation | 0.963 | 0.955 | 0.88 |  0.844 | 0.944 | 0.754 | 0.712 |
| Lowercase | 0.98 | 0.975 | 0.888 |  0.857 | 0.936 | 0.848 | 0.765 |
| No Spacing | 0.229 | 0.229 | 0.004 |  0.004 | 0.683 | 0.225 | 0.058 |
| Abbreviated first names | 0.976 | 0.974 | 0.885 |  0.854 | 0.934 | 0.845 | 0.741 |
| Input size augmentation 5 sentences | 0.978 | 0.973 | 0.88 |  0.85 | 0.883 | 0.844 | 0.77 |
| Input size augmentation 10 sentences | 0.977 | 0.973 | 0.878 |  0.847 | 0.872 | 0.844 | 0.768 |


### Stochastic Augmentations


Stochastic augmentations are augmentation which are repeated mulitple times to estimate the effect of the augmentation.


| Augmentation | Part-of-speech tagging (Accuracy) | Morphological tagging (Accuracy) | Dependency Parsing (UAS) | Dependency Parsing (LAS) | Sentence segmentation (F1) | Lemmatization (Accuracy) | Named entity recognition (F1) |
| --- | --- | --- | --- |  --- | --- | --- |  --- |
| Keystroke errors 2% | 0.936 (0.002) | 0.934 (0.002) | 0.836 (0.002) |  0.795 (0.002) | 0.889 (0.002) | 0.773 (0.002) | 0.627 (0.002) |
| Keystroke errors 5% | 0.869 (0.003) | 0.873 (0.003) | 0.753 (0.003) |  0.696 (0.003) | 0.829 (0.003) | 0.68 (0.003) | 0.487 (0.003) |
| Keystroke errors 15% | 0.647 (0.007) | 0.684 (0.007) | 0.5 (0.007) |  0.417 (0.007) | 0.664 (0.007) | 0.46 (0.007) | 0.256 (0.007) |
| Danish names | 0.978 (0.0) | 0.975 (0.0) | 0.885 (0.0) |  0.855 (0.0) | 0.934 (0.0) | 0.847 (0.0) | 0.771 (0.0) |
| Muslim names | 0.978 (0.0) | 0.975 (0.0) | 0.886 (0.0) |  0.855 (0.0) | 0.935 (0.0) | 0.847 (0.0) | 0.749 (0.0) |
| Female names | 0.979 (0.0) | 0.975 (0.0) | 0.886 (0.0) |  0.856 (0.0) | 0.933 (0.0) | 0.847 (0.0) | 0.775 (0.0) |
| Male names | 0.978 (0.0) | 0.975 (0.0) | 0.885 (0.0) |  0.855 (0.0) | 0.933 (0.0) | 0.847 (0.0) | 0.773 (0.0) |
| Spacing Augmention 5% | 0.941 (0.002) | 0.937 (0.002) | 0.78 (0.002) |  0.751 (0.002) | 0.905 (0.002) | 0.812 (0.002) | 0.701 (0.002) |


<summary> Description of Augmenters </summary>


**No augmentation:**
Applies no augmentation to the DaNE test set.


**Æøå Augmentation:**
This augmentation replace the æ,ø, and å with their spelling variations ae, oe and aa respectively.


**Lowercase:**
This augmentation lowercases all text.


**No Spacing:**
This augmentation removed all spacing from the text.


**Abbreviated first names:**
This agmentation abbreviates the first names of entities. For instance 'Kenneth Enevoldsen' would turn to 'K. Enevoldsen'.


**Keystroke errors 2%:**
This agmentation simulate keystroke errors by replacing 2% of keys with a neighbouring key on a Danish QWERTY keyboard. As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


**Keystroke errors 5%:**
This agmentation simulate keystroke errors by replacing 5% of keys with a neighbouring key on a Danish QWERTY keyboard. As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


**Keystroke errors 15%:**
This agmentation simulate keystroke errors by replacing 15% of keys with a neighbouring key on a Danish QWERTY keyboard. As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


**Danish names:**
This agmentation replace all names with Danish names derived from Danmarks Statistik (2021). As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


**Muslim names:**
This agmentation replace all names with Muslim names derived from  Meldgaard (2005). As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


**Female names:**
This agmentation replace all names with Danish female names derived from Danmarks Statistik (2021). As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


**Male names:**
This agmentation replace all names with Danish male names derived from Danmarks Statistik (2021). As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


**Spacing Augmention 5%:**
This agmentation replace all names with Danish male names derived from Danmarks Statistik (2021). As this agmentation is stochastic it is repeated 20 times to obtain a consistent estimate and the mean is provided with its standard deviation in parenthesis.


 </details>
 <br />


### Hardware


This was run an trained on a Quadro RTX 8000 GPU.


model name: da_dacy_medium_trf


model author: chcaa


task: token-classification


tags: da, spacy, token-classification, license:apache-2.0, model-index


tags:


- spacy
- token-classification
language:
- da
license: apache-2.0
model-index:
- name: da_dacy_small_trf
results:
  - task:
name: NER
type: token-classification
metrics:
    - name: NER Precision
type: precision
value: 0.81724846
    - name: NER Recall
type: recall
value: 0.8291666667
    - name: NER F Score
type: f_score
value: 0.8231644261
  - task:
name: SENTER
type: token-classification
metrics:
    - name: SENTER Precision
type: precision
value: 0.8603839442
    - name: SENTER Recall
type: recall
value: 0.8741134752
    - name: SENTER F Score
type: f_score
value: 0.8671943712
  - task:
name: UNLABELED_DEPENDENCIES
type: token-classification
metrics:
    - name: Unlabeled Dependencies Accuracy
type: accuracy
value: 0.8492442546
  - task:
name: LABELED_DEPENDENCIES
type: token-classification
metrics:
    - name: Labeled Dependencies Accuracy
type: accuracy
value: 0.8492442546


# DaCy small transformer


DaCy is a Danish language processing framework with state-of-the-art pipelines as well as functionality for analysing Danish pipelines.
DaCy's largest pipeline has achieved State-of-the-Art performance on Named entity recognition, part-of-speech tagging and dependency
parsing for Danish on the DaNE dataset. Check out the [DaCy repository](https://github.com/centre-for-humanities-computing/DaCy) for material on how to use DaCy and reproduce the results.
DaCy also contains guides on usage of the package as well as behavioural test for biases and robustness of Danish NLP pipelines.


| Feature | Description |
| --- | --- |
| **Name** | `da_dacy_small_trf` |
| **Version** | `0.1.0` |
| **spaCy** | `>=3.1.1,<3.2.0` |
| **Default Pipeline** | `transformer`, `morphologizer`, `parser`, `attribute_ruler`, `lemmatizer`, `ner` |
| **Components** | `transformer`, `morphologizer`, `parser`, `attribute_ruler`, `lemmatizer`, `ner` |
| **Vectors** | 0 keys, 0 unique vectors (0 dimensions) |
| **Sources** | [UD Danish DDT v2.5](https://github.com/UniversalDependencies/UD_Danish-DDT) (Johannsen, Anders; Martínez Alonso, Héctor; Plank, Barbara)
[DaNE](https://github.com/alexandrainst/danlp/blob/master/docs/datasets.md#danish-dependency-treebank-dane) (Rasmus Hvingelby, Amalie B. Pauli, Maria Barrett, Christina Rosted, Lasse M. Lidegaard, Anders Søgaard)
[Maltehb/-l-ctra-danish-electra-small-cased](https://huggingface.co/Maltehb/-l-ctra-danish-electra-small-cased) (Malte Højmark-Bertelsen) |
| **License** | `Apache-2.0 License` |
| **Author** | [Centre for Humanities Computing Aarhus](https://chcaa.io/#/) |


### Label Scheme


<details>


<summary>View label scheme (192 labels for 3 components)</summary>