CNEC_xlm-roberta-large

This model is a fine-tuned version of FacebookAI/xlm-roberta-large on the cnec dataset. It achieves the following results on the evaluation set:

• Loss: 0.1471
• Precision: 0.8567
• Recall: 0.9047
• F1: 0.8800
• Accuracy: 0.9772

Model description

The entities are described as:

• 'O' = Outside of a named entity
• 'B-A' = Beginning of a complex address number (Postal code, street number, even phone number)
• 'I-A' = Inside of a number in the address
• 'B-G' = Beginning of a geographical name
• 'I-G' = Inside of a geographical name
• 'B-I' = Beginning of an institution name
• 'I-I' = Inside of an institution name
• 'B-M' = Beginning of a media name (email, server, website, tv series, etc.)
• 'I-M' = Inside of a media name
• 'B-O' = Beginning of an artifact name (book, old movies, etc.)
• 'I-O' = Inside of an artifact name
• 'B-P' = Beginning of a person's name
• 'I-P' = Inside of a person's name
• 'B-T' = Beginning of a time expression
• 'I-T' = Inside of a time expression

Intended uses & limitations

CNEC or Czech named entity corpus is a dataset aimed at the Czech language. This is an edited version of the dataset with only 7 supertypes and 1 type for non-entity.

Training and evaluation data

The model was trained with an increased dropout rate to 0.2 for hidden_dropout_prob and 0.15 for attention_probs_dropout_prob

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• weight_decay = 0.01
• num_epochs: 10

Training results

Framework versions

• Transformers 4.36.2
• Pytorch 2.1.2+cu121
• Datasets 2.16.1
• Tokenizers 0.15.0