ltg/mt0-definition-no-xl

mT0-Definition-No XL

This model is a version of mT0 XL finetuned on Bokmålsordboka, a dataset of Norwegian definitions and usage examples.

It generates definitions of Norwegian words in context. Its input is the usage example and the instruction question "Hva betyr TARGET_WORD?"

Models for other languages:

• English: mT0-Definition-En XL
• Norwegian: mT0-Definition-No XL
• Russian: mT0-Definition-Ru XL

Model description

See details in the paper Enriching Word Usage Graphs with Cluster Definitions (LREC-COLING'2024) by Mariia Fedorova, Andrey Kutuzov, Nikolay Arefyev and Dominik Schlechtweg.

Intended uses & limitations

The model is intended for research purposes, as a source of contextualized dictionary-like lexical definitions. Generated definitions can contain all sorts of biases and stereotypes, stemming from the underlying language model.

Training and evaluation data

Bokmålsordboka by The Norwegian Language Council and the University of Bergen.

Training results

mT0-Definition-No XL achieves the following results on the evaluation set:

• Loss: 2.0358
• Rouge1: 28.3491
• Rouge2: 14.2699
• Rougel: 27.7602
• Rougelsum: 27.752
• Gen Len: 10.0765

Training procedure

mT0-Definition-No XL was fine-tuned in a sequence-to-sequence mode on examples of contextualized dictionary definitions.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 4
• total_train_batch_size: 128
• total_eval_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 20.0

Framework versions

• Transformers 4.37.1
• Pytorch 1.13.1+rocm5.2
• Datasets 2.16.1
• Tokenizers 0.15.1

Citation