RoBERTa-kaz-large

Model Description

roberta-kaz-large is a RoBERTa-based language model for the Kazakh language, trained from scratch using the RobertaForMaskedLM architecture. It has been trained on the "kz-transformers/multidomain-kazakh-dataset" from Hugging Face, which covers diverse domains to ensure broad generalization capabilities.

Usage

The model can be used with the Hugging Face Transformers library:

from transformers import RobertaTokenizerFast, RobertaForMaskedLM

tokenizer = RobertaTokenizerFast.from_pretrained('nur-dev/roberta-kaz-large')
model = RobertaForMaskedLM.from_pretrained('nur-dev/roberta-kaz-large')

Or directly with a pipeline for MLM:

from transformers import pipeline
pipe = pipeline('fill-mask', model='nur-dev/roberta-kaz-large')
predicted = pipe("Қазіргі <mask> әлемдік деңгейдегі <mask> университеттері сапалы білім, зияткерлік және мәдени <mask> беретін <mask> <mask> <mask> ғана емес, сонымен қатар мемлекет үшін <mask> қабілетті адами капиталды құратын <mask>, ғылым және өндірісті интеграциялаудың <mask> <mask> болып табылады.")

for t in predicted:
  print(t[0]['score'], t[0]['token_str'])

Training procedure

The model was trained using two NVIDIA A100 GPUs on over 5.3 million examples from the "kz-transformers/multidomain-kazakh-dataset." We conducted training across 10 epochs, handling large batches of data efficiently through gradient accumulation. The learning setup included a slow build-up in the learning rate to maximize learning stability and was optimized over 208,100 steps, focusing on improving the model’s ability to understand and generate the Kazakh language.

Limitations and Bias

As with any language model, roberta-kaz-large may inherently learn biases present in the training data. Users should be cautious and evaluate the model in diverse contexts to ensure it performs as expected, especially in sensitive applications.

Model Authors

Name: Kadyrbek Nurgali

• Email: nurgaliqadyrbek@gmail.com
• LinkedIn: Kadyrbek Nurgali