README.md

---
language:
- tr
arXiv: 2403.01308
library_name: transformers
pipeline_tag: text2text-generation
inference:
  parameters:
    max_new_tokens: 32
    num_beams: 8
    do_sample: false
widget:
- text: >-
    Soru yarat: Alan Mathison Turing İngiliz matematikçi, bilgisayar bilimcisi
    ve kriptolog. II. Dünya Savaşı sırasında Alman şifrelerinin kırılmasında çok
    önemli bir rol oynadığı için savaş kahramanı sayılmıştır. Ayrıca Manchester
    Üniversitesi'nde çalıştığı yıllarda, Turing makinesi denilen algoritma
    tanımı ile modern bilgisayarların kavramsal temelini atmıştır.
  example_title: Question generation
- text: >-
    Soru cevapla: Turing makinesi denilen algoritma tanımı ile modern
    bilgisayarların kavramsal temelini atan bilim insanı kimdir? kaynak: Alan
    Mathison Turing İngiliz matematikçi, bilgisayar bilimcisi ve kriptolog. II.
    Dünya Savaşı sırasında Alman şifrelerinin kırılmasında çok önemli bir rol
    oynadığı için savaş kahramanı sayılmıştır. Ayrıca Manchester
    Üniversitesi'nde çalıştığı yıllarda, Turing makinesi denilen algoritma
    tanımı ile modern bilgisayarların kavramsal temelini atmıştır.
  example_title: Question answering
- text: >-
    yanıtları çıkar: Alan Mathison Turing İngiliz matematikçi, bilgisayar
    bilimcisi ve kriptolog.  II. Dünya Savaşı sırasında Alman şifrelerinin
    kırılmasında çok önemli bir rol oynadığı için savaş kahramanı sayılmıştır.
    <hl>  Ayrıca Manchester Üniversitesi'nde çalıştığı yıllarda, Turing makinesi
    denilen algoritma tanımı ile modern bilgisayarların kavramsal temelini
    atmıştır <hl> .
  example_title: Answer Extraction
license: cc-by-nc-sa-4.0
datasets:
- vngrs-ai/vngrs-web-corpus
---
# VBART Model Card

## Model Description  

VBART is the first sequence-to-sequence LLM pre-trained on Turkish corpora from scratch on a large scale. It was pre-trained by VNGRS in February 2023.  
The model is capable of conditional text generation tasks such as text summarization, paraphrasing, and title generation when fine-tuned.
It outperforms its multilingual counterparts, albeit being much smaller than other implementations.

VBART-XLarge is created by adding extra Transformer layers between the layers of VBART-Large. Hence it was able to transfer learned weights from the smaller model while doublings its number of layers.
VBART-XLarge improves the results compared to VBART-Large albeit in small margins.

This repository contains fine-tuned TensorFlow and Safetensors weights of VBART for question-answering and generation tasks described in the [paper](https://doi.org/10.55730/1300-0632.3914).

- **Developed by:** [VNGRS-AI](https://vngrs.com/ai/)
- **Model type:**  Transformer encoder-decoder based on mBART architecture
- **Language(s) (NLP):** Turkish
- **License:** CC BY-NC-SA 4.0
- **Finetuned from:** VBART-XLarge
- **Paper:** [arXiv](https://arxiv.org/abs/2403.01308)
## How to Get Started with the Model  
```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("vngrs-ai/VBART-XLarge-QAQG",
                            model_input_names=['input_ids', 'attention_mask'])
# Uncomment the device_map kwarg and delete the closing bracket to use model for inference on GPU
model = AutoModelForSeq2SeqLM.from_pretrained("vngrs-ai/VBART-XLarge-QAQG")#, device_map="auto")

context="..."
question="..."
highlighted_context="..."

# Prompt for question generation
qg_prompt = f'Soru yarat: cevap: {context}'
# Prompt for question answering
qa_prompt = f'Soru cevapla: {question} kaynak: {context}'
# Prompt for answer extraction
ae_prompt = f'yanıtları çıkar: {highlighted_context}'


token_input = tokenizer(ae_prompt, return_tensors="pt")#.to('cuda')
outputs = model.generate(**token_input)
print(tokenizer.decode(outputs[0]))
```
  
## Training Details  
### Fine-tuning prompt
This model is fine-tuned on three tasks: 
- question answering: Answer a question in a given context. Prompted with
    ```Soru cevapla: <question> kaynak: <context>```
- question generation: Generate a question from a given context. Will accept a highlight token (`<hl>`, without spaces) to specify the answer to the question generated. Prompted with
    ```Soru yarat: <context>```
- answer extraction: Will extract possible answers from a highlighted range (using the same highlight token). Prompted with
    ``` yanıtları çıkar: <context with highlighted parts>```

### Training Data  
The base model is pre-trained on [vngrs-web-corpus](https://huggingface.co/datasets/vngrs-ai/vngrs-web-corpus). It is curated by cleaning and filtering Turkish parts of [OSCAR-2201](https://huggingface.co/datasets/oscar-corpus/OSCAR-2201) and [mC4](https://huggingface.co/datasets/mc4) datasets. These datasets consist of documents of unstructured web crawl data. More information about the dataset can be found on their respective pages. Data is filtered using a set of heuristics and certain rules, explained in the appendix of our [paper](https://arxiv.org/abs/2403.01308).

The fine-tuning dataset is [TQuAD](https://github.com/obss/turkish-question-generation), which has two versions. We have concatenated them and dropped duplicate samples. More information about this process can be found in Appendix B of our [paper](https://arxiv.org/abs/2403.01308). 

### Limitations
This model is fine-tuned for question-answering and question-generation tasks with specific prompts. It is not intended to be used in any other case and can not be fine-tuned to any other task with full performance of the base model. It is also not guaranteed that this model will work without specified prompts.

### Training Procedure  
Pre-trained for 8 days and for a total of 84B tokens. Finally, finetuned for 55 epochs.
#### Hardware
- **GPUs**: 8 x Nvidia A100-80 GB
#### Software
- TensorFlow
#### Hyperparameters  
##### Pretraining
- **Training regime:** fp16 mixed precision
- **Training objective**: Sentence permutation and span masking (using mask lengths sampled from Poisson distribution λ=3.5, masking 30% of tokens)
- **Optimizer** : Adam optimizer (β1 = 0.9, β2 = 0.98, Ɛ = 1e-6)
- **Scheduler**: Custom scheduler from the original Transformers paper (20,000 warm-up steps)
- **Weight Initialization**: Model Enlargement from VBART-Large. See the related section in the [paper](https://arxiv.org/abs/2403.01308) for the details. 
- **Dropout**: 0.1 (dropped to 0.05 and then to 0 in the last 80K and 80k steps, respectively)
- **Initial Learning rate**: 5e-6
- **Training tokens**: 84B

##### Fine-tuning
- **Training regime:** fp16 mixed precision
- **Optimizer** : Adam optimizer (β1 = 0.9, β2 = 0.98, Ɛ = 1e-6)
- **Scheduler**: Linear decay scheduler
- **Dropout**: 0.1 
-  **Learning rate**: 5e-6
-  **Fine-tune epochs**: 55

#### Metrics
![image/png](https://cdn-uploads.huggingface.co/production/uploads/62f8b3c84588fe31f435a92b/D-Epasj5C4icAu0ykqt10.png)

## Citation  
```
@article{turker2024vbart,
  title={VBART: The Turkish LLM},
  author={Turker, Meliksah and Ari, Erdi and Han, Aydin},
  journal={arXiv preprint arXiv:2403.01308},
  year={2024}
}
```