---
language: ar
datasets:
- Marefa-NER
widget:
- text: "في استاد القاهرة، بدأ حفل افتتاح بطولة كأس الأمم الأفريقية بحضور رئيس الجمهورية و رئيس الاتحاد الدولي لكرة القدم"
---

# Tebyan تبيـان
## Marefa Arabic Named Entity Recognition Model
## نموذج المعرفة لتصنيف أجزاء النص

</p?

---------
**Version**: 1.3

**Last Update:** 3-12-2021

## Model description

**Marefa-NER** is a Large Arabic Named Entity Recognition (NER) model built on a completely new dataset and targets to extract up to 9 different types of entities
```
Person, Location, Organization, Nationality, Job, Product, Event, Time, Art-Work
```

نموذج المعرفة لتصنيف أجزاء النص. نموذج جديد كليا من حيث البيانات المستخدمة في تدريب النموذج.
كذلك يستهدف النموذج تصنيف حتى 9 أنواع مختلفة من أجزاء النص
```
شخص - مكان - منظمة - جنسية - وظيفة - منتج - حدث - توقيت - عمل إبداعي
```

## How to use كيف تستخدم النموذج

*You can test the model quickly by checking this [Colab notebook](https://colab.research.google.com/drive/1OGp9Wgm-oBM5BBhTLx6Qow4dNRSJZ-F5?usp=sharing)*

----

Install the following Python packages

`$ pip3 install transformers==4.8.0 nltk==3.5 protobuf==3.15.3 torch==1.9.0 `

> If you are using `Google Colab`, please restart your runtime after installing the packages.

-----------

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

import numpy as np
import nltk
nltk.download('punkt')
from nltk.tokenize import word_tokenize

custom_labels = ["O", "B-job", "I-job", "B-nationality", "B-person", "I-person", "B-location","B-time", "I-time", "B-event", "I-event", "B-organization", "I-organization", "I-location", "I-nationality", "B-product", "I-product", "B-artwork", "I-artwork"]

def _extract_ner(text: str, model: AutoModelForTokenClassification,
                 tokenizer: AutoTokenizer, start_token: str="▁"):
    tokenized_sentence = tokenizer([text], padding=True, truncation=True, return_tensors="pt")
    tokenized_sentences = tokenized_sentence['input_ids'].numpy()

    with torch.no_grad():
        output = model(**tokenized_sentence)

    last_hidden_states = output[0].numpy()
    label_indices = np.argmax(last_hidden_states[0], axis=1)
    tokens = tokenizer.convert_ids_to_tokens(tokenized_sentences[0])
    special_tags = set(tokenizer.special_tokens_map.values())

    grouped_tokens = []
    for token, label_idx in zip(tokens, label_indices):
        if token not in special_tags:
            if not token.startswith(start_token) and len(token.replace(start_token,"").strip()) > 0:
                grouped_tokens[-1]["token"] += token
            else:
                grouped_tokens.append({"token": token, "label": custom_labels[label_idx]})

    # extract entities
    ents = []
    prev_label = "O"
    for token in grouped_tokens:
        label = token["label"].replace("I-","").replace("B-","")
        if token["label"] != "O":

            if label != prev_label:
                ents.append({"token": [token["token"]], "label": label})
            else:
                ents[-1]["token"].append(token["token"])

        prev_label = label

    # group tokens
    ents = [{"token": "".join(rec["token"]).replace(start_token," ").strip(), "label": rec["label"]}  for rec in ents ]

    return ents

model_cp = "marefa-nlp/marefa-ner"

tokenizer = AutoTokenizer.from_pretrained(model_cp)
model = AutoModelForTokenClassification.from_pretrained(model_cp, num_labels=len(custom_labels))

samples = [
    "تلقى تعليمه في الكتاب ثم انضم الى الأزهر عام 1873م. تعلم على يد السيد جمال الدين الأفغاني والشيخ محمد عبده",
    "بعد عودته إلى القاهرة، التحق نجيب الريحاني فرقة جورج أبيض، الذي كان قد ضمَّ - قُبيل ذلك - فرقته إلى فرقة سلامة حجازي . و منها ذاع صيته",
    "في استاد القاهرة، قام حفل افتتاح بطولة كأس الأمم الأفريقية بحضور رئيس الجمهورية و رئيس الاتحاد الدولي لكرة القدم",
    "من فضلك أرسل هذا البريد الى صديقي جلال الدين في تمام الساعة الخامسة صباحا في يوم الثلاثاء القادم",
    "امبارح اتفرجت على مباراة مانشستر يونايتد مع ريال مدريد في غياب الدون كرستيانو رونالدو",
    "لا تنسى تصحيني الساعة سبعة, و ضيف في الجدول اني احضر مباراة نادي النصر غدا",
]

# [optional]
samples = [ " ".join(word_tokenize(sample.strip())) for sample in samples if sample.strip() != "" ]

for sample in samples:
    ents = _extract_ner(text=sample, model=model, tokenizer=tokenizer, start_token="▁")

    print(sample)
    for ent in ents:
        print("\t",ent["token"],"==>",ent["label"])
    print("========\n")

```

Output

```
تلقى تعليمه في الكتاب ثم انضم الى الأزهر عام 1873م . تعلم على يد السيد جمال الدين الأفغاني والشيخ محمد عبده
	 الأزهر ==> organization
	 عام 1873م ==> time
	 السيد جمال الدين الأفغاني ==> person
	 محمد عبده ==> person
========

بعد عودته إلى القاهرة، التحق نجيب الريحاني فرقة جورج أبيض، الذي كان قد ضمَّ - قُبيل ذلك - فرقته إلى فرقة سلامة حجازي . و منها ذاع صيته
	 القاهرة، ==> location
	 نجيب الريحاني ==> person
	 فرقة جورج أبيض، ==> organization
	 فرقة سلامة حجازي ==> organization
========

في استاد القاهرة، قام حفل افتتاح بطولة كأس الأمم الأفريقية بحضور رئيس الجمهورية و رئيس الاتحاد الدولي لكرة القدم
	 استاد القاهرة، ==> location
	 بطولة كأس الأمم الأفريقية ==> event
	 رئيس الجمهورية ==> job
	 رئيس ==> job
	 الاتحاد الدولي لكرة القدم ==> organization
========

من فضلك أرسل هذا البريد الى صديقي جلال الدين في تمام الساعة الخامسة صباحا في يوم الثلاثاء القادم
	 جلال الدين ==> person
	 الساعة الخامسة صباحا ==> time
	 يوم الثلاثاء القادم ==> time
========

امبارح اتفرجت على مباراة مانشستر يونايتد مع ريال مدريد في غياب الدون كرستيانو رونالدو
	 مانشستر يونايتد ==> organization
	 ريال مدريد ==> organization
	 كرستيانو رونالدو ==> person
========

لا تنسى تصحيني الساعة سبعة , و ضيف في الجدول اني احضر مباراة نادي النصر غدا
	 الساعة سبعة ==> time
	 نادي النصر ==> organization
	 غدا ==> time
========
```

## Fine-Tuning

Check this [notebook](https://colab.research.google.com/drive/1WUYrnmDFFEItqGMvbyjqZEJJqwU7xQR-?usp=sharing) to fine-tune the NER model

## Evaluation

We tested the model agains a test set of 1959 sentences. The results is in the follwing table

| type         |   f1-score |   precision |   recall |   support |
|:-------------|-----------:|------------:|---------:|----------:|
| person       |   0.93298  |    0.931479 | 0.934487 |      4335 |
| location     |   0.891537 |    0.896926 | 0.886212 |      4939 |
| time         |   0.873003 |    0.876087 | 0.869941 |      1853 |
| nationality  |   0.871246 |    0.843153 | 0.901277 |      2350 |
| job          |   0.837656 |    0.79912  | 0.880097 |      2477 |
| organization |   0.781317 |    0.773328 | 0.789474 |      2299 |
| event        |   0.686695 |    0.733945 | 0.645161 |       744 |
| artwork      |   0.653552 |    0.678005 | 0.630802 |       474 |
| product      |   0.625483 |    0.553531 | 0.718935 |       338 |
| **weighted avg** |   0.859008 |    0.852365 | 0.86703  |     19809 |
| **micro avg**    |   0.858771 |    0.850669 | 0.86703  |     19809 |
| **macro avg**   |   0.79483  |    0.787286 | 0.806265 |     19809 |

## Acknowledgment شكر و تقدير

قام بإعداد البيانات التي تم تدريب النموذج عليها, مجموعة من المتطوعين الذين قضوا ساعات يقومون بتنقيح البيانات و مراجعتها

- على سيد عبد الحفيظ - إشراف
- نرمين محمد عطيه
- صلاح خيرالله
- احمد علي عبدربه
- عمر بن عبد العزيز سليمان
- محمد ابراهيم الجمال
- عبدالرحمن سلامه خلف
- إبراهيم كمال محمد سليمان
- حسن مصطفى حسن
- أحمد فتحي سيد
- عثمان مندو
- عارف الشريف
- أميرة محمد محمود
- حسن سعيد حسن
- عبد العزيز علي البغدادي
- واثق عبدالملك الشويطر
- عمرو رمضان عقل الحفناوي
- حسام الدين أحمد على
- أسامه أحمد محمد محمد
- حاتم محمد المفتي
- عبد الله دردير
- أدهم البغدادي
- أحمد صبري
- عبدالوهاب محمد محمد
- أحمد محمد عوض


{'C0': 'audit_rights',
 'C1': 'licensee_indemnity',
 'C2': 'licensor_indemnity',
 'C3': 'license_grant',
 'C4': 'eula_others',
 'C5': 'licensee_infringement_indemnity',
 'C6': 'licensor_exemption_liability',
 'C7': 'licensor_limit_liabilty',
 'C8': 'software_warranty'}


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: trained_model2
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# trained_model2

This model is a fine-tuned version of [distilgpt2](https://huggingface.co/distilgpt2) on an unknown dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 2
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 15.0

### Training results

### Framework versions

- Transformers 4.13.0.dev0
- Pytorch 1.9.1
- Datasets 1.14.0
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- wmt16
metrics:
- bleu
model-index:
- name: opus-mt-en-de-finetuned-en-to-de-model-2
  results:
  - task:
      name: Sequence-to-sequence Language Modeling
      type: text2text-generation
    dataset:
      name: wmt16
      type: wmt16
      args: de-en
    metrics:
    - name: Bleu
      type: bleu
      value: 27.7086
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# opus-mt-en-de-finetuned-en-to-de-model-2

This model is a fine-tuned version of [Helsinki-NLP/opus-mt-en-de](https://huggingface.co/Helsinki-NLP/opus-mt-en-de) on the wmt16 dataset.
It achieves the following results on the evaluation set:
- Loss: 1.5389
- Bleu: 27.7086
- Gen Len: 24.7238

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step   | Validation Loss | Bleu    | Gen Len |
|:-------------:|:-----:|:------:|:---------------:|:-------:|:-------:|
| 1.9613        | 1.0   | 284306 | 1.5389          | 27.7086 | 24.7238 |

### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0+cu102
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- wmt16
metrics:
- bleu
model-index:
- name: opus-mt-en-de-finetuned-en-to-de
  results:
  - task:
      name: Sequence-to-sequence Language Modeling
      type: text2text-generation
    dataset:
      name: wmt16
      type: wmt16
      args: de-en
    metrics:
    - name: Bleu
      type: bleu
      value: 29.4312
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# opus-mt-en-de-finetuned-en-to-de

This model is a fine-tuned version of [Helsinki-NLP/opus-mt-en-de](https://huggingface.co/Helsinki-NLP/opus-mt-en-de) on the wmt16 dataset.
It achieves the following results on the evaluation set:
- Loss: 1.4083
- Bleu: 29.4312
- Gen Len: 24.746

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step   | Validation Loss | Bleu    | Gen Len |
|:-------------:|:-----:|:------:|:---------------:|:-------:|:-------:|
| 1.978         | 1.0   | 568611 | 1.4083          | 29.4312 | 24.746  |

### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0+cu102
- Datasets 1.16.1
- Tokenizers 0.10.3


---

language: en

license: mit

---

Import it using pipeline

    from transformers import pipeline
    text_generation = pipeline('text-generation' , model='sparki/kinkyfurs-gpt2')

Then use it

    prefix_text = input()
    text_generation(prefix_text, max_length=50, num_beams=5,no_repeat_ngram_size=2,early_stopping=True)


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: run_6
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# run_6

This model is a fine-tuned version of [t5-large](https://huggingface.co/t5-large) on the None dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 1
- eval_batch_size: 8
- seed: 121
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 30

### Training results

### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0
- Datasets 1.15.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
metrics:
- bleu
model-index:
- name: t5-base_Run_X2
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# t5-base_Run_X2

This model is a fine-tuned version of [t5-base](https://huggingface.co/t5-base) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.6185
- Bleu: 45.2719

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 121
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.33
- num_epochs: 50
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Bleu    |
|:-------------:|:-----:|:----:|:---------------:|:-------:|
| No log        | 10.0  | 50   | 3.2932          | 2.2530  |
| No log        | 20.0  | 100  | 1.3323          | 15.1192 |
| No log        | 30.0  | 150  | 0.7797          | 43.9378 |
| No log        | 40.0  | 200  | 0.6185          | 46.7000 |
| No log        | 50.0  | 250  | 0.5643          | 46.0571 |

### Framework versions

- Transformers 4.14.1
- Pytorch 1.10.1+cu102
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
metrics:
- bleu
model-index:
- name: t5-base_Run_X4
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# t5-base_Run_X4

This model is a fine-tuned version of [t5-base](https://huggingface.co/t5-base) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0634
- Bleu: 94.2179

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 121
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 30
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Bleu    |
|:-------------:|:-----:|:----:|:---------------:|:-------:|
| No log        | 2.78  | 50   | 1.1874          | 17.8807 |
| No log        | 5.56  | 100  | 0.3570          | 54.7073 |
| No log        | 8.33  | 150  | 0.1895          | 57.8991 |
| No log        | 11.11 | 200  | 0.1108          | 63.2364 |
| No log        | 13.89 | 250  | 0.0887          | 63.1200 |
| No log        | 16.67 | 300  | 0.0773          | 65.8133 |
| No log        | 19.44 | 350  | 0.0734          | 66.8279 |
| No log        | 22.22 | 400  | 0.0676          | 67.5501 |
| No log        | 25.0  | 450  | 0.0653          | 66.8709 |
| 0.484         | 27.78 | 500  | 0.0634          | 65.7688 |

### Framework versions

- Transformers 4.14.1
- Pytorch 1.10.1+cu102
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: Test_run_1
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Test_run_1

This model is a fine-tuned version of [t5-base](https://huggingface.co/t5-base) on the None dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 5
- eval_batch_size: 8
- seed: 121
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10

### Training results

### Framework versions

- Transformers 4.13.0
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
metrics:
- bleu
model-index:
- name: t5-small_Run_X
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# t5-small_Run_X

This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.2410
- Bleu: 76.8898

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 7.5e-05
- train_batch_size: 5
- eval_batch_size: 1
- seed: 121
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10

### Training results

| Training Loss | Epoch | Step | Validation Loss | Bleu    |
|:-------------:|:-----:|:----:|:---------------:|:-------:|
| No log        | 1.0   | 32   | 2.1699          | 1.2538  |
| No log        | 2.0   | 64   | 0.9729          | 6.1092  |
| No log        | 3.0   | 96   | 0.5267          | 11.1118 |
| No log        | 4.0   | 128  | 0.4019          | 12.6861 |
| No log        | 5.0   | 160  | 0.3371          | 12.4613 |
| No log        | 6.0   | 192  | 0.2971          | 12.9702 |
| No log        | 7.0   | 224  | 0.2718          | 12.8378 |
| No log        | 8.0   | 256  | 0.2524          | 12.9908 |
| No log        | 9.0   | 288  | 0.2446          | 13.0304 |
| No log        | 10.0  | 320  | 0.2410          | 13.0611 |

### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0
- Datasets 1.15.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
metrics:
- bleu
model-index:
- name: Run_8
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Run_8

This model is a fine-tuned version of [google/t5-v1_1-large](https://huggingface.co/google/t5-v1_1-large) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: nan
- Bleu: 0.0843

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 7.5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 121
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0
- Datasets 1.15.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
metrics:
- bleu
model-index:
- name: t5-base_Run_X5
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# t5-base_Run_X5

This model is a fine-tuned version of [t5-base](https://huggingface.co/t5-base) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0011
- Bleu: 99.8146

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 121
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 30
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Bleu    |
|:-------------:|:-----:|:----:|:---------------:|:-------:|
| No log        | 0.42  | 50   | 3.3251          | 0.8615  |
| No log        | 0.83  | 100  | 1.5917          | 4.7904  |
| No log        | 1.25  | 150  | 0.7086          | 41.1927 |
| No log        | 1.67  | 200  | 0.3970          | 49.1976 |
| No log        | 2.08  | 250  | 0.2014          | 54.0266 |
| No log        | 2.5   | 300  | 0.1058          | 56.0118 |
| No log        | 2.92  | 350  | 0.0686          | 58.9581 |
| No log        | 3.33  | 400  | 0.0441          | 60.0015 |
| No log        | 3.75  | 450  | 0.0239          | 61.0581 |
| 1.0284        | 4.17  | 500  | 0.0191          | 61.4971 |
| 1.0284        | 4.58  | 550  | 0.0129          | 62.0168 |
| 1.0284        | 5.0   | 600  | 0.0128          | 61.9730 |
| 1.0284        | 5.42  | 650  | 0.0082          | 61.8373 |
| 1.0284        | 5.83  | 700  | 0.0054          | 62.7842 |
| 1.0284        | 6.25  | 750  | 0.0054          | 62.9966 |
| 1.0284        | 6.67  | 800  | 0.0045          | 63.0655 |
| 1.0284        | 7.08  | 850  | 0.0031          | 63.2890 |
| 1.0284        | 7.5   | 900  | 0.0028          | 63.6213 |
| 1.0284        | 7.92  | 950  | 0.0024          | 63.4271 |
| 0.0254        | 8.33  | 1000 | 0.0026          | 63.3434 |
| 0.0254        | 8.75  | 1050 | 0.0023          | 63.4567 |
| 0.0254        | 9.17  | 1100 | 0.0019          | 63.4567 |
| 0.0254        | 9.58  | 1150 | 0.0018          | 63.4567 |
| 0.0254        | 10.0  | 1200 | 0.0019          | 63.6718 |
| 0.0254        | 10.42 | 1250 | 0.0014          | 63.5642 |
| 0.0254        | 10.83 | 1300 | 0.0017          | 63.4567 |
| 0.0254        | 11.25 | 1350 | 0.0024          | 63.3006 |
| 0.0254        | 11.67 | 1400 | 0.0015          | 63.4567 |
| 0.0254        | 12.08 | 1450 | 0.0015          | 63.3786 |
| 0.0082        | 12.5  | 1500 | 0.0012          | 63.5642 |
| 0.0082        | 12.92 | 1550 | 0.0012          | 63.4082 |
| 0.0082        | 13.33 | 1600 | 0.0011          | 63.6718 |
| 0.0082        | 13.75 | 1650 | 0.0009          | 63.6718 |
| 0.0082        | 14.17 | 1700 | 0.0013          | 63.4567 |
| 0.0082        | 14.58 | 1750 | 0.0014          | 63.5642 |
| 0.0082        | 15.0  | 1800 | 0.0011          | 63.5642 |
| 0.0082        | 15.42 | 1850 | 0.0010          | 63.5158 |
| 0.0082        | 15.83 | 1900 | 0.0011          | 63.3301 |
| 0.0082        | 16.25 | 1950 | 0.0015          | 63.4567 |
| 0.0052        | 16.67 | 2000 | 0.0015          | 63.2225 |

### Framework versions

- Transformers 4.14.1
- Pytorch 1.10.1+cu102
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
metrics:
- bleu
model-index:
- name: t5-base_Run_X6
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# t5-base_Run_X6

This model is a fine-tuned version of [t5-base](https://huggingface.co/t5-base) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0011
- Bleu: 99.8146

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 121
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 30
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Bleu    |
|:-------------:|:-----:|:----:|:---------------:|:-------:|
| No log        | 0.42  | 50   | 3.3251          | 0.8615  |
| No log        | 0.83  | 100  | 1.5917          | 4.7904  |
| No log        | 1.25  | 150  | 0.7086          | 41.1927 |
| No log        | 1.67  | 200  | 0.3970          | 49.1976 |
| No log        | 2.08  | 250  | 0.2014          | 54.0266 |
| No log        | 2.5   | 300  | 0.1058          | 56.0118 |
| No log        | 2.92  | 350  | 0.0686          | 58.9581 |
| No log        | 3.33  | 400  | 0.0441          | 60.0015 |
| No log        | 3.75  | 450  | 0.0239          | 61.0581 |
| 1.0284        | 4.17  | 500  | 0.0191          | 61.4971 |
| 1.0284        | 4.58  | 550  | 0.0129          | 62.0168 |
| 1.0284        | 5.0   | 600  | 0.0128          | 61.9730 |
| 1.0284        | 5.42  | 650  | 0.0082          | 61.8373 |
| 1.0284        | 5.83  | 700  | 0.0054          | 62.7842 |
| 1.0284        | 6.25  | 750  | 0.0054          | 62.9966 |
| 1.0284        | 6.67  | 800  | 0.0045          | 63.0655 |
| 1.0284        | 7.08  | 850  | 0.0031          | 63.2890 |
| 1.0284        | 7.5   | 900  | 0.0028          | 63.6213 |
| 1.0284        | 7.92  | 950  | 0.0024          | 63.4271 |
| 0.0254        | 8.33  | 1000 | 0.0026          | 63.3434 |
| 0.0254        | 8.75  | 1050 | 0.0023          | 63.4567 |
| 0.0254        | 9.17  | 1100 | 0.0019          | 63.4567 |
| 0.0254        | 9.58  | 1150 | 0.0018          | 63.4567 |
| 0.0254        | 10.0  | 1200 | 0.0019          | 63.6718 |
| 0.0254        | 10.42 | 1250 | 0.0014          | 63.5642 |
| 0.0254        | 10.83 | 1300 | 0.0017          | 63.4567 |
| 0.0254        | 11.25 | 1350 | 0.0024          | 63.3006 |
| 0.0254        | 11.67 | 1400 | 0.0015          | 63.4567 |
| 0.0254        | 12.08 | 1450 | 0.0015          | 63.3786 |
| 0.0082        | 12.5  | 1500 | 0.0012          | 63.5642 |
| 0.0082        | 12.92 | 1550 | 0.0012          | 63.4082 |
| 0.0082        | 13.33 | 1600 | 0.0011          | 63.6718 |
| 0.0082        | 13.75 | 1650 | 0.0009          | 63.6718 |
| 0.0082        | 14.17 | 1700 | 0.0013          | 63.4567 |
| 0.0082        | 14.58 | 1750 | 0.0014          | 63.5642 |
| 0.0082        | 15.0  | 1800 | 0.0011          | 63.5642 |
| 0.0082        | 15.42 | 1850 | 0.0010          | 63.5158 |
| 0.0082        | 15.83 | 1900 | 0.0011          | 63.3301 |
| 0.0082        | 16.25 | 1950 | 0.0015          | 63.4567 |
| 0.0052        | 16.67 | 2000 | 0.0015          | 63.2225 |

### Framework versions

- Transformers 4.14.1
- Pytorch 1.10.1+cu102
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: run_5
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# run_5

This model is a fine-tuned version of [t5-base](https://huggingface.co/t5-base) on the None dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.001
- train_batch_size: 1
- eval_batch_size: 8
- seed: 121
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10

### Training results

### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0
- Datasets 1.15.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
metrics:
- bleu
model-index:
- name: t5-large_Run_X3
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# t5-large_Run_X3

This model is a fine-tuned version of [t5-large](https://huggingface.co/t5-large) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: nan
- Bleu: 0.4892

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 121
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.33
- training_steps: 500
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Bleu   |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| No log        | 2.5   | 50   | nan             | 0.5945 |
| No log        | 5.0   | 100  | nan             | 0.5945 |
| No log        | 7.5   | 150  | nan             | 0.5945 |
| No log        | 10.0  | 200  | nan             | 0.5945 |
| No log        | 12.5  | 250  | nan             | 0.5945 |

### Framework versions

- Transformers 4.14.1
- Pytorch 1.10.1+cu102
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
metrics:
- bleu
model-index:
- name: Run_7
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Run_7

This model is a fine-tuned version of [t5-large](https://huggingface.co/t5-large) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: nan
- Bleu: 0.1076

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 7.5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 121
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0
- Datasets 1.15.1
- Tokenizers 0.10.3


---
license: apache-2.0
datasets:
- null
model_index:
- name: dear-jarvis-v5
  results:
  - task:
      name: Text Classification
      type: text-classification
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# dear-jarvis-v5

This model is a fine-tuned version of [distilbert-base-cased](https://huggingface.co/distilbert-base-cased) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.3148

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 470  | 0.3106          |
| 0.3452        | 2.0   | 940  | 0.3064          |
| 0.2692        | 3.0   | 1410 | 0.3148          |

### Framework versions

- Transformers 4.7.0
- Pytorch 1.9.0+cu102
- Datasets 1.8.0
- Tokenizers 0.10.3


Labels are based on Plutchik's model of emotions and may be combined:
![image](https://user-images.githubusercontent.com/12978899/122398897-f60d2500-cf97-11eb-8991-61e68f4ea1fc.png)


## Model description
This model was trained on the XED dataset and achieved
validation loss: 0.5995
validation acc: 84.28% (ROC-AUC)

Labels are based on Plutchik's model of emotions and may be combined:
![image](https://user-images.githubusercontent.com/12978899/122398897-f60d2500-cf97-11eb-8991-61e68f4ea1fc.png)

### Framework versions
- Transformers 4.6.1
- Pytorch 1.8.1+cu101
- Datasets 1.8.0
- Tokenizers 0.10.3


---
language: en
license: mit
datasets:
- bookcorpus
- wikipedia
---

# XLNet (base-sized model)

XLNet model pre-trained on English language. It was introduced in the paper [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237) by Yang et al. and first released in [this repository](https://github.com/zihangdai/xlnet/).

Disclaimer: The team releasing XLNet did not write a model card for this model so this model card has been written by the Hugging Face team.

## Model description

XLNet is a new unsupervised language representation learning method based on a novel generalized permutation language modeling objective. Additionally, XLNet employs Transformer-XL as the backbone model, exhibiting excellent performance for language tasks involving long context. Overall, XLNet achieves state-of-the-art (SOTA) results on various downstream language tasks including question answering, natural language inference, sentiment analysis, and document ranking.

## Intended uses & limitations

The model is mostly intended to be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?search=xlnet) to look for fine-tuned versions on a task that interests you.

Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) to make decisions, such as sequence classification, token classification or question answering. For tasks such as text generation, you should look at models like GPT2.

## Usage

Here is how to use this model to get the features of a given text in PyTorch:

```python
from transformers import XLNetTokenizer, XLNetModel

tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
model = XLNetModel.from_pretrained('xlnet-base-cased')

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state
```

### BibTeX entry and citation info

```bibtex
@article{DBLP:journals/corr/abs-1906-08237,
  author    = {Zhilin Yang and
               Zihang Dai and
               Yiming Yang and
               Jaime G. Carbonell and
               Ruslan Salakhutdinov and
               Quoc V. Le},
  title     = {XLNet: Generalized Autoregressive Pretraining for Language Understanding},
  journal   = {CoRR},
  volume    = {abs/1906.08237},
  year      = {2019},
  url       = {http://arxiv.org/abs/1906.08237},
  eprinttype = {arXiv},
  eprint    = {1906.08237},
  timestamp = {Mon, 24 Jun 2019 17:28:45 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1906-08237.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
```


---
language: "en"
tags:
- paraphrase-generation
- text-generation
- Conditional Generation
inference: false
---

# Paraphrase-Generation

## Model description

T5 Model for generating paraphrases of english sentences. Trained on the [Google PAWS](https://github.com/google-research-datasets/paws) dataset.

## How to use

PyTorch and TF models available

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Vamsi/T5_Paraphrase_Paws")
model = AutoModelForSeq2SeqLM.from_pretrained("Vamsi/T5_Paraphrase_Paws")

sentence = "This is something which i cannot understand at all"

text =  "paraphrase: " + sentence + " </s>"

encoding = tokenizer.encode_plus(text,pad_to_max_length=True, return_tensors="pt")
input_ids, attention_masks = encoding["input_ids"].to("cuda"), encoding["attention_mask"].to("cuda")

outputs = model.generate(
    input_ids=input_ids, attention_mask=attention_masks,
    max_length=256,
    do_sample=True,
    top_k=120,
    top_p=0.95,
    early_stopping=True,
    num_return_sequences=5
)

for output in outputs:
    line = tokenizer.decode(output, skip_special_tokens=True,clean_up_tokenization_spaces=True)
    print(line)

```

For more reference on training your own T5 model or using this model, do check out [Paraphrase Generation](https://github.com/Vamsi995/Paraphrase-Generator).


---
license: apache-2.0
tags:
- generated_from_trainer
metrics:
- rouge
model-index:
- name: t5-small-herblabels
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# t5-small-herblabels

This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4823
- Rouge1: 3.0759
- Rouge2: 1.0495
- Rougel: 3.0758
- Rougelsum: 3.0431
- Gen Len: 18.9716

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:------:|:---------:|:-------:|
| No log        | 1.0   | 264  | 1.6010          | 2.4276 | 0.5658 | 2.3546 | 2.3099    | 18.9091 |
| 2.5052        | 2.0   | 528  | 1.0237          | 2.9016 | 0.3395 | 2.8221 | 2.783     | 18.9673 |
| 2.5052        | 3.0   | 792  | 0.7793          | 2.962  | 0.3091 | 2.9375 | 2.8786    | 18.9588 |
| 1.1552        | 4.0   | 1056 | 0.6530          | 2.98   | 0.4375 | 2.9584 | 2.8711    | 18.9588 |
| 1.1552        | 5.0   | 1320 | 0.5863          | 3.0023 | 0.5882 | 2.987  | 2.9155    | 18.9588 |
| 0.8659        | 6.0   | 1584 | 0.5428          | 3.0576 | 0.8019 | 3.0494 | 2.9989    | 18.9716 |
| 0.8659        | 7.0   | 1848 | 0.5145          | 3.0808 | 0.9476 | 3.0719 | 3.0237    | 18.9716 |
| 0.747         | 8.0   | 2112 | 0.4962          | 3.0748 | 1.0032 | 3.0683 | 3.0359    | 18.9716 |
| 0.747         | 9.0   | 2376 | 0.4856          | 3.0702 | 1.0196 | 3.0665 | 3.0328    | 18.9716 |
| 0.6987        | 10.0  | 2640 | 0.4823          | 3.0759 | 1.0495 | 3.0758 | 3.0431    | 18.9716 |

### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0
- Datasets 1.16.1
- Tokenizers 0.10.3


---
language:
- en
thumbnail: https://cogcomp.seas.upenn.edu/images/logo.png
tags:
- text-classification
- bart
- xsum
license: cc-by-sa-4.0
datasets:
- xsum
widget:
- text: "<s> Ban Ki-moon was elected for a second term in 2007. </s></s> Ban Ki-Moon was re-elected for a second term by the UN General Assembly, unopposed and unanimously, on 21 June 2011."
- text: "<s> Ban Ki-moon was elected for a second term in 2011. </s></s> Ban Ki-Moon was re-elected for a second term by the UN General Assembly, unopposed and unanimously, on 21 June 2011."

---

# bart-faithful-summary-detector

## Model description

A BART (base) model trained to classify whether a summary is *faithful* to the original article. See our [paper in NAACL'21](https://www.seas.upenn.edu/~sihaoc/static/pdf/CZSR21.pdf) for details.

## Usage
Concatenate a summary and a source document as input (note that the summary needs to be the **first** sentence).

Here's an example usage (with PyTorch)
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("CogComp/bart-faithful-summary-detector")
model = AutoModelForSequenceClassification.from_pretrained("CogComp/bart-faithful-summary-detector")

article = "Ban Ki-Moon was re-elected for a second term by the UN General Assembly, unopposed and unanimously, on 21 June 2011."

bad_summary = "Ban Ki-moon was elected for a second term in 2007."
good_summary = "Ban Ki-moon was elected for a second term in 2011."

bad_pair = tokenizer(text=bad_summary, text_pair=article, return_tensors='pt')
good_pair = tokenizer(text=good_summary, text_pair=article, return_tensors='pt')

bad_score = model(**bad_pair)
good_score = model(**good_pair)

print(good_score[0][:, 1] > bad_score[0][:, 1]) # True, label mapping: "0" -> "Hallucinated" "1" -> "Faithful"
```

### BibTeX entry and citation info

```bibtex
@inproceedings{CZSR21,
    author = {Sihao Chen and Fan Zhang and Kazoo Sone and Dan Roth},
    title = {{Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection}},
    booktitle = {NAACL},
    year = {2021}
}
```


Don’t worry we will make available extremely beautiful and Sensuous Mahipalpur escorts to make your stay in this area exciting and thrilling. The feminine charm of curvy Site: - Mahipalpur Escorts

Site: - https://www.sarakaur.com/mahipalpur-escorts.php
Site: - https://www.missdelhiescort.com/mahipalpur-escorts.html

https://cutt.ly/yUZ81up     https://cutt.ly/uUZ4rZa   https://cutt.ly/NOlaLek


---
language:
- en
thumbnail: https://raw.githubusercontent.com/JetRunner/BERT-of-Theseus/master/bert-of-theseus.png
tags:
- sequence
- classification
license: apache-2.0
datasets:
- imdb
metrics:
- accuracy
---


This is the BERT-Medium model from Google: https://github.com/google-research/bert#bert. A BERT model with 8 layers, 512 hidden unit size, and 8 attention heads.


# albert-small-v2

This is a 6 layer version of [albert-base-v2](https://huggingface.co/albert-base-v2).


# MiniLMv2
This is a MiniLMv2 model from: [https://github.com/microsoft/unilm](https://github.com/microsoft/unilm/tree/master/minilm)


---
license: mit
---

## MiniLM: 3 Layer Version

This is a 3 layer version of [microsoft/MiniLM-L12-H384-uncased](https://huggingface.co/microsoft/MiniLM-L12-H384-uncased/) by keeping only the layer [3, 7, 11].


# Multilingual MiniLMv2
This is a MiniLMv2 model from: [https://github.com/microsoft/unilm](https://github.com/microsoft/unilm/tree/master/minilm)


# BERT-Small-L-4_H-512_A-8
This is a port of the [BERT-Small model](https://github.com/google-research/bert) to Pytorch. It uses 4 layers, a hidden size of 512 and 8 attention heads.


This is the [General_TinyBERT_v2(6layer-768dim)](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERT) ported to Huggingface transformers.


This is the BERT-Medium model from Google: https://github.com/google-research/bert#bert. A BERT model with 4 layers, 256 hidden unit size, and 4 attention heads.


This is the [General_TinyBERT_v2(4layer-312dim)](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERT) ported to Huggingface transformers.


---
license: mit
---

## MiniLM: 6 Layer Version

This is a 6 layer version of [microsoft/MiniLM-L12-H384-uncased](https://huggingface.co/microsoft/MiniLM-L12-H384-uncased/) by keeping only every second layer.


This is the BERT-Medium model from Google: https://github.com/google-research/bert#bert. A BERT model with 2 layers, 128 hidden unit size, and 2 attention heads.


This is the zero-shot baseline model in the paper ["GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval"](https://arxiv.org/abs/2112.07577)

The training setup:
1. Start from `distilbert-base-uncased`;
2. Mine 50 hard negatives for each query on MS MARCO with `sentence-transformers/msmarco-distilbert-base-v3` and `sentence-transformers/msmarco-MiniLM-L-6-v3`;
3. Do Margin-MSE training on the tuples (including queries, gold relevant, and hard negatives) with the teacher model `cross-encoder/ms-marco-MiniLM-L-6-v2` for 70K steps with batch size 75, max. sequence-length 350.


Naming pattern:

1. `GPL/${dataset}-msmarco-distilbert-gpl`: Model with training order of (1) MarginMSE on MSMARCO -> (2) GPL on ${dataset};
2. `GPL/${dataset}-tsdae-msmarco-distilbert-gpl`: Model with training order of (1) TSDAE on ${dataset} -> (2) MarginMSE on MSMARCO -> (3) GPL on ${dataset};
3. `GPL/msmarco-distilbert-margin-mse`: Model trained on MSMARCO with MarginMSE;
4. `GPL/${dataset}-tsdae-msmarco-distilbert-margin-mse`: Model with training order of (1) TSDAE on ${dataset} -> (2) MarginMSE on MSMARCO;

Actually, models in 1. and 2. are built on top of 3. and 4., respectively.


---
language: multilingual
tags:
- LID
- spoken language recognition
license: apache-2.0
datasets:
- VoxLingua107
metrics:
- ER
inference: false
---

# Spoken Language Identification Model

## Model description

The model can classify a speech utterance according to the language spoken.
It covers following different languages (
English,
Indonesian,
Japanese,
Korean,
Thai,
Vietnamese,
Mandarin Chinese).


---
language: multilingual
thumbnail:
tags:
- audio-classification
- speechbrain
- embeddings
- Language
- Identification
- pytorch
- ECAPA-TDNN
- TDNN
- VoxLingua107
license: "apache-2.0"
datasets:
- VoxLingua107
metrics:
- Accuracy
widget:
- example_title: English Sample
  src: https://cdn-media.huggingface.co/speech_samples/LibriSpeech_61-70968-0000.flac
---

# VoxLingua107 ECAPA-TDNN Spoken Language Identification Model

## Model description

This is a spoken language recognition model trained on the VoxLingua107 dataset using SpeechBrain.
The model uses the ECAPA-TDNN architecture that has previously been used for speaker recognition.

The model can classify a speech utterance according to the language spoken.
It covers 107 different languages (
Abkhazian,
Afrikaans,
Amharic,
Arabic,
Assamese,
Azerbaijani,
Bashkir,
Belarusian,
Bulgarian,
Bengali,
Tibetan,
Breton,
Bosnian,
Catalan,
Cebuano,
Czech,
Welsh,
Danish,
German,
Greek,
English,
Esperanto,
Spanish,
Estonian,
Basque,
Persian,
Finnish,
Faroese,
French,
Galician,
Guarani,
Gujarati,
Manx,
Hausa,
Hawaiian,
Hindi,
Croatian,
Haitian,
Hungarian,
Armenian,
Interlingua,
Indonesian,
Icelandic,
Italian,
Hebrew,
Japanese,
Javanese,
Georgian,
Kazakh,
Central Khmer,
Kannada,
Korean,
Latin,
Luxembourgish,
Lingala,
Lao,
Lithuanian,
Latvian,
Malagasy,
Maori,
Macedonian,
Malayalam,
Mongolian,
Marathi,
Malay,
Maltese,
Burmese,
Nepali,
Dutch,
Norwegian Nynorsk,
Norwegian,
Occitan,
Panjabi,
Polish,
Pushto,
Portuguese,
Romanian,
Russian,
Sanskrit,
Scots,
Sindhi,
Sinhala,
Slovak,
Slovenian,
Shona,
Somali,
Albanian,
Serbian,
Sundanese,
Swedish,
Swahili,
Tamil,
Telugu,
Tajik,
Thai,
Turkmen,
Tagalog,
Turkish,
Tatar,
Ukrainian,
Urdu,
Uzbek,
Vietnamese,
Waray,
Yiddish,
Yoruba,
Mandarin Chinese).

## Intended uses & limitations

The model has two uses:

  - use 'as is' for spoken language recognition
  - use as an utterance-level feature (embedding) extractor, for creating a dedicated language ID model on your own data

The model is trained on automatically collected YouTube data. For more
information about the dataset, see [here](http://bark.phon.ioc.ee/voxlingua107/).

#### How to use

```python
import torchaudio
from speechbrain.pretrained import EncoderClassifier
language_id = EncoderClassifier.from_hparams(source="TalTechNLP/voxlingua107-epaca-tdnn", savedir="tmp")
# Download Thai language sample from Omniglot and cvert to suitable form
signal = language_id.load_audio("https://omniglot.com/soundfiles/udhr/udhr_th.mp3")
prediction =  language_id.classify_batch(signal)
print(prediction)
  (tensor([[0.3210, 0.3751, 0.3680, 0.3939, 0.4026, 0.3644, 0.3689, 0.3597, 0.3508,
           0.3666, 0.3895, 0.3978, 0.3848, 0.3957, 0.3949, 0.3586, 0.4360, 0.3997,
           0.4106, 0.3886, 0.4177, 0.3870, 0.3764, 0.3763, 0.3672, 0.4000, 0.4256,
           0.4091, 0.3563, 0.3695, 0.3320, 0.3838, 0.3850, 0.3867, 0.3878, 0.3944,
           0.3924, 0.4063, 0.3803, 0.3830, 0.2996, 0.4187, 0.3976, 0.3651, 0.3950,
           0.3744, 0.4295, 0.3807, 0.3613, 0.4710, 0.3530, 0.4156, 0.3651, 0.3777,
           0.3813, 0.6063, 0.3708, 0.3886, 0.3766, 0.4023, 0.3785, 0.3612, 0.4193,
           0.3720, 0.4406, 0.3243, 0.3866, 0.3866, 0.4104, 0.4294, 0.4175, 0.3364,
           0.3595, 0.3443, 0.3565, 0.3776, 0.3985, 0.3778, 0.2382, 0.4115, 0.4017,
           0.4070, 0.3266, 0.3648, 0.3888, 0.3907, 0.3755, 0.3631, 0.4460, 0.3464,
           0.3898, 0.3661, 0.3883, 0.3772, 0.9289, 0.3687, 0.4298, 0.4211, 0.3838,
           0.3521, 0.3515, 0.3465, 0.4772, 0.4043, 0.3844, 0.3973, 0.4343]]), tensor([0.9289]), tensor([94]), ['th'])
# The scores in the prediction[0] tensor can be interpreted as cosine scores between
# the languages and the given utterance (i.e., the larger the better)
# The identified language ISO code is given in prediction[3]
print(prediction[3])
  ['th']

# Alternatively, use the utterance embedding extractor:
emb =  language_id.encode_batch(signal)
print(emb.shape)
  torch.Size([1, 1, 256])
```

#### Limitations and bias

Since the model is trained on VoxLingua107, it has many limitations and biases, some of which are:

 - Probably it's accuracy on smaller languages  is quite limited
 - Probably it works worse on female speech than male speech (because YouTube data includes much more male speech)
 - Based on subjective experiments, it doesn't work well on speech with a foreign accent
 - Probably it doesn't work well on children's speech and on persons with speech disorders

## Training data

The model is trained on [VoxLingua107](http://bark.phon.ioc.ee/voxlingua107/).

VoxLingua107 is a speech dataset for training spoken language identification models.
The dataset consists of short speech segments automatically extracted from YouTube videos and labeled according the language of the video title and description, with some post-processing steps to filter out false positives.

VoxLingua107 contains data for 107 languages. The total amount of speech in the training set is 6628 hours.
The average amount of data per language is 62 hours. However, the real amount per language varies a lot. There is also a seperate development set containing 1609 speech segments from 33 languages, validated by at least two volunteers to really contain the given language.

## Training procedure

We used [SpeechBrain](https://github.com/speechbrain/speechbrain) to train the model.
Training recipe will be published soon.

## Evaluation results

Error rate: 7% on the development dataset

### BibTeX entry and citation info

```bibtex
@inproceedings{valk2021slt,
  title={{VoxLingua107}: a Dataset for Spoken Language Recognition},
  author={J{\"o}rgen Valk and Tanel Alum{\"a}e},
  booktitle={Proc. IEEE SLT Workshop},
  year={2021},
}
```


---
language: pt
tags:
- portuguese
- brazil
- pt_BR
widget:
- text: Brasilia é a capital do <mask>
---

``` python
from transformers import pipeline
unmasker = pipeline('fill-mask', model='josu/roberta-pt-br')
text = 'Brasilia é a capital do <mask>'

[{'sequence': 'Brasilia é a capital do Brasil',
  'score': 0.24386335909366608,
  'token': 707,
  'token_str': ' Brasil'},
 {'sequence': 'Brasilia é a capital do estado',
  'score': 0.2320091277360916,
  'token': 1031,
  'token_str': ' estado'},
 {'sequence': 'Brasilia é a capital do país',
  'score': 0.0665697380900383,
  'token': 998,
  'token_str': ' país'},
 {'sequence': 'Brasilia é a capital do Rio',
  'score': 0.05980581417679787,
  'token': 993,
  'token_str': ' Rio'},
 {'sequence': 'Brasilia é a capital do capital',
  'score': 0.058453518897295,
  'token': 2027,
  'token_str': ' capital'}]

  ```


---
language: vi
datasets:
- common_voice
- FOSD: https://data.mendeley.com/datasets/k9sxg2twv4/4
metrics:
- wer
tags:
- language-modeling
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: MT5 Fix Asr Vietnamese by Ontocord
  results:
  - task:
      name: Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice vi
      type: common_voice
      args: vi
    metrics:
       - name: Test WER
         type: wer
         value: 25.207182
---


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: finetune-data-skills
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# finetune-data-skills

This model is a fine-tuned version of [bert-base-uncased](https://huggingface.co/bert-base-uncased) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 2.1058

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 2.7239        | 1.0   | 3926  | 2.2459          |
| 2.3113        | 2.0   | 7852  | 2.1255          |
| 2.197         | 3.0   | 11778 | 2.0966          |

### Framework versions

- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.18.0
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: finetune-clm-employment
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# finetune-clm-employment

This model is a fine-tuned version of [distilroberta-base](https://huggingface.co/distilroberta-base) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 1.8445

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 2.3283        | 1.0   | 3989  | 1.9578          |
| 2.0824        | 2.0   | 7978  | 1.9013          |
| 1.9936        | 3.0   | 11967 | 1.8625          |

### Framework versions

- Transformers 4.14.1
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- squad
model-index:
- name: albert-large-v2_squad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# albert-large-v2_squad

This model is a fine-tuned version of [albert-large-v2](https://huggingface.co/albert-large-v2) on the **squadV1** dataset.

- "eval_exact_match": 84.80605487228004
- "eval_f1": 91.80638438705844
- "eval_samples": 10808

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 16
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Training results

### Framework versions

- Transformers 4.14.1
- Pytorch 1.9.0
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- squad
model-index:
- name: albert-base-v2_squad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# albert-base-v2_squad

This model is a fine-tuned version of [albert-base-v2](https://huggingface.co/albert-base-v2) on the **squadV1** dataset.
- "eval_exact_match": 82.69631031220435
- "eval_f1": 90.10806626207174
- "eval_samples": 10808
## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 16
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Training results

### Framework versions

- Transformers 4.14.1
- Pytorch 1.9.0
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: mit
tags:
- generated_from_trainer
datasets:
- squad
model-index:
- name: microsoft_deberta-base_squad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# microsoft_deberta-base_squad

This model is a fine-tuned version of [microsoft/deberta-base](https://huggingface.co/microsoft/deberta-base) on the **squadV1** dataset.
- "eval_exact_match": 86.30085146641439
- "eval_f1": 92.68502275661561
- "eval_samples": 10788

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 12
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Training results

### Framework versions

- Transformers 4.14.1
- Pytorch 1.9.0
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- squad
model-index:
- name: google_electra-small-discriminator_squad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# google_electra-small-discriminator_squad

This model is a fine-tuned version of [google/electra-small-discriminator](https://huggingface.co/google/electra-small-discriminator) on the **squadV1** dataset.

- "eval_exact_match": 76.95364238410596
- "eval_f1": 84.98869246841396
- "eval_samples": 10784

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 16
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Training results

### Framework versions

- Transformers 4.14.1
- Pytorch 1.9.0
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- squad
model-index:
- name: distilroberta-base_squad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilroberta-base_squad

This model is a fine-tuned version of [distilroberta-base](https://huggingface.co/distilroberta-base) on the **squadV1** dataset.

- "eval_exact_match": 80.97445600756859
- "eval_f1": 88.0153886332912
- "eval_samples": 10790

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Training results

### Framework versions

- Transformers 4.14.1
- Pytorch 1.9.0
- Datasets 1.16.1
- Tokenizers 0.10.3


---
tags:
- generated_from_trainer
datasets:
- squad
model-index:
- name: xlm-roberta-base_squad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# eval

This model is a fine-tuned version of [xlm-roberta-large](https://huggingface.co/xlm-roberta-large) on the squad dataset.

- eval_exact_match": 85.96026490066225
- "eval_f1": 92.25000664341768
- "eval_samples": 10918

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 12
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 0.67

### Framework versions

- Transformers 4.14.1
- Pytorch 1.9.0
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- squad
model-index:
- name: google_electra-base-discriminator_squad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# google_electra-base-discriminator_squad

This model is a fine-tuned version of [google/electra-base-discriminator](https://huggingface.co/google/electra-base-discriminator) on the **squadV1** dataset.
- "eval_exact_match": 85.33585619678335
- "eval_f1": 91.97363450387108
- "eval_samples": 10784

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 16
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Training results

### Framework versions

- Transformers 4.14.1
- Pytorch 1.9.0
- Datasets 1.16.1
- Tokenizers 0.10.3


---
license: mit
tags:
- generated_from_trainer
datasets:
- squad
model-index:
- name: xlm-roberta-base_squad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# xlm-roberta-base_squad

This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the squad dataset.
- "eval_exact_match": 82.69631031220435
- "eval_f1": 89.4562841806503
- "eval_samples": 10918

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Training results

### Framework versions

- Transformers 4.14.1
- Pytorch 1.9.0
- Datasets 1.16.1
- Tokenizers 0.10.3


---
tags:
- generated_from_trainer
datasets:
- squad
model-index:
- name: microsoft-deberta-large
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# microsoft-deberta-large

This model is a fine-tuned version of [microsoft_deberta-large](https://huggingface.co/microsoft/deberta-large) on the **squadV1** dataset.

- "eval_exact_match": 87.89025543992432
- "eval_f1": 93.8755152147345
- "eval_samples": 10788

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 12
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Framework versions

- Transformers 4.14.1
- Pytorch 1.9.0
- Datasets 1.16.1
- Tokenizers 0.10.3


---
widget:
- text: "Even the Dwarves"
- text: "The secrets of"
---

# Model name
Magic The Generating

## Model description

This is a fine tuned GPT-2 model trained on a corpus of all available English language Magic the Gathering card flavour texts.

## Intended uses & limitations

This is intended only for use in generating new, novel, and sometimes surprising, MtG like flavour texts.

#### How to use

```python
from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained("rjbownes/Magic-The-Generating")

model = GPT2LMHeadModel.from_pretrained("rjbownes/Magic-The-Generating")

```

#### Limitations and bias

The training corpus was surprisingly small, only ~29000 cards, I had suspected there were more. This might mean there is a real limit to the number of entirely original strings this will generate.
This is also only based on the 117M parameter GPT2, it's a pretty obvious upgrade to retrain with medium, large or XL models. However, despite this, the outputs I tested were very convincing!

## Training data

The data was 29222 MtG card flavour texts. The model was based on the "gpt2" pretrained transformer: https://huggingface.co/gpt2.

## Training procedure

Only English language MtG flavour texts were scraped from the [Scryfall](https://scryfall.com/) API. Empty strings and any non-UTF-8 encoded tokens were removed leaving 29222 entries.
This was trained using google Colab with a T4 instance. 4 epochs, adamW optimizer with default parameters and a batch size of 32. Token embedding lengths were capped at 98 tokens as this was the longest string and an attention mask was added to the training model to ignore all padding tokens.

## Eval results

Average Training Loss: 0.44866578806635815.
Validation loss: 0.5606984243444775.

Sample model outputs:

1. "Every branch a crossroads, every vine a swift steed."
	—Gwendlyn Di Corci

2. "The secrets of this world will tell their masters where to strike if need be."
	—Noyan Dar, Tazeem roilmage

3. "The secrets of nature are expensive. You'd be better off just to have more freedom."

4. "Even the Dwarves knew to leave some stones unturned."

5. "The wise always keep an ear open to the whispers of power."

### BibTeX entry and citation info

```bibtex
@article{BownesLM,
  title={Fine Tuning GPT-2 for Magic the Gathering flavour text generation.},
  author={Richard J. Bownes},
  journal={Medium},
  year={2020}
}

```


---
tags:
- conversational
---

#Okarin Bot


---
license: mit
tags:
- generated_from_trainer
model-index:
- name: bert-base-italian-uncased-finetuned-ComunaliRoma
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bert-base-italian-uncased-finetuned-ComunaliRoma

This model is a fine-tuned version of [dbmdz/bert-base-italian-uncased](https://huggingface.co/dbmdz/bert-base-italian-uncased) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 3.0398

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 156  | 3.1907          |
| No log        | 2.0   | 312  | 3.0522          |
| No log        | 3.0   | 468  | 3.0203          |

### Framework versions

- Transformers 4.11.3
- Pytorch 1.9.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


---
license: mit
tags:
- generated_from_trainer
model-index:
- name: bert-base-italian-xxl-uncased-finetuned-ComunaliRoma
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bert-base-italian-xxl-uncased-finetuned-ComunaliRoma

This model is a fine-tuned version of [dbmdz/bert-base-italian-xxl-uncased](https://huggingface.co/dbmdz/bert-base-italian-xxl-uncased) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 2.5095

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 2.6717        | 1.0   | 1014 | 2.6913          |
| 2.4869        | 2.0   | 2028 | 2.5843          |
| 2.3411        | 3.0   | 3042 | 2.5095          |

### Framework versions

- Transformers 4.11.3
- Pytorch 1.9.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


# Text classifier using DistilBERT to determine Partisanship
## This is one of the single-class partisan detecting models. (see leftpartisan/leftcenterpartisan/rightcenterpartisan/centerpartisan)

label_0 refers to "other" while label_1 refers to "right" (right as in right-leaning).

This was trained with 40,000 articles.

### Best Practices
This model was optimized for 512 token-length text. Any text below 150 tokens will result in inaccurate results.


# Text classifier using DistilBERT to determine Partisanship
## This is one of many single-class partisanship models

label_0 refers to "left" while label_1 refers to "other".

This model was trained on 40,000 articles.

### Best Practices
This model was optimized for 512 token-length text. Any text below 150 tokens will result in inaccurate results.


---
license: apache-2.0
tags:
- generated_from_trainer
- ta
- robust-speech-event
datasets:
- common_voice
model-index:
- name: Tamil-Wav2Vec-xls-r-300m-Tamil-colab
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Tamil-Wav2Vec-xls-r-300m-Tamil-colab

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 30
- mixed_precision_training: Native AMP

### Framework versions

- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.18.3
- Tokenizers 0.10.3


---
license: mit
tags:
- generated_from_keras_callback
model-index:
- name: roberta-model-english
  results: []
---

<!-- This model card has been generated automatically according to the information Keras had access to. You should
probably proofread and complete it, then remove this comment. -->

# roberta-model-english

This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset.
It achieves the following results on the evaluation set:
- Train Loss: 0.1140
- Train Accuracy: 0.9596
- Validation Loss: 0.2166
- Validation Accuracy: 0.9301
- Epoch: 2

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- optimizer: {'name': 'Adam', 'learning_rate': 5e-05, 'decay': 0.0, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-07, 'amsgrad': False}
- training_precision: float32

### Training results

| Train Loss | Train Accuracy | Validation Loss | Validation Accuracy | Epoch |
|:----------:|:--------------:|:---------------:|:-------------------:|:-----:|
| 0.2922     | 0.8804         | 0.2054          | 0.9162              | 0     |
| 0.1710     | 0.9352         | 0.1879          | 0.9353              | 1     |
| 0.1140     | 0.9596         | 0.2166          | 0.9301              | 2     |

### Framework versions

- Transformers 4.16.2
- TensorFlow 2.7.0
- Tokenizers 0.11.0


---
license: apache-2.0
tags:
- generated_from_keras_callback
model-index:
- name: bert-model-english
  results: []
---

<!-- This model card has been generated automatically according to the information Keras had access to. You should
probably proofread and complete it, then remove this comment. -->

# bert-model-english

This model is a fine-tuned version of [bert-base-cased](https://huggingface.co/bert-base-cased) on an unknown dataset.
It achieves the following results on the evaluation set:
- Train Loss: 0.1408
- Train Sparse Categorical Accuracy: 0.9512
- Validation Loss: nan
- Validation Sparse Categorical Accuracy: 0.0
- Epoch: 4

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- optimizer: {'name': 'Adam', 'learning_rate': 5e-05, 'decay': 0.0, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-07, 'amsgrad': False}
- training_precision: float32

### Training results

| Train Loss | Train Sparse Categorical Accuracy | Validation Loss | Validation Sparse Categorical Accuracy | Epoch |
|:----------:|:---------------------------------:|:---------------:|:--------------------------------------:|:-----:|
| 0.2775     | 0.8887                            | nan             | 0.0                                    | 0     |
| 0.1702     | 0.9390                            | nan             | 0.0                                    | 1     |
| 0.1300     | 0.9555                            | nan             | 0.0                                    | 2     |
| 0.1346     | 0.9544                            | nan             | 0.0                                    | 3     |
| 0.1408     | 0.9512                            | nan             | 0.0                                    | 4     |

### Framework versions

- Transformers 4.16.2
- TensorFlow 2.7.0
- Datasets 1.18.3
- Tokenizers 0.11.0


---
license: apache-2.0
tags:
- generated_from_keras_callback
model-index:
- name: bert-model-english1
  results: []
---

<!-- This model card has been generated automatically according to the information Keras had access to. You should
probably proofread and complete it, then remove this comment. -->

# bert-model-english1

This model is a fine-tuned version of [bert-base-cased](https://huggingface.co/bert-base-cased) on an unknown dataset.
It achieves the following results on the evaluation set:
- Train Loss: 0.0274
- Train Accuracy: 0.9914
- Validation Loss: 0.3493
- Validation Accuracy: 0.9303
- Epoch: 2

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- optimizer: {'name': 'Adam', 'learning_rate': 5e-05, 'decay': 0.0, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-07, 'amsgrad': False}
- training_precision: float32

### Training results

| Train Loss | Train Accuracy | Validation Loss | Validation Accuracy | Epoch |
|:----------:|:--------------:|:---------------:|:-------------------:|:-----:|
| 0.0366     | 0.9885         | 0.3013          | 0.9299              | 0     |
| 0.0261     | 0.9912         | 0.3445          | 0.9351              | 1     |
| 0.0274     | 0.9914         | 0.3493          | 0.9303              | 2     |

### Framework versions

- Transformers 4.16.2
- TensorFlow 2.7.0
- Datasets 1.18.3
- Tokenizers 0.11.0


---
tags:
- generated_from_trainer
metrics:
- accuracy
model-index:
- name: finetuned-model
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# finetuned-model

This model is a fine-tuned version of [cardiffnlp/twitter-roberta-base-sentiment](https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.8601
- Accuracy: 0.6117

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2

### Training results

| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| No log        | 1.0   | 84   | 0.8663          | 0.5914   |
| No log        | 2.0   | 168  | 0.8601          | 0.6117   |

### Framework versions

- Transformers 4.16.1
- Pytorch 1.10.0+cu111
- Datasets 1.18.2
- Tokenizers 0.11.0


---
language:
- en
license: mit
widget:
- text: "Nonviolence is justified because"

---
GPT2 fine-tuned on a dataset of pacifist philosophy and foreign policy texts.


---
language:
- ja
tags:
- text generation
- pytorch
- causal-lm
- japanese
license: apache-2.0
datasets:
- oscar
- cc100
- wikipedia
---

# GPT-Neo 1.3B pre-trained model for Japanese

## Model Description

GPT2/GPT3 like model trained on Japanese.corpus.

## Training data

- cc100 ja
- oscar ja
- wikipedia ja

## How to use

```
from transformers import pipeline
>>> generator = pipeline('text-generation', model='yellowback/gpt-neo-japanese-1.3B')
>>> generator("こんばんは、徳川家康です。", do_sample=True, max_length=50, num_return_sequences=3)

[{'generated_text': 'こんばんは、徳川家康です。 世の中を見渡してみても、残念なことだけれども、まぎれもなく「世のなか...\n5月になりました、家康です。 ゴールデンウィークも終ってしまい、世間では'},
 {'generated_text': 'こんばんは、徳川家康です。さあ今日は昨晩から降り続いた雨は上がりましたが、まだまだ雨脚は強いですが、晴れるところは晴れて欲しいですね。昨日の夜は仕事だったので、今日の夕'},
 {'generated_text': 'こんばんは、徳川家康です。 今回は、『世界史再考──日本史再考』という本を書いたあと、『世界史再考──日本史再考』の6~8章'}]
```


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: wav2vec2-base-timit-demo-colab
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-base-timit-demo-colab

This model is a fine-tuned version of [facebook/wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base) on the None dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 1
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


---
tags:
- conversational
---

#Eren Yeager DialoGPT Model


---
language: id
datasets:
- oscar
---
# IndoBERT (Indonesian BERT Model)

## Model description
IndoBERT is a pre-trained language model based on BERT architecture for the Indonesian Language.

This model is base-uncased version which use bert-base config.

## Intended uses & limitations

#### How to use

```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("sarahlintang/IndoBERT")
model = AutoModel.from_pretrained("sarahlintang/IndoBERT")
tokenizer.encode("hai aku mau makan.")
[2, 8078, 1785, 2318, 1946, 18, 4]
```

## Training data

This model was pre-trained on 16 GB of raw text ~2 B words from Oscar Corpus (https://oscar-corpus.com/).

This model is equal to bert-base model which has 32,000 vocabulary size.

## Training procedure

The training of the model has been performed using Google’s original Tensorflow code on eight core Google Cloud TPU v2.
We used a Google Cloud Storage bucket, for persistent storage of training data and models.

## Eval results

We evaluate this model on three Indonesian NLP downstream task:
- some extractive summarization model
- sentiment analysis
- Part-of-Speech Tagger
it was proven that this model outperforms multilingual BERT for all downstream tasks.


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- financial_phrasebank
metrics:
- recall
- accuracy
- precision
model-index:
- name: financial_sentiment_model
  results:
  - task:
      name: Text Classification
      type: text-classification
    dataset:
      name: financial_phrasebank
      type: financial_phrasebank
      args: sentences_50agree
    metrics:
    - name: Recall
      type: recall
      value: 0.8839956357328868
    - name: Accuracy
      type: accuracy
      value: 0.8804123711340206
    - name: Precision
      type: precision
      value: 0.8604175202419276
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# financial_sentiment_model

This model is a fine-tuned version of [deepmind/language-perceiver](https://huggingface.co/deepmind/language-perceiver) on the financial_phrasebank dataset.
It achieves the following results on the evaluation set:
- Loss: 0.3467
- Recall: 0.8840
- Accuracy: 0.8804
- Precision: 0.8604

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- distributed_type: tpu
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 4

### Training results

| Training Loss | Epoch | Step | Validation Loss | Recall | Accuracy | Precision |
|:-------------:|:-----:|:----:|:---------------:|:------:|:--------:|:---------:|
| 0.4481        | 1.0   | 273  | 0.4035          | 0.8526 | 0.8433   | 0.7955    |
| 0.4069        | 2.0   | 546  | 0.4478          | 0.8683 | 0.8289   | 0.8123    |
| 0.2225        | 3.0   | 819  | 0.3167          | 0.8747 | 0.8680   | 0.8387    |
| 0.1245        | 4.0   | 1092 | 0.3467          | 0.8840 | 0.8804   | 0.8604    |

### Framework versions

- Transformers 4.15.0
- Pytorch 1.9.0+cu102
- Datasets 1.17.0
- Tokenizers 0.10.3


---
language: en
tags:
- grammar
- text2text-generation
license: cc-by-nc-sa-4.0
datasets:
- jfleg
---

# T5 Grammar Correction

This model generates a revised version of inputted text with the goal of containing fewer grammatical errors.
It was trained with [Happy Transformer](https://github.com/EricFillion/happy-transformer)
using a dataset called [JFLEG](https://arxiv.org/abs/1702.04066). Here's a [full article](https://www.vennify.ai/fine-tune-grammar-correction/) on how to train a similar model.

## Usage

`pip install happytransformer `

```python
from happytransformer import HappyTextToText, TTSettings

happy_tt = HappyTextToText("T5", "vennify/t5-base-grammar-correction")

args = TTSettings(num_beams=5, min_length=1)

# Add the prefix "grammar: " before each input
result = happy_tt.generate_text("grammar: This sentences has has bads grammar.", args=args)

print(result.text) # This sentence has bad grammar.

```


---
language: "mr"
---

# Welcome to Roberta-Marathi-MLM

## Model Description

> This is a small language model for [Marathi](https://en.wikipedia.org/wiki/Marathi) language with 1M data samples taken from
  [OSCAR page](https://oscar-public.huma-num.fr/shuffled/mr_dedup.txt.gz)

## Training params

- **Dataset** - 1M data samples are used to train this model from OSCAR page(https://oscar-corpus.com/) eventhough data set is of 2.7 GB due to resource constraint to train
I have picked only 1M data from the total 2.7GB data set. If you are interested in collaboration and have computational resources to train on you are most welcome to do so.

- **Preprocessing** - ByteLevelBPETokenizer is used to tokenize the sentences at character level and vocabulary size is set to 52k as per standard values given by ðŸ¤—
<!-- - **Hyperparameters** - __ByteLevelBPETokenizer__ : vocabulary size = 52_000 and  min_frequency = 2
                        __Trainer__ :               num_train_epochs=12 - trained for 12 epochs
                                                    per_gpu_train_batch_size=64 - batch size for the datasamples is 64
                                                    save_steps=10_000 - save model for every 10k steps
                                                    save_total_limit=2 - save limit is set for 2 -->

**Intended uses & limitations**
  this is for anyone who wants to make use of marathi language models for various tasks like language generation, translation and many more use cases.

**Whatever else is helpful!**
  If you are intersted in collaboration feel free to reach  me [Deepam](mailto:deepam8155@gmail.com)


---
language: pt
tags:
- portuguese
- brazil
- pt_BR
widget:
- text: gostei muito dessa <mask>
---

# BR_BERTo

Portuguese (Brazil) model for text inference.

## Params

Trained on a corpus of 6_993_330 sentences.

- Vocab size: 150_000
- RobertaForMaskedLM  size : 512
- Num train epochs: 3
- Time to train: ~10days (on GCP with a Nvidia T4)

I follow the great tutorial from HuggingFace team:

[How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train)

More infor here:

[BR_BERTo](https://github.com/rdenadai/BR-BERTo)


---
license: apache-2.0
tags:
- generated_from_trainer
metrics:
- precision
- recall
model-index:
- name: distilbert-magazine-classifier
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert-magazine-classifier

This model is a fine-tuned version of [distilbert-base-cased](https://huggingface.co/distilbert-base-cased) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 1.8377
- Precision: 0.25
- Recall: 0.125
- Fscore: 0.1667

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Training results

| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | Fscore |
|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|
| 0.1779        | 1.0   | 2    | 1.7584          | 0.2222    | 0.3333 | 0.2667 |
| 0.1635        | 2.0   | 4    | 1.7585          | 0.25      | 0.125  | 0.1667 |
| 0.1405        | 3.0   | 6    | 1.8377          | 0.25      | 0.125  | 0.1667 |

### Framework versions

- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


---
language: ko
tags:
  - korean
  - klue
mask_token: "[MASK]"
widget:
  - text: 대한민국의 수도는 [MASK] 입니다.
---

# KLUE RoBERTa small

Pretrained RoBERTa Model on Korean Language. See [Github](https://github.com/KLUE-benchmark/KLUE) and [Paper](https://arxiv.org/abs/2105.09680) for more details.

## How to use

_NOTE:_ Use `BertTokenizer` instead of RobertaTokenizer. (`AutoTokenizer` will load `BertTokenizer`)

```python
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("klue/roberta-small")
tokenizer = AutoTokenizer.from_pretrained("klue/roberta-small")
```

## BibTeX entry and citation info

```bibtex
@misc{park2021klue,
      title={KLUE: Korean Language Understanding Evaluation},
      author={Sungjoon Park and Jihyung Moon and Sungdong Kim and Won Ik Cho and Jiyoon Han and Jangwon Park and Chisung Song and Junseong Kim and Yongsook Song and Taehwan Oh and Joohong Lee and Juhyun Oh and Sungwon Lyu and Younghoon Jeong and Inkwon Lee and Sangwoo Seo and Dongjun Lee and Hyunwoo Kim and Myeonghwa Lee and Seongbo Jang and Seungwon Do and Sunkyoung Kim and Kyungtae Lim and Jongwon Lee and Kyumin Park and Jamin Shin and Seonghyun Kim and Lucy Park and Alice Oh and Jungwoo Ha and Kyunghyun Cho},
      year={2021},
      eprint={2105.09680},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


---
language: ko
tags:
  - korean
  - klue
mask_token: "[MASK]"
widget:
  - text: 대한민국의 수도는 [MASK] 입니다.
---

# KLUE RoBERTa base

Pretrained RoBERTa Model on Korean Language. See [Github](https://github.com/KLUE-benchmark/KLUE) and [Paper](https://arxiv.org/abs/2105.09680) for more details.

## How to use

_NOTE:_ Use `BertTokenizer` instead of RobertaTokenizer. (`AutoTokenizer` will load `BertTokenizer`)

```python
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("klue/roberta-base")
tokenizer = AutoTokenizer.from_pretrained("klue/roberta-base")
```

## BibTeX entry and citation info

```bibtex
@misc{park2021klue,
      title={KLUE: Korean Language Understanding Evaluation},
      author={Sungjoon Park and Jihyung Moon and Sungdong Kim and Won Ik Cho and Jiyoon Han and Jangwon Park and Chisung Song and Junseong Kim and Yongsook Song and Taehwan Oh and Joohong Lee and Juhyun Oh and Sungwon Lyu and Younghoon Jeong and Inkwon Lee and Sangwoo Seo and Dongjun Lee and Hyunwoo Kim and Myeonghwa Lee and Seongbo Jang and Seungwon Do and Sunkyoung Kim and Kyungtae Lim and Jongwon Lee and Kyumin Park and Jamin Shin and Seonghyun Kim and Lucy Park and Alice Oh and Jungwoo Ha and Kyunghyun Cho},
      year={2021},
      eprint={2105.09680},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


---
language: ko
tags:
  - korean
  - klue
mask_token: "[MASK]"
widget:
  - text: 대한민국의 수도는 [MASK] 입니다.
---

# KLUE BERT base

Pretrained BERT Model on Korean Language. See [Github](https://github.com/KLUE-benchmark/KLUE) and [Paper](https://arxiv.org/abs/2105.09680) for more details.

## How to use

```python
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("klue/bert-base")
tokenizer = AutoTokenizer.from_pretrained("klue/bert-base")
```

## BibTeX entry and citation info

```bibtex
@misc{park2021klue,
      title={KLUE: Korean Language Understanding Evaluation},
      author={Sungjoon Park and Jihyung Moon and Sungdong Kim and Won Ik Cho and Jiyoon Han and Jangwon Park and Chisung Song and Junseong Kim and Yongsook Song and Taehwan Oh and Joohong Lee and Juhyun Oh and Sungwon Lyu and Younghoon Jeong and Inkwon Lee and Sangwoo Seo and Dongjun Lee and Hyunwoo Kim and Myeonghwa Lee and Seongbo Jang and Seungwon Do and Sunkyoung Kim and Kyungtae Lim and Jongwon Lee and Kyumin Park and Jamin Shin and Seonghyun Kim and Lucy Park and Alice Oh and Jungwoo Ha and Kyunghyun Cho},
      year={2021},
      eprint={2105.09680},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


---
language: ko
tags:
  - korean
  - klue
mask_token: "[MASK]"
widget:
  - text: 대한민국의 수도는 [MASK] 입니다.
---

# KLUE RoBERTa large

Pretrained RoBERTa Model on Korean Language. See [Github](https://github.com/KLUE-benchmark/KLUE) and [Paper](https://arxiv.org/abs/2105.09680) for more details.

## How to use

_NOTE:_ Use `BertTokenizer` instead of RobertaTokenizer. (`AutoTokenizer` will load `BertTokenizer`)

```python
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("klue/roberta-large")
tokenizer = AutoTokenizer.from_pretrained("klue/roberta-large")
```

## BibTeX entry and citation info

```bibtex
@misc{park2021klue,
      title={KLUE: Korean Language Understanding Evaluation},
      author={Sungjoon Park and Jihyung Moon and Sungdong Kim and Won Ik Cho and Jiyoon Han and Jangwon Park and Chisung Song and Junseong Kim and Yongsook Song and Taehwan Oh and Joohong Lee and Juhyun Oh and Sungwon Lyu and Younghoon Jeong and Inkwon Lee and Sangwoo Seo and Dongjun Lee and Hyunwoo Kim and Myeonghwa Lee and Seongbo Jang and Seungwon Do and Sunkyoung Kim and Kyungtae Lim and Jongwon Lee and Kyumin Park and Jamin Shin and Seonghyun Kim and Lucy Park and Alice Oh and Jungwoo Ha and Kyunghyun Cho},
      year={2021},
      eprint={2105.09680},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


---
tags:
- conversational
---

# Moragna DialoGPT Model