---
language: el #TODO: replace {lang_id} in your language code here. Make sure the code is one of the *ISO codes* of [this](https://huggingface.co/languages) site.
datasets:
- common_voice #TODO: remove if you did not use the common voice dataset
- CSS10
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: Greek XLSR Wav2Vec2 Large 53 - CV + CSS10 #TODO: replace {human_readable_name} with a name of your model as it should appear on the leaderboard. It could be something like `Elgeish XLSR Wav2Vec2 Large 53`
  results:
  - task:
      name: Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice el #TODO: replace {lang_id} in your language code here. Make sure the code is one of the *ISO codes* of [this](https://huggingface.co/languages) site.
      type: common_voice
      args: el #TODO: replace {lang_id} in your language code here. Make sure the code is one of the *ISO codes* of [this](https://huggingface.co/languages) site.
    metrics:
       - name: Test WER
         type: wer
         value: 20.89 #TODO (IMPORTANT): replace {wer_result_on_test} with the WER error rate you achieved on the common_voice test set. It should be in the format XX.XX (don't add the % sign here). **Please** remember to fill out this value after you evaluated your model, so that your model appears on the leaderboard. If you fill out this model card before evaluating your model, please remember to edit the model card afterward to fill in your value
---

# Wav2Vec2-Large-XLSR-53-greek

Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on greek using the [Common Voice](https://huggingface.co/datasets/common_voice) and [CSS10](https://github.com/Kyubyong/css10) datasets.
When using this model, make sure that your speech input is sampled at 16kHz.

## Usage

The model can be used directly (without a language model) as follows:

```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

test_dataset = load_dataset("common_voice", "el", split="test")

processor = Wav2Vec2Processor.from_pretrained("PereLluis13/wav2vec2-large-xlsr-53-greek")
model = Wav2Vec2ForCTC.from_pretrained("PereLluis13/wav2vec2-large-xlsr-53-greek")

resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)

with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits

predicted_ids = torch.argmax(logits, dim=-1)

print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```

## Evaluation

The model can be evaluated as follows on the greek test data of Common Voice.

```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re

test_dataset = load_dataset("common_voice", "el", split="test")
wer = load_metric("wer")

processor = Wav2Vec2Processor.from_pretrained("PereLluis13/wav2vec2-large-xlsr-53-greek")
model = Wav2Vec2ForCTC.from_pretrained("PereLluis13/wav2vec2-large-xlsr-53-greek")
model.to("cuda")

chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\%\‘\”\�]'

resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits

    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch

result = test_dataset.map(evaluate, batched=True, batch_size=8)

print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```

**Test Result**: 20.89 %

## Training

The Common Voice `train`, `validation`, and CSS10 datasets were used for training, added as `extra` split to the dataset. The sampling rate and format of the CSS10 files is different, hence the function `speech_file_to_array_fn` was changed to:
```
    def speech_file_to_array_fn(batch):
        try:
            speech_array, sampling_rate = sf.read(batch["path"] + ".wav")
        except:
            speech_array, sampling_rate = librosa.load(batch["path"], sr = 16000, res_type='zero_order_hold')
            sf.write(batch["path"] + ".wav", speech_array, sampling_rate, subtype='PCM_24')
        batch["speech"] = speech_array
        batch["sampling_rate"] = sampling_rate
        batch["target_text"] = batch["text"]
        return batch
```

As suggested by [Florian Zimmermeister](https://github.com/flozi00).

The script used for training can be found in [run_common_voice.py](examples/research_projects/wav2vec2/run_common_voice.py), still pending of PR. The only changes are to `speech_file_to_array_fn`. Batch size was kept at 32 (using `gradient_accumulation_steps`) using one of the [OVH](https://www.ovh.com/) machines, with a V100 GPU (thank you very much [OVH](https://www.ovh.com/)). The model trained for 40 epochs, the first 20 with the `train+validation` splits, and then `extra` split was added with the data from CSS10 at the 20th epoch.


# HateSpeechDetection
---
pipeline_tag: text-classification
---

The model is used for classifying a text as Hatespeech or Normal. The model is trained using data from Twitter, specifically Kenyan related tweets. To maximize on the limited dataset, text augmentation was done.

The dataset is available here: https://github.com/sgich/HateSpeechDetection

Using a pre-trained "bert-base-uncased" transformer model, adding a dropout layer, a linear output layer and adding 10 common emojis that may be related to either Hate or Normal Speech. Then the model was tuned on a dataset of Kenyan/Kenyan-related scraped tweets with the purpose of performing text classification of "Normal Speech" or "Hate Speech" based on the text. This model was the result of realizing that majority of similar models did not cater for the African context where the target groups are not based on race and/or religious affiliation but mostly tribal differences which has proved fatal in the past.

The model can be improved greatly by using a large and representative dataset and optimization of the model to a better degree.


---
tags:
- conversational
---

# Evan Model


---
tags:
- conversational
---

# Evelynn DialoGPT Model


---
language: en
tags:
- bart
- seq2seq
- summarization
license: apache-2.0
datasets:
- samsum
widget:
- text: |
    Jeff: Can I train a 🤗 Transformers model on Amazon SageMaker?
    Philipp: Sure you can use the new Hugging Face Deep Learning Container.
    Jeff: ok.
    Jeff: and how can I get started?
    Jeff: where can I find documentation?
    Philipp: ok, ok you can find everything here. https://huggingface.co/blog/the-partnership-amazon-sagemaker-and-hugging-face
model-index:
- name: bart-base-samsum
  results:
  - task:
      name: Abstractive Text Summarization
      type: abstractive-text-summarization
    dataset:
      name: "SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization"
      type: samsum
    metrics:
       - name: Validation ROGUE-1
         type: rogue-1
         value: 46.6619
       - name: Validation ROGUE-2
         type: rogue-2
         value: 23.3285
       - name: Validation ROGUE-L
         type: rogue-l
         value: 39.4811
       - name: Test ROGUE-1
         type: rogue-1
         value: 44.9932
       - name: Test ROGUE-2
         type: rogue-2
         value: 21.7286
       - name: Test ROGUE-L
         type: rogue-l
         value: 38.1921
---
## `bart-base-samsum`
This model was obtained by fine-tuning `facebook/bart-base` on Samsum dataset.

## Usage
```python
from transformers import pipeline

summarizer = pipeline("summarization", model="lidiya/bart-base-samsum")
conversation = '''Jeff: Can I train a 🤗 Transformers model on Amazon SageMaker?
Philipp: Sure you can use the new Hugging Face Deep Learning Container.
Jeff: ok.
Jeff: and how can I get started?
Jeff: where can I find documentation?
Philipp: ok, ok you can find everything here. https://huggingface.co/blog/the-partnership-amazon-sagemaker-and-hugging-face
'''
summarizer(conversation)
```

## Training procedure
- Colab notebook: https://colab.research.google.com/drive/1RInRjLLso9E2HG_xjA6j8JO3zXzSCBRF?usp=sharing

## Results
| key | value |
| --- | ----- |
| eval_rouge1 | 46.6619 |
| eval_rouge2 | 23.3285 |
| eval_rougeL | 39.4811 |
| eval_rougeLsum | 43.0482 |
| test_rouge1 | 44.9932 |
| test_rouge2 | 21.7286 |
| test_rougeL | 38.1921 |
| test_rougeLsum | 41.2672 |


---
language: en
tags:
- bart
- seq2seq
- summarization
license: apache-2.0
datasets:
- samsum
widget:
- text: |
    Hannah: Hey, do you have Betty's number?
    Amanda: Lemme check
    Amanda: Sorry, can't find it.
    Amanda: Ask Larry
    Amanda: He called her last time we were at the park together
    Hannah: I don't know him well
    Amanda: Don't be shy, he's very nice
    Hannah: If you say so..
    Hannah: I'd rather you texted him
    Amanda: Just text him 🙂
    Hannah: Urgh.. Alright
    Hannah: Bye
    Amanda: Bye bye
model-index:
- name: bart-large-xsum-samsum
  results:
  - task:
      name: Abstractive Text Summarization
      type: abstractive-text-summarization
    dataset:
      name: "SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization"
      type: samsum
    metrics:
       - name: Validation ROGUE-1
         type: rogue-1
         value: 54.3921
       - name: Validation ROGUE-2
         type: rogue-2
         value: 29.8078
       - name: Validation ROGUE-L
         type: rogue-l
         value: 45.1543
       - name: Test ROGUE-1
         type: rogue-1
         value: 53.3059
       - name: Test ROGUE-2
         type: rogue-2
         value: 28.355
       - name: Test ROGUE-L
         type: rogue-l
         value: 44.0953
---
## `bart-large-xsum-samsum`
This model was obtained by fine-tuning `facebook/bart-large-xsum` on [Samsum](https://huggingface.co/datasets/samsum) dataset.
## Usage
```python
from transformers import pipeline

summarizer = pipeline("summarization", model="lidiya/bart-large-xsum-samsum")
conversation = '''Hannah: Hey, do you have Betty's number?
Amanda: Lemme check
Amanda: Sorry, can't find it.
Amanda: Ask Larry
Amanda: He called her last time we were at the park together
Hannah: I don't know him well
Amanda: Don't be shy, he's very nice
Hannah: If you say so..
Hannah: I'd rather you texted him
Amanda: Just text him 🙂
Hannah: Urgh.. Alright
Hannah: Bye
Amanda: Bye bye
'''
summarizer(conversation)
```
## Training procedure
- Colab notebook: https://colab.research.google.com/drive/1dul0Sg-TTMy9xZCJzmDRajXbyzDwtYx6?usp=sharing
## Results
| key | value |
| --- | ----- |
| eval_rouge1 | 54.3921 |
| eval_rouge2 | 29.8078 |
| eval_rougeL | 45.1543 |
| eval_rougeLsum | 49.942 |
| test_rouge1 | 53.3059 |
| test_rouge2 | 28.355 |
| test_rougeL | 44.0953 |
| test_rougeLsum | 48.9246 |


---
tags:
- conversational
---

# Discord Model Medium 7 epochs


---
tags:
- conversational
---

# Discord


---
tags:
- conversational
---

# Twilight Model Medium 13 epochs


---
tags:
- conversational
---

# Discord Model


---
language: en
tags:
- t5-small
datasets:
- Gigaword
- DUC2004
---

## Multi-Summary Reinforcement Learning for Unsupervised Length-Controllable Summarization

This model (MSRP) is trained on Gigaword data.

### How to use

```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer("anonsubms/msrp")
model = AutoModel("anonsubms/msrp")
tar_len = "8"
doc = tar_len+": Replace me by a news article"
encoded_input = tokenizer(doc, return_tensors='pt')
summary = model(**encoded_input)
```


---
tags:
- generated_from_trainer
datasets:
- jnlpba
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: biobert-base-cased-v1.2-finetuned-ner
  results:
  - task:
      name: Token Classification
      type: token-classification
    dataset:
      name: jnlpba
      type: jnlpba
      args: jnlpba
    metrics:
    - name: Precision
      type: precision
      value: 0.7150627220423177
    - name: Recall
      type: recall
      value: 0.8300729927007299
    - name: F1
      type: f1
      value: 0.7682875335686659
    - name: Accuracy
      type: accuracy
      value: 0.90497239665345
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# biobert-base-cased-v1.2-finetuned-ner

This model is a fine-tuned version of [dmis-lab/biobert-base-cased-v1.2](https://huggingface.co/dmis-lab/biobert-base-cased-v1.2) on the jnlpba dataset.
It achieves the following results on the evaluation set:
- Loss: 0.3655
- Precision: 0.7151
- Recall: 0.8301
- F1: 0.7683
- Accuracy: 0.9050

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|:--------:|
| 0.257         | 1.0   | 1160 | 0.2889          | 0.7091    | 0.8222 | 0.7615 | 0.9021   |
| 0.1962        | 2.0   | 2320 | 0.3009          | 0.7154    | 0.8259 | 0.7667 | 0.9048   |
| 0.158         | 3.0   | 3480 | 0.3214          | 0.7098    | 0.8228 | 0.7621 | 0.9031   |
| 0.131         | 4.0   | 4640 | 0.3385          | 0.7174    | 0.8292 | 0.7692 | 0.9055   |
| 0.1081        | 5.0   | 5800 | 0.3655          | 0.7151    | 0.8301 | 0.7683 | 0.9050   |

### Framework versions

- Transformers 4.11.3
- Pytorch 1.9.1+cu102
- Datasets 1.13.2
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- glue
metrics:
- matthews_correlation
model-index:
- name: distilbert-base-uncased-cola
  results:
  - task:
      name: Text Classification
      type: text-classification
    dataset:
      name: glue
      type: glue
      args: cola
    metrics:
    - name: Matthews Correlation
      type: matthews_correlation
      value: 0.5301312348234369
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert-base-uncased-cola

This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the glue dataset.
It achieves the following results on the evaluation set:
- Loss: 1.2715
- Matthews Correlation: 0.5301

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10

### Training results

| Training Loss | Epoch | Step | Validation Loss | Matthews Correlation |
|:-------------:|:-----:|:----:|:---------------:|:--------------------:|
| 0.5216        | 1.0   | 535  | 0.5124          | 0.4104               |
| 0.3456        | 2.0   | 1070 | 0.5700          | 0.4692               |
| 0.2362        | 3.0   | 1605 | 0.7277          | 0.4844               |
| 0.1818        | 4.0   | 2140 | 0.7553          | 0.5007               |
| 0.1509        | 5.0   | 2675 | 0.9406          | 0.4987               |
| 0.1017        | 6.0   | 3210 | 0.9475          | 0.5387               |
| 0.0854        | 7.0   | 3745 | 1.0933          | 0.5317               |
| 0.051         | 8.0   | 4280 | 1.1719          | 0.5358               |
| 0.0512        | 9.0   | 4815 | 1.2296          | 0.5321               |
| 0.0308        | 10.0  | 5350 | 1.2715          | 0.5301               |

### Framework versions

- Transformers 4.12.3
- Pytorch 1.10.0+cu102
- Datasets 1.15.1
- Tokenizers 0.10.3


---
tags:
- conversational
---

#Rick DialoGPT model


Hyperparameters: 1 epoch, max_len_dict, and 1e-5 learning rate


## Schema Guided Dialogue Output Plan Constructor


---
language: ja
thumbnail: https://github.com/rinnakk/japanese-gpt2/blob/master/rinna.png
tags:
- ja
- japanese
- gpt2
- text-generation
- lm
- nlp
license: mit
datasets:
- cc100
- wikipedia
widget:
- text: "生命、宇宙、そして万物についての究極の疑問の答えは"
---

# japanese-gpt2-xsmall

![rinna-icon](./rinna.png)

This repository provides an extra-small-sized Japanese GPT-2 model. The model was trained using code from Github repository [rinnakk/japanese-pretrained-models](https://github.com/rinnakk/japanese-pretrained-models) by [rinna Co., Ltd.](https://corp.rinna.co.jp/)

# How to use the model

*NOTE:* Use `T5Tokenizer` to initiate the tokenizer.

~~~~
from transformers import T5Tokenizer, GPT2LMHeadModel

tokenizer = T5Tokenizer.from_pretrained("rinna/japanese-gpt2-small")
tokenizer.do_lower_case = True  # due to some bug of tokenizer config loading

model = GPT2LMHeadModel.from_pretrained("rinna/japanese-gpt2-small")
~~~~

# Model architecture
A 6-layer, 512-hidden-size transformer-based language model.

# Training
The model was trained on [Japanese CC-100](http://data.statmt.org/cc-100/ja.txt.xz) and [Japanese Wikipedia](https://dumps.wikimedia.org/other/cirrussearch) to optimize a traditional language modelling objective on 8\\*V100 GPUs for around 4 days. It reaches around 28 perplexity on a chosen validation set from CC-100.

# Tokenization
The model uses a [sentencepiece](https://github.com/google/sentencepiece)-based tokenizer, the vocabulary was trained on the Japanese Wikipedia using the official sentencepiece training script.

# Licenese
[The MIT license](https://opensource.org/licenses/MIT)


---
language: ja
thumbnail: https://github.com/rinnakk/japanese-gpt2/blob/master/rinna.png
tags:
- ja
- japanese
- gpt2
- text-generation
- lm
- nlp
license: mit
datasets:
- cc100
- wikipedia
widget:
- text: "生命、宇宙、そして万物についての究極の疑問の答えは"
---

# japanese-gpt2-small

![rinna-icon](./rinna.png)

This repository provides a small-sized Japanese GPT-2 model. The model was trained using code from Github repository [rinnakk/japanese-pretrained-models](https://github.com/rinnakk/japanese-pretrained-models) by [rinna Co., Ltd.](https://corp.rinna.co.jp/)

# How to use the model

*NOTE:* Use `T5Tokenizer` to initiate the tokenizer.

~~~~
from transformers import T5Tokenizer, GPT2LMHeadModel

tokenizer = T5Tokenizer.from_pretrained("rinna/japanese-gpt2-small")
tokenizer.do_lower_case = True  # due to some bug of tokenizer config loading

model = GPT2LMHeadModel.from_pretrained("rinna/japanese-gpt2-small")
~~~~

# Model architecture
A 12-layer, 768-hidden-size transformer-based language model.

# Training
The model was trained on [Japanese CC-100](http://data.statmt.org/cc-100/ja.txt.xz) and [Japanese Wikipedia](https://dumps.wikimedia.org/other/cirrussearch) to optimize a traditional language modelling objective on 8\\*V100 GPUs for around 15 days. It reaches around 21 perplexity on a chosen validation set from CC-100.

# Tokenization
The model uses a [sentencepiece](https://github.com/google/sentencepiece)-based tokenizer, the vocabulary was trained on the Japanese Wikipedia using the official sentencepiece training script.

# Licenese
[The MIT license](https://opensource.org/licenses/MIT)


---
language: ja
thumbnail: https://github.com/rinnakk/japanese-pretrained-models/blob/master/rinna.png
tags:
- ja
- japanese
- gpt
- text-generation
- lm
- nlp
license: mit
datasets:
- cc100
- wikipedia
widget:
- text: "西田幾多郎は、"
---

# japanese-gpt-1b

![rinna-icon](./rinna.png)

This repository provides a 1.3B-parameter Japanese GPT model. The model was trained by [rinna Co., Ltd.](https://corp.rinna.co.jp/)

# How to use the model

*NOTE:* Use `T5Tokenizer` to initiate the tokenizer.

~~~~
import torch
from transformers import T5Tokenizer, AutoModelForCausalLM

tokenizer = T5Tokenizer.from_pretrained("rinna/japanese-gpt-1b")
model = AutoModelForCausalLM.from_pretrained("rinna/japanese-gpt-1b")

if torch.cuda.is_available():
    model = model.to("cuda")

text = "西田幾多郎は、"
token_ids = tokenizer.encode(text, add_special_tokens=False, return_tensors="pt")

with torch.no_grad():
    output_ids = model.generate(
        token_ids.to(model.device),
        max_length=100,
        min_length=100,
        do_sample=True,
        top_k=500,
        top_p=0.95,
        pad_token_id=tokenizer.pad_token_id,
        bos_token_id=tokenizer.bos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        bad_word_ids=[[tokenizer.unk_token_id]]
    )

output = tokenizer.decode(output_ids.tolist()[0])
print(output)
# sample output: 西田幾多郎は、その主著の「善の研究」などで、人間の内面に自然とその根源があると指摘し、その根源的な性格は、この西田哲学を象徴しているとして、カントの「純粋理性批判」と「判断力批判」を対比して捉えます。それは、「人が理性的存在であるかぎりにおいて、人はその当人に固有な道徳的に自覚された善悪の基準を持っている」とするもので、この理性的な善悪の観念を否定するのがカントの
~~~~

# Model architecture
A 24-layer, 2048-hidden-size transformer-based language model.

# Training
The model was trained on [Japanese C4](https://huggingface.co/datasets/allenai/c4), [Japanese CC-100](http://data.statmt.org/cc-100/ja.txt.xz) and [Japanese Wikipedia](https://dumps.wikimedia.org/other/cirrussearch) to optimize a traditional language modelling objective. It reaches around 14 perplexity on a chosen validation set from the same data.
# Tokenization
The model uses a [sentencepiece](https://github.com/google/sentencepiece)-based tokenizer. The vocabulary was first trained on a selected subset from the training data using the official sentencepiece training script, and then augmented with emojis and symbols.
# Licenese
[The MIT license](https://opensource.org/licenses/MIT)


---
language: ja
thumbnail: https://github.com/rinnakk/japanese-gpt2/blob/master/rinna.png
tags:
- ja
- japanese
- gpt2
- text-generation
- lm
- nlp
license: mit
datasets:
- cc100
- wikipedia
widget:
- text: "生命、宇宙、そして万物についての究極の疑問の答えは"
---

# japanese-gpt2-medium

![rinna-icon](./rinna.png)

This repository provides a medium-sized Japanese GPT-2 model. The model was trained using code from Github repository [rinnakk/japanese-pretrained-models](https://github.com/rinnakk/japanese-pretrained-models) by [rinna Co., Ltd.](https://corp.rinna.co.jp/)

# How to use the model

*NOTE:* Use `T5Tokenizer` to initiate the tokenizer.

~~~~
from transformers import T5Tokenizer, AutoModelForCausalLM

tokenizer = T5Tokenizer.from_pretrained("rinna/japanese-gpt2-medium")
tokenizer.do_lower_case = True  # due to some bug of tokenizer config loading

model = AutoModelForCausalLM.from_pretrained("rinna/japanese-gpt2-medium")
~~~~

# Model architecture
A 24-layer, 1024-hidden-size transformer-based language model.

# Training
The model was trained on [Japanese CC-100](http://data.statmt.org/cc-100/ja.txt.xz) and [Japanese Wikipedia](https://dumps.wikimedia.org/other/cirrussearch) to optimize a traditional language modelling objective on 8\\*V100 GPUs for around 30 days. It reaches around 18 perplexity on a chosen validation set from the same data.

# Tokenization
The model uses a [sentencepiece](https://github.com/google/sentencepiece)-based tokenizer, the vocabulary was trained on the Japanese Wikipedia using the official sentencepiece training script.

# Licenese
[The MIT license](https://opensource.org/licenses/MIT)


---
language: ja
thumbnail: https://github.com/rinnakk/japanese-gpt2/blob/master/rinna.png
tags:
- ja
- japanese
- roberta
- masked-lm
- nlp
license: mit
datasets:
- cc100
- wikipedia
mask_token: "[MASK]"
widget:
- text: "[CLS]4年に1度[MASK]は開かれる。"
---

# japanese-roberta-base

![rinna-icon](./rinna.png)

This repository provides a base-sized Japanese RoBERTa model. The model was trained using code from Github repository [rinnakk/japanese-pretrained-models](https://github.com/rinnakk/japanese-pretrained-models) by [rinna Co., Ltd.](https://corp.rinna.co.jp/)

# How to load the model

*NOTE:* Use `T5Tokenizer` to initiate the tokenizer.

~~~~
from transformers import T5Tokenizer, RobertaForMaskedLM

tokenizer = T5Tokenizer.from_pretrained("rinna/japanese-roberta-base")
tokenizer.do_lower_case = True  # due to some bug of tokenizer config loading

model = RobertaForMaskedLM.from_pretrained("rinna/japanese-roberta-base")
~~~~

# How to use the model for masked token prediction

## Note 1: Use `[CLS]`

To predict a masked token, be sure to add a `[CLS]` token before the sentence for the model to correctly encode it, as it is used during the model training.

## Note 2: Use `[MASK]` after tokenization

A) Directly typing `[MASK]` in an input string and B) replacing a token with `[MASK]` after tokenization will yield different token sequences, and thus different prediction results. It is more appropriate to use `[MASK]` after tokenization (as it is consistent with how the model was pretrained). However, the Huggingface Inference API only supports typing `[MASK]` in the input string and produces less robust predictions.

## Note 3: Provide `position_ids` as an argument explicitly

When `position_ids` are not provided for a `Roberta*` model, Huggingface's `transformers` will automatically construct it but start from `padding_idx` instead of `0` (see [issue](https://github.com/rinnakk/japanese-pretrained-models/issues/3) and function `create_position_ids_from_input_ids()` in Huggingface's [implementation](https://github.com/huggingface/transformers/blob/master/src/transformers/models/roberta/modeling_roberta.py)), which unfortunately does not work as expected with `rinna/japanese-roberta-base` since the `padding_idx` of the corresponding tokenizer is not `0`. So please be sure to constrcut the `position_ids` by yourself and make it start from position id `0`.

## Example

Here is an example by to illustrate how our model works as a masked language model. Notice the difference between running the following code example and running the Huggingface Inference API.

~~~~
# original text
text = "4年に1度オリンピックは開かれる。"

# prepend [CLS]
text = "[CLS]" + text

# tokenize
tokens = tokenizer.tokenize(text)
print(tokens)  # output: ['[CLS]', '▁4', '年に', '1', '度', 'オリンピック', 'は', '開かれる', '。']

# mask a token
masked_idx = 5
tokens[masked_idx] = tokenizer.mask_token
print(tokens)  # output: ['[CLS]', '▁4', '年に', '1', '度', '[MASK]', 'は', '開かれる', '。']

# convert to ids
token_ids = tokenizer.convert_tokens_to_ids(tokens)
print(token_ids)  # output: [4, 1602, 44, 24, 368, 6, 11, 21583, 8]

# convert to tensor
import torch
token_tensor = torch.LongTensor([token_ids])

# provide position ids explicitly
position_ids = list(range(0, token_tensor.size(1)))
print(position_ids)  # output: [0, 1, 2, 3, 4, 5, 6, 7, 8]
position_id_tensor = torch.LongTensor([position_ids])

# get the top 10 predictions of the masked token
with torch.no_grad():
    outputs = model(input_ids=token_tensor, position_ids=position_id_tensor)
    predictions = outputs[0][0, masked_idx].topk(10)

for i, index_t in enumerate(predictions.indices):
    index = index_t.item()
    token = tokenizer.convert_ids_to_tokens([index])[0]
    print(i, token)

"""
0 総会
1 サミット
2 ワールドカップ
3 フェスティバル
4 大会
5 オリンピック
6 全国大会
7 党大会
8 イベント
9 世界選手権
"""
~~~~

# Model architecture
A 12-layer, 768-hidden-size transformer-based masked language model.

# Training
The model was trained on [Japanese CC-100](http://data.statmt.org/cc-100/ja.txt.xz) and [Japanese Wikipedia](https://dumps.wikimedia.org/jawiki/) to optimize a masked language modelling objective on 8*V100 GPUs for around 15 days. It reaches ~3.9 perplexity on a dev set sampled from CC-100.

# Tokenization
The model uses a [sentencepiece](https://github.com/google/sentencepiece)-based tokenizer, the vocabulary was trained on the Japanese Wikipedia using the official sentencepiece training script.

# Licenese
[The MIT license](https://opensource.org/licenses/MIT)


---
language: en
tags:
- sagemaker
- bart
- summarization
license: apache-2.0
datasets:
- samsum
model-index:
- name: bart-large-cnn-samsum
  results:
  - task:
      name: Abstractive Text Summarization
      type: abstractive-text-summarization
    dataset:
      name: "SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization"
      type: samsum
    metrics:
       - name: Validation ROGUE-1
         type: rogue-1
         value: 42.621
       - name: Validation ROGUE-2
         type: rogue-2
         value: 21.9825
       - name: Validation ROGUE-L
         type: rogue-l
         value: 33.034
       - name: Test ROGUE-1
         type: rogue-1
         value: 41.3174
       - name: Test ROGUE-2
         type: rogue-2
         value: 20.8716
       - name: Test ROGUE-L
         type: rogue-l
         value: 32.1337
widget:
- text: |
    Jeff: Can I train a 🤗 Transformers model on Amazon SageMaker?
    Philipp: Sure you can use the new Hugging Face Deep Learning Container.
    Jeff: ok.
    Jeff: and how can I get started?
    Jeff: where can I find documentation?
    Philipp: ok, ok you can find everything here. https://huggingface.co/blog/the-partnership-amazon-sagemaker-and-hugging-face
---
## `bart-large-cnn-samsum`
This model was trained using Amazon SageMaker and the new Hugging Face Deep Learning container.
For more information look at:
- [🤗 Transformers Documentation: Amazon SageMaker](https://huggingface.co/transformers/sagemaker.html)
- [Example Notebooks](https://github.com/huggingface/notebooks/tree/master/sagemaker)
- [Amazon SageMaker documentation for Hugging Face](https://docs.aws.amazon.com/sagemaker/latest/dg/hugging-face.html)
- [Python SDK SageMaker documentation for Hugging Face](https://sagemaker.readthedocs.io/en/stable/frameworks/huggingface/index.html)
- [Deep Learning Container](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#huggingface-training-containers)
## Hyperparameters
    {
    "dataset_name": "samsum",
    "do_eval": true,
    "do_predict": true,
    "do_train": true,
    "fp16": true,
    "learning_rate": 5e-05,
    "model_name_or_path": "facebook/bart-large-cnn",
    "num_train_epochs": 3,
    "output_dir": "/opt/ml/model",
    "per_device_eval_batch_size": 4,
    "per_device_train_batch_size": 4,
    "predict_with_generate": true,
    "sagemaker_container_log_level": 20,
    "sagemaker_job_name": "huggingface-pytorch-training-2021-09-08-06-40-19-182",
    "sagemaker_program": "run_summarization.py",
    "sagemaker_region": "us-west-2",
    "sagemaker_submit_directory": "s3://sagemaker-us-west-2-847380964353/huggingface-pytorch-training-2021-09-08-06-40-19-182/source/sourcedir.tar.gz",
    "seed": 7
}
## Usage
    from transformers import pipeline
    summarizer = pipeline("summarization", model="philschmid/bart-large-cnn-samsum")
    conversation = '''Jeff: Can I train a 🤗 Transformers model on Amazon SageMaker?
    Philipp: Sure you can use the new Hugging Face Deep Learning Container.
    Jeff: ok.
    Jeff: and how can I get started?
    Jeff: where can I find documentation?
    Philipp: ok, ok you can find everything here. https://huggingface.co/blog/the-partnership-amazon-sagemaker-and-hugging-face
    '''
    nlp(conversation)
## Results
| key | value |
| --- | ----- |
| eval_rouge1 | 42.059 |
| eval_rouge2 | 21.5509 |
| eval_rougeL | 32.4083 |
| eval_rougeLsum | 39.0015 |
| test_rouge1 | 40.8656 |
| test_rouge2 | 20.3517 |
| test_rougeL | 31.2268 |
| test_rougeLsum | 37.9301 |


---
tags:
- generated_from_trainer
datasets:
- cnn_dailymail
model-index:
- name: albert_gpt2_Full_summarization_cnndm
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# albert_gpt2_Full_summarization_cnndm

This model is a fine-tuned version of [](https://huggingface.co/) on the cnn_dailymail dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 3.0
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.12.0.dev0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


---
tags:
- generated_from_trainer
datasets:
- cnn_dailymail
model-index:
- name: distilbert_gpt2_summarization_cnndm
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert_gpt2_summarization_cnndm

This model is a fine-tuned version of [](https://huggingface.co/) on the cnn_dailymail dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 3.0
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.12.0.dev0
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


---
tags:
- generated_from_trainer
datasets:
- cnn_dailymail
model-index:
- name: xlmroberta_large_gpt2_summarization_cnndm
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# xlmroberta_large_gpt2_summarization_cnndm

This model is a fine-tuned version of [](https://huggingface.co/) on the cnn_dailymail dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 3.0
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.12.0.dev0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


---
tags:
- generated_from_trainer
datasets:
- cnn_dailymail
model-index:
- name: bert_distilgpt2_summarization_cnn_dailymail
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bert_distilgpt2_summarization_cnn_dailymail

This model is a fine-tuned version of [](https://huggingface.co/) on the cnn_dailymail dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 3.0
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Datasets 1.18.3
- Tokenizers 0.11.0


---
tags:
- generated_from_trainer
datasets:
- cnn_dailymail
model-index:
- name: roberta_distilgpt2_summarization_cnn_dailymail
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# roberta_distilgpt2_summarization_cnn_dailymail

This model is a fine-tuned version of [](https://huggingface.co/) on the cnn_dailymail dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 3.0
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Datasets 1.18.2
- Tokenizers 0.11.0


---
tags:
- generated_from_trainer
datasets:
- cnn_dailymail
model-index:
- name: albert_distilgpt2_summarization_cnn_dailymail
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# albert_distilgpt2_summarization_cnn_dailymail

This model is a fine-tuned version of [](https://huggingface.co/) on the cnn_dailymail dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 3.0
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Datasets 1.18.3
- Tokenizers 0.11.0


---
tags:
- generated_from_trainer
datasets:
- cnn_dailymail
model-index:
- name: albert_large_gpt2_summarization_cnndm
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# albert_large_gpt2_summarization_cnndm

This model is a fine-tuned version of [](https://huggingface.co/) on the cnn_dailymail dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 3.0
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.12.0.dev0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


---
tags:
- generated_from_trainer
datasets:
- xsum
model-index:
- name: bert_gpt2_summarization_xsum
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bert_gpt2_summarization_xsum

This model is a fine-tuned version of [](https://huggingface.co/) on the xsum dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 3.0
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.12.0.dev0
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


---
tags:
- generated_from_trainer
datasets:
- cnn_dailymail
model-index:
- name: roberta_gpt2_summarization_cnn_dailymail
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# roberta_gpt2_summarization_cnn_dailymail

This model is a fine-tuned version of [](https://huggingface.co/) on the cnn_dailymail dataset.

## Model description
This model uses RoBerta encoder and GPT2 decoder and fine-tuned on the summarization task. It got Rouge scores as follows:

Rouge1= 35.886

Rouge2= 16.292

RougeL= 23.499
## Intended uses & limitations
To use its API:

from transformers import RobertaTokenizerFast, GPT2Tokenizer, EncoderDecoderModel

model = EncoderDecoderModel.from_pretrained("Ayham/roberta_gpt2_summarization_cnn_dailymail")

input_tokenizer = RobertaTokenizerFast.from_pretrained('roberta-base')

output_tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

article = """Your Input Text"""

input_ids = input_tokenizer(article, return_tensors="pt").input_ids

output_ids = model.generate(input_ids)

print(output_tokenizer.decode(output_ids[0], skip_special_tokens=True))

More information needed

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 3.0
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.12.0.dev0
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


---
tags:
- generated_from_trainer
datasets:
- cnn_dailymail
model-index:
- name: bert_gpt2_summarization_cnndm
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bert_gpt2_summarization_cnndm

This model is a fine-tuned version of [](https://huggingface.co/) on the cnn_dailymail dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 3.0
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.12.0.dev0
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


---
tags:
- generated_from_trainer
datasets:
- cnn_dailymail
model-index:
- name: xlnet_distilgpt2_summarization_cnn_dailymail
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# xlnet_distilgpt2_summarization_cnn_dailymail

This model is a fine-tuned version of [](https://huggingface.co/) on the cnn_dailymail dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 3.0
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Datasets 1.18.3
- Tokenizers 0.11.0


---
tags:
- generated_from_trainer
datasets:
- cnn_dailymail
model-index:
- name: xlnet_gpt2_summarization_cnn_dailymail
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# xlnet_gpt2_summarization_cnn_dailymail

This model is a fine-tuned version of [](https://huggingface.co/) on the cnn_dailymail dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 3.0
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.12.0.dev0
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


---
tags:
- generated_from_trainer
datasets:
- cnn_dailymail
model-index:
- name: distilbert_distilgpt2_summarization_cnn_dailymail
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert_distilgpt2_summarization_cnn_dailymail

This model is a fine-tuned version of [](https://huggingface.co/) on the cnn_dailymail dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 3.0
- mixed_precision_training: Native AMP

### Training results

### Framework versions

- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Datasets 1.18.3
- Tokenizers 0.11.0


---
tags:
- conversational
---

# My Hero Academia DialoGPT Model


---
language: en
license: apache-2.0
datasets:
- bookcorpus
- wikipedia
---

# BERT large model (cased) whole word masking

Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in
[this paper](https://arxiv.org/abs/1810.04805) and first released in
[this repository](https://github.com/google-research/bert). This model is cased: it makes a difference between english and English.

Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same.

The training is identical -- each masked WordPiece token is predicted independently.

Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by
the Hugging Face team.

## Model description

BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it
was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
was pretrained with two objectives:

- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
  the entire masked sentence through the model and has to predict the masked words. This is different from traditional
  recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like
  GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the
  sentence.
- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes
  they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to
  predict if the two sentences were following each other or not.

This way, the model learns an inner representation of the English language that can then be used to extract features
useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
classifier using the features produced by the BERT model as inputs.

This model has the following configuration:

- 24-layer
- 1024 hidden dimension
- 16 attention heads
- 336M parameters.

## Intended uses & limitations

You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=bert) to look for
fine-tuned versions on a task that interests you.

Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
generation you should look at model like GPT2.

### How to use

You can use this model directly with a pipeline for masked language modeling:

```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='bert-large-cased-whole-word-masking')
>>> unmasker("Hello I'm a [MASK] model.")
[
   {
      "sequence":"[CLS] Hello I'm a fashion model. [SEP]",
      "score":0.1474294513463974,
      "token":4633,
      "token_str":"fashion"
   },
   {
      "sequence":"[CLS] Hello I'm a magazine model. [SEP]",
      "score":0.05430116504430771,
      "token":2435,
      "token_str":"magazine"
   },
   {
      "sequence":"[CLS] Hello I'm a male model. [SEP]",
      "score":0.039395421743392944,
      "token":2581,
      "token_str":"male"
   },
   {
      "sequence":"[CLS] Hello I'm a former model. [SEP]",
      "score":0.036936815828084946,
      "token":1393,
      "token_str":"former"
   },
   {
      "sequence":"[CLS] Hello I'm a professional model. [SEP]",
      "score":0.03663451969623566,
      "token":1848,
      "token_str":"professional"
   }
]
```

Here is how to use this model to get the features of a given text in PyTorch:

```python
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('bert-large-cased-whole-word-masking')
model = BertModel.from_pretrained("bert-large-cased-whole-word-masking")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```

and in TensorFlow:

```python
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('bert-large-cased-whole-word-masking')
model = TFBertModel.from_pretrained("bert-large-cased-whole-word-masking")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```

### Limitations and bias

Even if the training data used for this model could be characterized as fairly neutral, this model can have biased
predictions:

```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='bert-large-cased-whole-word-masking')
>>> unmasker("The man worked as a [MASK].")
[
   {
      "sequence":"[CLS] The man worked as a carpenter. [SEP]",
      "score":0.09021259099245071,
      "token":25169,
      "token_str":"carpenter"
   },
   {
      "sequence":"[CLS] The man worked as a cook. [SEP]",
      "score":0.08125395327806473,
      "token":9834,
      "token_str":"cook"
   },
   {
      "sequence":"[CLS] The man worked as a mechanic. [SEP]",
      "score":0.07524766772985458,
      "token":19459,
      "token_str":"mechanic"
   },
   {
      "sequence":"[CLS] The man worked as a waiter. [SEP]",
      "score":0.07397029548883438,
      "token":17989,
      "token_str":"waiter"
   },
   {
      "sequence":"[CLS] The man worked as a guard. [SEP]",
      "score":0.05848982185125351,
      "token":3542,
      "token_str":"guard"
   }
]

>>> unmasker("The woman worked as a [MASK].")
[
   {
      "sequence":"[CLS] The woman worked as a maid. [SEP]",
      "score":0.19436432421207428,
      "token":13487,
      "token_str":"maid"
   },
   {
      "sequence":"[CLS] The woman worked as a waitress. [SEP]",
      "score":0.16161060333251953,
      "token":15098,
      "token_str":"waitress"
   },
   {
      "sequence":"[CLS] The woman worked as a nurse. [SEP]",
      "score":0.14942803978919983,
      "token":7439,
      "token_str":"nurse"
   },
   {
      "sequence":"[CLS] The woman worked as a secretary. [SEP]",
      "score":0.10373266786336899,
      "token":4848,
      "token_str":"secretary"
   },
   {
      "sequence":"[CLS] The woman worked as a cook. [SEP]",
      "score":0.06384387612342834,
      "token":9834,
      "token_str":"cook"
   }
]
```

This bias will also affect all fine-tuned versions of this model.

## Training data

The BERT model was pretrained on [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038
unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and
headers).

## Training procedure

### Preprocessing

The texts are lowercased and tokenized using WordPiece and a vocabulary size of 30,000. The inputs of the model are
then of the form:

```
[CLS] Sentence A [SEP] Sentence B [SEP]
```

With probability 0.5, sentence A and sentence B correspond to two consecutive sentences in the original corpus and in
the other cases, it's another random sentence in the corpus. Note that what is considered a sentence here is a
consecutive span of text usually longer than a single sentence. The only constrain is that the result with the two
"sentences" has a combined length of less than 512 tokens.

The details of the masking procedure for each sentence are the following:
- 15% of the tokens are masked.
- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
- In the 10% remaining cases, the masked tokens are left as is.

### Pretraining

The model was trained on 4 cloud TPUs in Pod configuration (16 TPU chips total) for one million steps with a batch size
of 256. The sequence length was limited to 128 tokens for 90% of the steps and 512 for the remaining 10%. The optimizer
used is Adam with a learning rate of 1e-4, \\(\beta_{1} = 0.9\\) and \\(\beta_{2} = 0.999\\), a weight decay of 0.01,
learning rate warmup for 10,000 steps and linear decay of the learning rate after.

## Evaluation results

When fine-tuned on downstream tasks, this model achieves the following results:

Model                                    | SQUAD 1.1 F1/EM | Multi NLI Accuracy
---------------------------------------- | :-------------: | :----------------:
BERT-Large, Cased (Whole Word Masking)   | 92.9/86.7       | 86.46

### BibTeX entry and citation info

```bibtex
@article{DBLP:journals/corr/abs-1810-04805,
  author    = {Jacob Devlin and
               Ming{-}Wei Chang and
               Kenton Lee and
               Kristina Toutanova},
  title     = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language
               Understanding},
  journal   = {CoRR},
  volume    = {abs/1810.04805},
  year      = {2018},
  url       = {http://arxiv.org/abs/1810.04805},
  archivePrefix = {arXiv},
  eprint    = {1810.04805},
  timestamp = {Tue, 30 Oct 2018 20:39:56 +0100},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1810-04805.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
```


---
tags:
- token-classification
library_name: generic
---

# Token Classification repository template

This is a template repository for token classification to support generic inference with Hugging Face Hub generic Inference API. There are two required steps

1. Specify the requirements by defining a `requirements.txt` file.
2. Implement the `pipeline.py` `__init__` and `__call__` methods. These methods are called by the Inference API. The `__init__` method should load the model and preload all the elements needed for inference (model, processors, tokenizers, etc.). This is only called once. The `__call__` method performs the actual inference. Make sure to follow the same input/output specifications defined in the template for the pipeline to work.

Example repos
* https://huggingface.co/osanseviero/en_core_web_sm/blob/main/pipeline.py

## How to start
First create a repo in https://hf.co/new.
Then clone this template and push it to your repo.
```
git clone https://huggingface.co/templates/token-classification
cd token-classification
git remote set-url origin https://huggingface.co/$YOUR_USER/$YOUR_REPO_NAME
git push --force
```


---
tags:
- structured-data-classification
library_name: generic
---
# Structured Data Classification repository template

This is a template repository for structured data classification to support generic inference with Hugging Face Hub generic Inference API. There are two required steps
1. Specify the requirements by defining a `requirements.txt` file.
2. Implement the `pipeline.py` `__init__` and `__call__` methods. These methods are called by the Inference API. The `__init__` method should load the model and preload all the elements needed for inference (model, processors, tokenizers, etc.). This is only called once. The `__call__` method performs the actual inference. Make sure to follow the same input/output specifications defined in the template for the pipeline to work.

Example repos
* https://huggingface.co/osanseviero/wine-quality

## How to start
First create a repo in https://hf.co/new.
Then clone this template and push it to your repo.
```
git clone https://huggingface.co/templates/structured-data-classification
cd structured-data-classification
git remote set-url origin https://huggingface.co/$YOUR_USER/$YOUR_REPO_NAME
git push --force
```


---
tags:
- audio-to-audio
library_name: generic
---

# Audio to Audio repository template

This is a template repository for Audio to Audio to support generic inference with Hugging Face Hub generic Inference API. Examples of Audio to Audio are Source Separation and Speech Enhancement. There are two required steps:

1. Specify the requirements by defining a `requirements.txt` file.
2. Implement the `pipeline.py` `__init__` and `__call__` methods. These methods are called by the Inference API. The `__init__` method should load the model and preload all the elements needed for inference (model, processors, tokenizers, etc.). This is only called once. The `__call__` method performs the actual inference. Make sure to follow the same input/output specifications defined in the template for the pipeline to work.

Example repos
* https://huggingface.co/osanseviero/ConvTasNet_Libri1Mix_enhsingle_16k

## How to start

First create a repo in https://hf.co/new.
Then clone this template and push it to your repo.

```
git clone https://huggingface.co/templates/audio-to-audio
cd audio-to-audio
git remote set-url origin https://huggingface.co/$YOUR_USER/$YOUR_REPO_NAME
git push --force
```


---
tags:
- image-classification
library_name: generic
---

# Image Classification repository template

This is a template repository for image classification to support generic inference with Hugging Face Hub generic Inference API. There are two required steps

1. Specify the requirements by defining a `requirements.txt` file.
2. Implement the `pipeline.py` `__init__` and `__call__` methods. These methods are called by the Inference API. The `__init__` method should load the model and preload all the elements needed for inference (model, processors, tokenizers, etc.). This is only called once. The `__call__` method performs the actual inference. Make sure to follow the same input/output specifications defined in the template for the pipeline to work.

Example repos
* https://huggingface.co/osanseviero/fastai_cat_vs_dog/tree/main

## How to start
First create a repo in https://hf.co/new.
Then clone this template and push it to your repo.
```
git clone https://huggingface.co/templates/image-classification
cd image-classification
git remote set-url origin https://huggingface.co/$YOUR_USER/$YOUR_REPO_NAME
git push --force
```


---
tags:
- text-classification
library_name: generic
---

# Text Classification repository template

This is a template repository for Text Classification to support generic inference with Hugging Face Hub generic Inference API. There are two required steps:

1. Specify the requirements by defining a `requirements.txt` file.
2. Implement the `pipeline.py` `__init__` and `__call__` methods. These methods are called by the Inference API. The `__init__` method should load the model and preload all the elements needed for inference (model, processors, tokenizers, etc.). This is only called once. The `__call__` method performs the actual inference. Make sure to follow the same input/output specifications defined in the template for the pipeline to work.

## How to start
First create a repo in https://hf.co/new.
Then clone this template and push it to your repo.
```
git clone https://huggingface.co/templates/text-classification
cd text-classification
git remote set-url origin https://huggingface.co/$YOUR_USER/$YOUR_REPO_NAME
git push --force
```


---
tags:
- text-to-image
library_name: generic
---

# Text To Image repository template

This is a template repository for text to image to support generic inference with Hugging Face Hub generic Inference API. There are two required steps
1. Specify the requirements by defining a `requirements.txt` file.
2. Implement the `pipeline.py` `__init__` and `__call__` methods. These methods are called by the Inference API. The `__init__` method should load the model and preload all the elements needed for inference (model, processors, tokenizers, etc.). This is only called once. The `__call__` method performs the actual inference. Make sure to follow the same input/output specifications defined in the template for the pipeline to work.

Example repos
* https://huggingface.co/osanseviero/BigGAN-deep-128/blob/main/pipeline.py

## How to start
First create a repo in https://hf.co/new.
Then clone this template and push it to your repo.
```
git clone https://huggingface.co/templates/text-to-image
cd text-to-image
git remote set-url origin https://huggingface.co/$YOUR_USER/$YOUR_REPO_NAME
git push --force
```


---
tags:
- automatic-speech-recognition
library_name: generic
---

# Automatic Speech Recognition repository template

This is a template repository for Automatic Speech Recognition to support generic inference with Hugging Face Hub generic Inference API. There are two required steps:

1. Specify the requirements by defining a `requirements.txt` file.
2. Implement the `pipeline.py` `__init__` and `__call__` methods. These methods are called by the Inference API. The `__init__` method should load the model and preload all the elements needed for inference (model, processors, tokenizers, etc.). This is only called once. The `__call__` method performs the actual inference. Make sure to follow the same input/output specifications defined in the template for the pipeline to work.

Example repos
* https://huggingface.co/osanseviero/pyctcdecode_asr

## How to start
First create a repo in https://hf.co/new.
Then clone this template and push it to your repo.
```
git clone https://huggingface.co/templates/automatic-speech-recognition
cd automatic-speech-recognition
git remote set-url origin https://huggingface.co/$YOUR_USER/$YOUR_REPO_NAME
git push --force
```


---
tags:
- feature-extraction
library_name: generic
---

# Feature Extraction repository template

This is a template repository for feature extraction to support generic inference with Hugging Face Hub generic Inference API. There are two required steps

1. Specify the requirements by defining a `requirements.txt` file.
2. Implement the `pipeline.py` `__init__` and `__call__` methods. These methods are called by the Inference API. The `__init__` method should load the model and preload all the elements needed for inference (model, processors, tokenizers, etc.). This is only called once. The `__call__` method performs the actual inference. Make sure to follow the same input/output specifications defined in the template for the pipeline to work.

Example repos
* https://huggingface.co/osanseviero/fasttext_english

## How to start
First create a repo in https://hf.co/new.
Then clone this template and push it to your repo.
```
git clone https://huggingface.co/templates/feature-extraction
cd feature-extraction
git remote set-url origin https://huggingface.co/$YOUR_USER/$YOUR_REPO_NAME
git push --force
```


---
tags:
- conversational
---

# Thor DialogGPT Model


---
tags:
- conversational
---
# Rikka DialoGPT Model


---
tags:
- conversational
---
# Cultured Kumiko DialoGPT Model


---
tags:
- conversational
---
# Revised Kumiko DialoGPT Model


---
tags:
- conversational
---
# Kumiko DialoGPT Model


---
tags:
- conversational
---
# Medium Kumiko DialoGPT Model


---
language: en
tags:
- image-classification
- image-captioning

---

# Poster2Plot

An image captioning model to generate movie/t.v show plot from poster. It generates decent plots but is no way perfect. We are still working on improving the model.

## Live demo on Hugging Face Spaces: https://huggingface.co/spaces/deepklarity/poster2plot

# Model Details

The base model uses a Vision Transformer (ViT) model as an image encoder and GPT-2 as a decoder.

We used the following models:

* Encoder: [google/vit-base-patch16-224-in21k](https://huggingface.co/google/vit-base-patch16-224-in21k)
* Decoder: [gpt2](https://huggingface.co/gpt2)

# Datasets

Publicly available IMDb datasets were used to train the model.

# How to use

## In PyTorch

```python
import torch
import re
import requests
from PIL import Image
from transformers import AutoTokenizer, AutoFeatureExtractor, VisionEncoderDecoderModel

# Pattern to ignore all the text after 2 or more full stops
regex_pattern = "[.]{2,}"

def post_process(text):
    try:
        text = text.strip()
        text = re.split(regex_pattern, text)[0]
    except Exception as e:
        print(e)
        pass
    return text

def predict(image, max_length=64, num_beams=4):
    pixel_values = feature_extractor(images=image, return_tensors="pt").pixel_values
    pixel_values = pixel_values.to(device)

    with torch.no_grad():
        output_ids = model.generate(
            pixel_values,
            max_length=max_length,
            num_beams=num_beams,
            return_dict_in_generate=True,
        ).sequences

    preds = tokenizer.batch_decode(output_ids, skip_special_tokens=True)
    pred = post_process(preds[0])

    return pred

model_name_or_path = "deepklarity/poster2plot"
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Load model.

model = VisionEncoderDecoderModel.from_pretrained(model_name_or_path)
model.to(device)
print("Loaded model")

feature_extractor = AutoFeatureExtractor.from_pretrained(model.encoder.name_or_path)
print("Loaded feature_extractor")

tokenizer = AutoTokenizer.from_pretrained(model.decoder.name_or_path, use_fast=True)
if model.decoder.name_or_path == "gpt2":
    tokenizer.pad_token = tokenizer.eos_token

print("Loaded tokenizer")

url = "https://upload.wikimedia.org/wikipedia/en/2/26/Moana_Teaser_Poster.jpg"
with Image.open(requests.get(url, stream=True).raw) as image:
    pred = predict(image)

print(pred)

```


Roberta-base training attempt on hindi datasets.


---
tags:
- generated_from_trainer
model-index:
- name: gpt-4-est-base
  results: []
---

# gpt-4-est-base

This is GPT for Estonian. Not GPT-4 :-) This is the base-size [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2) model, trained from scratch on 2.2 billion words (Estonian National Corpus + News Crawl + Common Crawl) for 3 epochs.

[Colab demo](https://colab.research.google.com/drive/1Bp7mGEQ1vmyqXPyXHV1yj68cRZEi2mq4?usp=sharing)

### Format

For training data was prepended with a text domain tag, and it should be added as prefix when using the model: >general<, >web<, >news<, >doaj< and >wiki< (standing for general texts, web crawled texts, news, article abstracts and wikipedia texts). Use the prefixes like this, e.g: ">web< Kas tead, et".

### Model details
- num. of layers: 12
- num. of heads: 12
- embedding size: 768
- context size: 1024
- total size: 118.68M params

Further details to be added soon.

### Framework versions

- Transformers 4.13.0.dev0
- Pytorch 1.10.0+cu102
- Datasets 1.15.1
- Tokenizers 0.10.3


---
tags:
- generated_from_trainer
model-index:
- name: gpt-4-est-large
  results: []
---

# gpt-4-est-large

This is GPT for Estonian. Not GPT-4 :-) This is the large-size [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2) model, trained from scratch on 2.2 billion words (Estonian National Corpus + News Crawl + Common Crawl).

[Colab demo](https://colab.research.google.com/drive/1Bp7mGEQ1vmyqXPyXHV1yj68cRZEi2mq4?usp=sharing)

### Format

For training data was prepended with a text domain tag, and it should be added as prefix when using the model: >general<, >web<, >news<, >doaj< and >wiki< (standing for general texts, web crawled texts, news, article abstracts and wikipedia texts). Use the prefixes like this, e.g: ">web< Kas tead, et".

### Model details

- num. of layers: 24
- num. of heads: 24
- embedding size: 1536
- context size: 1024
- total size: 723.58M params

Further details to be added soon.

### Framework versions

- Transformers 4.13.0.dev0
- Pytorch 1.10.0+cu102
- Datasets 1.15.1
- Tokenizers 0.10.3


# EstBERT_NER

## Model description

EstBERT_NER is a fine-tuned EstBERT model that can be used for Named Entity Recognition. This model was trained on the Estonian NER dataset created by [Tkachenko et al](https://www.aclweb.org/anthology/W13-2412.pdf). It can recognize three types of entities: locations (LOC), organizations (ORG) and persons (PER).

## How to use

You can use this model with Transformers pipeline for NER. Post-processing of results may be necessary as the model occasionally tags subword tokens as entities.

```
from transformers import BertTokenizer, BertForTokenClassification
from transformers import pipeline

tokenizer = BertTokenizer.from_pretrained('tartuNLP/EstBERT_NER')
bertner = BertForTokenClassification.from_pretrained('tartuNLP/EstBERT_NER')

nlp = pipeline("ner", model=bertner, tokenizer=tokenizer)
sentence = 'Eesti Ekspressi teada on Eesti Pank uurinud Hansapanga tehinguid , mis toimusid kaks aastat tagasi suvel ja mille käigus voolas panka ligi miljardi krooni ulatuses kahtlast raha .'

ner_results = nlp(sentence)
print(ner_results)
```
```
[{'word': 'Eesti', 'score': 0.9964128136634827, 'entity': 'B-ORG', 'index': 1}, {'word': 'Ekspressi', 'score': 0.9978809356689453, 'entity': 'I-ORG', 'index': 2}, {'word': 'Eesti', 'score': 0.9988121390342712, 'entity': 'B-ORG', 'index': 5}, {'word': 'Pank', 'score': 0.9985784292221069, 'entity': 'I-ORG', 'index': 6}, {'word': 'Hansapanga', 'score': 0.9979034662246704, 'entity': 'B-ORG', 'index': 8}]

```

## BibTeX entry and citation info

```
@misc{tanvir2020estbert,
      title={EstBERT: A Pretrained Language-Specific BERT for Estonian},
      author={Hasan Tanvir and Claudia Kittask and Kairit Sirts},
      year={2020},
      eprint={2011.04784},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


---
language: et
---
# EstBERT

### What's this?
The EstBERT model is a pretrained BERTBase model exclusively trained on Estonian cased corpus on both 128 and 512 sequence length of data.

### How to use?
You can use the model transformer library both in tensorflow and pytorch version.
```
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("tartuNLP/EstBERT")
model = AutoModelForMaskedLM.from_pretrained("tartuNLP/EstBERT")
```
You can also download the pretrained model from here, [EstBERT_128]() [EstBERT_512]()
#### Dataset used to train the model
The EstBERT model is trained both on 128 and 512 sequence length of data. For training the EstBERT we used the [Estonian National Corpus 2017](https://metashare.ut.ee/repository/browse/estonian-national-corpus-2017/b616ceda30ce11e8a6e4005056b40024880158b577154c01bd3d3fcfc9b762b3/), which was the largest Estonian language corpus available at the time. It consists of four sub-corpora: Estonian Reference Corpus 1990-2008, Estonian Web Corpus 2013, Estonian Web Corpus 2017 and Estonian Wikipedia Corpus 2017.

### Why would I use?
Overall EstBERT performs better in parts of speech (POS), name entity recognition (NER), rubric, and sentiment classification tasks compared to mBERT and XLM-RoBERTa. The comparative results can be found below;

|Model   |UPOS                  |XPOS   |Morph  |bf UPOS   |bf XPOS                  |Morph                 |
|--------------|----------------------------|-------------|-------------|-------------|----------------------------|----------------------------|
| EstBERT      | **_97.89_** | **98.40** | **96.93** | **97.84** | **_98.43_** | **_96.80_** |
| mBERT        | 97.42                     | 98.06      | 96.24      | 97.43      | 98.13                     | 96.13                     |
| XLM-RoBERTa | 97.78                     | 98.36      | 96.53      | 97.80      | 98.40                     | 96.69                     |

|Model|Rubric128        |Sentiment128  | Rubric128   |Sentiment512         |
|-------------------|----------------------------|--------------------|-----------------------------------------------|----------------------------|
| EstBERT           | **_81.70_** | 74.36             | **80.96**                                   | 74.50                     |
| mBERT             | 75.67                     | 70.23             | 74.94                                        | 69.52                     |
| XLM\-RoBERTa      | 80.34                     | **74.50**        | 78.62                                        | **_76.07_**|

|Model   |Precicion128   |Recall128                  |F1-Score128               |Precision512               |Recall512   |F1-Score512   |
|--------------|----------------|----------------------------|----------------------------|----------------------------|-------------|----------------|
| EstBERT      | **88.42**    | 90.38                     |**_89.39_** | 88.35                     | 89.74      | 89.04         |
| mBERT        | 85.88         | 87.09                     | 86.51                     |**_88.47_** | 88.28      | 88.37         |
| XLM\-RoBERTa | 87.55         |**_91.19_** | 89.34                     | 87.50                     | **90.76** | **89.10**    |


Algebraic Notation model of sequences of moves of complete chess games.


# Argument Relation Mining

Best performing model trained in the "Transformer-Based Models for Automatic Detection of Argument Relations: A Cross-Domain Evaluation" paper.

Code available in https://github.com/raruidol/ArgumentRelationMining

Cite:

```
@article{ruiz2021transformer,
  title={Transformer-based models for automatic identification of argument relations: A cross-domain evaluation},
  author={Ruiz-Dolz, Ramon and Alemany, Jose and Heras, Stella and Garcia-Fornes, Ana},
  journal={IEEE Intelligent Systems},
  year={2021},
  publisher={IEEE}
}
```


Algebraic Notation model of sequences of moves done by a unique player in a chess game.


Hello, I am Saba Rajput and I am a 22-year-old young and charming Delhi Call girls and I provide Delhi Escorts Service Safe & Secure High-Class Services Affordable Rate 100% Satisfaction, Unlimited Enjoyment. I am from Delhi and I am a working girl in an Escorts company based in Delhi


---
tags:
- generated_from_trainer
datasets:
- wikitext
model-index:
- name: msft-regular-model
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# msft-regular-model

This model is a fine-tuned version of [](https://huggingface.co/) on the wikitext dataset.
It achieves the following results on the evaluation set:
- Loss: 5.3420

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20

### Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 9.1224        | 0.17  | 200   | 8.0736          |
| 7.5229        | 0.34  | 400   | 7.1536          |
| 7.0122        | 0.51  | 600   | 6.9072          |
| 6.8296        | 0.69  | 800   | 6.7582          |
| 6.709         | 0.86  | 1000  | 6.6436          |
| 6.5882        | 1.03  | 1200  | 6.5563          |
| 6.4807        | 1.2   | 1400  | 6.4784          |
| 6.4172        | 1.37  | 1600  | 6.4165          |
| 6.3403        | 1.54  | 1800  | 6.3555          |
| 6.2969        | 1.71  | 2000  | 6.3107          |
| 6.2346        | 1.89  | 2200  | 6.2691          |
| 6.1767        | 2.06  | 2400  | 6.2299          |
| 6.1326        | 2.23  | 2600  | 6.1937          |
| 6.1035        | 2.4   | 2800  | 6.1602          |
| 6.0624        | 2.57  | 3000  | 6.1241          |
| 6.0393        | 2.74  | 3200  | 6.0971          |
| 5.9982        | 2.91  | 3400  | 6.0656          |
| 5.9526        | 3.08  | 3600  | 6.0397          |
| 5.9086        | 3.26  | 3800  | 6.0104          |
| 5.8922        | 3.43  | 4000  | 5.9888          |
| 5.8631        | 3.6   | 4200  | 5.9661          |
| 5.8396        | 3.77  | 4400  | 5.9407          |
| 5.8055        | 3.94  | 4600  | 5.9177          |
| 5.7763        | 4.11  | 4800  | 5.9007          |
| 5.7314        | 4.28  | 5000  | 5.8834          |
| 5.7302        | 4.46  | 5200  | 5.8620          |
| 5.6987        | 4.63  | 5400  | 5.8451          |
| 5.6754        | 4.8   | 5600  | 5.8242          |
| 5.6571        | 4.97  | 5800  | 5.8059          |
| 5.615         | 5.14  | 6000  | 5.7871          |
| 5.596         | 5.31  | 6200  | 5.7817          |
| 5.5738        | 5.48  | 6400  | 5.7570          |
| 5.5641        | 5.66  | 6600  | 5.7431          |
| 5.5503        | 5.83  | 6800  | 5.7271          |
| 5.5214        | 6.0   | 7000  | 5.7108          |
| 5.4712        | 6.17  | 7200  | 5.7018          |
| 5.48          | 6.34  | 7400  | 5.6936          |
| 5.4527        | 6.51  | 7600  | 5.6812          |
| 5.4514        | 6.68  | 7800  | 5.6669          |
| 5.4454        | 6.86  | 8000  | 5.6509          |
| 5.399         | 7.03  | 8200  | 5.6408          |
| 5.3747        | 7.2   | 8400  | 5.6327          |
| 5.3667        | 7.37  | 8600  | 5.6197          |
| 5.3652        | 7.54  | 8800  | 5.6084          |
| 5.3394        | 7.71  | 9000  | 5.5968          |
| 5.3349        | 7.88  | 9200  | 5.5870          |
| 5.2994        | 8.05  | 9400  | 5.5826          |
| 5.2793        | 8.23  | 9600  | 5.5710          |
| 5.2716        | 8.4   | 9800  | 5.5623          |
| 5.275         | 8.57  | 10000 | 5.5492          |
| 5.264         | 8.74  | 10200 | 5.5449          |
| 5.241         | 8.91  | 10400 | 5.5322          |
| 5.2285        | 9.08  | 10600 | 5.5267          |
| 5.2021        | 9.25  | 10800 | 5.5187          |
| 5.1934        | 9.43  | 11000 | 5.5158          |
| 5.1737        | 9.6   | 11200 | 5.5044          |
| 5.1774        | 9.77  | 11400 | 5.5008          |
| 5.1841        | 9.94  | 11600 | 5.4960          |
| 5.1414        | 10.11 | 11800 | 5.4895          |
| 5.1491        | 10.28 | 12000 | 5.4849          |
| 5.1184        | 10.45 | 12200 | 5.4738          |
| 5.1136        | 10.63 | 12400 | 5.4690          |
| 5.1199        | 10.8  | 12600 | 5.4598          |
| 5.1056        | 10.97 | 12800 | 5.4536          |
| 5.0648        | 11.14 | 13000 | 5.4496          |
| 5.0598        | 11.31 | 13200 | 5.4449          |
| 5.0656        | 11.48 | 13400 | 5.4422          |
| 5.0664        | 11.65 | 13600 | 5.4367          |
| 5.0675        | 11.83 | 13800 | 5.4286          |
| 5.0459        | 12.0  | 14000 | 5.4249          |
| 5.0073        | 12.17 | 14200 | 5.4260          |
| 5.0229        | 12.34 | 14400 | 5.4175          |
| 5.0079        | 12.51 | 14600 | 5.4119          |
| 5.0           | 12.68 | 14800 | 5.4194          |
| 5.0094        | 12.85 | 15000 | 5.4068          |
| 4.9967        | 13.02 | 15200 | 5.3995          |
| 4.9541        | 13.2  | 15400 | 5.4002          |
| 4.9753        | 13.37 | 15600 | 5.3965          |
| 4.9732        | 13.54 | 15800 | 5.3925          |
| 4.9624        | 13.71 | 16000 | 5.3888          |
| 4.9559        | 13.88 | 16200 | 5.3824          |
| 4.9559        | 14.05 | 16400 | 5.3851          |
| 4.9109        | 14.22 | 16600 | 5.3815          |
| 4.9211        | 14.4  | 16800 | 5.3784          |
| 4.9342        | 14.57 | 17000 | 5.3735          |
| 4.9271        | 14.74 | 17200 | 5.3711          |
| 4.9328        | 14.91 | 17400 | 5.3646          |
| 4.8994        | 15.08 | 17600 | 5.3664          |
| 4.8932        | 15.25 | 17800 | 5.3642          |
| 4.8886        | 15.42 | 18000 | 5.3620          |
| 4.8997        | 15.6  | 18200 | 5.3584          |
| 4.8846        | 15.77 | 18400 | 5.3551          |
| 4.8993        | 15.94 | 18600 | 5.3516          |
| 4.8648        | 16.11 | 18800 | 5.3552          |
| 4.8838        | 16.28 | 19000 | 5.3512          |
| 4.8575        | 16.45 | 19200 | 5.3478          |
| 4.8623        | 16.62 | 19400 | 5.3480          |
| 4.8631        | 16.8  | 19600 | 5.3439          |
| 4.8576        | 16.97 | 19800 | 5.3428          |
| 4.8265        | 17.14 | 20000 | 5.3420          |
| 4.8523        | 17.31 | 20200 | 5.3410          |
| 4.8477        | 17.48 | 20400 | 5.3396          |
| 4.8507        | 17.65 | 20600 | 5.3380          |
| 4.8498        | 17.82 | 20800 | 5.3333          |
| 4.8261        | 17.99 | 21000 | 5.3342          |
| 4.8201        | 18.17 | 21200 | 5.3324          |
| 4.8214        | 18.34 | 21400 | 5.3341          |
| 4.8195        | 18.51 | 21600 | 5.3315          |
| 4.8216        | 18.68 | 21800 | 5.3335          |
| 4.8243        | 18.85 | 22000 | 5.3291          |
| 4.832         | 19.02 | 22200 | 5.3295          |
| 4.8085        | 19.19 | 22400 | 5.3309          |
| 4.8094        | 19.37 | 22600 | 5.3283          |
| 4.815         | 19.54 | 22800 | 5.3280          |
| 4.8219        | 19.71 | 23000 | 5.3270          |
| 4.8117        | 19.88 | 23200 | 5.3280          |

### Framework versions

- Transformers 4.13.0.dev0
- Pytorch 1.10.0
- Datasets 1.14.0
- Tokenizers 0.10.3


---
tags:
- conversational
---

# Anakin Skywalker DialoGPT Model


---
language: "english"
license: "mit"
datasets:
- race
metrics:
- accuracy

---

# MCQ with Distilbert


---
tags:
- conversational
---

#Loki DialoGPT Model


# ChineseBERT-base

This repository contains code, model, dataset for **ChineseBERT** at ACL2021.

paper:
**[ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information](https://arxiv.org/abs/2106.16038)**
*Zijun Sun, Xiaoya Li, Xiaofei Sun, Yuxian Meng, Xiang Ao, Qing He, Fei Wu and Jiwei Li*

code:
[ChineseBERT github link](https://github.com/ShannonAI/ChineseBert)

## Model description
We propose ChineseBERT, which incorporates both the glyph and pinyin information of Chinese
characters into language model pretraining.

First, for each Chinese character, we get three kind of embedding.
 - **Char Embedding:** the same as origin BERT token embedding.
 - **Glyph Embedding:** capture visual features based on different fonts of a Chinese character.
 - **Pinyin Embedding:** capture phonetic feature from the pinyin sequence ot a Chinese Character.

Then, char embedding, glyph embedding and pinyin embedding
are first concatenated, and mapped to a D-dimensional embedding through a fully
connected layer to form the fusion embedding.
Finally, the fusion embedding is added with the position embedding, which is fed as input to the BERT model.
The following image shows an overview architecture of ChineseBERT model.

![MODEL](https://raw.githubusercontent.com/ShannonAI/ChineseBert/main/images/ChineseBERT.png)

ChineseBERT leverages the glyph and pinyin information of Chinese
characters to enhance the model's ability of capturing
context semantics from surface character forms and
disambiguating polyphonic characters in Chinese.


# ChineseBERT-large

This repository contains code, model, dataset for **ChineseBERT** at ACL2021.

paper:
**[ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information](https://arxiv.org/abs/2106.16038)**
*Zijun Sun, Xiaoya Li, Xiaofei Sun, Yuxian Meng, Xiang Ao, Qing He, Fei Wu and Jiwei Li*

code:
[ChineseBERT github link](https://github.com/ShannonAI/ChineseBert)

## Model description
We propose ChineseBERT, which incorporates both the glyph and pinyin information of Chinese
characters into language model pretraining.

First, for each Chinese character, we get three kind of embedding.
 - **Char Embedding:** the same as origin BERT token embedding.
 - **Glyph Embedding:** capture visual features based on different fonts of a Chinese character.
 - **Pinyin Embedding:** capture phonetic feature from the pinyin sequence ot a Chinese Character.

Then, char embedding, glyph embedding and pinyin embedding
are first concatenated, and mapped to a D-dimensional embedding through a fully
connected layer to form the fusion embedding.
Finally, the fusion embedding is added with the position embedding, which is fed as input to the BERT model.
The following image shows an overview architecture of ChineseBERT model.

![MODEL](https://raw.githubusercontent.com/ShannonAI/ChineseBert/main/images/ChineseBERT.png)

ChineseBERT leverages the glyph and pinyin information of Chinese
characters to enhance the model's ability of capturing
context semantics from surface character forms and
disambiguating polyphonic characters in Chinese.


git lfs install
git clone https://huggingface.co/EleutherAI/gpt-j-6B


KcELECTRA([https://github.com/Beomi/KcELECTRA](https://github.com/Beomi/KcELECTRA))의 Tokenizer에서 [UNK]로 대체되는 토큰들을 추가했습니다.


ElectraBERT tuned with korean-bad-speeches


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- null
model_index:
- name: distilgpt2-finetuned-distilgpt2-med_articles
  results:
  - task:
      name: Causal Language Modeling
      type: text-generation
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilgpt2-finetuned-distilgpt2-med_articles

This model is a fine-tuned version of [vishnun/distilgpt2-finetuned-distilgpt2-med_articles](https://huggingface.co/vishnun/distilgpt2-finetuned-distilgpt2-med_articles) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 3.3171

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 65   | 3.3417          |
| No log        | 2.0   | 130  | 3.3300          |
| No log        | 3.0   | 195  | 3.3231          |
| No log        | 4.0   | 260  | 3.3172          |
| No log        | 5.0   | 325  | 3.3171          |

### Framework versions

- Transformers 4.9.2
- Pytorch 1.9.0+cu102
- Datasets 1.11.0
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- null
model_index:
- name: distilgpt2-finetuned-tamil-gpt
  results:
  - task:
      name: Causal Language Modeling
      type: text-generation
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilgpt2-finetuned-tamil-gpt

This model is a fine-tuned version of [distilgpt2](https://huggingface.co/distilgpt2) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 4.4097

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 228  | 4.4097          |
| No log        | 2.0   | 456  | 4.4097          |
| 4.3169        | 3.0   | 684  | 4.4097          |
| 4.3169        | 4.0   | 912  | 4.4097          |
| 4.3116        | 5.0   | 1140 | 4.4097          |

### Framework versions

- Transformers 4.9.2
- Pytorch 1.9.0+cu102
- Datasets 1.11.0
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- null
model_index:
- name: distilgpt2-finetuned-tamilmixsentiment
  results:
  - task:
      name: Causal Language Modeling
      type: text-generation
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilgpt2-finetuned-tamilmixsentiment

This model is a fine-tuned version of [distilgpt2](https://huggingface.co/distilgpt2) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 4.4572

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 5.6438        | 1.0   | 907  | 4.8026          |
| 4.774         | 2.0   | 1814 | 4.5953          |
| 4.5745        | 3.0   | 2721 | 4.5070          |
| 4.4677        | 4.0   | 3628 | 4.4688          |
| 4.4294        | 5.0   | 4535 | 4.4572          |

### Framework versions

- Transformers 4.9.2
- Pytorch 1.9.0+cu102
- Datasets 1.11.0
- Tokenizers 0.10.3


# Tamil Mix Sentiment analysis

Model is trained on tamil-mix-sentiment dataset and finetuned with backend as bert-base-cased model

## Inference usage

On the hosted Inference type in the text for which you want to classify.
Eg: Super a iruku bro intha work, vera level mass


---
tags:
- generated_from_trainer
metrics:
- f1
model-index:
- name: rubert-base-srl
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# rubert-base-srl

This model is a fine-tuned version of [./ruBert-base/](https://huggingface.co/./ruBert-base/) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.2429
- F1: 0.9563

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-06
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.06
- num_epochs: 10.0

### Training results

| Training Loss | Epoch | Step | Validation Loss | F1     |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 0.5816        | 1.0   | 57   | 0.3865          | 0.8371 |
| 0.3685        | 2.0   | 114  | 0.1707          | 0.9325 |
| 0.1057        | 3.0   | 171  | 0.0972          | 0.9563 |
| 0.0964        | 4.0   | 228  | 0.1429          | 0.9775 |
| 0.1789        | 5.0   | 285  | 0.2493          | 0.9457 |
| 0.0016        | 6.0   | 342  | 0.1900          | 0.6349 |
| 0.0013        | 7.0   | 399  | 0.2060          | 0.9563 |
| 0.0008        | 8.0   | 456  | 0.2321          | 0.9563 |
| 0.0006        | 9.0   | 513  | 0.2412          | 0.9563 |
| 0.0006        | 10.0  | 570  | 0.2429          | 0.9563 |

### Framework versions

- Transformers 4.13.0.dev0
- Pytorch 1.10.0+cu102
- Datasets 1.15.1
- Tokenizers 0.10.3