A simple model trained on dialogues of characters in NBC series, `The Office`. The model can do a binary classification between `Michael Scott` and `Dwight Shrute`'s dialogues.


<style type="text/css">
.tg  {border-collapse:collapse;border-spacing:0;}
.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
  overflow:hidden;padding:10px 5px;word-break:normal;}
.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
  font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}
.tg .tg-c3ow{border-color:inherit;text-align:center;vertical-align:top}
</style>


    Label Definitions

    Label 0
    Michael

    Label 1
    Dwight


model name: bert_classification_lm


model author: Alireza1044


task: text-classification


tags: pytorch, bert, text-classification, transformers, infinity_compatible


language: Bengali
datasets:


- OpenSLR
metrics:
- wer
tags:
- bn
- audio
- automatic-speech-recognition
- speech
license: cc-by-sa-4.0
model-index:
- name: XLSR Wav2Vec2 Bengali by Arijit
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: OpenSLR
type: OpenSLR
args: ben
metrics:
    - name: Test WER
type: wer
value: 32.45


# Wav2Vec2-Large-XLSR-Bengali


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) Bengali using a subset of 40,000 utterances from [Bengali ASR training data set containing ~196K utterances](https://www.openslr.org/53/). Tested WER using ~4200 held out from training.
When using this model, make sure that your speech input is sampled at 16kHz.
Train Script can be Found at : train.py
Data Prep Notebook : https://colab.research.google.com/drive/1JMlZPU-DrezXjZ2t7sOVqn7CJjZhdK2q?usp=sharing
Inference Notebook : https://colab.research.google.com/drive/1uKC2cK9JfUPDTUHbrNdOYqKtNozhxqgZ?usp=sharing


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
processor = Wav2Vec2Processor.from_pretrained("arijitx/wav2vec2-large-xlsr-bengali")
model = Wav2Vec2ForCTC.from_pretrained("arijitx/wav2vec2-large-xlsr-bengali")
# model = model.to("cuda")
resampler = torchaudio.transforms.Resample(TEST_AUDIO_SR, 16_000)
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch)
    speech =  resampler(speech_array).squeeze().numpy()
    return speech
speech_array = speech_file_to_array_fn("test_file.wav")
inputs = processor(speech_array, sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
    logits = model(inputs.input_values).logits

predicted_ids = torch.argmax(logits, dim=-1)
preds = processor.batch_decode(predicted_ids)[0]
print(preds.replace("[PAD]",""))
```


**Test Result**: WER on ~4200 utterance : 32.45 %


model name: asr-en-bn-test


model author: YasinShihab


task: automatic-speech-recognition


tags: Bengali, dataset:OpenSLR, bn, audio, automatic-speech-recognition, speech, license:cc-by-sa-4.0, model-index


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: wav2vec2-base-timit-demo-colab
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-base-timit-demo-colab


This model is a fine-tuned version of [facebook/wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 3.3780
- Wer: 1.0


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 1
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| No log        | 0.08  | 10   | 14.0985         | 1.0    |
| No log        | 0.16  | 20   | 13.8638         | 1.0004 |
| No log        | 0.24  | 30   | 13.5135         | 1.0023 |
| No log        | 0.32  | 40   | 12.8708         | 1.0002 |
| No log        | 0.4   | 50   | 11.6927         | 1.0    |
| No log        | 0.48  | 60   | 10.2733         | 1.0    |
| No log        | 0.56  | 70   | 8.1396          | 1.0    |
| No log        | 0.64  | 80   | 5.3503          | 1.0    |
| No log        | 0.72  | 90   | 3.7975          | 1.0    |
| No log        | 0.8   | 100  | 3.4275          | 1.0    |
| No log        | 0.88  | 110  | 3.3596          | 1.0    |
| No log        | 0.96  | 120  | 3.3780          | 1.0    |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: wav2vec2-base-timit-demo-colab


model author: hiraki


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, transformers, license:apache-2.0, generated_from_trainer, model-index


XLM-RoBERTa base (`xlm-roberta-base`) finetuned on squad v1.1.


**Training-specifications:**


- training_epochs: 3.0
- max_seq_length: 384
- batch_size: 16
- dataset_name: squad
- doc_stride 128


**Train-results:**


```
{
    "epoch": 3.0,
    "init_mem_cpu_alloc_delta": 991453184,
    "init_mem_cpu_peaked_delta": 0,
    "init_mem_gpu_alloc_delta": 1109893120,
    "init_mem_gpu_peaked_delta": 0,
    "train_mem_cpu_alloc_delta": 14753792,
    "train_mem_cpu_peaked_delta": 0,
    "train_mem_gpu_alloc_delta": 3330195456,
    "train_mem_gpu_peaked_delta": 8287144960,
    "train_runtime": 11376.3034,
    "train_samples": 89597,
    "train_samples_per_second": 1.477
}
```


**Eval-results:**


```
{
    "epoch": 3.0,
    "eval_samples": 10918,
    "exact_match": 82.06244087038789,
    "f1": 89.09539709124654
}
```


model name: xlm_roberta_squad_v1.1


model author: jakobwes


task: question-answering


tags: pytorch, xlm-roberta, question-answering, transformers, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: distilbert-base-uncased-finetuned-squad
results: []


# distilbert-base-uncased-finetuned-squad


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the squad dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Framework versions


- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-squad


model author: kenlevine


task: question-answering


tags: pytorch, tensorboard, distilbert, question-answering, dataset:squad, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


language:


- en
thumbnail:
tags:
- gpt2
- conversational
license: cc-by-4.0
widget:
- text: Hello there
context: 'Gandalf'


# GPT2 Persona Chatbot based on Movie Characters


Model used for https://www.metayazar.com/chatbot


GPT2 Small Trained on movie scripts (especially Sci-fi)


Usual HF api will not work see HF Spaces for demo usage https://huggingface.co/spaces/gorkemgoknar/moviechatbot


This work is based on Persona Chatbot originally done by Hugging Face team (https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313)


For cleaning movie scripts I also provide cleaner code
https://github.com/gorkemgoknar/moviescriptcleaner


Example persona how to:
https://gist.github.com/gorkemgoknar/ae29bf9d14fa814e6a64d0e57a4a4ed7


For obvious reasons I cannot share raw personafile but you can check above gist for example how to create it.


A working "full" demo can be seen in https://www.metayazar.com/chatbot


For Turkish version (with limited training) https://www.metayazar.com/chatbot_tr


Due to double LM head standart hugging face interface will not work. But if you follow huggingface tutorial should be same.
Except each persona is encoded as "My name is XXXX"


Use model, tokenizer and parameters within a class and call in below functions to trigger model.
Some of the available personas:


| Macleod | Moran | Brenda | Ramirez | Peter Parker | Quentin Beck | Andy
| Red | Norton | Willard | Chief | Chef | Kilgore | Kurtz | Westley | Buttercup
| Vizzini | Fezzik | Inigo | Man In Black | Taylor | Zira | Zaius | Cornelius
| Bud | Lindsey | Hippy | Erin | Ed | George | Donna | Trinity | Agent Smith
| Morpheus | Neo | Tank | Meryl | Truman | Marlon | Christof | Stromboli | Bumstead
| Schreber | Walker | Korben | Cornelius | Loc Rhod | Anakin | Obi-Wan | Palpatine
| Padme | Superman | Luthor | Dude | Walter | Donny | Maude | General | Starkiller
| Indiana | Willie | Short Round | John | Sarah | Terminator | Miller | Sarge | Reiben
| Jackson | Upham | Chuckie | Will | Lambeau | Sean | Skylar | Saavik | Spock
| Kirk | Bones | Khan | Kirk | Spock | Sybok | Scotty | Bourne | Pamela | Abbott


```python
    def get_answer(self, input_text, personality, history, params=None):

        ##Check length of history (to save 1 computation!)
        if len(history)>0:
            #mostly it will be empty list so need a length check for performance
            #would do string check also but just assume it is list of list of strings, as not public

            new_hist = []
            for ele in history:
                new_hist.append( self.tokenizer.encode(ele) )
            history = new_hist.copy()

        history.append(self.tokenizer.encode(input_text))

        with torch.no_grad():
            out_ids = self.sample_sequence(personality, history, self.tokenizer, self.model, params=params)
        history.append(out_ids)
        history = history[-(2*self.parameters['max_history']+1):]
        out_text = self.tokenizer.decode(out_ids, skip_special_tokens=True)
        #print(out_text)

        history_decoded = []
        for ele in history:
            history_decoded.append(self.tokenizer.decode(ele))

        return out_text, history_decoded, self.parameters

```


model name: gpt2chatbotenglish


model author: gorkemgoknar


task: conversational


tags: pytorch, gpt2, text-generation, en, transformers, conversational, license:cc-by-4.0


language:


- tr
datasets:
- common_voice
- movies
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: XLSR Wav2Vec2 Large Turkish with extended dataset by Gorkem Goknar
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice tr
type: common_voice
args: tr
metrics:
    - name: Test WER
type: wer
value: 50.41


# Wav2Vec2-Large-XLSR-53-Turkish


Note: This model is trained with 5 Turkish movies additional to common voice dataset.
Although WER is high (50%) per common voice test dataset,  performance from "other sources " seems pretty good.


Disclaimer: Please use another wav2vec2-tr model in hub for "clean environment" dialogues as they tend to do better in clean sounds with less background noise.


Dataset building from csv and merging code can be found on below of this Readme.


Please try speech yourself on the right side to see its performance.


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Turkish using the [Common Voice](https://huggingface.co/datasets/common_voice) and 5 Turkish movies that include background noise/talkers .


When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
import pydub
from pydub.utils import mediainfo
import array
from pydub import AudioSegment
from pydub.utils import get_array_type
import numpy as np

from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
test_dataset = load_dataset("common_voice", "tr", split="test[:2%]")
processor = Wav2Vec2Processor.from_pretrained("gorkemgoknar/wav2vec2-large-xlsr-53-turkish")
model = Wav2Vec2ForCTC.from_pretrained("gorkemgoknar/wav2vec2-large-xlsr-53-turkish")

new_sample_rate = 16000

def audio_resampler(batch, new_sample_rate = 16000):

    #not working without complex library compilation in windows for mp3
    #speech_array, sampling_rate = torchaudio.load(batch["path"])
    #speech_array, sampling_rate = librosa.load(batch["path"])

    #sampling_rate =  pydub.utils.info['sample_rate']  ##gets current samplerate

    sound = pydub.AudioSegment.from_file(file=batch["path"])
    sampling_rate = new_sample_rate
    sound = sound.set_frame_rate(new_sample_rate)
    left = sound.split_to_mono()[0]
    bit_depth = left.sample_width * 8
    array_type = pydub.utils.get_array_type(bit_depth)

    numeric_array = np.array(array.array(array_type, left._data) )

    speech_array = torch.FloatTensor(numeric_array)

    batch["speech"] = numeric_array
    batch["sampling_rate"] = sampling_rate
    #batch["target_text"] = batch["sentence"]

    return batch

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch = audio_resampler(batch, new_sample_rate = new_sample_rate)
    return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
predicted_ids = torch.argmax(logits, dim=-1)
print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


The model can be evaluated as follows on the Turkish test data of Common Voice.


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re
import pydub
import array
import numpy as np

test_dataset = load_dataset("common_voice", "tr", split="test")
wer = load_metric("wer")
processor = Wav2Vec2Processor.from_pretrained("gorkemgoknar/wav2vec2-large-xlsr-53-turkish")
model = Wav2Vec2ForCTC.from_pretrained("gorkemgoknar/wav2vec2-large-xlsr-53-turkish")
model.to("cuda")

#Note: Not ignoring "'"  on this one
#Note: Not ignoring "'"  on this one
chars_to_ignore_regex = '[\\,\\?\\.\\!\\-\\;\\:\\"\\“\\%\\‘\\”\\�\\#\\>\\<\\_\\’\\[\\]\\{\\}]'

#resampler = torchaudio.transforms.Resample(48_000, 16_000)
#using custom load and transformer for audio  -> see audio_resampler
new_sample_rate = 16000

def audio_resampler(batch, new_sample_rate = 16000):

    #not working without complex library compilation in windows for mp3
    #speech_array, sampling_rate = torchaudio.load(batch["path"])
    #speech_array, sampling_rate = librosa.load(batch["path"])
    #sampling_rate =  pydub.utils.info['sample_rate']  ##gets current samplerate

    sound = pydub.AudioSegment.from_file(file=batch["path"])

    sound = sound.set_frame_rate(new_sample_rate)
    left = sound.split_to_mono()[0]
    bit_depth = left.sample_width * 8
    array_type = pydub.utils.get_array_type(bit_depth)

    numeric_array = np.array(array.array(array_type, left._data) )

    speech_array = torch.FloatTensor(numeric_array)

    return speech_array, new_sample_rate

def remove_special_characters(batch):

    ##this one comes from subtitles if additional timestamps not processed  -> 00:01:01   00:01:01,33
    batch["sentence"] = re.sub('\\b\\d{2}:\\d{2}:\\d{2}(,+\\d{2})?\\b', ' ', batch["sentence"])
    ##remove all caps in text [AÇIKLAMA] etc, do it before..
    batch["sentence"] = re.sub('\\[(\\b[A-Z]+\\])', '', batch["sentence"])
    ##replace three dots (that are inside string with single)
    batch["sentence"] = re.sub("([a-zA-Z]+)\\.\\.\\.", r"\\1.", batch["sentence"])
    #standart ignore list
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower() + " "

    return batch

# Preprocessing the datasets.
# We need to read the aduio files as arrays
new_sample_rate = 16000
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    ##speech_array, sampling_rate = torchaudio.load(batch["path"])
    ##load and conversion done in resampler , takes and returns batch
    speech_array, sampling_rate = audio_resampler(batch, new_sample_rate = new_sample_rate)
    batch["speech"] = speech_array
    batch["sampling_rate"] = sampling_rate
    batch["target_text"] = batch["sentence"]

    return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):

    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits
    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch

print("EVALUATING:")

##for 8GB RAM on GPU best is batch_size 2 for windows,  4 may fit in linux only
result = test_dataset.map(evaluate, batched=True, batch_size=2)
print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))

```


**Test Result**: 50.41 %


## Training


The Common Voice `train` and `validation` datasets were used for training. Additional 5 Turkish movies with subtitles also used for training.
Similar training model used as base fine-tuning, additional audio resampler is on above code.


Putting model building and merging code below for reference


```python
import pandas as pd
from datasets import load_dataset, load_metric

import os
from pathlib import Path
from datasets import Dataset
import csv

#Walk all subdirectories of base_set_path and find csv files
base_set_path = r'C:\\dataset_extracts'
csv_files = []
for path, subdirs, files in os.walk(base_set_path):
    for name in files:
        if name.endswith(".csv"):
            deckfile= os.path.join(path, name)
            csv_files.append(deckfile)

def get_dataset_from_csv_file(csvfilename,names=['sentence', 'path']):
  path = Path(csvfilename)
  csv_delimiter="\\t"  ##tab seperated, change if something else

  ##Pandas has bug reading non-ascii file names, make sure use open with encoding
  df=pd.read_csv(open(path, 'r', encoding='utf-8'), delimiter=csv_delimiter,header=None , names=names, encoding='utf8')
  return Dataset.from_pandas(df)

custom_datasets= []
for csv_file in csv_files:
  this_dataset=get_dataset_from_csv_file(csv_file)
  custom_datasets.append(this_dataset)

from datasets import concatenate_datasets, load_dataset
from datasets import load_from_disk

# Merge datasets together (from csv files)
dataset_file_path = ".\\dataset_file"
custom_datasets_concat = concatenate_datasets( [dset for dset in custom_datasets] )

#save this one to disk
custom_datasets_concat.save_to_disk( dataset_file_path )

#load back from disk
custom_datasets_from_disk = load_from_disk(dataset_file_path)
```


model name: wav2vec2-large-xlsr-53-turkish


model author: gorkemgoknar


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, tr, dataset:common_voice, dataset:movies, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


language:


- tr
thumbnail:
tags:
- gpt2
- turkish


license: apache-2.0
datasets:


- wikipedia-turkish
metrics:
- perplexity
- accuracy


widget:


- text: Bu yazıyı bir bilgisayar yazdı. Yazarken
context: ''
- text: İnternete kolay erişim sayesinde dünya daha da küçüldü. Bunun sonucunda
context: ''


# Turkish GPT2 Model Finetuned


# Türkçe GPT2 Modeli


## Model description


This is a GPT2-Small English based model finetuned and additionaly trainied with Wikipedia Articles in Turkish as of 28-10-2020


Live demo based on this work at : https://www.metayazar.com/


Fine tuned writer on this model: https://huggingface.co/gorkemgoknar/gpt2-turkish-writer


Work has been done on Pierre Guillou tutorial as on this page.
(https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb)


Code is converted to work with Fastai 2.X .


Using Google Colab for training.


Additional tutorial and source will be in https://github.com/gorkemgoknar in later stage.


Current accuracy 33 %  , Perplexity : 51.88


Models are available:


* [gpt2-small-tuned-tr] (https://huggingface.co/gorkemgoknar/gpt2-small-turkish)
* [gpt2-small-turkish-writer] (https://huggingface.co/gorkemgoknar/gpt2-turkish-writer)


## Intended uses & limitations


#### How to use


#### Install


```python
from transformers import AutoTokenizer, AutoModelWithLMHead
import torch

tokenizer = AutoTokenizer.from_pretrained("gorkemgoknar/gpt2-small-turkish")
model = AutoModelWithLMHead.from_pretrained("gorkemgoknar/gpt2-small-turkish")

# Get sequence length max of 1024
tokenizer.model_max_length=1024

model.eval()  # disable dropout (or leave in train mode to finetune)

```


#### Generate 1 word


```python
# input sequence
text = "Bu yazıyı bilgisayar yazdı."
inputs = tokenizer(text, return_tensors="pt")

# model output
outputs = model(**inputs, labels=inputs["input_ids"])
loss, logits = outputs[:2]
predicted_index = torch.argmax(logits[0, -1, :]).item()
predicted_text = tokenizer.decode([predicted_index])

# results
print('input text:', text)
print('predicted text:', predicted_text)

# input text:
# predicted text:

```


#### Generate Full Sequence


```python
# input sequence
text = "Bu yazıyı bilgisayar yazdı."
inputs = tokenizer(text, return_tensors="pt")

# model output using Top-k sampling text generation method
sample_outputs = model.generate(inputs.input_ids,
                                pad_token_id=50256,
                                do_sample=True,
                                max_length=50, # put the token number you want
                                top_k=40,
                                num_return_sequences=1)

# generated sequence
for i, sample_output in enumerate(sample_outputs):
    print(">> Generated text {}\\\\
\\\\
{}".format(i+1, tokenizer.decode(sample_output.tolist())))

# >> Generated text
#

```


#### Limitations and bias


The training data used for this model come from Turkish Wikipedia. We know it contains a lot of unfiltered content from the internet, which is far from neutral.


## Training data


Wikipedia Turkish article dump as of 28-10-2020


## Training procedure


## Eval results


| epoch\\\\t|train_loss\\\\t|valid_loss\\\\t|accuracy\\\\t|perplexity\\\\t|time   |
| ----- | --------      |---------      | ----------    | ---------     | ----- |
|0\\\\t|4.777015\\\\t|4.621834\\\\t|0.292547\\\\t|101.680367\\\\t|2:42:05|
|1\\\\t|4.509412\\\\t|4.403999\\\\t|0.305574\\\\t|81.777267\\\\t|1:09:38|
|2\\\\t|4.169529\\\\t|4.120755\\\\t|0.324908\\\\t|61.605747\\\\t|1:07:45|
|3\\\\t|4.293973\\\\t|4.177899\\\\t|0.317211\\\\t|65.228653\\\\t|1:07:02|
|4\\\\t|4.049848\\\\t|3.949103\\\\t|0.338347\\\\t|51.888783\\\\t|1:05:53|


#Epoch 0 on Tesla T4, others on V100


model name: gpt2-small-turkish


model author: gorkemgoknar


task: text-generation


tags: pytorch, jax, gpt2, text-generation, tr, dataset:wikipedia-turkish, transformers, turkish, license:apache-2.0


language:


- tr
thumbnail:
tags:
- gpt2
- turkish
- aiwriter
- finetuned


license: apache-2.0
datasets:


- wikipedia-turkish
- custom-book-corpus
metrics:
- perplexity
- accuracy


widget:


- text: Bir zaman topu olan ama köpeği olmayan bir çocuk vardı. Parkta
context: ''
- text: 'Uzun uzun sahile doğru baktı. Düşündüklerinden '
context: ''
- text: Çok uzun zaman önce galaksinin uzak bir köşesinde...
context: ''
- text: "'Bugün kendimi çok hasta hissediyorum' dedi. Karşısında "
context: ''


# Turkish AI Writer based on GPT2-Small


# Türkçe Yapay Zeka Yazarı


## Model description


This model is enhanced version of gpt2-small-turkish finetuned version. In addition to 28-10-2020 Wikipedia Turkish article dump this model is trained with more than 400 classic novels and plays in Turkish (Including Dostoyevski, Shaekspeare, Dumas)


Base work has been done on Pierre Guillou tutorial as on this page.
(https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb)


Note that Since Turkish language is not close to English as in Porteguese instead  of training last 2 layers, last 3 layers are trained.


Code is converted to work with Fastai 2.X .
Using Google Colab for training.


Current accuracy 36.3 %  , Perplexity : 44.75


Demo (using CPU inference) is available on: http://www.metayazar.com


Models are available:


* [gpt2-small-tuned-tr] (https://huggingface.co/gorkemgoknar/gpt2-small-turkish)
* [gpt2-small-turkish-writer] (https://huggingface.co/gorkemgoknar/gpt2-turkish-writer)


## Intended uses & limitations


#### How to use


#### Install


```python
from transformers import AutoTokenizer, AutoModelWithLMHead
import torch

tokenizer = AutoTokenizer.from_pretrained("gorkemgoknar/gpt2-turkish-writer")
model = AutoModelWithLMHead.from_pretrained("gorkemgoknar/gpt2-turkish-writer")

# Get sequence length max of 1024
tokenizer.model_max_length=1024

model.eval()  # disable dropout (or leave in train mode to finetune)

```


#### Generate 1 word


```python
# input sequence
text = "Bu yazıyı bilgisayar yazdı."
inputs = tokenizer(text, return_tensors="pt")

# model output
outputs = model(**inputs, labels=inputs["input_ids"])
loss, logits = outputs[:2]
predicted_index = torch.argmax(logits[0, -1, :]).item()
predicted_text = tokenizer.decode([predicted_index])

# results
print('input text:', text)
print('predicted text:', predicted_text)

# input text:
# predicted text:

```


#### Generate Full Sequence


```python
# input sequence
text = "Bu yazıyı bilgisayar yazdı."
inputs = tokenizer(text, return_tensors="pt")

# model output using Top-k sampling text generation method
sample_outputs = model.generate(inputs.input_ids,
                                pad_token_id=50256,
                                do_sample=True,
                                max_length=50, # put the token number you want
                                top_k=40,
                                num_return_sequences=1)

# generated sequence
for i, sample_output in enumerate(sample_outputs):
    print(">> Generated text {}\n\n{}".format(i+1, tokenizer.decode(sample_output.tolist())))

# >> Generated text
#

```


#### Limitations and bias


The training data used for this model come from Turkish Wikipedia and books. We know it contains a lot of unfiltered content from the internet, which is far from neutral. Also not much pre-processing was done on books hence chapter names and page numbers can be seen on some cases. This is a work in progress.


## Training data


Wikipedia Turkish article dump as of 28-10-2020
Turkish book dataset of >400 classic novels


## Training procedure


## Eval results


| epoch	|train_loss	|valid_loss	|accuracy	|perplexity	|time   |
| ----- | --------      |---------      | ----------    | ---------     | ----- |
|0	|4.497828	|4.549605	|0.277328	|94.595070	|2:09:58|
|1	|4.503929	|4.519456	|0.275071	|91.785645	|2:04:30|
|2	|3.612716	|3.921146	|0.344802	|50.458256	|2:03:22|
|3	|3.777645	|4.072006	|0.326130	|58.674530	|1:56:14|
|4	|2.934462	|3.801303	|0.363719	|44.759476	|1:58:55|


Note: 1cycle rule training is used and epochs are at different times


model name: gpt2-turkish-writer


model author: gorkemgoknar


task: text-generation


tags: pytorch, jax, gpt2, text-generation, tr, dataset:wikipedia-turkish, dataset:custom-book-corpus, transformers, turkish, aiwriter, finetuned, license:apache-2.0


thumbnail: https://huggingface.co/front/thumbnails/dialogpt.png
tags:


- conversational
license: mit


# DialoGPT Trained on the Speech of a Game Character


This is an instance of [microsoft/DialoGPT-medium](https://huggingface.co/microsoft/DialoGPT-small) trained on a game character, Joshua from [The World Ends With You](https://en.wikipedia.org/wiki/The_World_Ends_with_You). The data comes from [a Kaggle game script dataset](https://www.kaggle.com/ruolinzheng/twewy-game-script).
I built a Discord AI chatbot based on this model. [Check out my GitHub repo.](https://github.com/RuolinZheng08/twewy-discord-chatbot)
Chat with the model:


```python
from transformers import AutoTokenizer, AutoModelWithLMHead

tokenizer = AutoTokenizer.from_pretrained("scottastrong/DialogGPT-small-Scott")
model = AutoModelWithLMHead.from_pretrained("scottastrong/DialogGPT-small-Scott")
# Let's chat for 4 lines
for step in range(4):
    # encode the new user input, add the eos_token and return a tensor in Pytorch
    new_user_input_ids = tokenizer.encode(input(">> User:") + tokenizer.eos_token, return_tensors='pt')
    # print(new_user_input_ids)
    # append the new user input tokens to the chat history
    bot_input_ids = torch.cat([chat_history_ids, new_user_input_ids], dim=-1) if step > 0 else new_user_input_ids
    # generated a response while limiting the total chat history to 1000 tokens,
    chat_history_ids = model.generate(
        bot_input_ids, max_length=200,
        pad_token_id=tokenizer.eos_token_id,
        no_repeat_ngram_size=3,
        do_sample=True,
        top_k=100,
        top_p=0.7,
        temperature=0.8
    )

    # pretty print last ouput tokens from bot
    print("JoshuaBot: {}".format(tokenizer.decode(chat_history_ids[:, bot_input_ids.shape[-1]:][0], skip_special_tokens=True)))
```


model name: DialogGPT-small-Scott


model author: ScottaStrong


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational, license:mit


This is an instance of [microsoft/DialoGPT-medium](https://huggingface.co/microsoft/DialoGPT-medium) trained on a game character, Joshua from [The World Ends With You](https://en.wikipedia.org/wiki/The_World_Ends_with_You). The data comes from [a Kaggle game script dataset](https://www.kaggle.com/ruolinzheng/twewy-game-script).
I built a Discord AI chatbot based on this model. [Check out my GitHub repo.](https://github.com/RuolinZheng08/twewy-discord-chatbot)
Chat with the model:


```python
from transformers import AutoTokenizer, AutoModelWithLMHead

tokenizer = AutoTokenizer.from_pretrained("scottastrong/DialogGPT-medium-joshua")
model = AutoModelWithLMHead.from_pretrained("scottastrong/DialogGPT-medium-joshua")
# Let's chat for 4 lines
for step in range(4):
    # encode the new user input, add the eos_token and return a tensor in Pytorch
    new_user_input_ids = tokenizer.encode(input(">> User:") + tokenizer.eos_token, return_tensors='pt')
    # print(new_user_input_ids)
    # append the new user input tokens to the chat history
    bot_input_ids = torch.cat([chat_history_ids, new_user_input_ids], dim=-1) if step > 0 else new_user_input_ids
    # generated a response while limiting the total chat history to 1000 tokens,
    chat_history_ids = model.generate(
        bot_input_ids, max_length=200,
        pad_token_id=tokenizer.eos_token_id,
        no_repeat_ngram_size=3,
        do_sample=True,
        top_k=100,
        top_p=0.7,
        temperature=0.8
    )

    # pretty print last ouput tokens from bot
    print("JoshuaBot: {}".format(tokenizer.decode(chat_history_ids[:, bot_input_ids.shape[-1]:][0], skip_special_tokens=True)))
```


model name: DialogGPT-medium-joshua


```python
from transformers import AutoTokenizer, AutoModelWithLMHead

tokenizer = AutoTokenizer.from_pretrained("scottastrong/DialogGPT-small-joshua")
model = AutoModelWithLMHead.from_pretrained("scottastrong/DialogGPT-small-joshua")
# Let's chat for 4 lines
for step in range(4):
    # encode the new user input, add the eos_token and return a tensor in Pytorch
    new_user_input_ids = tokenizer.encode(input(">> User:") + tokenizer.eos_token, return_tensors='pt')
    # print(new_user_input_ids)
    # append the new user input tokens to the chat history
    bot_input_ids = torch.cat([chat_history_ids, new_user_input_ids], dim=-1) if step > 0 else new_user_input_ids
    # generated a response while limiting the total chat history to 1000 tokens,
    chat_history_ids = model.generate(
        bot_input_ids, max_length=200,
        pad_token_id=tokenizer.eos_token_id,
        no_repeat_ngram_size=3,
        do_sample=True,
        top_k=100,
        top_p=0.7,
        temperature=0.8
    )

    # pretty print last ouput tokens from bot
    print("JoshuaBot: {}".format(tokenizer.decode(chat_history_ids[:, bot_input_ids.shape[-1]:][0], skip_special_tokens=True)))
```


model name: DialogGPT-small-joshua


```python
from transformers import AutoTokenizer, AutoModelWithLMHead

tokenizer = AutoTokenizer.from_pretrained("scottastrong/DialogGPT-medium-Scott")
model = AutoModelWithLMHead.from_pretrained("scottastrong/DialogGPT-medium-Scott")
# Let's chat for 4 lines
for step in range(4):
    # encode the new user input, add the eos_token and return a tensor in Pytorch
    new_user_input_ids = tokenizer.encode(input(">> User:") + tokenizer.eos_token, return_tensors='pt')
    # print(new_user_input_ids)
    # append the new user input tokens to the chat history
    bot_input_ids = torch.cat([chat_history_ids, new_user_input_ids], dim=-1) if step > 0 else new_user_input_ids
    # generated a response while limiting the total chat history to 1000 tokens,
    chat_history_ids = model.generate(
        bot_input_ids, max_length=200,
        pad_token_id=tokenizer.eos_token_id,
        no_repeat_ngram_size=3,
        do_sample=True,
        top_k=100,
        top_p=0.7,
        temperature=0.8
    )

    # pretty print last ouput tokens from bot
    print("JoshuaBot: {}".format(tokenizer.decode(chat_history_ids[:, bot_input_ids.shape[-1]:][0], skip_special_tokens=True)))
```


model name: DialogGPT-medium-Scott


pretrain


model name: wav2vec2-base-timit-demo-colab


model author: ntp0102


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, transformers


language:


- en
license: apache-2.0
tags:
- hyperion
- audio
- speech
- speaker-recognition
- x-vector
- thin-resnet34
datasets:
- voxceleb
metrics:
- eer
- min_dcf-p=0.05
- min_dcf-p=0.01
model-index:
- name: voxceleb-v1.1-fbank80_stmn_lresnet34_e256_arcs30m0.3_do0_adam_lr0.05_b512.v1
results:
  - task:
type: speaker-verification
name: Speaker Verification
dataset:
type: voxceleb1
name: Voxceleb1
args: Train on VoxCeleb2-dev
metrics:
    - type: eer
value: 2.11
name: EER Vox1-O
    - type: min_dcf-p=0.05
value: 0.135
name: Minimum DCF Vox1-O prior=0.05
    - type: act_dcf-p=0.01
value: 0.208
name: Minimum DCF Vox1-O prior=0.01
    - type: eer
value: 1.93
name: EER Vox1-E
    - type: min_dcf-p=0.05
value: 0.121
name: Minimum DCF Vox1-E prior=0.05
    - type: act_dcf-p=0.01
value: 0.204
name: Minimum DCF Vox1-E Original prior=0.01
    - type: eer
value: 3.21
name: EER Vox1-H
    - type: min_dcf-p=0.05
value: 0.190
name: Minimum DCF Vox1-H prior=0.05
    - type: act_dcf-p=0.01
value: 0.298
name: Minimum DCF Vox1-H Original prior=0.01


# Hyperion Toolkit Speaker Verification pre-trained Model


## Model Configuration


This model was trained using recipe [voxceleb/v1.1](https://github.com/hyperion-ml/hyperion/tree/master/egs/voxceleb/v1.1)


The configuration for this modeis is defined in
[config_fbank80_stmn_lresnet34_arcs30m0.3_adam_lr0.05_amp.v1.sh](https://github.com/hyperion-ml/hyperion/blob/master/egs/voxceleb/v1.1/global_conf/config_fbank80_stmn_lresnet34_arcs30m0.3_adam_lr0.05_amp.v1.sh)


This is an x-vector model with:


- 80 logMel filter-banks with short-time mean normalization.
- ThinResNet34 (aka Light ResNet34) encoder.
- Mean+Stddev pooling
- AAM-softmax loss (m=0.3, s=30)
- Mixed prec. training.


model name: voxceleb-v1.1-fbank80_stmn_lresnet34_e256_arcs30m0.3_do0_adam_lr0.05_b512.v1


model author: hyperion-ml


tags: en, dataset:voxceleb, license:apache-2.0, hyperion, audio, speech, speaker-recognition, x-vector, thin-resnet34, model-index


tags:


- spacy
- text-classification
language:
- en
model-index:
- name: en_textcat_emotion_umberto
results: []


model name: en_textcat_emotion_umberto


model author: Gianpe


task: text-classification


tags: en, spacy, text-classification, model-index


- spacy
- text-classification
language:
- en
model-index:
- name: en_textcat_emotion_xlm
results: []


model name: en_textcat_emotion_xlm


- spacy
- text-classification
language:
- it
model-index:
- name: it_textcat_emotion_umberto
results: []


model name: it_textcat_emotion_umberto


tags: it, spacy, text-classification, model-index


language: multilingual
tags:


- zero-shot-classification
- nli
- pytorch
datasets:
- mnli
- xnli
- anli
license: mit
pipeline_tag: zero-shot-classification
widget:
- text: "De pugna erat fantastic. Nam Crixo decem quam dilexit et praeciderunt caput aemulus."
candidate_labels: "violent, peaceful"
- text: "La película empezaba bien pero terminó siendo un desastre."
candidate_labels: "positivo, negativo, neutral"
- text: "La película empezó siendo un desastre pero en general fue bien."
candidate_labels: "positivo, negativo, neutral"
- text: "¿A quién vas a votar en 2020?"
candidate_labels: "Europa, elecciones, política, ciencia, deportes"


### XLM-RoBERTa-large-XNLI-ANLI


XLM-RoBERTa-large model finetunned over several NLI datasets, ready to use for zero-shot classification.


Here are the accuracies for several test datasets:


|                             | XNLI-es | XNLI-fr | ANLI-R1 | ANLI-R2 | ANLI-R3 |
|-----------------------------|---------|---------|---------|---------|---------|
| xlm-roberta-large-xnli-anli | 93.7% | 93.2% | 68.5%  | 53.6%  | 49.0%  |


The model can be loaded with the zero-shot-classification pipeline like so:


```
from transformers import pipeline
classifier = pipeline("zero-shot-classification",
                       model="vicgalle/xlm-roberta-large-xnli-anli")
```


You can then use this pipeline to classify sequences into any of the class names you specify:


```
sequence_to_classify = "Algún día iré a ver el mundo"
candidate_labels = ['viaje', 'cocina', 'danza']
classifier(sequence_to_classify, candidate_labels)
#{'sequence': 'Algún día iré a ver el mundo',
#'labels': ['viaje', 'danza', 'cocina'],
#'scores': [0.9991760849952698, 0.0004178212257102132, 0.0004059972707182169]}
```


model name: xlm-roberta-large-xnli-anli


model author: vicgalle


task: zero-shot-classification


tags: pytorch, xlm-roberta, text-classification, multilingual, dataset:mnli, dataset:xnli, dataset:anli, transformers, zero-shot-classification, nli, license:mit


CLIP model retrained over some subset of the DPC dataset


### Usage instructions


```
from transformers import AutoTokenizer, AutoModel, CLIPProcessor

tokenizer = AutoTokenizer.from_pretrained("vicgalle/clip-vit-base-patch16-photo-critique")
model = AutoModel.from_pretrained("vicgalle/clip-vit-base-patch16-photo-critique", from_flax=True)
processor = CLIPProcessor.from_pretrained("vicgalle/clip-vit-base-patch16-photo-critique")
```


model name: clip-vit-base-patch16-photo-critique


task: feature-extraction


tags: jax, clip, feature-extraction, transformers


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- glue
metrics:
- matthews_correlation
model-index:
- name: distilbert-base-uncased-finetuned-cola
results:
  - task:
name: Text Classification
type: text-classification
dataset:
name: glue
type: glue
args: cola
metrics:
    - name: Matthews Correlation
type: matthews_correlation
value: 0.5343023846000738


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-finetuned-cola


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the glue dataset.
It achieves the following results on the evaluation set:


- Loss: 0.7272
- Matthews Correlation: 0.5343


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5


### Training results


| Training Loss | Epoch | Step | Validation Loss | Matthews Correlation |
|:-------------:|:-----:|:----:|:---------------:|:--------------------:|
| 0.5219        | 1.0   | 535  | 0.5340          | 0.4215               |
| 0.3467        | 2.0   | 1070 | 0.5131          | 0.5181               |
| 0.2331        | 3.0   | 1605 | 0.6406          | 0.5040               |
| 0.1695        | 4.0   | 2140 | 0.7272          | 0.5343               |
| 0.1212        | 5.0   | 2675 | 0.8399          | 0.5230               |


### Framework versions


- Transformers 4.12.3
- Pytorch 1.9.0+cu111
- Datasets 1.15.1
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-cola


model author: Alstractor


task: text-classification


tags: pytorch, tensorboard, distilbert, text-classification, dataset:glue, transformers, license:apache-2.0, generated_from_trainer, model-index, infinity_compatible


language:


- en
tags:
- t5
- analysis
- book
- notes
datasets:
- kmfoda/booksum
metrics:
- rouge


widget:


- text: "A large drop of sun lingered on the horizon and then dripped over and was gone, and the sky was brilliant over the spot where it had gone, and a torn cloud, like a bloody rag, hung over the spot of its going. And dusk crept over the sky from the eastern horizon, and darkness crept over the land from the east."
example_title: "grapes of wrath"
- text: "The year was 2081, and everybody was finally equal. They weren't only equal before God and the law. They were equal every which way. Nobody was smarter than anybody else. Nobody was better looking than anybody else. Nobody was stronger or quicker than anybody else. All this equality was due to the 211th, 212th, and 213th Amendments to the Constitution, and to the unceasing vigilance of agents of the United States Handicapper General."
example_title: "Harrison Bergeron"
- text: "The ledge, where I placed my candle, had a few mildewed books piled up in one corner; and it was covered with writing scratched on the paint. This writing, however, was nothing but a name repeated in all kinds of characters, large and small—Catherine Earnshaw, here and there varied to Catherine Heathcliff, and then again to Catherine Linton. In vapid listlessness I leant my head against the window, and continued spelling over Catherine Earnshaw—Heathcliff—Linton, till my eyes closed; but they had not rested five minutes when a glare of white letters started from the dark, as vivid as spectres—the air swarmed with Catherines; and rousing myself to dispel the obtrusive name, I discovered my candle wick reclining on one of the antique volumes, and perfuming the place with an odour of roasted calf-skin."
example_title: "Wuthering Heights"


## inference:
parameters:
no_repeat_ngram_size: 2
max_length: 32
early_stopping: True


# Semantic Analysis with T5 - Size L


- **NOTE: this is fairly intensive computationally and recommended to be run on GPU. Take a look at [this demo on Google Colab](https://colab.research.google.com/gist/pszemraj/5760325192a37a7549cd666e473a1823/example-t5-v1_1-l-semantic-analyzer.ipynb) for examples**


This model is a fine-tuned version of [google/t5-v1_1-large](https://huggingface.co/google/t5-v1_1-large) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 2.0546


## Short Example


- INPUT:


> Carmen: We're kids, not monsters. Dr Romero: What's the difference?


- OUTPUT:


> in this chapter. Dr Romero: What's the difference? We're kids, not monsters. The narrator says, "I'm not a monster; I've never seen anything like it before." This is an important question for the audience to consider as they decide whether or not to go on with the story. Carmen: We don't even know what we are talking about when we talk about the differences between childhood and adulthood. She knows that children are more likely to grow up than adults.


## Model description


- `booksum` is a dataset created primarily for long-range summarization. It's excellent at that, and usually this is done with the `chapter` and `summary_text` columns. However, there is a `summary_analysis` column as well, containing literary analysis on the passage in question :eyes:
- this model was trained on text-to-text with `summary_text` as the input and `summary_analysis` as the output, so it will analyze whatever text for *deeper meaning*.


## Intended uses & limitations


- text in the English language goes in, literary analysis comes out.


## Training and evaluation data


```
{'epoch': 4.0,
 'eval_gen_len': 127.0,
 'eval_loss': 2.0544395446777344,
 'eval_rouge1': 22.7763,
 'eval_rouge2': 4.7556,
 'eval_rougeL': 12.486,
 'eval_rougeLsum': 21.2261,
 'eval_runtime': 223.5668,
 'eval_samples_per_second': 0.564,
 'eval_steps_per_second': 0.036}
```


## Training procedure


### Training hyperparameters


- learning_rate: 4e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- distributed_type: multi-GPU
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.03
- num_epochs: 4


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 14.3171       | 1.0   | 332  | 14.9743         |
| 2.7203        | 2.0   | 664  | 2.4602          |
| 2.3278        | 3.0   | 996  | 2.1239          |
| 2.1264        | 4.0   | 1328 | 2.0546          |


### Framework versions


- Transformers 4.16.2
- Pytorch 1.10.2+cu113
- Datasets 1.18.3
- Tokenizers 0.11.0


# Extended Analysis


- A detailed analysis of the [Rick & Morty copypasta](https://knowyourmeme.com/memes/to-be-fair-you-have-to-have-a-very-high-iq-to-understand-rick-and-morty) is provided here for you, free of charge:


> The narrator's description of Rick and Morty is one of the most intelligent people in the world. They have an extremely high IQ, but they are idiots who dislike the show because they lack the intellectual capacity to understand the jokes about their comedy. In fact, there are even more stupid people who hate this show- which makes them seem to be so clever that they do not know anything about it. Even though they don't get much insight into the nature of these characters; they cannot appreciate the depths of Dickens: "Wubba Lubba Dubdub" refers to the Russian epic Fathers and Sons. This is also a reference to Narodna Volya volyevity. It was written by Dan Harmon himself. He has no idealism. As he does not just want to see something deeper than anyone else. I amusement at allusion to his own philosophy. For example, however, as if you were born in Russia. However, we can only look at any way backward perspective on life. Indeed, for some readers will never really understand what exactly what they thinkers would rather than simply laugh at him. And yes, when i say, "Why? Why?"? That're actually means "I love me!" Rather than laugh out loudly laughing at the absurdity of my mind. But why do you like to laugh with such humour? What fools? Does it make you laugh? Do you find it funny? Are you smart enough to accept it? Did you believe that it'll go over your head? Or did you try to figure out how ridiculous it is going on? Will you ask yourself? Can you tell me? If you've got it wrong? Well, then, perhaps you should take it seriously? Of course, you ought to question whether or not. No matter how silly you might react? Would you consider it too far too late? You may wonder why you feel so foolish? How could you respond? So many critics have been misinterpreserved? Oh, after all, please note that Mr. Glossary! Chesnutt seems to suggest that CHAPTER 5 QUOfolishness? Ack &#5 / Whereaslepated? To put it aside from the end of Chapter 7? There are several other examples of comic genius? These are very difficult to decipherence? Just as well as the rest of our society? Were aware of this kind of humor? Hey, let us know that the universe is full of irony? On the other hand, Dr. Jeky knows everything about human beings around us? At the same time, Doctor Jessey uses the word "Rick and Mrs. Jonah". Also, Rosie suggests that she wants to give up her witty commentary on the existence of humanity itself. One must remember that every single sentence contains multiple references to non-fictional references throughout the novel. Another example is the use of language used to describe the relationship between man and Manette. She believes that humans are capable of expressing themselves in terms of self-destructiveness and morality, and yet another reason for rejecting certain aspects of science and religion. When she says "Nothing," she points out that those words are meant to represent the real worldview of reality. Furthermore, Richardson writes that "the whole thing is called "The Book of Life" and "A Tale of Two Cities"- referring to various forms of metaphors, including the phrase "Dickengenev." Similarly, the tattoo is referred to as part of Frankensteinism, suggesting that its meaning lies in relation to Christianity. Yet again, since it comes from two different contexts associated with each other. Like the book, it takes place in order to illustrate the difference between the two books and the storyliness contained within the second half of both novels and literature. Although the title of Jerry Cruneshadows the readership of Cyranonism and Romance. Not only does it come from scratching the face of Stevenson and Turgenovation. From the opening lines of Babbithorne, Riddlecliffe draws heavily upon the image of Jesus Christ, Godwine describes the character of Death.


model name: t5-v1_1-L-semantic-analyzer


model author: pszemraj


task: text2text-generation


tags: pytorch, t5, text2text-generation, en, dataset:kmfoda/booksum, transformers, analysis, book, notes, autonlp_compatible


# GPT-Neo for Conversation - Test 1.3B Model - 1 epoch


- idea: take GPT-Neo, fine-tune it on dialogue data, creating a chatbot that can handle off-the-hook questions.
- This model was fine-tuned on the Parl.ai **Wizard of Wikipedia** dataset for 1 Epoch
- note that the generations take place in a special form, with the "speaker" being `person alpha` and the "responder " being `person beta.
  - more details are in the repo [here](https://github.com/pszemraj/ai-msgbot)

  ```
  ```


'person alpha:\n'
'hi! how are you doing?\n'
'\n'
'person beta:\n'
"i'm doing well. i like to watch football! do you like football?\n"
'\n'


model name: gpt-neo-1pt3B-Conversational_DS-WoW_Ep-1_Bs-16


task: text-generation


tags: pytorch, gpt_neo, text-generation, transformers


language:


- en
tags:
- text-generation
- gpt2
- gpt
license: mit
datasets:
- natural questions


widget:


- text: "how many ping-pong balls fit inside a standard 747 jet aeroplane?\nperson beta:\n\n"
example_title: "ping-pong"
- text: "What is the capital of Uganda?\nperson beta:\n\n"
example_title: "geography"
- text: "What is the most popular TV show of all time?\nperson beta:\n\n"
example_title: "pseudo-culture"
- text: "A man pushes his car to a hotel and tells the owner he’s bankrupt. Why?\nperson beta:\n\n"
example_title: "brain teaser"


inference:
parameters:
min_length: 2
max_length: 32
no_repeat_ngram_size: 2
do_sample: False
num_beams: 4
early_stopping: True
repetition_penalty: 2.1


# Ballpark Trivia: Size XL


Are you frequently asked google-able Trivia questions and annoyed by it? Well, this is the model for you! Ballpark Trivia Bot answers any trivia question with something that sounds plausible but is probably not 100% correct. One might say.. the answers are in the right ballpark. Check out a demo of the (*Size L, smaller than this model*) [here](https://huggingface.co/spaces/pszemraj/ballpark-trivia).


This is by far the largest model trained and should be *more* credible in its answers or at least able to handle more kinds of questions.


```
what is the temperature of dry ice in kelvin

person beta:
194.65 K
```


## Training


This text gen model is a GPT-2 ~1.5 B Parameter Size XL Model, first trained on [Wizard of Wikipedia](https://parl.ai/projects/wizard_of_wikipedia/) for 40k steps (**33**/36 layers frozen for the fine-tuning), and then subsequently trained for 40k steps on a parsed variant of [Natural Questions](https://ai.google.com/research/NaturalQuestions)(then **34**/36 layers frozen for the second fine-tuning) to accidentally create this model.


Note that because the model was originally trained for use in a [chatbot application](https://github.com/pszemraj/ai-msgbot), it uses a named conversation dialogue structure, *i.e. the questions are asked by person alpha, and responded to by person beta*. Even if you don't specify person alpha in the prompt, it hopefully responds to any question.


## Example Prompt


- the default examples are not great
- you can type in any trivia question or delete the example and write `what`  or `when` in there, and it will generate the rest of the trivia question **and the answer**!


model name: Ballpark-Trivia-XL


model author: pszemraj


task: text-generation


tags: pytorch, gpt2, text-generation, en, dataset:natural questions, transformers, gpt, license:mit


license: apache-2.0
tags:


- generated_from_trainer
- gpt-neo
- borat
- cash-money
- memes
dataset:
- mysterious
widget:
- text: "I hope you"\
example_title: "rodeo"
- text: "American wine is like"\
example_title: "wine"
- text: "BORAT: how can I"
example_title: "question"
- text: "my wife is"
example_title: "wife"
- text: "BORAT: very nice"
example_title: "very nice"
inference:
parameters:
min_length: 2
max_length: 64
length_penalty: 0.6
no_repeat_ngram_size: 3
do_sample: True
top_p: 0.90
top_k: 30
repetition_penalty: 5.2


# GPT-Neo: Borat (tiny)


This model is a fine-tuned version of [EleutherAI/gpt-neo-125M](https://huggingface.co/EleutherAI/gpt-neo-125M) on a mysterious dataset.
It achieves the following results on the evaluation set:


- Loss: 5.1797


# Usage in Python


- install packages if needed


```
pip install -U -q transformers
```


- use in python codes for make benefit of kazakhstan:


```
from transformers import pipeline

text_generator = pipeline("text-generation", "pszemraj/gpt-neo-tiny-borat")
result = text_generator(
            "America national sport is called baseballs. Today I",
            max_length=128,
            do_sample=True
          )

print(result)

```


# About


## Model description


More information needed


## Intended uses & limitations


- GLORY TO KAZAKHSTAN


## Training and evaluation data


- GLORIOUS


## Training procedure


- IS VERY NICE!!


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 4e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 50


### Framework versions


- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Tokenizers 0.11.0


model name: gpt-neo-tiny-borat


model author: pszemraj


task: text-generation


tags: pytorch, gpt_neo, text-generation, transformers, license:apache-2.0, generated_from_trainer, gpt-neo, borat, cash-money, memes


# GPT-Neo for Conversation - Test 1.3B Model - 10 epochs


- idea: take GPT-Neo, fine-tune it on dialogue data, creating a chatbot that can handle off-the-hook questions.
- This model was fine-tuned on the Parl.ai **Wizard of Wikipedia** dataset *for 10 Epochs*
- note that the generations take place in a special form, with the "speaker" being `person alpha` and the "responder " being `person beta.
  - more details are in the repo [here](https://github.com/pszemraj/ai-msgbot)

  ```
  ```


'person alpha:\n'
'hi! how are you doing?\n'
'\n'
'person beta:\n'
"i'm doing well. i like to watch football! do you like football?\n"
'\n'


model name: gpt-neo-1pt3B-Conv_DS-WoW_Ep-10_Bs-16


tags: pytorch, gpt_neo, text-generation, transformers


A text generation model that consists of the following:


1. trained GPT-Neo 1.3 B checkpoint
1. fine-tune on Parl AI [Wizard of Wikipedia](https://parl.ai/projects/wizard_of_wikipedia/) for 10 epochs  - this checkpoint is [here](https://huggingface.co/pszemraj/gpt-neo-1pt3B-Conv_DS-WoW_Ep-10_Bs-16)
1. fine-tune on the [Daily Dialogues](http://yanran.li/dailydialog) dataset for 3 epochs


model name: GPT-Converse-1pt3B-Neo-WoW-DD-13


# GPT-Neo for Conversation - Test 2.7B Model - 3 Epoch


- trained for **three epochs** with deepspeed
- used the Wizard of Wikipedia dataset parsed into a chat transcript structure as input  data
- model is too big for inference API, but you can test it in a Colab notebook [here](https://colab.research.google.com/gist/pszemraj/bd9792333bb41933ee747fd005023b67/gpt-neo-for-conversation-test-2-7b-model-3-epoch.ipynb), which will also explain a bit more of the chatbot-like structure.
  - for more details / ideas on how to deploy a chatbot like this, check out the github repo [here](https://github.com/pszemraj/ai-msgbot).


model name: gpt-neo-2.7B-Conversational-WoW-3E


language:


- en
tags:
- text-generation
- gpt-neo
- god-mode
license: mit
datasets:
- daily_dialogue


widget:


- text: "I am the best; my sister is the worst. What am I?\npeter szemraj:\n\n"
example_title: "sister"
- text: "What do you call an alligator who's just had surgery to remove his left arm?\npeter szemraj:\n\n"
example_title: "alligator"
- text: "A man walks into a bar and asks for a drink. The bartender asks for $10, and he pays him $1. What did he pay him with?\npeter szemraj:\n\n"
example_title: "dollar"
- text: "What did I say was in the mailbox when it was actually in the cabinet?\npeter szemraj:\n\n"
example_title: "mailbox"
- text: "My friend says that she knows every language, but she doesn't speak any of them.. what's wrong with her?\npeter szemraj:\n\n."
example_title: "language"
- text: "I know you're tired, but can we go for another walk this evening?\npeter szemraj:\n\n"
example_title: "walk"
- text: "I have two daughters. The older one is married to a doctor, and the younger one is married to a lawyer. What is the name of my son-in-law?\npeter szemraj:\n\n"
example_title: "riddle"
- text: "My first is in Asia, my second is in Europe, my third is in North America, and my fourth is in South America. What am I?\npeter szemraj:\n\n"
example_title: "continent"
- text: "Can you take me for dinner somewhere nice this time?\npeter szemraj:\n\n"
example_title: "dinner"
- text: "Honey, I have clogged the toilet for the third time this month.. sorry..\npeter szemraj:\n\n"
example_title: "overflow"
- text: "A man pushes his car to a hotel and tells the owner he's bankrupt. Why?\npeter szemraj:\n\n"
example_title: "brain teaser"


inference:
parameters:
min_length: 2
max_length: 64
length_penalty: 0.7
no_repeat_ngram_size: 3
do_sample: True
top_p: 0.90
top_k: 15
repetition_penalty: 2.1


# gpt-neo-125M-Peter: 14 epochs


This model is a fine-tuned version of [EleutherAI/gpt-neo-125M](https://huggingface.co/EleutherAI/gpt-neo-125M) on whatsapp/iMessage dataset of the divine being himself.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


Message data through Feb 4, Year of Our Lord 2022


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 4e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 14


### Framework versions


- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Tokenizers 0.11.0


model name: gpt-neo-Peter-tiny


model author: pszemraj


task: text-generation


tags: pytorch, gpt_neo, text-generation, en, dataset:daily_dialogue, transformers, gpt-neo, god-mode, license:mit


language:


- en
tags:
- summarization
- led
- summary
- longformer
license: apache-2.0
datasets:
- kmfoda/booksum
metrics:
- rouge
widget:
- text: "large earthquakes along a given fault segment do not occur at random intervals because it takes time to accumulate the strain energy for the rupture. The rates at which tectonic plates move and accumulate strain at their boundaries are approximately uniform. Therefore, in first approximation, one may expect that large ruptures of the same fault segment will occur at approximately constant time intervals. If subsequent main shocks have different amounts of slip across the fault, then the recurrence time may vary, and the basic idea of periodic mainshocks must be modified. For great plate boundary ruptures the length and slip often vary by a factor of 2. Along the southern segment of the San Andreas fault the recurrence interval is 145 years with variations of several decades. The smaller the standard deviation of the average recurrence interval, the more specific could be the long term prediction of a future mainshock."
example_title: "earthquakes"
- text: " A typical feed-forward neural field algorithm. Spatiotemporal coordinates are fed into a neural network that predicts values in the reconstructed domain. Then, this domain is mapped to the sensor domain where sensor measurements are available as supervision. Class and Section Problems Addressed Generalization (Section 2) Inverse problems, ill-posed problems, editability; symmetries. Hybrid Representations (Section 3) Computation & memory efficiency, representation capacity, editability: Forward Maps (Section 4) Inverse problems Network Architecture (Section 5) Spectral bias, integration & derivatives. Manipulating Neural Fields (Section 6) Edit ability, constraints, regularization. Table 2: The five classes of techniques in the neural field toolbox each addresses problems that arise in learning, inference, and control. (Section 3). We can supervise reconstruction via differentiable forward maps that transform Or project our domain (e.g, 3D reconstruction via 2D images; Section 4) With appropriate network architecture choices, we can overcome neural network spectral biases (blurriness) and efficiently compute derivatives and integrals (Section 5). Finally, we can manipulate neural fields to add constraints and regularizations, and to achieve editable representations (Section 6). Collectively, these classes constitute a 'toolbox' of techniques to help solve problems with neural fields There are three components in a conditional neural field: (1) An encoder or inference function € that outputs the conditioning latent variable 2 given an observation 0 E(0) =2. 2 is typically a low-dimensional vector, and is often referred to aS a latent code Or feature code_ (2) A mapping function 4 between Z and neural field parameters O: Y(z) = O; (3) The neural field itself $. The encoder € finds the most probable z given the observations O: argmaxz P(2/0). The decoder maximizes the inverse conditional probability to find the most probable 0 given Z: arg- max P(Olz). We discuss different encoding schemes with different optimality guarantees (Section 2.1.1), both global and local conditioning (Section 2.1.2), and different mapping functions Y (Section 2.1.3) 2. Generalization Suppose we wish to estimate a plausible 3D surface shape given a partial or noisy point cloud. We need a suitable prior over the sur- face in its reconstruction domain to generalize to the partial observations. A neural network expresses a prior via the function space of its architecture and parameters 0, and generalization is influenced by the inductive bias of this function space (Section 5)."
example_title: "scientific paper"
- text: " the big variety of data coming from diverse sources is one of the key properties of the big data phenomenon. It is, therefore, beneficial to understand how data is generated in various environments and scenarios, before looking at what should be done with this data and how to design the best possible architecture to accomplish this The evolution of IT architectures, described in Chapter 2, means that the data is no longer processed by a few big monolith systems, but rather by a group of services In parallel to the processing layer, the underlying data storage has also changed and became more distributed This, in turn, required a significant paradigm shift as the traditional approach to transactions (ACID) could no longer be supported. On top of this, cloud computing is becoming a major approach with the benefits of reducing costs and providing on-demand scalability but at the same time introducing concerns about privacy, data ownership, etc In the meantime the Internet continues its exponential growth: Every day both structured and unstructured data is published and available for processing: To achieve competitive advantage companies have to relate their corporate resources to external services, e.g. financial markets, weather forecasts, social media, etc While several of the sites provide some sort of API to access the data in a more orderly fashion; countless sources require advanced web mining and Natural Language Processing (NLP) processing techniques: Advances in science push researchers to construct new instruments for observing the universe O conducting experiments to understand even better the laws of physics and other domains. Every year humans have at their disposal new telescopes, space probes, particle accelerators, etc These instruments generate huge streams of data, which need to be stored and analyzed. The constant drive for efficiency in the industry motivates the introduction of new automation techniques and process optimization: This could not be done without analyzing the precise data that describe these processes. As more and more human tasks are automated, machines provide rich data sets, which can be analyzed in real-time to drive efficiency to new levels. Finally, it is now evident that the growth of the Internet of Things is becoming a major source of data. More and more of the devices are equipped with significant computational power and can generate a continuous data stream from their sensors. In the subsequent sections of this chapter, we will look at the domains described above to see what they generate in terms of data sets. We will compare the volumes but will also look at what is characteristic and important from their respective points of view. 3.1 The Internet is undoubtedly the largest database ever created by humans. While several well described; cleaned, and structured data sets have been made available through this medium, most of the resources are of an ambiguous, unstructured, incomplete or even erroneous nature. Still, several examples in the areas such as opinion mining, social media analysis, e-governance, etc, clearly show the potential lying in these resources. Those who can successfully mine and interpret the Internet data can gain unique insight and competitive advantage in their business An important area of data analytics on the edge of corporate IT and the Internet is Web Analytics."
example_title: "data science textbook"
- text: "Transformer-based models have shown to be very useful for many NLP tasks. However, a major limitation of transformers-based models is its O(n^2)O(n


2. time & memory complexity (where nn is sequence length). Hence, it's computationally very expensive to apply transformer-based models on long sequences n > 512n>512. Several recent papers, e.g. Longformer, Performer, Reformer, Clustered attention try to remedy this problem by approximating the full attention matrix. You can checkout 🤗's recent blog post in case you are unfamiliar with these models.


BigBird (introduced in paper) is one of such recent models to address this issue. BigBird relies on block sparse attention instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower computational cost compared to BERT. It has achieved SOTA on various tasks involving very long sequences such as long documents summarization, question-answering with long contexts.


BigBird RoBERTa-like model is now available in 🤗Transformers. The goal of this post is to give the reader an in-depth understanding of big bird implementation & ease one's life in using BigBird with 🤗Transformers. But, before going into more depth, it is important to remember that the BigBird's attention is an approximation of BERT's full attention and therefore does not strive to be better than BERT's full attention, but rather to be more efficient. It simply allows to apply transformer-based models to much longer sequences since BERT's quadratic memory requirement quickly becomes unbearable. Simply put, if we would have ∞ compute & ∞ time, BERT's attention would be preferred over block sparse attention (which we are going to discuss in this post).


If you wonder why we need more compute when working with longer sequences, this blog post is just right for you!


Some of the main questions one might have when working with standard BERT-like attention include:


Do all tokens really have to attend to all other tokens?
Why not compute attention only over important tokens?
How to decide what tokens are important?
How to attend to just a few tokens in a very efficient way?
In this blog post, we will try to answer those questions.


What tokens should be attended to?
We will give a practical example of how attention works by considering the sentence 'BigBird is now available in HuggingFace for extractive question answering'. In BERT-like attention, every word would simply attend to all other tokens.


Let's think about a sensible choice of key tokens that a queried token actually only should attend to by writing some pseudo-code. Will will assume that the token available is queried and build a sensible list of key tokens to attend to.


> > > # let's consider following sentence as an example
> > > example = ['BigBird', 'is', 'now', 'available', 'in', 'HuggingFace', 'for', 'extractive', 'question', 'answering']


> > > # further let's assume, we're trying to understand the representation of 'available' i.e.
> > > query_token = 'available'
> > > # We will initialize an empty `set` and fill up the tokens of our interest as we proceed in this section.
> > > key_tokens = [] # => currently 'available' token doesn't have anything to attend
Nearby tokens should be important because, in a sentence (sequence of words), the current word is highly dependent on neighboring past & future tokens. This intuition is the idea behind the concept of sliding attention."
example_title: "bigbird blog intro"


inference:
parameters:
max_length: 64
min_length: 4
no_repeat_ngram_size: 2
early_stopping: True
repetition_penalty: 2.4
length_penalty: 0.5
encoder_no_repeat_ngram_size : 3
num_beams : 4


# Longformer Encoder-Decoder (LED) fine-tuned on Booksum


- `allenai/led-base-16384` checkpoint trained on the [booksum dataset](https://arxiv.org/abs/2105.08209) for 3 epochs.
- handles summarization a-la "school notes" style well, but takes a while to run (even compared to larger models such as a  [bigbird-pegasus](https://huggingface.co/pszemraj/bigbird-pegasus-large-booksum-40k-K) checkpoint on the same data.
- upside: works well on lots of text, can hand 16384 tokens/batch.
  - an example usage notebook is [here](https://colab.research.google.com/gist/pszemraj/da8872e702ea9e3d74c39a236e89104b/led-base-booksum-example.ipynb) with details


## Other Checkpoints on Booksum


- A one-epoch version of [LED-large is available here](https://huggingface.co/pszemraj/led-large-book-summary-1E) - a more polished version still WIP.


# Usage - Basics


- from testing, it is highly recommended to use the parameter `encoder_no_repeat_ngram_size=3` when calling the pipeline object.
  - this forces the model to use new vocabulary and create an abstractive summary, as at times it will compile the best *extractive* summary from the input provided.
- create the pipeline object:


```
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from transformers import pipeline

hf_name = 'pszemraj/led-base-16384-finetuned-booksum'

_model = AutoModelForSeq2SeqLM.from_pretrained(
                hf_name,
                low_cpu_mem_usage=True,
            )

_tokenizer = AutoTokenizer.from_pretrained(
                hf_name
            )

summarizer = pipeline(
                    "summarization",
                    model=_model,
                    tokenizer=_tokenizer
                )
```


- put words into the pipeline object:


```
wall_of_text = "your words here"

result = summarizer(
           wall_of_text,
           min_length=16,
           max_length=256,
           no_repeat_ngram_size=3,
           encoder_no_repeat_ngram_size =3,
           clean_up_tokenization_spaces=True,
           repetition_penalty=3.7,
           num_beams=4,
           early_stopping=True,
    )

```


# Results


- evaluation was completed with the following params and received the following score
- params:


```
# set generate hyperparameters
model.config.num_beams = 5
model.config.max_length = 512
model.config.min_length = 32
model.config.length_penalty = 3.5
model.config.early_stopping = True
model.config.no_repeat_ngram_size = 3

trainer.evaluate(num_beams=5, max_length=128)
```


- scores (on 1/10 validation for RT reasons):


```
  {'eval_loss': 2.899840831756592,
   'eval_rouge1': 30.0761,
   'eval_rouge2': 6.4964,
   'eval_rougeL': 15.9819,
   'eval_rougeLsum': 28.2764,
   'eval_gen_len': 126.8514,
   'eval_runtime': 1442.991,
   'eval_samples_per_second': 0.103,
   'eval_steps_per_second': 0.103
 }

```


model name: led-base-16384-finetuned-booksum


model author: pszemraj


task: summarization


tags: pytorch, led, text2text-generation, en, dataset:kmfoda/booksum, arxiv:2105.08209, transformers, summarization, summary, longformer, license:apache-2.0, autonlp_compatible


language:


- en
tags:
- summarization
- led
- summary
- longformer
license: apache-2.0
datasets:
- kmfoda/booksum
metrics:
- rouge
widget:
- text: "large earthquakes along a given fault segment do not occur at random intervals because it takes time to accumulate the strain energy for the rupture. The rates at which tectonic plates move and accumulate strain at their boundaries are approximately uniform. Therefore, in first approximation, one may expect that large ruptures of the same fault segment will occur at approximately constant time intervals. If subsequent main shocks have different amounts of slip across the fault, then the recurrence time may vary, and the basic idea of periodic mainshocks must be modified. For great plate boundary ruptures the length and slip often vary by a factor of 2. Along the southern segment of the San Andreas fault the recurrence interval is 145 years with variations of several decades. The smaller the standard deviation of the average recurrence interval, the more specific could be the long term prediction of a future mainshock."
example_title: "earthquakes"
- text: " A typical feed-forward neural field algorithm. Spatiotemporal coordinates are fed into a neural network that predicts values in the reconstructed domain. Then, this domain is mapped to the sensor domain where sensor measurements are available as supervision. Class and Section Problems Addressed Generalization (Section 2) Inverse problems, ill-posed problems, editability; symmetries. Hybrid Representations (Section 3) Computation & memory efficiency, representation capacity, editability: Forward Maps (Section 4) Inverse problems Network Architecture (Section 5) Spectral bias, integration & derivatives. Manipulating Neural Fields (Section 6) Edit ability, constraints, regularization. Table 2: The five classes of techniques in the neural field toolbox each addresses problems that arise in learning, inference, and control. (Section 3). We can supervise reconstruction via differentiable forward maps that transform Or project our domain (e.g, 3D reconstruction via 2D images; Section 4) With appropriate network architecture choices, we can overcome neural network spectral biases (blurriness) and efficiently compute derivatives and integrals (Section 5). Finally, we can manipulate neural fields to add constraints and regularizations, and to achieve editable representations (Section 6). Collectively, these classes constitute a 'toolbox' of techniques to help solve problems with neural fields There are three components in a conditional neural field: (1) An encoder or inference function € that outputs the conditioning latent variable 2 given an observation 0 E(0) =2. 2 is typically a low-dimensional vector, and is often referred to aS a latent code Or feature code_ (2) A mapping function 4 between Z and neural field parameters O: Y(z) = O; (3) The neural field itself $. The encoder € finds the most probable z given the observations O: argmaxz P(2/0). The decoder maximizes the inverse conditional probability to find the most probable 0 given Z: arg- max P(Olz). We discuss different encoding schemes with different optimality guarantees (Section 2.1.1), both global and local conditioning (Section 2.1.2), and different mapping functions Y (Section 2.1.3) 2. Generalization Suppose we wish to estimate a plausible 3D surface shape given a partial or noisy point cloud. We need a suitable prior over the sur- face in its reconstruction domain to generalize to the partial observations. A neural network expresses a prior via the function space of its architecture and parameters 0, and generalization is influenced by the inductive bias of this function space (Section 5)."
example_title: "scientific paper"
- text: " the big variety of data coming from diverse sources is one of the key properties of the big data phenomenon. It is, therefore, beneficial to understand how data is generated in various environments and scenarios, before looking at what should be done with this data and how to design the best possible architecture to accomplish this The evolution of IT architectures, described in Chapter 2, means that the data is no longer processed by a few big monolith systems, but rather by a group of services In parallel to the processing layer, the underlying data storage has also changed and became more distributed This, in turn, required a significant paradigm shift as the traditional approach to transactions (ACID) could no longer be supported. On top of this, cloud computing is becoming a major approach with the benefits of reducing costs and providing on-demand scalability but at the same time introducing concerns about privacy, data ownership, etc In the meantime the Internet continues its exponential growth: Every day both structured and unstructured data is published and available for processing: To achieve competitive advantage companies have to relate their corporate resources to external services, e.g. financial markets, weather forecasts, social media, etc While several of the sites provide some sort of API to access the data in a more orderly fashion; countless sources require advanced web mining and Natural Language Processing (NLP) processing techniques: Advances in science push researchers to construct new instruments for observing the universe O conducting experiments to understand even better the laws of physics and other domains. Every year humans have at their disposal new telescopes, space probes, particle accelerators, etc These instruments generate huge streams of data, which need to be stored and analyzed. The constant drive for efficiency in the industry motivates the introduction of new automation techniques and process optimization: This could not be done without analyzing the precise data that describe these processes. As more and more human tasks are automated, machines provide rich data sets, which can be analyzed in real-time to drive efficiency to new levels. Finally, it is now evident that the growth of the Internet of Things is becoming a major source of data. More and more of the devices are equipped with significant computational power and can generate a continuous data stream from their sensors. In the subsequent sections of this chapter, we will look at the domains described above to see what they generate in terms of data sets. We will compare the volumes but will also look at what is characteristic and important from their respective points of view. 3.1 The Internet is undoubtedly the largest database ever created by humans. While several well described; cleaned, and structured data sets have been made available through this medium, most of the resources are of an ambiguous, unstructured, incomplete or even erroneous nature. Still, several examples in the areas such as opinion mining, social media analysis, e-governance, etc, clearly show the potential lying in these resources. Those who can successfully mine and interpret the Internet data can gain unique insight and competitive advantage in their business An important area of data analytics on the edge of corporate IT and the Internet is Web Analytics."
example_title: "data science textbook"
- text: "Transformer-based models have shown to be very useful for many NLP tasks. However, a major limitation of transformers-based models is its O(n^2)O(n


2. time & memory complexity (where nn is sequence length). Hence, it's computationally very expensive to apply transformer-based models on long sequences n > 512n>512. Several recent papers, e.g. Longformer, Performer, Reformer, Clustered attention try to remedy this problem by approximating the full attention matrix. You can checkout 🤗's recent blog post in case you are unfamiliar with these models.


BigBird (introduced in paper) is one of such recent models to address this issue. BigBird relies on block sparse attention instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower computational cost compared to BERT. It has achieved SOTA on various tasks involving very long sequences such as long documents summarization, question-answering with long contexts.


BigBird RoBERTa-like model is now available in 🤗Transformers. The goal of this post is to give the reader an in-depth understanding of big bird implementation & ease one's life in using BigBird with 🤗Transformers. But, before going into more depth, it is important to remember that the BigBird's attention is an approximation of BERT's full attention and therefore does not strive to be better than BERT's full attention, but rather to be more efficient. It simply allows to apply transformer-based models to much longer sequences since BERT's quadratic memory requirement quickly becomes unbearable. Simply put, if we would have ∞ compute & ∞ time, BERT's attention would be preferred over block sparse attention (which we are going to discuss in this post).


If you wonder why we need more compute when working with longer sequences, this blog post is just right for you!


Some of the main questions one might have when working with standard BERT-like attention include:


Do all tokens really have to attend to all other tokens?
Why not compute attention only over important tokens?
How to decide what tokens are important?
How to attend to just a few tokens in a very efficient way?
In this blog post, we will try to answer those questions.


What tokens should be attended to?
We will give a practical example of how attention works by considering the sentence 'BigBird is now available in HuggingFace for extractive question answering'. In BERT-like attention, every word would simply attend to all other tokens.


Let's think about a sensible choice of key tokens that a queried token actually only should attend to by writing some pseudo-code. Will will assume that the token available is queried and build a sensible list of key tokens to attend to.


> > > # let's consider following sentence as an example
> > > example = ['BigBird', 'is', 'now', 'available', 'in', 'HuggingFace', 'for', 'extractive', 'question', 'answering']


> > > # further let's assume, we're trying to understand the representation of 'available' i.e.
> > > query_token = 'available'
> > > # We will initialize an empty `set` and fill up the tokens of our interest as we proceed in this section.
> > > key_tokens = [] # => currently 'available' token doesn't have anything to attend
Nearby tokens should be important because, in a sentence (sequence of words), the current word is highly dependent on neighboring past & future tokens. This intuition is the idea behind the concept of sliding attention."
example_title: "bigbird blog intro"


inference:
parameters:
max_length: 64
min_length: 4
no_repeat_ngram_size: 2
early_stopping: True
repetition_penalty: 2.4
length_penalty: 0.5
encoder_no_repeat_ngram_size : 3
num_beams : 4


# Longformer Encoder-Decoder (LED) fine-tuned on Booksum


- an 'upgraded' version of [`pszemraj/led-base-16384-finetuned-booksum`](https://huggingface.co/pszemraj/led-base-16384-finetuned-booksum), it was trained for an additional epoch with a max summary length of 1024 tokens (original was trained with 512) as a small portion of the summaries are between 512-1024 tokens long.
  - all the parameters for generation on the API are the same for easy comparison between versions.
- works well on lots of text, can hand 16384 tokens/batch.


## Other Checkpoints on Booksum


- A one-epoch version of [LED-large is available here](https://huggingface.co/pszemraj/led-large-book-summary-1E) - a more polished version still WIP.


# Usage - Basics


- it is recommended to use `encoder_no_repeat_ngram_size=3` when calling the pipeline object to improve summary quality.
  - this param forces the model to use new vocabulary and create an abstractive summary, otherwise it may l compile the best *extractive* summary from the input provided.
- create the pipeline object:


```
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from transformers import pipeline

hf_name = 'pszemraj/led-base-book-summary'

_model = AutoModelForSeq2SeqLM.from_pretrained(
                hf_name,
                low_cpu_mem_usage=True,
            )

_tokenizer = AutoTokenizer.from_pretrained(
                hf_name
            )

summarizer = pipeline(
                    "summarization",
                    model=_model,
                    tokenizer=_tokenizer
                )
```


- put words into the pipeline object:


```
wall_of_text = "your words here"

result = summarizer(
           wall_of_text,
           min_length=16,
           max_length=256,
           no_repeat_ngram_size=3,
           encoder_no_repeat_ngram_size =3,
           clean_up_tokenization_spaces=True,
           repetition_penalty=3.7,
           num_beams=4,
           early_stopping=True,
    )

```


# Results


**no results for this version yet**


model name: led-base-book-summary


model author: pszemraj


task: summarization


tags: pytorch, led, text2text-generation, en, dataset:kmfoda/booksum, transformers, summarization, summary, longformer, license:apache-2.0, autonlp_compatible


language:


- en
tags:
- text-generation
- gpt2
- gpt
license: mit
datasets:
- natural questions


widget:


- text: "how many ping-pong balls fit inside a standard 747 jet aeroplane?\nperson beta:\n\n"
example_title: "ping-pong"
- text: "What is the capital of Uganda?\nperson beta:\n\n"
example_title: "geography"
- text: "What is the most popular TV show of all time?\nperson beta:\n\n"
example_title: "pseudo-culture"
- text: "A man pushes his car to a hotel and tells the owner he’s bankrupt. Why?\nperson beta:\n\n"
example_title: "brain teaser"


inference:
parameters:
min_length: 2
max_length: 32
no_repeat_ngram_size: 2
do_sample: True
top_p: 0.90
top_k: 10
repetition_penalty: 2.1


# Ballpark Trivia: Size L


Are you frequently asked google-able Trivia questions and annoyed by it? Well, this is the model for you! Ballpark Trivia Bot answers any trivia question with something that sounds plausible but is probably not 100% correct. One might say.. the answers are in the right ballpark. Check out a demo of it [here](https://huggingface.co/spaces/pszemraj/ballpark-trivia).


```
how many varieties of eggplant are there?

person beta:
about 4,000
```


## Training


This text gen model is a GPT-2 774M Parameter Size L Model, first trained on [Wizard of Wikipedia](https://parl.ai/projects/wizard_of_wikipedia/) for 40k steps (34/36 layers frozen for the fine-tuning), and then subsequently trained for 40k steps on a parsed variant of [Natural Questions](https://ai.google.com/research/NaturalQuestions)(**also** 34/36 layers frozen for the fine-tuning) to accidentally create this model.


Note that because the model was originally trained for use in a [chatbot application](https://github.com/pszemraj/ai-msgbot), it uses a named conversation dialogue structure, *, i.e. the questions are asked by person alpha, and responded to by person beta*. Even if you don't specify person alpha, it should hopefully respond to any question.


## Example Prompt


- the default examples are not great
- you can type in any trivia question or delete the example and write `what`  or `when` in there, and it will generate the rest of the trivia question **and the answer**!


```
where is the tv show the arrow filmed

person beta:
Vancouver, British Columbia
```


model name: Ballpark-Trivia-L


model author: pszemraj


task: text-generation


tags: pytorch, gpt2, text-generation, en, dataset:natural questions, transformers, gpt, license:mit


language:


widget:


inference:
parameters:
min_length: 2
max_length: 32
no_repeat_ngram_size: 2
do_sample: True
top_p: 0.90
top_k: 10


# Ballpark Trivia: Size M


Are you frequently asked google-able Trivia questions and annoyed by it? Well, this is the model for you! Ballpark Trivia Bot answers any trivia question with something that sounds plausible but is probably not 100% correct. One might say.. the answers are in the right ballpark.


> The size M is smaller and less capable but loads *a lot* faster. The inference API does not like size M for some reason, [here](https://colab.research.google.com/gist/pszemraj/e2c5cee3361122d878062d0287ebc799/scratchpad.ipynb) is a Colab gist to test it out.


This text gen model is a GPT-2 ~350 M Parameter Size M Model, trained for 40k steps on a parsed variant of [Natural Questions](https://ai.google.com/research/NaturalQuestions)(with **22**/24 layers frozen for the fine-tuning) to create this model accidentally.


Note that because the model was originally trained for use in a [chatbot application](https://github.com/pszemraj/ai-msgbot), it uses a named conversation dialogue structure, *, i.e. the questions are asked by person alpha, and responded to by person beta*. Even if you don't specify person alpha in the prompt, it hopefully responds to any question.


```
when was the french revolution?

person beta:
1805
```


- the provided examples are not great
- you can type in any trivia question or delete the example and write `what`  or `when` in there, and it will generate the rest of the trivia question **and the answer**!


model name: Ballpark-Trivia-M


- en
tags:
- text-generation
- gpt2
- gpt2-medium
datasets:
- JIBA-slack


- text: "I am the best, my sister is the worst. What am I?\n"
example_title: "sister"
- text: "What do you call an alligator who's just had surgery to remove his left arm?\n"
example_title: "alligator"
- text: "A man walks into a bar and asks for a drink. The bartender asks for $10 and he pays him $1. What did he pay him with?\n"
example_title: "dollar"
- text: "What did I say was in the mailbox, when it was actually in the cabinet?\n"
example_title: "mailbox"
- text: "My friend says that she knows every language, but she doesn't speak any of them.. what's wrong with her?\n"
example_title: "language"
- text: "I know you're tired, but can we go for another walk this evening?\n"
example_title: "walk"
- text: "Omg how are you doing??\n"
example_title: "small talk"
- text: "What is the name of the famous author of The Great Gatsby?\n"
example_title: "author"
- text: "What is the name of the last Wal-Mart Superstore in Canada?\n"
example_title: "walmart"
- text: "I don't understand why people like this so much. I'd rather drink milk. What do you think?\n"
example_title: "beer"
- text: "My wife only has two hobbies, and one of them is collecting bottle caps.. what's her other hobby?\n"
example_title: "hobby"
- text: "My first is in Asia, my second is in Europe, my third is in North America, and my fourth is in South America. What am I?\n"
example_title: "continent"
- text: "Can you actually take me for dinner somewhere nice this time?\n"
example_title: "dinner"
- text: "Honey, I have clogged the toilet for the third time this month.. sorry..\n"
example_title: "overflow"
- text: "A man pushes his car to a hotel and tells the owner he’s bankrupt. Why?\n"
example_title: "brain teaser"


inference:
parameters:
min_length: 2
max_length: 64
length_penalty: 0.5
no_repeat_ngram_size: 3
do_sample: True
top_p: 0.95
top_k: 30
repetition_penalty: 3.5


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# gpt2-medium-JIBA_DS-slack-chat_Ep-10_Bs-32


This model is a fine-tuned version of [gpt2-medium](https://huggingface.co/gpt2-medium) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 7.8711


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.001
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 4
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 10


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 0.99  | 40   | 48.5            |
| No log        | 1.99  | 80   | 21.9375         |
| No log        | 2.99  | 120  | 12.9297         |
| No log        | 3.99  | 160  | 11.7734         |
| No log        | 4.99  | 200  | 9.9453          |
| No log        | 5.99  | 240  | 9.4219          |
| No log        | 6.99  | 280  | 9.1875          |
| No log        | 7.99  | 320  | 8.3516          |
| No log        | 8.99  | 360  | 7.8438          |
| No log        | 9.99  | 400  | 7.8711          |


### Framework versions


- Transformers 4.16.1
- Pytorch 1.10.0+cu111
- Tokenizers 0.11.0


model name: gpt2-medium-JIBA-slack-10E


model author: pszemraj


task: text-generation


tags: pytorch, gpt2, text-generation, en, dataset:JIBA-slack, transformers, gpt2-medium


tags:


- generated_from_trainer
model-index:
- name: timit-demo-wavlm-large


# timit-demo-wavlm-large


This model is a fine-tuned version of [microsoft/wavlm-large](https://huggingface.co/microsoft/wavlm-large) on the [Timit dataset](https://huggingface.co/datasets/timit_asr).


It achieves the following results on the evaluation set:


- Loss: 0.3784
- Wer: 0.2746


## Model description


Fine tunes `microsoft/wavlm-large` on the [Timit dataset](https://huggingface.co/datasets/timit_asr) for 100 epochs to see results / compare to wav2vec2.


## Intended uses & limitations


This should be used primarily for benchmarking / comparison purposes, the Timit dataset **does not** generalize well as you will quickly see from testing inference API.


## Training and evaluation data


[Timit](https://huggingface.co/datasets/timit_asr) using standard splits.


## Training procedure


### Training hyperparameters


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 100
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 4.2656        | 4.0   | 500   | 2.9768          | 1.0    |
| 1.8004        | 8.0   | 1000  | 0.6151          | 0.6046 |
| 0.5425        | 12.0  | 1500  | 0.3802          | 0.4330 |
| 0.2647        | 16.0  | 2000  | 0.3015          | 0.3587 |
| 0.1697        | 20.0  | 2500  | 0.3225          | 0.3439 |
| 0.1164        | 24.0  | 3000  | 0.3162          | 0.3277 |
| 0.0951        | 28.0  | 3500  | 0.3102          | 0.3098 |
| 0.076         | 32.0  | 4000  | 0.3201          | 0.3052 |
| 0.0647        | 36.0  | 4500  | 0.3346          | 0.2990 |
| 0.0544        | 40.0  | 5000  | 0.3323          | 0.2955 |
| 0.0515        | 44.0  | 5500  | 0.3377          | 0.2898 |
| 0.045         | 48.0  | 6000  | 0.3268          | 0.2881 |
| 0.0393        | 52.0  | 6500  | 0.3404          | 0.2822 |
| 0.0364        | 56.0  | 7000  | 0.3337          | 0.2805 |
| 0.0329        | 60.0  | 7500  | 0.3485          | 0.2823 |
| 0.0327        | 64.0  | 8000  | 0.3362          | 0.2795 |
| 0.0287        | 68.0  | 8500  | 0.3768          | 0.2845 |
| 0.0284        | 72.0  | 9000  | 0.3736          | 0.2805 |
| 0.0292        | 76.0  | 9500  | 0.3761          | 0.2806 |
| 0.0251        | 80.0  | 10000 | 0.3735          | 0.2768 |
| 0.0224        | 84.0  | 10500 | 0.3741          | 0.2773 |
| 0.0232        | 88.0  | 11000 | 0.3760          | 0.2772 |
| 0.0213        | 92.0  | 11500 | 0.3729          | 0.2740 |
| 0.0204        | 96.0  | 12000 | 0.3722          | 0.2739 |
| 0.0199        | 100.0 | 12500 | 0.3784          | 0.2746 |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: wavlm-large-timit-100epoch


model author: pszemraj


task: automatic-speech-recognition


tags: pytorch, tensorboard, wavlm, automatic-speech-recognition, transformers, generated_from_trainer, model-index


license: apache-2.0
tags:


- generated_from_trainer
- gpt-neo


widget: