```python
from multitask_model import BertForSequenceClassification
from transformers import AutoTokenizer
import torch
model = BertForSequenceClassification.from_pretrained(
        "shahrukhx01/bert-multitask-query-classifiers",
        task_labels_map={"quora_keyword_pairs": 2, "spaadia_squad_pairs": 2},
    )
tokenizer = AutoTokenizer.from_pretrained("shahrukhx01/bert-multitask-query-classifiers")

## Keyword vs Statement/Question Classifier
input = ["keyword query", "is this a keyword query?"]
task_name="quora_keyword_pairs"
sequence = tokenizer(input, padding=True, return_tensors="pt")['input_ids']
logits = model(sequence, task_name=task_name)[0]
predictions = torch.argmax(torch.softmax(logits, dim=1).detach().cpu(), axis=1)
for input, prediction in zip(input, predictions):
  print(f"task: {task_name}, input: {input} \n prediction=> {prediction}")
  print()

## Statement vs Question Classifier
input = ["where is berlin?", "is this a keyword query?", "Berlin is in Germany."]
task_name="spaadia_squad_pairs"
sequence = tokenizer(input, padding=True, return_tensors="pt")['input_ids']
logits = model(sequence, task_name=task_name)[0]
predictions = torch.argmax(torch.softmax(logits, dim=1).detach().cpu(), axis=1)
for input, prediction in zip(input, predictions):
  print(f"task: {task_name}, input: {input} \n prediction=> {prediction}")
  print()
```


model name: bert-multitask-query-classifiers


model author: shahrukhx01


task: text-classification


tags: pytorch, bert, text-classification, transformers, infinity_compatible


tags:


- fuzzy-matching
- fuzzy-search
- entity-resolution
- record-linking
- structured-data-search


A Siamese BERT architecture trained at character levels tokens for embedding based Fuzzy matching.


## Usage (Sentence-Transformers)


Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:


```
pip install -U sentence-transformers
```


Then you can use the model like this:


```python
from sentence_transformers import SentenceTransformer, util
word1 = "fuzzformer"
word1 = " ".join([char for char in word1]) ## divide the word to char level to fuzzy match
word2 = "fizzformer"
word2 = " ".join([char for char in word2]) ## divide the word to char level to fuzzy match
words = [word1, word2]

model = SentenceTransformer('shahrukhx01/paraphrase-mpnet-base-v2-fuzzy-matcher')
fuzzy_embeddings = model.encode(words)

print("Fuzzy Match score:")
print(util.cos_sim(fuzzy_embeddings[0], fuzzy_embeddings[1]))
```


## Usage (HuggingFace Transformers)


```python
import torch
from transformers import AutoTokenizer, AutoModel
from torch import Tensor, device

def cos_sim(a: Tensor, b: Tensor):
    """
    borrowed from sentence transformers repo
    Computes the cosine similarity cos_sim(a[i], b[j]) for all i and j.
    :return: Matrix with res[i][j]  = cos_sim(a[i], b[j])
    """
    if not isinstance(a, torch.Tensor):
        a = torch.tensor(a)

    if not isinstance(b, torch.Tensor):
        b = torch.tensor(b)

    if len(a.shape) == 1:
        a = a.unsqueeze(0)

    if len(b.shape) == 1:
        b = b.unsqueeze(0)

    a_norm = torch.nn.functional.normalize(a, p=2, dim=1)
    b_norm = torch.nn.functional.normalize(b, p=2, dim=1)
    return torch.mm(a_norm, b_norm.transpose(0, 1))

#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

# Words we want fuzzy embeddings for
word1 = "fuzzformer"
word1 = " ".join([char for char in word1]) ## divide the word to char level to fuzzy match
word2 = "fizzformer"
word2 = " ".join([char for char in word2]) ## divide the word to char level to fuzzy match
words = [word1, word2]
# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('shahrukhx01/paraphrase-mpnet-base-v2-fuzzy-matcher')
model = AutoModel.from_pretrained('shahrukhx01/paraphrase-mpnet-base-v2-fuzzy-matcher')

# Tokenize sentences
encoded_input = tokenizer(words, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, max pooling.
fuzzy_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Fuzzy Match score:")
print(cos_sim(fuzzy_embeddings[0], fuzzy_embeddings[1]))
```


## ACKNOWLEDGEMENT


A big thank you to [Sentence Transformers](https://github.com/UKPLab/sentence-transformers) as their implementation really expedited the implementation of Fuzzformer.


## Citation


To cite FuzzTransformer in your work, please use the following bibtex reference:
@misc{shahrukhkhan2021fuzzTransformer,

author       = {Shahrukh Khan},

title        = {FuzzTransformer: A character level embedding based Siamese transformer for fuzzy string matching.},

year         = 2021,

publisher    = {Coming soon},

doi          = {Coming soon},

url          = {Coming soon}

}


model name: paraphrase-mpnet-base-v2-fuzzy-matcher


model author: shahrukhx01


task: feature-extraction


tags: pytorch, mpnet, feature-extraction, transformers, fuzzy-matching, fuzzy-search, entity-resolution, record-linking, structured-data-search


tags:


- wikisql
- text2sql


```python
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig
model = BartForConditionalGeneration.from_pretrained('shahrukhx01/schema-aware-distilbart-cnn-12-6-text2sql')
tokenizer = BartTokenizer.from_pretrained('shahrukhx01/schema-aware-distilbart-cnn-12-6-text2sql')
## add NL query with table schema
question = "What is terrence ross' nationality? </s> <col0> Player : text <col1> No. : text <col2> Nationality : text <col3> Position : text <col4> Years in Toronto : text <col5>  School/Club Team : text"

inputs = tokenizer([question], max_length=1024, return_tensors='pt')

# Generate SQL
text_query_ids = model.generate(inputs['input_ids'], num_beams=4, min_length=0, max_length=125, early_stopping=True)
prediction = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in text_query_ids][0]
print(prediction)
```


model name: schema-aware-distilbart-cnn-12-6-text2sql


task: text2text-generation


tags: pytorch, bart, text2text-generation, transformers, wikisql, text2sql, autonlp_compatible


language: "en"
tags:


- neural-search-query-classification
- neural-search
widget:
- text: "what did you eat in lunch?"


# KEYWORD STATEMENT VS QUESTION CLASSIFIER FOR NEURAL SEARCH


```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("shahrukhx01/question-vs-statement-classifier")

model = AutoModelForSequenceClassification.from_pretrained("shahrukhx01/question-vs-statement-classifier")
```


Trained to add the feature for classifying queries between Question Query vs Statement Query using classification in [Haystack](https://github.com/deepset-ai/haystack/issues/611)


model name: question-vs-statement-classifier


task: text-classification


tags: pytorch, bert, text-classification, en, transformers, neural-search-query-classification, neural-search, infinity_compatible


## Multiple Prediction Heads


* ExtractiveQA Head
* Three Class Classification Head, classes => (yes, no, extra_qa) to answer binary questions or direct to ExtractiveQA Head


## BoolQ Validation dataset Evaluation: <br/>


support => 3270

accuracy => 0.73

macro f1 => 0.71


## SQuAD Validation dataset Evaluation: <br/>


eval_HasAns_exact      = 78.0196

eval_HasAns_f1         = 84.0327

eval_HasAns_total      =    5928

eval_NoAns_exact       = 81.8167

eval_NoAns_f1          = 81.8167

eval_NoAns_total       =    5945

eval_best_exact        = 79.9208

eval_best_f1           = 82.9231

eval_exact             = 79.9208

eval_f1                = 82.9231

eval_samples           =   12165

eval_total             =   11873


## Uasge in transformers


Import the script from [here](https://huggingface.co/shahrukhx01/roberta-base-squad2-boolq-baseline/blob/main/multitask_model.py)


```python
from multitask_model import RobertaForMultitaskQA
from transformers import RobertaTokenizerFast
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = RobertaForMultitaskQA.from_pretrained(
        "shahrukhx01/roberta-base-squad2-boolq-baseline",
        task_labels_map={"squad_v2": 2, "boolq": 3},
    ).to(device)
tokenizer = RobertaTokenizerFast.from_pretrained("shahrukhx01/roberta-base-squad2-boolq-baseline")
```


model name: roberta-base-squad2-boolq-baseline


model author: shahrukhx01


task: question-answering


tags: pytorch, roberta, question-answering, transformers, autonlp_compatible, infinity_compatible


```python
from transformers import (
    AutoModelForSequenceClassification,
    AutoTokenizer,
)

model = AutoModelForSequenceClassification.from_pretrained("shahrukhx01/roberta-base-boolq")
model.to(device)
#model.push_to_hub("roberta-base-boolq")

tokenizer = AutoTokenizer.from_pretrained("shahrukhx01/roberta-base-boolq")

def predict(question, passage):
  sequence = tokenizer.encode_plus(question, passage, return_tensors="pt")['input_ids'].to(device)

  logits = model(sequence)[0]
  probabilities = torch.softmax(logits, dim=1).detach().cpu().tolist()[0]
  proba_yes = round(probabilities[1], 2)
  proba_no = round(probabilities[0], 2)

  print(f"Question: {question}, Yes: {proba_yes}, No: {proba_no}")

passage = """Berlin is the capital and largest city of Germany by both area and population. Its 3.8 million inhabitants make it the European Union's most populous city,
                        according to population within city limits."""

question = "Is Berlin the smallest city of Germany?"
predict(s_question, passage)
```


model name: roberta-base-boolq


task: text-classification


tags: pytorch, roberta, text-classification, transformers, infinity_compatible


language: "en"
tags:


- neural-search-query-classification
- neural-search
widget:
- text: "keyword query."


# KEYWORD QUERY VS STATEMENT/QUESTION CLASSIFIER FOR NEURAL SEARCH


| Train Loss    | Validation Acc.| Test Acc.|
| ------------- |:-------------: | -----:   |
| 0.000806      | 0.99  | 0.997    |


```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("shahrukhx01/bert-mini-finetune-question-detection")

model = AutoModelForSequenceClassification.from_pretrained("shahrukhx01/bert-mini-finetune-question-detection")
```


Trained to add feature for classifying queries between Keyword Query or Question + Statement Query using classification in [Haystack](https://github.com/deepset-ai/haystack/issues/611)


Problem Statement:
One common challenge that we saw in deployments: We need to distinguish between real questions and keyword queries that come in. We only want to route questions to the Reader branch in order to maximize the accuracy of results and minimize computation efforts/costs.


Baseline:
https://www.kaggle.com/shahrukhkhan/question-v-statement-detection


Dataset:
https://www.kaggle.com/stefanondisponibile/quora-question-keyword-pairs


Kaggle Notebook:
https://www.kaggle.com/shahrukhkhan/question-vs-statement-classification-mini-bert/


model name: bert-mini-finetune-question-detection


model author: shahrukhx01


tags: pytorch, bert, text-classification, en, transformers, neural-search-query-classification, neural-search, infinity_compatible


- schema-aware-text2sql
- text2sql
- wikisql
widget:
- text: "What is terrence ross' nationality? </s> <col0> Player : text <col1> No. : text <col2> Nationality : text <col3> Position : text <col4> Years in Toronto : text <col5>  School/Club Team : text"


```python
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig
model = BartForConditionalGeneration.from_pretrained('shahrukhx01/schema-aware-denoising-bart-large-cnn-text2sql')
tokenizer = BartTokenizer.from_pretrained('shahrukhx01/schema-aware-denoising-bart-large-cnn-text2sql')
## add NL query with table schema
question = "What is terrence ross' nationality? </s> <col0> Player : text <col1> No. : text <col2> Nationality : text <col3> Position : text <col4> Years in Toronto : text <col5>  School/Club Team : text"

inputs = tokenizer([question], max_length=1024, return_tensors='pt')

# Generate SQL
text_query_ids = model.generate(inputs['input_ids'], num_beams=4, min_length=0, max_length=125, early_stopping=True)
prediction = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in text_query_ids][0]
print(prediction)
```


model name: schema-aware-denoising-bart-large-cnn-text2sql


task: text2text-generation


tags: pytorch, bart, text2text-generation, en, transformers, schema-aware-text2sql, text2sql, wikisql, autonlp_compatible


The distilbart-cnn-12-6-text2sql is fine-tuned on WIKISQL dataset.


```python
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig

model = BartForConditionalGeneration.from_pretrained('shahrukhx01/distilbart-cnn-12-6-text2sql')
tokenizer = BartTokenizer.from_pretrained('shahrukhx01/distilbart-cnn-12-6-text2sql')

TEXT_QUERY = "what is the temperature of berlin "
inputs = tokenizer([TEXT_QUERY], max_length=1024, return_tensors='pt')

# Generate SQL
text_query_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=5, early_stopping=True)
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in text_query_ids])
```


model name: distilbart-cnn-12-6-text2sql


tags: pytorch, bart, text2text-generation, transformers, autonlp_compatible


language: de
license: mit
thumbnail: "https://raw.githubusercontent.com/German-NLP-Group/german-transformer-training/master/model_cards/german-electra-logo.png"
tags:


- electra
- commoncrawl
- uncased
- umlaute
- umlauts
- german
- deutsch


# German Electra Uncased


<img width="300px" src="https://raw.githubusercontent.com/German-NLP-Group/german-transformer-training/master/model_cards/german-electra-logo.png">
[¹]


## Version 2 Release


We released an improved version of this model. Version 1 was trained for 766,000 steps. For this new version we continued the training for an additional 734,000 steps. It therefore follows that version 2 was trained on a total of 1,500,000 steps. See "Evaluation of Version 2: GermEval18 Coarse" below for details.


## Model Info


This Model is suitable for training on many downstream tasks in German (Q&A, Sentiment Analysis, etc.).


It can be used as a drop-in replacement for **BERT** in most down-stream tasks (**ELECTRA** is even implemented as an extended **BERT** Class).


At the time of release (August 2020) this model is the best performing publicly available German NLP model on various German evaluation metrics (CONLL03-DE, GermEval18 Coarse, GermEval18 Fine). For GermEval18 Coarse results see below. More will be published soon.


## Installation


This model has the special feature that it is **uncased** but does **not strip accents**.
This possibility was added by us with [PR #6280](https://github.com/huggingface/transformers/pull/6280).
To use it you have to use Transformers version 3.1.0 or newer.


```bash
pip install transformers -U
```


## Uncase and Umlauts ('Ö', 'Ä', 'Ü')


This model is uncased. This helps especially for domains where colloquial terms with uncorrect capitalization is often used.


The special characters 'ö', 'ü', 'ä' are included through the `strip_accent=False` option, as this leads to an improved precision.


## Creators


This model was trained and open sourced in conjunction with the [**German NLP Group**](https://github.com/German-NLP-Group) in equal parts by:


- [**Philip May**](https://May.la) - [T-Systems on site services GmbH](https://www.t-systems-onsite.de/)
- [**Philipp Reißel**](https://www.reissel.eu) - [ambeRoad](https://amberoad.de/)


## Evaluation of Version 2: GermEval18 Coarse


We evaluated all language models on GermEval18 with the F1 macro score. For each model we did an extensive automated hyperparameter search. With the best hyperparmeters we did fit the moodel multiple times on GermEval18. This is done to cancel random effects and get results of statistical relevance.


![GermEval18 Coarse Model Evaluation for Version 2](https://raw.githubusercontent.com/German-NLP-Group/german-transformer-training/master/model_cards/model-eval-v2.png)


## Checkpoint evaluation


Since it it not guaranteed that the last checkpoint is the best, we evaluated the checkpoints on GermEval18. We found that the last checkpoint is indeed the best. The training was stable and did not overfit the text corpus.


## Pre-training details


### Data


- Cleaned Common Crawl Corpus 2019-09 German: [CC_net](https://github.com/facebookresearch/cc_net) (Only head coprus and filtered for language_score > 0.98) - 62 GB
- German Wikipedia Article Pages Dump (20200701) - 5.5 GB
- German Wikipedia Talk Pages Dump (20200620) - 1.1 GB
- Subtitles - 823 MB
- News 2018 - 4.1 GB


The sentences were split with [SojaMo](https://github.com/tsproisl/SoMaJo). We took the German Wikipedia Article Pages Dump 3x to oversample. This approach was also used in a similar way in GPT-3 (Table 2.2).


More Details can be found here [Preperaing Datasets for German Electra Github](https://github.com/German-NLP-Group/german-transformer-training)


### Electra Branch no_strip_accents


Because we do not want to stip accents in our training data we made a change to Electra and used this repo [Electra no_strip_accents](https://github.com/PhilipMay/electra/tree/no_strip_accents) (branch `no_strip_accents`). Then created the tf dataset with:


```bash
python build_pretraining_dataset.py --corpus-dir <corpus_dir> --vocab-file <dir>/vocab.txt --output-dir ./tf_data --max-seq-length 512 --num-processes 8 --do-lower-case --no-strip-accents
```


### The training


The training itself can be performed with the Original Electra Repo (No special case for this needed).
We run it with the following Config:


The exact Training Config

debug False

disallow_correct False

disc_weight 50.0

do_eval False

do_lower_case True

do_train True

electra_objective True

embedding_size 768

eval_batch_size 128

gcp_project None

gen_weight 1.0

generator_hidden_size 0.33333

generator_layers 1.0

iterations_per_loop 200

keep_checkpoint_max 0

learning_rate 0.0002

lr_decay_power 1.0

mask_prob 0.15

max_predictions_per_seq 79

max_seq_length 512

model_dir gs://XXX

model_hparam_overrides {}

model_name 02_Electra_Checkpoints_32k_766k_Combined

model_size base

num_eval_steps 100

num_tpu_cores 8

num_train_steps 766000

num_warmup_steps 10000

pretrain_tfrecords gs://XXX

results_pkl gs://XXX

results_txt gs://XXX

save_checkpoints_steps 5000

temperature 1.0

tpu_job_name None

tpu_name electrav5

tpu_zone None

train_batch_size 256

uniform_generator False

untied_generator True

untied_generator_embeddings False

use_tpu True

vocab_file gs://XXX

vocab_size 32767

weight_decay_rate 0.01


![Training Loss](https://raw.githubusercontent.com/German-NLP-Group/german-transformer-training/master/model_cards/loss.png)


Please Note: *Due to the GAN like strucutre of Electra the loss is not that meaningful*


It took about 7 Days on a preemtible TPU V3-8. In total, the Model went through approximately 10 Epochs. For an automatically recreation of a cancelled TPUs we used [tpunicorn](https://github.com/shawwn/tpunicorn). The total cost of training summed up to about 450 $ for one run. The Data-pre processing and Vocab Creation needed approximately 500-1000 CPU hours. Servers were fully provided by [T-Systems on site services GmbH](https://www.t-systems-onsite.de/), [ambeRoad](https://amberoad.de/).
Special thanks to [Stefan Schweter](https://github.com/stefan-it) for your feedback and providing parts of the text corpus.


[¹]: Source for the picture [Pinterest](https://www.pinterest.cl/pin/371828512984142193/)


### Negative Results


We tried the following approaches which we found had no positive influence:


- **Increased Vocab Size**: Leads to more parameters and thus reduced examples/sec while no visible Performance gains were measured
- **Decreased Batch-Size**: The original Electra was trained with a Batch Size per TPU Core of 16 whereas this Model was trained with 32 BS / TPU Core. We found out that 32 BS leads to better results when you compare metrics over computation time


## License - The MIT License


Copyright 2020-2021 Philip May<br>
Copyright 2020-2021 Philipp Reissel


Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:


The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.


THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


model name: electra-base-german-uncased


model author: german-nlp-group


tags: pytorch, electra, pretraining, de, transformers, license:mit, commoncrawl, uncased, umlaute, umlauts, german, deutsch, infinity_compatible


# TEST


# huggingface model


model name: dummy


model author: blackface


tags:


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: distilbert-base-multilingual-cased-finetuned-squad
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-multilingual-cased-finetuned-squad


This model is a fine-tuned version of [distilbert-base-multilingual-cased](https://huggingface.co/distilbert-base-multilingual-cased) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 0.6587


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.923         | 1.0   | 579  | 0.8439          |
| 0.8479        | 2.0   | 1158 | 0.6784          |
| 0.6148        | 3.0   | 1737 | 0.6587          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: distilbert-base-multilingual-cased-finetuned-squad


model author: Khanh


task: question-answering


tags: pytorch, tensorboard, distilbert, question-answering, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: distilbert-base-multilingual-cased-finetuned-viquad
results: []


# distilbert-base-multilingual-cased-finetuned-viquad


- Loss: 3.4241


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 65   | 4.0975          |
| No log        | 2.0   | 130  | 3.9315          |
| No log        | 3.0   | 195  | 3.6742          |
| No log        | 4.0   | 260  | 3.4878          |
| No log        | 5.0   | 325  | 3.4241          |


### Framework versions


model name: distilbert-base-multilingual-cased-finetuned-viquad


model author: Khanh


task: question-answering


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: bert-base-multilingual-cased-finetuned-viquad
results: []


# bert-base-multilingual-cased-finetuned-viquad


This model is a fine-tuned version of [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 1.9815


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 65   | 2.5534          |
| No log        | 2.0   | 130  | 2.1165          |
| No log        | 3.0   | 195  | 1.9815          |


### Framework versions


model name: bert-base-multilingual-cased-finetuned-viquad


model author: Khanh


task: question-answering


tags: pytorch, tensorboard, bert, question-answering, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: mit
tags:


- generated_from_trainer
model-index:
- name: xlm-roberta-base-finetuned-viquad
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# xlm-roberta-base-finetuned-viquad


This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 2.3761


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 259  | 2.9945          |
| 3.3665        | 2.0   | 518  | 2.3761          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: xlm-roberta-base-finetuned-viquad


model author: Khanh


task: question-answering


tags: pytorch, tensorboard, xlm-roberta, question-answering, transformers, license:mit, generated_from_trainer, model-index, autonlp_compatible


license: mit
tags:


- generated_from_trainer
model-index:
- name: xlm-roberta-base-finetuned-squad
results: []


# xlm-roberta-base-finetuned-squad


- Loss: 0.5539


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.7665        | 1.0   | 2295 | 0.5231          |
| 0.5236        | 2.0   | 4590 | 0.5539          |


### Framework versions


model name: xlm-roberta-base-finetuned-squad


model author: Khanh


task: question-answering


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: bert-base-multilingual-cased-finetuned-squad
results: []


# bert-base-multilingual-cased-finetuned-squad


This model is a fine-tuned version of [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4919


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.1782        | 1.0   | 579  | 0.5258          |
| 0.4938        | 2.0   | 1158 | 0.4639          |
| 0.32          | 3.0   | 1737 | 0.4919          |


### Framework versions


model name: bert-base-multilingual-cased-finetuned-squad


model author: Khanh


task: question-answering


tags: pytorch, tensorboard, bert, question-answering, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: distilbert-base-uncased-finetuned-squad
results:
  - task:
name: Question Answering
type: question-answering


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-finetuned-squad


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on an unknown dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


### Framework versions


- Transformers 4.10.2
- Pytorch 1.9.0+cu102
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-squad


model author: tarikul


task: question-answering


tags: pytorch, tensorboard, distilbert, question-answering, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


language: ti
widget:


- text: "ድምጻዊ ኣብርሃም ኣፈወርቂ ንዘልኣለም ህያው ኮይኑ ኣብ ልብና ይነብር"
datasets:
- TLMD
- NTC
metrics:
- f1
- precision
- recall
- accuracy
model-index:
- name: tiroberta-base-pos
results:
  - task:
name: Token Classification
type: token-classification
metrics:
    - name: F1
type: f1
value: 0.9562
    - name: Precision
type: precision
value: 0.9562
    - name: Recall
type: recall
value: 0.9562
    - name: Accuracy
type: accuracy
value: 0.9562


# Tigrinya POS tagging with TiRoBERTa


This model is a fine-tuned version of [TiRoBERTa](https://huggingface.co/fgaim/tiroberta) on the NTC-v1 dataset (Tedla et al. 2016).


## Training


### Hyperparameters


- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10.0


### Results


The model achieves the following results on the test set:


- Loss: 0.3194
- Adj Precision: 0.9219
- Adj Recall: 0.9335
- Adj F1: 0.9277
- Adj Number: 1670
- Adv Precision: 0.8297
- Adv Recall: 0.8554
- Adv F1: 0.8423
- Adv Number: 484
- Con Precision: 0.9844
- Con Recall: 0.9763
- Con F1: 0.9804
- Con Number: 972
- Fw Precision: 0.7895
- Fw Recall: 0.5357
- Fw F1: 0.6383
- Fw Number: 28
- Int Precision: 0.6552
- Int Recall: 0.7308
- Int F1: 0.6909
- Int Number: 26
- N Precision: 0.9650
- N Recall: 0.9662
- N F1: 0.9656
- N Number: 3992
- Num Precision: 0.9747
- Num Recall: 0.9665
- Num F1: 0.9706
- Num Number: 239
- N Prp Precision: 0.9308
- N Prp Recall: 0.9447
- N Prp F1: 0.9377
- N Prp Number: 470
- N V Precision: 0.9854
- N V Recall: 0.9736
- N V F1: 0.9794
- N V Number: 416
- Pre Precision: 0.9722
- Pre Recall: 0.9625
- Pre F1: 0.9673
- Pre Number: 907
- Pro Precision: 0.9448
- Pro Recall: 0.9236
- Pro F1: 0.9341
- Pro Number: 445
- Pun Precision: 1.0
- Pun Recall: 0.9994
- Pun F1: 0.9997
- Pun Number: 1607
- Unc Precision: 1.0
- Unc Recall: 0.875
- Unc F1: 0.9333
- Unc Number: 16
- V Precision: 0.8780
- V Recall: 0.9231
- V F1: 0.9
- V Number: 78
- V Aux Precision: 0.9685
- V Aux Recall: 0.9878
- V Aux F1: 0.9780
- V Aux Number: 654
- V Ger Precision: 0.9388
- V Ger Recall: 0.9571
- V Ger F1: 0.9479
- V Ger Number: 513
- V Imf Precision: 0.9634
- V Imf Recall: 0.9497
- V Imf F1: 0.9565
- V Imf Number: 914
- V Imv Precision: 0.8793
- V Imv Recall: 0.7286
- V Imv F1: 0.7969
- V Imv Number: 70
- V Prf Precision: 0.8960
- V Prf Recall: 0.9082
- V Prf F1: 0.9020
- V Prf Number: 294
- V Rel Precision: 0.9678
- V Rel Recall: 0.9538
- V Rel F1: 0.9607
- V Rel Number: 757
- Overall Precision: 0.9562
- Overall Recall: 0.9562
- Overall F1: 0.9562
- Overall Accuracy: 0.9562


- Transformers 4.12.0.dev0
- Pytorch 1.9.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


## Citation


If you use this model in your product or research, please cite as follows:


```
@article{Fitsum2021TiPLMs,
  author={Fitsum Gaim and Wonsuk Yang and Jong C. Park},
  title={Monolingual Pre-trained Language Models for Tigrinya},
  year=2021,
  publisher={WiNLP 2021/EMNLP 2021}
}
```


## References


```
Tedla, Y., Yamamoto, K. & Marasinghe, A. 2016.
Tigrinya Part-of-Speech Tagging with Morphological Patterns and the New Nagaoka Tigrinya Corpus.
International Journal Of Computer Applications 146 pp. 33-41 (2016).
```


model name: tiroberta-pos


model author: fgaim


task: token-classification


tags: pytorch, roberta, token-classification, ti, dataset:TLMD, dataset:NTC, transformers, model-index, autonlp_compatible, infinity_compatible


language:


- en
datasets:
- c4
- squad
tags:
- text2text-generation
widget:
- text: "question: What is the atomic number for oxygen? context: Oxygen is a chemical element with symbol O and atomic number 8."
- text: "question: What is the chemical symbol of Oxygen? context: Oxygen is a chemical element with symbol O and atomic number 8."


## license: apache-2.0


## T5-small for QA


[Google's T5-small](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) pre-trained on the [C4](https://huggingface.co/datasets/c4) dataset, fine-tuned for Question-Answering on [SQuAD v2](https://huggingface.co/datasets/squad_v2) with the following hyperparameters:


```
  optimizer=adamw_hf
  learning_rate=3e-5
  adam_beta1=0.9
  adam_beta2=0.999
  adam_epsilon=1e-08
  num_train_epochs=2
  per_device_train_batch_size=12
```


## Usage


The input [context and question] has to be prepared in a specific way as follows:


```python
from transformers import pipeline

def prep_input(_context, _question):
    return " ".join(["question:", _question.strip(), "context:", _context.strip()])

t5qa = pipeline("text2text-generation", "fgaim/t5-small-squad-v2")

context = """
Oxygen is a chemical element with symbol O and atomic number 8. It is a member of the chalcogen group on the periodic table and is a highly reactive nonmetal and oxidizing agent that readily forms compounds (notably oxides) with most elements. By mass, oxygen is the third-most abundant element in the universe, after hydrogen and helium. At standard temperature and pressure, two atoms of the element bind to form dioxygen, a colorless and odorless diatomic gas with the formula O.
"""

t5qa(prep_input(context, "How many atoms combine to form dioxygen?"))
# [{'generated_text': 'two'}]

t5qa(prep_input(context, "What element makes up almost half of the earth's crust by mass?"))
# [{'generated_text': 'oxygen'}]

t5qa(prep_input(context, "What are the most abundent elements of the universe by mass?"))
# [{'generated_text': 'hydrogen and helium'}]
```


model name: t5-small-squad-v2


task: text2text-generation


tags: pytorch, t5, text2text-generation, en, dataset:c4, dataset:squad, transformers, license:apache-2.0, autonlp_compatible


language: ti
widget:


- text: "ዓቕሚ መንእሰይ ኤርትራ [MASK] ተራእዩ"


# Pre-trained ELECTRA small for Tigrinya Language


We pre-train ELECTRA small on the [TLMD](https://zenodo.org/record/5139094) dataset, with over 40 million tokens.


Contained are trained Flax and PyTorch models.


## Hyperparameters


The hyperparameters corresponding to model sizes mentioned above are as follows:


| Model Size | L  | AH | HS  | FFN  | P    | Seq  |
|------------|----|----|-----|------|------|------|
| SMALL      | 12 | 4  | 256 | 1024 | 14M  | 512  |


(L = number of layers; AH = number of attention heads; HS = hidden size; FFN = feedforward network dimension; P = number of parameters; Seq = maximum sequence length.)


model name: tielectra-small


model author: fgaim


task: fill-mask


tags: pytorch, jax, electra, fill-mask, ti, transformers, autonlp_compatible, infinity_compatible


- text: "ድምጻዊ ኣብርሃም ኣፈወርቂ ንዘልኣለም ህያው ኮይኑ ኣብ ልብና ይነብር"
metrics:
- f1
- precision
- recall
- accuracy
model-index:
- name: tielectra-small-sentiment
results:
  - task:
name: Text Classification
type: text-classification
metrics:
    - name: F1
type: f1
value: 0.8228962818003914
    - name: Precision
type: precision
value: 0.8055555555555556
    - name: Recall
type: recall
value: 0.841
    - name: Accuracy
type: accuracy
value: 0.819


# Sentiment Analysis for Tigrinya with TiELECTRA small


This model is a fine-tuned version of [TiELECTRA small](https://huggingface.co/fgaim/tielectra-small) on a YouTube comments Sentiment Analysis dataset for Tigrinya (Tela et al. 2020).


## Basic usage


```python
from transformers import pipeline

ti_sent = pipeline("sentiment-analysis", model="fgaim/tielectra-small-sentiment")
ti_sent("ድምጻዊ ኣብርሃም ኣፈወርቂ ንዘልኣለም ህያው ኮይኑ ኣብ ልብና ይነብር")
```


## Training


### Hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0


### Results


The model achieves the following results on the evaluation set:


- F1: 0.8229
- Precision: 0.8056
- Recall: 0.841
- Accuracy: 0.819
- Loss: 0.4299


### Framework versions


- Transformers 4.10.3
- Pytorch 1.9.0+cu111
- Datasets 1.10.2
- Tokenizers 0.10.1


## Citation


If you use this model in your product or research, please cite as follows:


```
@article{Fitsum2021TiPLMs,
  author={Fitsum Gaim and Wonsuk Yang and Jong C. Park},
  title={Monolingual Pre-trained Language Models for Tigrinya},
  year=2021,
  publisher= {WiNLP 2021/EMNLP 2021}
}
```


## References


```
Tela, A., Woubie, A. and Hautamäki, V. 2020.
Transferring Monolingual Model to Low-Resource Language: The Case of Tigrinya.
ArXiv, abs/2006.07698.
```


model name: tielectra-small-sentiment


model author: fgaim


task: text-classification


tags: pytorch, electra, text-classification, ti, transformers, model-index, infinity_compatible


language: ti
widget:


- text: "ድምጻዊ ኣብርሃም ኣፈወርቂ ንዘልኣለም ህያው ኮይኑ ኣብ ልብና ይነብር"
datasets:
- TLMD
metrics:
- accuracy
- f1
- precision
- recall
model-index:
- name: tiroberta-sentiment
results:
  - task:
name: Text Classification
type: text-classification
metrics:
    - name: Accuracy
type: accuracy
value: 0.828
    - name: F1
type: f1
value: 0.8476527900797165
    - name: Precision
type: precision
value: 0.760731319554849
    - name: Recall
type: recall
value: 0.957


# Sentiment Analysis for Tigrinya with TiRoBERTa


This model is a fine-tuned version of [TiRoBERTa](https://huggingface.co/fgaim/roberta-base-tigrinya) on a YouTube comments Sentiment Analysis dataset for Tigrinya (Tela et al. 2020).


## Basic usage


```python
from transformers import pipeline

ti_sent = pipeline("sentiment-analysis", model="fgaim/tiroberta-sentiment")
ti_sent("ድምጻዊ ኣብርሃም ኣፈወርቂ ንዘልኣለም ህያው ኮይኑ ኣብ ልብና ይነብር")
```


## Training


### Hyperparameters


### Results


It achieves the following results on the evaluation set:


- F1: 0.8477
- Precision: 0.7607
- Recall: 0.957
- Accuracy: 0.828
- Loss: 0.6796


### Framework versions


## Citation


```
@article{Fitsum2021TiPLMs,
  author={Fitsum Gaim and Wonsuk Yang and Jong C. Park},
  title={Monolingual Pre-trained Language Models for Tigrinya},
  year=2021,
  publisher={WiNLP 2021/EMNLP 2021}
}
```


## References


model name: tiroberta-sentiment


model author: fgaim


task: text-classification


tags: pytorch, roberta, text-classification, ti, dataset:TLMD, transformers, model-index, infinity_compatible


language: ti
widget:


- text: "ዓቕሚ መንእሰይ ኤርትራ <mask> ተራእዩ"


# RoBERTa Pretrained for Tigrinya Language


We pretrain a RoBERTa base model for Tigrinya on a dataset of 40 million tokens trained for 40 epochs.


Contained in this repo is the original pretrained Flax model that was trained on a TPU v3.8 and it's corresponding PyTorch version.


## Hyperparameters


The hyperparameters corresponding to model sizes mentioned above are as follows:


| Model Size | L  | AH | HS  | FFN  | P    | Seq  |
|------------|----|----|-----|------|------|------|
| BASE       | 12 | 12 | 768 | 3072 | 125M | 512  |


(L = number of layers; AH = number of attention heads; HS = hidden size; FFN = feedforward network dimension; P = number of parameters; Seq = maximum sequence length.)


model name: tiroberta-base


task: fill-mask


tags: pytorch, jax, roberta, fill-mask, ti, transformers, autonlp_compatible, infinity_compatible


- text: "ድምጻዊ ኣብርሃም ኣፈወርቂ ንዘልኣለም ህያው ኮይኑ ኣብ ልብና ይነብር"
datasets:
- TLMD
- NTC
metrics:
- f1
- precision
- recall
- accuracy
model-index:
- name: tielectra-small-pos
results:
  - task:
name: Token Classification
type: token-classification
metrics:
    - name: F1
type: f1
value: 0.9456
    - name: Precision
type: precision
value: 0.9456
    - name: Recall
type: recall
value: 0.9456
    - name: Accuracy
type: accuracy
value: 0.9456


# Tigrinya POS tagging with TiELECTRA


This model is a fine-tuned version of [TiELECTRA](https://huggingface.co/fgaim/tielectra-small) on the NTC-v1 dataset (Tedla et al. 2016).


## Basic usage


```python
from transformers import pipeline

ti_pos = pipeline("token-classification", model="fgaim/tielectra-small-pos")
ti_pos("ድምጻዊ ኣብርሃም ኣፈወርቂ ንዘልኣለም ህያው ኮይኑ ኣብ ልብና ይነብር")
```


## Training


### Hyperparameters


The following hyperparameters were used during training:


- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10.0


### Results


The model achieves the following results on the test set:


- Loss: 0.2236
- Adj Precision: 0.9148
- Adj Recall: 0.9192
- Adj F1: 0.9170
- Adj Number: 1670
- Adv Precision: 0.8228
- Adv Recall: 0.8058
- Adv F1: 0.8142
- Adv Number: 484
- Con Precision: 0.9793
- Con Recall: 0.9743
- Con F1: 0.9768
- Con Number: 972
- Fw Precision: 0.5
- Fw Recall: 0.3214
- Fw F1: 0.3913
- Fw Number: 28
- Int Precision: 0.64
- Int Recall: 0.6154
- Int F1: 0.6275
- Int Number: 26
- N Precision: 0.9525
- N Recall: 0.9587
- N F1: 0.9556
- N Number: 3992
- Num Precision: 0.9825
- Num Recall: 0.9372
- Num F1: 0.9593
- Num Number: 239
- N Prp Precision: 0.9132
- N Prp Recall: 0.9404
- N Prp F1: 0.9266
- N Prp Number: 470
- N V Precision: 0.9667
- N V Recall: 0.9760
- N V F1: 0.9713
- N V Number: 416
- Pre Precision: 0.9645
- Pre Recall: 0.9592
- Pre F1: 0.9619
- Pre Number: 907
- Pro Precision: 0.9395
- Pro Recall: 0.9079
- Pro F1: 0.9234
- Pro Number: 445
- Pun Precision: 1.0
- Pun Recall: 0.9994
- Pun F1: 0.9997
- Pun Number: 1607
- Unc Precision: 0.9286
- Unc Recall: 0.8125
- Unc F1: 0.8667
- Unc Number: 16
- V Precision: 0.7609
- V Recall: 0.8974
- V F1: 0.8235
- V Number: 78
- V Aux Precision: 0.9581
- V Aux Recall: 0.9786
- V Aux F1: 0.9682
- V Aux Number: 654
- V Ger Precision: 0.9183
- V Ger Recall: 0.9415
- V Ger F1: 0.9297
- V Ger Number: 513
- V Imf Precision: 0.9473
- V Imf Recall: 0.9442
- V Imf F1: 0.9458
- V Imf Number: 914
- V Imv Precision: 0.8163
- V Imv Recall: 0.5714
- V Imv F1: 0.6723
- V Imv Number: 70
- V Prf Precision: 0.8927
- V Prf Recall: 0.8776
- V Prf F1: 0.8851
- V Prf Number: 294
- V Rel Precision: 0.9535
- V Rel Recall: 0.9485
- V Rel F1: 0.9510
- V Rel Number: 757
- Overall Precision: 0.9456
- Overall Recall: 0.9456
- Overall F1: 0.9456
- Overall Accuracy: 0.9456


### Framework versions


- Transformers 4.10.3
- Pytorch 1.9.0+cu111
- Datasets 1.10.2
- Tokenizers 0.10.1


## Citation


If you use this model in your product or research, please cite as follows:


```
@article{Fitsum2021TiPLMs,
  author= {Fitsum Gaim and Wonsuk Yang and Jong C. Park},
  title= {Monolingual Pre-trained Language Models for Tigrinya},
  year= 2021,
  publisher= {WiNLP 2021/EMNLP 2021}
}
```


## References


```
Tedla, Y., Yamamoto, K. & Marasinghe, A. 2016.
Tigrinya Part-of-Speech Tagging with Morphological Patterns and the New Nagaoka Tigrinya Corpus.
International Journal Of Computer Applications 146 pp. 33-41 (2016).
```


model name: tielectra-small-pos


model author: fgaim


task: token-classification


tags: pytorch, electra, token-classification, ti, dataset:TLMD, dataset:NTC, transformers, model-index, autonlp_compatible, infinity_compatible


license: mit
tags:


- generated_from_trainer
datasets:
- amazon_reviews_multi
model-index:
- name: xlm-roberta-base-finetuned-marc-en
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# xlm-roberta-base-finetuned-marc-en


This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the amazon_reviews_multi dataset.
It achieves the following results on the evaluation set:


- Loss: 0.9211
- Mae: 0.5122


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2


### Training results


| Training Loss | Epoch | Step | Validation Loss | Mae    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 1.1436        | 1.0   | 235  | 1.0181          | 0.5366 |
| 0.9756        | 2.0   | 470  | 0.9211          | 0.5122 |


### Framework versions


- Transformers 4.12.3
- Pytorch 1.9.0+cu111
- Datasets 1.15.1
- Tokenizers 0.10.3


model name: xlm-roberta-base-finetuned-marc-en


model author: tkesonia


task: text-classification


tags: pytorch, tensorboard, xlm-roberta, text-classification, dataset:amazon_reviews_multi, transformers, license:mit, generated_from_trainer, model-index


tags:


- espnet
- audio
- automatic-speech-recognition
language: en
datasets:
- librispeech
license: cc-by-4.0


## ESPnet2 ASR model


### `eml914/streaming_transformer_asr_librispeech`


This model was trained by Emiru Tsunoo using librispeech recipe in [espnet](https://github.com/espnet/espnet/).


### Demo: How to use in ESPnet2


```bash
cd espnet
git checkout 12eb132418a1f69548f7998e53273cd05d989ed9
pip install -e .
cd egs2/librispeech/asr1
./run.sh --skip_data_prep false --skip_train true --download_model eml914/streaming_transformer_asr_librispeech
```


<!-- Generated by scripts/utils/show_asr_result.sh -->


# RESULTS


## Environments


- date: `Wed Nov 17 18:18:46 JST 2021`
- python version: `3.8.11 (default, Aug  3 2021, 15:09:35)  [GCC 7.5.0]`
- espnet version: `espnet 0.10.5a1`
- pytorch version: `pytorch 1.4.0`
- Git hash: `12eb132418a1f69548f7998e53273cd05d989ed9`
  - Commit date: `Tue Nov 16 10:12:21 2021 +0900`


## asr_train_asr_streaming_fbank_pitch_en_bpe5000_sp


### WER


### CER


### TER


## ASR config


<details><summary>expand</summary>