In 2020, COVID-19 which is caused by a coronavirus called SARS-CoV-2 took over the world. It touched the lives of many people and caused a lot of hardship for humanity. There are still many questions in regards to COVID-19 and it is often difficult to get the right answers. The aim of this project is to finetune models for closed book question answering. In closed-book QA, we feed the model a question *without any context or access to external knowledge* and train it to predict the answer. Since the model doesn't receive any context, the primary way it can learn to answer these questions is based on the "knowledge" it obtained during pre-training [[1]](https://colab.research.google.com/github/google-research/text-to-text-transfer-transformer/blob/master/notebooks/t5-trivia.ipynb#scrollTo=zSeyoqE7WMwu)  [[2]](https://arxiv.org/abs/2002.08910).


The main goals of this project are:


1. Train a model for question answering in regards to COVID-19
1. Release the top performing models for further research and enhancement
1. Release all of the preprocessing and postprocessing scripts and findings for future research.


## TO DO LIST:


- [x] Team members met and the following was discussed:
  - Data preparation script is prepared that mixes CORD-19 and Pubmed.
  - Agreed to finalize the training scripts by 9pm PDT 7/9/2021.
  - Tokenizer is now trained.
- [ ] Setup the pretraining script
- [ ] Prepare the finetuning tasks inspired from [T5 Trivia Colab](https://colab.research.google.com/github/google-research/text-to-text-transfer-transformer/blob/master/notebooks/t5-trivia.ipynb)
  - What datasets we want to go with?
    - [Covid-QA](https://huggingface.co/datasets/covid_qa_deepset) (Maybe as test set?)
    - [Trivia](https://huggingface.co/datasets/covid_qa_deepset)
    - [CDC-QA](https://www.cdc.gov/coronavirus/2019-ncov/faq.html) (We can scrape quickly using beautiful soup or something)
    - [More Medical Datasets](https://aclanthology.org/2020.findings-emnlp.289.pdf) (See the dataset section for inspiratio)


## 1. Model


We will be using T5 model.


## 2. Datasets


The following datasets would be used for finetuning the model. Note that the last dataset is optional and the model is evaluated only using Covid-QA.


For **Intermediate Pre-Training**:


1. [CORD-19](https://allenai.org/data/cord-19)


For **Fine-Tuning** :


1. [Covid-QA](https://huggingface.co/datasets/covid_qa_deepset)
1. [CDC-QA](https://www.cdc.gov/coronavirus/2019-ncov/faq.html)
1. Optional - [Trivia-QA](https://nlp.cs.washington.edu/triviaqa/)


## 3. Training Scripts


We can make use of :


1. [For preprocessing and mixing datasets](https://colab.research.google.com/github/google-research/text-to-text-transfer-transformer/blob/master/notebooks/t5-trivia.ipynb#:~:text=In%20this%20notebook%2C%20we'll,it%20to%20predict%20the%20answer.)
1. [For T5 training](https://github.com/huggingface/transformers/blob/master/src/transformers/models/t5/modeling_flax_t5.py)


## 4. Additional Reading


- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://arxiv.org/pdf/2002.08910.pdf)


model name: t5-covid-qa


model author: flax-community


tags: arxiv:2002.08910


language: sv
license: cc-by-4.0
tags:


- swedish
- roberta
pipeline_tag: fill-mask
widget:
- text: Meninged med livet är <mask>.


# Swe Roberta Wiki Oscar


## Description


This Roberta model was trained on the Swedish Wikipedia and Oscar datasets


## Model series


This model is part of a series of models training on TPU with Flax Jax during Huggingface Flax/Jax challenge.


## Gpt models


## Swedish Gpt


## Swedish gpt wiki


# Nordic gpt wiki


## Dansk gpt wiki


## Norsk gpt wiki


## Roberta models


## Nordic Roberta Wiki


## Swe Roberta Wiki Oscar


## Roberta Swedish Scandi


## Roberta Swedish


## Swedish T5 model


model name: swe-roberta-wiki-oscar


model author: flax-community


task: fill-mask


tags: pytorch, jax, tensorboard, roberta, feature-extraction, sv, transformers, license:cc-by-4.0, swedish, fill-mask, autonlp_compatible, infinity_compatible


language:


- en
- python
license: mit
tags:
- gpt_neo
- code_synthesis
datasets:
- apps


# GPT-Neo-125M-APPS-all


> **Please refer to our new [GitHub Wiki](https://github.com/ncoop57/gpt-code-clippy/wiki) which documents our efforts in detail in creating the open source version of GitHub  Copilot**


## Model Description


GPT-Neo-125M-APPS-all is a GPT-Neo-125M finetuned on APPS dataset. This model is specialized to solve programming tasks.


## Training data


The model is trained on the [Automated Programming Progress Standard (APPS) dataset](https://github.com/hendrycks/apps). The dataset consists of 10,000 coding problems in total, with 131,836 test cases for checking solutions and 232,444 ground-truth solutions written by humans. Problems can be complicated, as the average length of a problem is 293.2 words. The data are split evenly into training and test sets, with 5,000 problems each.


This model is fine-tuned using most of the APPS dataset including both train and test split to explore the impact of this training task on model performance on other code synthesis evaluation metrics. A model fine-tuned on train set only can be found [here](https://huggingface.co/flax-community/gpt-neo-125M-apps).


## Training procedure


The training script used to train this model can be found [here](https://github.com/ncoop57/gpt-code-clippy/blob/camera-ready/training/run_clm_apps.py).


Training is done for 5 epochs using AdamW optimizer and leaner decay learning rate schedule with 800 warmup steps. To reproduce the training one can use this command with the above script:


```bash
python run_clm_apps.py \
    --output_dir $HOME/gpt-neo-125M-apps \
    --model_name_or_path EleutherAI/gpt-neo-125B \
    --dataset_name $HOME/gpt-code-clippy/data_processing/apps.py \
    --dataset_config_name formatted \
    --do_train --do_eval \
    --block_size="1024" \
    --per_device_train_batch_size="16" \
    --per_device_eval_batch_size="16" \
    --preprocessing_num_workers="16" \
    --learning_rate="8e-5" \
    --warmup_steps="800" \
    --adam_beta1="0.9" \
    --adam_beta2="0.98" \
    --weight_decay="0.1" \
    --overwrite_output_dir \
    --num_train_epochs="5" \
    --logging_steps="50" \
    --eval_steps="2000" \
    --report_to="wandb" \
    --dtype="bfloat16" \
    --save_strategy epoch \
    --gradient_accumulation_steps 2 \
    --all_data true \
```


## Intended Use and Limitations


The model is finetuned to solve programming problems given a text description and optional starter code.


### How to use


You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:


```py
from transformers import AutoModelForCausalLM, AutoTokenizer, FlaxAutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("flax-community/gpt-code-clippy-125M-apps-alldata")
tokenizer = AutoTokenizer.from_pretrained("flax-community/gpt-code-clippy-125M-apps-alldata")

prompt = """
A function to greet user. Given a user name it should say hello

def greet(name):

ANSWER:
"""

input_ids = tokenizer(prompt, return_tensors='pt').input_ids.to(device)
start = input_ids.size(1)
out = model.generate(input_ids, do_sample=True, max_length=50, num_beams=2,
                     early_stopping=True, eos_token_id=tokenizer.eos_token_id, )

print(tokenizer.decode(out[0][start:]))
```


### Limitations and Biases


The model is intended to be used for research purposes and comes with no guarantees of quality of generated code.


The paper ["Evaluating Large Language Models Trained on Code"](https://arxiv.org/abs/2107.03374) from OpenAI has a good discussion on what the impact of a large language model trained on code could be. Therefore, some parts of their discuss are highlighted here as it pertains to this dataset and models that may be trained from it. **As well as some differences in views from the paper, particularly around legal implications**.


1. **Over-reliance:** This model may generate plausible solutions that may appear correct, but are not necessarily the correct solution. Not properly evaluating the generated code may cause have negative consequences such as the introduction of bugs, or the introduction of security vulnerabilities. Therefore, it is important that users are aware of the limitations and potential negative consequences of using this language model.
1. **Economic and labor market impacts:** Large language models trained on large code datasets such as this one that are capable of generating high-quality code have the potential to automate part of the software development process. This may negatively impact software developers. However, as discussed in the paper, as shown in the Summary Report of software developers from [O*NET OnLine](https://www.onetonline.org/link/summary/15-1252.00), developers don't just write software.
1. **Biases:** The model is trained on data containing prompt questions formatted in specific way. The performance of the model can be worse if the prompt
formatting is different from that used in APPS dataset.


GPT-CC is finetuned GPT-Neo and might have inhereted biases and limitations from it. See [GPT-Neo model card](https://huggingface.co/EleutherAI/gpt-neo-125M#limitations-and-biases) for details.


## Eval results


Coming soon...


model name: gpt-neo-125M-apps-all


model author: flax-community


task: text-generation


tags: pytorch, jax, gpt_neo, text-generation, en, python, dataset:apps, arxiv:2107.03374, transformers, license:mit, code_synthesis


# CLIP-Vision-BERT Multilingual Pre-trained Model


Pretrained CLIP-Vision-BERT pre-trained on translated [Conceptual-12M](https://github.com/google-research-datasets/conceptual-12m) image-text pairs using a masked language modeling (MLM) objective. 10M cleaned image-text pairs are translated using [mBART-50 one-to-many model](https://huggingface.co/facebook/mbart-large-50-one-to-many-mmt) to 2.5M examples each in English, French, German and Spanish. This model is based on the VisualBERT which was introduced in
[this paper](https://arxiv.org/abs/1908.03557) and first released in
[this repository](https://github.com/uclanlp/visualbert). We trained CLIP-Vision-BERT model during community week hosted by Huggingface 🤗 using JAX/Flax.


This checkpoint is pre-trained for 60k steps.


## Model description


CLIP-Vision-BERT is a modified BERT model which takes in visual embeddings from CLIP-Vision transformer and concatenates them with BERT textual embeddings before passing them to the self-attention layers of BERT. This is done for deep cross-modal interaction between the two modes.


## Intended uses & limitations❗


You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.
Note that this model is primarily aimed at being fine-tuned on tasks such as visuo-linguistic sequence classification or visual question answering. We used this model to fine-tuned on a multi-translated version of the visual question answering task - [VQA v2](https://visualqa.org/challenge.html). Since Conceptual-12M is a dataset scraped from the internet, it will involve some biases which will also affect all fine-tuned versions of this model.


### How to use❓


You can use this model directly with a pipeline for masked language modeling. You will need to clone the model from [here](https://github.com/gchhablani/multilingual-vqa). An example of usage is shown below:


```python
>>> from torchvision.io import read_image
>>> import numpy as  np
>>> import os
>>> from transformers import CLIPProcessor, BertTokenizerFast
>>> from model.flax_clip_vision_bert.modeling_clip_vision_bert import FlaxCLIPVisionBertForMaskedLM
>>> image_path = os.path.join('images/val2014', os.listdir('images/val2014')[0])
>>> img = read_image(image_path)
>>> clip_processor = CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32')
ftfy or spacy is not installed using BERT BasicTokenizer instead of ftfy.
>>> clip_outputs = clip_processor(images=img)
>>> clip_outputs['pixel_values'][0] = clip_outputs['pixel_values'][0].transpose(1,2,0) # Need to transpose images as model expected channel last images.
>>> tokenizer = BertTokenizerFast.from_pretrained('bert-base-multilingual-uncased')
>>> model = FlaxCLIPVisionBertForMaskedLM.from_pretrained('flax-community/clip-vision-bert-cc12m-60k')
>>> text = "Three teddy [MASK] in a showcase."
>>> tokens = tokenizer([text], return_tensors="np")
>>> pixel_values = np.concatenate([clip_outputs['pixel_values']])
>>> outputs = model(pixel_values=pixel_values, **tokens)
>>> indices = np.where(tokens['input_ids']==tokenizer.mask_token_id)
>>> preds = outputs.logits[indices][0]
>>> sorted_indices = np.argsort(preds)[::-1] # Get reverse sorted scores
/home/crocoder/anaconda3/lib/python3.8/site-packages/jax/_src/numpy/lax_numpy.py:4615: UserWarning: 'kind' argument to argsort is ignored.
  warnings.warn("'kind' argument to argsort is ignored.")
>>> top_5_indices = sorted_indices[:5]
>>> top_5_tokens = tokenizer.convert_ids_to_tokens(top_5_indices)
>>> top_5_scores = preds[top_5_indices]
>>> print(dict(zip(top_5_tokens, top_5_scores)))
{'bears': 19.241959, 'bear': 17.700356, 'animals': 14.368396, 'girls': 14.343797, 'dolls': 14.274415}

```


## Training data 🏋🏻♂


The CLIP-Vision-BERT model was pre-trained on a translated version of the Conceptual-12m dataset in four languages using mBART-50: English, French, German and Spanish, with 2.5M image-text pairs in each.


The dataset captions and image urls can be downloaded from [flax-community/conceptual-12m-mbart-50-translated](https://huggingface.co/datasets/flax-community/conceptual-12m-mbart-50-multilingual).


## Data Cleaning 🧹


Though the original dataset contains 12M image-text pairs, a lot of the URLs are invalid now, and in some cases, images are corrupt or broken. We remove such examples from our data, which leaves us with approximately 10M image-text pairs.


**Splits**
We used 99% of the 10M examples as a train set, and the remaining ~ 100K examples as our validation set.


## Training procedure 👨🏻💻


### Preprocessing


The texts are lowercased and tokenized using WordPiece and a shared vocabulary size of approximately 110,000. The beginning of a new document is marked with `[CLS]` and the end of one by `[SEP]`
The details of the masking procedure for each sentence are the following:


- 15% of the tokens are masked.
- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
- In the 10% remaining cases, the masked tokens are left as is.


The visual embeddings are taken from the CLIP-Vision model and combined with the textual embeddings inside the BERT embedding layer. The padding is done in the middle. Here is an example of what the embeddings look like:


```
[CLS Emb] [Textual Embs] [SEP Emb] [Pad Embs] [Visual Embs]
```


A total length of 128 tokens, including the visual embeddings, is used. The texts are truncated or padded accordingly.


### Pretraining


The checkpoint of the model was trained on Google Cloud Engine TPUv3-8 machine (with 335 GB of RAM, 1000 GB of hard drive, 96 CPU cores) **8 v3 TPU cores** for 60k steps with a per device batch size of 64 and a max sequence length of 128. The optimizer used is Adafactor with a learning rate of 1e-4, learning rate warmup for 5,000 steps, and linear decay of the learning rate after.


We tracked experiments using TensorBoard. Here is the link to the main dashboard: [CLIP Vision BERT CC12M Pre-training Dashboard](https://huggingface.co/flax-community/multilingual-vqa-pt-ckpts/tensorboard)


#### **Pretraining Results 📊**


The model at this checkpoint reached **eval accuracy of 67.53%** and **with train loss at 1.793 and eval loss at 1.724**.


## Fine Tuning on downstream tasks


We performed fine-tuning on downstream tasks. We used the following datasets for visual question answering:


1. Multilingual of [Visual Question Answering (VQA) v2](https://visualqa.org/challenge.html) - We translated this dataset to the four languages using `Helsinki-NLP` Marian models. The translated data can be found at [flax-community/multilingual-vqa](https://huggingface.co/datasets/flax-community/multilingual-vqa).


The checkpoints for the fine-tuned model on this pre-trained checkpoint can be found [here](https://huggingface.co/flax-community/multilingual-vqa-pt-60k-ft/tensorboard).
The fine-tuned model achieves eval accuracy of 49% on our validation dataset.


## Team Members


- Gunjan Chhablani [@gchhablani](https://hf.co/gchhablani)
- Bhavitvya Malik[@bhavitvyamalik](https://hf.co/bhavitvyamalik)


## Acknowledgements


We thank [Nilakshan Kunananthaseelan](https://huggingface.co/knilakshan20) for helping us whenever he could get a chance. We also thank [Abheesht Sharma](https://huggingface.co/abheesht) for helping in the discussions in the initial phases. [Luke Melas](https://github.com/lukemelas) helped us get the CC-12M data on our TPU-VMs and we are very grateful to him.


This project would not be possible without the help of [Patrick](https://huggingface.co/patrickvonplaten) and [Suraj](https://huggingface.co/valhalla) who met with us frequently and helped review our approach and guided us throughout the project.


Huge thanks to Huggingface 🤗 & Google Jax/Flax team for such a wonderful community week and for answering our queries on the Slack channel, and for providing us with the TPU-VMs.


<img src=https://pbs.twimg.com/media/E443fPjX0AY1BsR.jpg:large>


model name: clip-vision-bert-cc12m-60k


model author: flax-community


task: fill-mask


tags: jax, clip-vision-bert, fill-mask, arxiv:1908.03557, transformers, autonlp_compatible


# Tokenizer


We trained our tokenizer using [sentencepiece](https://github.com/google/sentencepiece)'s unigram tokenizer. Then loaded the tokenizer as MT5TokenizerFast.


## Model


We used [MT5-base](https://huggingface.co/google/mt5-base) model.


## Datasets


We used [Code Search Net](https://huggingface.co/datasets/code_search_net)'s dataset and some scrapped data from internet to train the model. We maintained a list of datasets where each dataset had codes of same language.


## Plots


### Train loss


![train loss](https://i.ibb.co/x53Wm8n/train-loss.png)


### Evaluation loss


![eval loss](https://i.ibb.co/McB2jnf/eval-loss.png)


### Evaluation accuracy


![eval accuracy](https://i.ibb.co/YDGhLdn/eval-accuracy.png)


### Learning rate


![learning rate](https://i.ibb.co/CMStzWv/learning-rate.png)


## Fine tuning (WIP)


We fine tuned the model with [CodeXGLUE code-to-code-trans dataset](https://huggingface.co/datasets/code_x_glue_cc_code_to_code_trans), and scrapper data.


model name: code-mt5-base


model author: flax-community


task: text2text-generation


tags: pytorch, jax, tensorboard, mt5, text2text-generation, transformers, autonlp_compatible


# CLIP-Vision-Marian Seq2Seq Encoder-Decoder Model


Pretrained CLIP-Vision-Marian pre-trained on a subset of Spanish-translated Conceptual-12M image-text pairs using a seq2seq model training objective. 2.5M cleaned English image-text pairs are translated using Spanish Marian Model. We trained CLIP-Vision-Marian model during community week hosted by Huggingface 🤗 using JAX/Flax.


## Model description


CLIP-Vision-Marian is a modified transformers model which takes in visual embeddings from CLIP-Vision transformer and feeds into the `encoder_hidden_states` of a Marian decoder. This is done for deep cross-modal interaction via `cross-attention` between the two modes. The decoder then predicts logits for the `input_ids` provided and can be used for generation.


## Intended uses & limitations❗


You can use the raw model for encoder-decoder network where you want the encoder to encode images and the decoder to decode text.


Note that this model is primarily aimed at being fine-tuned on tasks like Spanish image captioning.


### How to use❓


You will need to clone the model from [here](https://github.com/bhavitvyamalik/spanish-image-captioning). An example of usage is shown below:


```python
>>> from torchvision.io import read_image
>>> import numpy as  np
>>> import wget
>>> import os
>>> from transformers import CLIPProcessor, MarianTokenizer
>>> from models.flax_clip_vision_marian.modeling_clip_vision_marian import FlaxCLIPVisionMarianMT
img = wget.download("https://huggingface.co/streamlitiframe/flax-community/spanish-image-captioning/+/media/55a8898e61131569cc0ed4e72a8b3092969d63c2dff4f47ed9ef0d89.jpeg")
>>> img = read_image(img) # reading image
>>> clip_processor = CLIPProcessor.from_pretrained('flax-community/clip-vit-base-patch32_marian')
>>> clip_outputs = clip_processor(images=img)
>>> clip_outputs['pixel_values'][0] = clip_outputs['pixel_values'][0].transpose(1,2,0) # Need to transpose images as model expected channel last images.
>>> tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-es')
>>> model = FlaxCLIPVisionMarianMT.from_pretrained('flax-community/clip-vit-base-patch32_marian-es')
>>> output_ids = model.generate(batch["pixel_values"], early_stopping=True, num_beams=4, max_length=64).sequences
>>> output_string = tokenizer.batch_decode(output_ids.reshape(-1, 64), skip_special_tokens=True, max_length=64)
>>> output_string
# Sopa de avena en un tazón blanco con arándanos frescos
```


## Training data 🏋🏻♂


The Spanish image captioning model was trained on a subset of Conceptual 12M dataset by Google:

[Conceptual 12M](https://github.com/google-research-datasets/conceptual-12m), Introduced by Changpinyo et al. in [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://arxiv.org/abs/2102.08981).


### Please update the dataset link here


The translated dataset can be downloaded from [conceptual-12m-multilingual-marian-es](https://huggingface.co/datasets/flax-community/conceptual-12m-multilingual-marian-es). We do not provide images as we do not own any of them. One can download images from the `image_url` section of the original Conceptual 12M dataset.


## Data Cleaning 🧹


Though the original dataset contains 12M image-text pairs, a lot of the URLs are invalid now, and in some cases, images are corrupt or broken. We remove such examples from our data, which leaves us with approximately 10M image-text pairs, out of which we took only 2.5M image, caption pairs.


#### **Train set:**


Total data:

2475000 captions

2475000 images


#### **Validation set**


Total data:

25000 captions

25000 images


## Training procedure 👨🏻💻


### Training


The model was trained on Google Cloud Engine TPUv3-8 machine (with 335 GB of RAM, 1000 GB of hard drive, 96 CPU cores) **8 v3 TPU cores** for 42K steps with a batch size of 128 and a sequence length of 128. The
optimizer used is Adam with a learning rate of 3e-4, β1 = 0.9, β2 = 0.98 and
ε = 1e-8, a weight decay of 0.01, learning rate warmup for 1,000 steps and linear decay of the learning
rate after.


We tracked experiments using Tensorboard which can be found in `Training Metrics` tab.


#### **Pretraining Results 📊**


Our model reached **eval loss of ~3.1** around ~20K steps. Here are the BLEU^ scores for different languages:


|Language                  |BLEU-1|BLEU-2|BLEU-3|BLEU-4|
|--------------------------|------|------|------|------|
|Spanish                   | 0.2015| 0.1348| 0.09982| 0.0748|


^BLEU scores are out of 1


## **App Demo**


You can try out our model on 🤗 Huggingface's spaces 🪐 :
[Streamlit app of Spanish Image Captioning model on Huggingface Spaces](https://huggingface.co/spaces/flax-community/spanish-image-captioning)


## Team Members


- Bhavitvya Malik [@bhavitvyamalik](https://github.com/bhavitvyamalik)
- Gunjan Chhablani [@gchhablani](https://github.com/gchhablani)


## Credits


Thanks to Huggingface 🤗 & Google JAX/Flax team for such a wonderful community week. Big thanks to [@patrickvonplaten](https://github.com/patrickvonplaten) and [@patil-suraj](https://github.com/patil-suraj) for helping us with our solution during the community week.


<img src=https://pbs.twimg.com/media/E443fPjX0AY1BsR.jpg:large>


model name: clip-vit-base-patch32_marian-es


model author: flax-community


tags: jax, tensorboard, clip-vision-marian, arxiv:2102.08981, transformers


language: si
tags:


- fill-mask
- sinhala
- roberta


## Sinhala Roberta model trained on MC4 Sinhala dataset (manually cleaned)


model name: Sinhala-roberta


task: feature-extraction


tags: pytorch, jax, tensorboard, roberta, feature-extraction, si, transformers, fill-mask, sinhala, autonlp_compatible, infinity_compatible


# CLIP-Vision-BERT Multilingual VQA Model


Fine-tuned CLIP-Vision-BERT on translated [VQAv2](https://visualqa.org/challenge.html) image-text pairs using sequence classification objective. We translate the dataset to three other languages other than English: French, German, and Spanish using the [MarianMT Models](https://huggingface.co/transformers/model_doc/marian.html). This model is based on the VisualBERT which was introduced in
[this paper](https://arxiv.org/abs/1908.03557) and first released in
[this repository](https://github.com/uclanlp/visualbert). The output is 3129 class logits, the same classes as used by VisualBERT authors.


The initial weights are loaded from the Conceptual-12M 60k [checkpoints](https://huggingface.co/flax-community/clip-vision-bert-cc12m-60k).


We trained the CLIP-Vision-BERT VQA model during community week hosted by Huggingface 🤗 using JAX/Flax.


## Model description


CLIP-Vision-BERT is a modified BERT model which takes in visual embeddings from the CLIP-Vision transformer and concatenates them with BERT textual embeddings before passing them to the self-attention layers of BERT. This is done for deep cross-modal interaction between the two modes.


## Intended uses & limitations❗


This model is fine-tuned on a multi-translated version of the visual question answering task - [VQA v2](https://visualqa.org/challenge.html). Since VQAv2 is a dataset scraped from the internet, it will involve some biases which will also affect all fine-tuned versions of this model.


### How to use❓


You can use this model directly on visual question answering. You will need to clone the model from [here](https://github.com/gchhablani/multilingual-vqa). An example of usage is shown below:


```python
>>> from torchvision.io import read_image
>>> import numpy as  np
>>> import os
>>> from transformers import CLIPProcessor, BertTokenizerFast
>>> from model.flax_clip_vision_bert.modeling_clip_vision_bert import FlaxCLIPVisionBertForSequenceClassification
>>> image_path = os.path.join('images/val2014', os.listdir('images/val2014')[0])
>>> img = read_image(image_path)
>>> clip_processor = CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32')
ftfy or spacy is not installed using BERT BasicTokenizer instead of ftfy.
>>> clip_outputs = clip_processor(images=img)
>>> clip_outputs['pixel_values'][0] = clip_outputs['pixel_values'][0].transpose(1,2,0) # Need to transpose images as model expected channel last images.
>>> tokenizer = BertTokenizerFast.from_pretrained('bert-base-multilingual-uncased')
>>> model = FlaxCLIPVisionBertForSequenceClassification.from_pretrained('flax-community/clip-vision-bert-vqa-ft-6k')
>>> text = "Are there teddy bears in the image?"
>>> tokens = tokenizer([text], return_tensors="np")
>>> pixel_values = np.concatenate([clip_outputs['pixel_values']])
>>> outputs = model(pixel_values=pixel_values, **tokens)
>>> preds = outputs.logits[0]
>>> sorted_indices = np.argsort(preds)[::-1] # Get reverse sorted scores
>>> top_5_indices = sorted_indices[:5]
>>> top_5_tokens = list(map(model.config.id2label.get,top_5_indices))
>>> top_5_scores = preds[top_5_indices]
>>> print(dict(zip(top_5_tokens, top_5_scores)))
{'yes': 15.809224, 'no': 7.8785815, '<unk>': 4.622649, 'very': 4.511462, 'neither': 3.600822}
```


## Training data 🏋🏻♂


The CLIP-Vision-BERT model was fine-tuned on the translated version of the VQAv2 dataset in four languages using Marian: English, French, German and Spanish. Hence, the dataset is four times the original English questions.


The dataset questions and image URLs/paths can be downloaded from [flax-community/multilingual-vqa](https://huggingface.co/datasets/flax-community/multilingual-vqa).


## Data Cleaning 🧹


Though the original dataset contains 443,757  train and 214,354 validation image-question pairs. We only use the `multiple_choice_answer`. The answers which are not present in the 3129 classes are mapped to the `<unk>` label.


**Splits**
We use the original train-val splits from the VQAv2 dataset. After translation, we get 1,775,028 train image-text pairs, and 857,416 validation image-text pairs.


## Training procedure 👨🏻💻


### Preprocessing


The texts are lowercased and tokenized using WordPiece and a shared vocabulary size of approximately 110,000. The beginning of a new document is marked with `[CLS]` and the end of one by `[SEP]`.


### Fine-tuning


The checkpoint of the model was trained on Google Cloud Engine TPUv3-8 machine (with 335 GB of RAM, 1000 GB of hard drive, 96 CPU cores) **8 v3 TPU cores** for 6k steps with a per device batch size of 128 and a max sequence length of 128. The optimizer used is AdamW with a learning rate of 5e-5, learning rate warmup for 1600 steps, and linear decay of the learning rate after.


We tracked experiments using TensorBoard. Here is link to main dashboard: [CLIP Vision BERT VQAv2 Fine-tuning Dashboard](https://huggingface.co/flax-community/multilingual-vqa-pt-60k-ft/tensorboard)


#### **Fine-tuning Results 📊**


The model at this checkpoint reached **eval accuracy of 0.49** on our multilingual VQAv2 dataset.


## Team Members


- Gunjan Chhablani [@gchhablani](https://hf.co/gchhablani)
- Bhavitvya Malik[@bhavitvyamalik](https://hf.co/bhavitvyamalik)


## Acknowledgements


We thank [Nilakshan Kunananthaseelan](https://huggingface.co/knilakshan20) for helping us whenever he could get a chance. We also thank [Abheesht Sharma](https://huggingface.co/abheesht) for helping in the discussions in the initial phases. [Luke Melas](https://github.com/lukemelas) helped us get the CC-12M data on our TPU-VMs and we are very grateful to him.


This project would not be possible without the help of [Patrick](https://huggingface.co/patrickvonplaten) and [Suraj](https://huggingface.co/valhalla) who met with us frequently and helped review our approach and guided us throughout the project.


Huge thanks to Huggingface 🤗 & Google Jax/Flax team for such a wonderful community week and for answering our queries on the Slack channel, and for providing us with the TPU-VMs.


<img src=https://pbs.twimg.com/media/E443fPjX0AY1BsR.jpg:large>


model name: clip-vision-bert-vqa-ft-6k


model author: flax-community


task: text-classification


tags: jax, clip-vision-bert, text-classification, arxiv:1908.03557, transformers


# Scandinavian RoBERTa


A RoBERTa-large model trained from scratch on Swedish, Danish, Norwegian,
Icelandic (and maybe also Faroese if we can get enough data) using
[Flax](https://github.com/google/flax), including training scripts.


This is part of the
[Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104),
organised by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google.


## Team members


- saattrupdan | Dan Saattrup Nielsen | [saattrupdan@gmail.com](mailto:saattrupdan@gmail.com)
- MortenKP | Morten Kloster Pedersen
- Maltehb | Malte Højmark-Bertelsen
- pere | Per
- versae | Javier de la Rosa
- Juunge | Kasper Junge
- Gabriel | Gabriel Borg
- ?xxzyx? | Xander Nordentord
- ?grofte? | Morten Grøftehauge


## Useful links


- [Community Week timeline](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104#summary-timeline-calendar-6)
- [Community Week README](https://github.com/huggingface/transformers/blob/master/examples/research_projects/jax-projects/README.md)
- [Community Week thread](https://discuss.huggingface.co/t/scandinavian-roberta/7533)
- [Masked Language Modelling example scripts](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)
- [HuggingFace Model Repository](https://huggingface.co/saattrupdan/scandinavian-roberta)


model name: roberta-large-scandi


task: fill-mask


tags: jax, tensorboard, roberta, fill-mask, transformers, autonlp_compatible, infinity_compatible


# bengali-t5-base


**bengali-t5-base** is a model trained on the Bengali portion of MT5 dataset. We used the `T5-base` model for this model.


[Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organized by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google.


The model is trained on around ~11B tokens (64 size batch, 512 tokens, 350k steps).


## load tokenizer


```
>>> tokenizer = transformers.AutoTokenizer.from_pretrained("flax-community/bengali-t5-base")
>>> tokenizer.encode("আমি বাংলার গান গাই")
>>> tokenizer.decode([93, 1912, 814, 5995, 3, 1])
```


```
[93, 1912, 814, 5995, 3, 1]
'আমি বাংলার গান গাই </s>'
```


## load model


```
>>> config  = T5Config.from_pretrained("flax-community/bengali-t5-base")
>>> model = FlaxT5ForConditionalGeneration.from_pretrained("flax-community/bengali-t5-base", config=config)
```


The model is trained on `de-noising` objectives followed by the script [here](https://huggingface.co/flax-community/bengali-t5-base/blob/main/run_t5_mlm_flax.py) and [here](https://huggingface.co/flax-community/bengali-t5-base/blob/main/run.sh). Currently This model doesn't have any generation capability. If you want this model to have generation capability, please do a finetuning on `prefix-LM` objective mentioned in the [paper](https://arxiv.org/abs/1910.10683).


See the tensorboard log in `Training metrics` tab.


Please note that we haven't finetuned the model in any downstream task.


## Proposal


- [Project Proposal](https://discuss.huggingface.co/t/pretrain-t5-from-scratch-in-bengali/7121)


## Participants


- [Ibraheem Muhammad Moosa](https://huggingface.co/ibraheemmoosa)
- [Tasnim Mohiuddin](https://huggingface.co/tasnim)
- [Khalid Saifullah](https://huggingface.co/khalidsaifullaah)
- [Tahsin Mayeesha](https://tahsin-mayeesha.github.io/)
- [M Saiful Bari](https://huggingface.co/sbmaruf)


## Useful links


- [Community Week timeline](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104#summary-timeline-calendar-6)
- [Community Week README](https://github.com/huggingface/transformers/blob/master/examples/research_projects/jax-projects/README.md)
- [Masked Language Modelling example scripts](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)
- [Model Repository](https://huggingface.co/flax-community/roberta-base-als-demo)


model name: bengali-t5-base


model author: flax-community


task: text2text-generation


tags: pytorch, jax, tensorboard, mt5, text2text-generation, arxiv:1910.10683, transformers, autonlp_compatible


# CLIP-Vision-BERT Multilingual Pre-trained Model


Pretrained CLIP-Vision-BERT pre-trained on translated [Conceptual-12M](https://github.com/google-research-datasets/conceptual-12m) image-text pairs using a masked language modeling (MLM) objective. 10M cleaned image-text pairs are translated using [mBART-50 one-to-many model](https://huggingface.co/facebook/mbart-large-50-one-to-many-mmt) to 2.5M examples each in English, French, German and Spanish. This model is based on the VisualBERT which was introduced in
[this paper](https://arxiv.org/abs/1908.03557) and first released in
[this repository](https://github.com/uclanlp/visualbert). We trained CLIP-Vision-BERT model during community week hosted by Huggingface 🤗 using JAX/Flax.


This checkpoint is pre-trained for 70k steps.


## Model description


CLIP-Vision-BERT is a modified BERT model which takes in visual embeddings from CLIP-Vision transformer and concatenates them with BERT textual embeddings before passing them to the self-attention layers of BERT. This is done for deep cross-modal interaction between the two modes.


## Intended uses & limitations❗


You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.
Note that this model is primarily aimed at being fine-tuned on tasks such as visuo-linguistic sequence classification or visual question answering. We used this model to fine-tuned on a multi-translated version of the visual question answering task - [VQA v2](https://visualqa.org/challenge.html). Since Conceptual-12M is a dataset scraped from the internet, it will involve some biases which will also affect all fine-tuned versions of this model.


### How to use❓


You can use this model directly with a pipeline for masked language modeling. You will need to clone the model from [here](https://github.com/gchhablani/multilingual-vqa). An example of usage is shown below:


```python
>>> from torchvision.io import read_image
>>> import numpy as  np
>>> import os
>>> from transformers import CLIPProcessor, BertTokenizerFast
>>> from model.flax_clip_vision_bert.modeling_clip_vision_bert import FlaxCLIPVisionBertForMaskedLM
>>> image_path = os.path.join('images/val2014', os.listdir('images/val2014')[0])
>>> img = read_image(image_path)
>>> clip_processor = CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32')
ftfy or spacy is not installed using BERT BasicTokenizer instead of ftfy.
>>> clip_outputs = clip_processor(images=img)
>>> clip_outputs['pixel_values'][0] = clip_outputs['pixel_values'][0].transpose(1,2,0) # Need to transpose images as model expected channel last images.
>>> tokenizer = BertTokenizerFast.from_pretrained('bert-base-multilingual-uncased')
>>> model = FlaxCLIPVisionBertForMaskedLM.from_pretrained('flax-community/clip-vision-bert-cc12m-70k')
>>> text = "Three teddy [MASK] in a showcase."
>>> tokens = tokenizer([text], return_tensors="np")
>>> pixel_values = np.concatenate([clip_outputs['pixel_values']])
>>> outputs = model(pixel_values=pixel_values, **tokens)
>>> indices = np.where(tokens['input_ids']==tokenizer.mask_token_id)
>>> preds = outputs.logits[indices][0]
>>> sorted_indices = np.argsort(preds)[::-1] # Get reverse sorted scores
>>> top_5_indices = sorted_indices[:5]
>>> top_5_tokens = tokenizer.convert_ids_to_tokens(top_5_indices)
>>> top_5_scores = preds[top_5_indices]
>>> print(dict(zip(top_5_tokens, top_5_scores)))
{'bears': 19.400345, 'bear': 17.866995, 'animals': 14.453735, 'dogs': 14.427426, 'girls': 14.097499}
```


## Training data 🏋🏻♂


The CLIP-Vision-BERT model was pre-trained on a translated version of the Conceptual-12m dataset in four languages using mBART-50: English, French, German and Spanish, with 2.5M image-text pairs in each.


The dataset captions and image urls can be downloaded from [flax-community/conceptual-12m-mbart-50-translated](https://huggingface.co/datasets/flax-community/conceptual-12m-mbart-50-multilingual).


## Data Cleaning 🧹


Though the original dataset contains 12M image-text pairs, a lot of the URLs are invalid now, and in some cases, images are corrupt or broken. We remove such examples from our data, which leaves us with approximately 10M image-text pairs.


**Splits**
We used 99% of the 10M examples as a train set, and the remaining ~ 100K examples as our validation set.


## Training procedure 👨🏻💻


### Preprocessing


The texts are lowercased and tokenized using WordPiece and a shared vocabulary size of approximately 110,000. The beginning of a new document is marked with `[CLS]` and the end of one by `[CLS]`
The details of the masking procedure for each sentence are the following:


- 15% of the tokens are masked.
- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
- In the 10% remaining cases, the masked tokens are left as is.


The visual embeddings are taken from the CLIP-Vision model and combined with the textual embeddings inside the BERT embedding layer. The padding is done in the middle. Here is an example of what the embeddings look like:


```
[CLS Emb] [Textual Embs] [SEP Emb] [Pad Embs] [Visual Embs]
```


A total length of 128 tokens, including the visual embeddings, is used. The texts are truncated or padded accordingly.


### Pretraining


The checkpoint of the model was trained on Google Cloud Engine TPUv3-8 machine (with 335 GB of RAM, 1000 GB of hard drive, 96 CPU cores) **8 v3 TPU cores** for 70k steps with a per device batch size of 64 and a max sequence length of 128. The optimizer used is Adafactor with a learning rate of 1e-4, learning rate warmup for 1,000 steps, and linear decay of the learning rate after.


We tracked experiments using TensorBoard. Here is the link to the main dashboard: [CLIP Vision BERT CC12M Pre-training Dashboard](https://huggingface.co/flax-community/multilingual-vqa-pt-ckpts/tensorboard)


#### **Pretraining Results 📊**


The model at this checkpoint reached **eval accuracy of 67.85%** and **with train loss at 1.756 and eval loss at 1.706**.


## Team Members


- Gunjan Chhablani [@gchhablani](https://hf.co/gchhablani)
- Bhavitvya Malik[@bhavitvyamalik](https://hf.co/bhavitvyamalik)


## Acknowledgements


We thank [Nilakshan Kunananthaseelan](https://huggingface.co/knilakshan20) for helping us whenever he could get a chance. We also thank [Abheesht Sharma](https://huggingface.co/abheesht) for helping in the discussions in the initial phases. [Luke Melas](https://github.com/lukemelas) helped us get the CC-12M data on our TPU-VMs and we are very grateful to him.


This project would not be possible without the help of [Patrick](https://huggingface.co/patrickvonplaten) and [Suraj](https://huggingface.co/valhalla) who met with us frequently and helped review our approach and guided us throughout the project.


Huge thanks to Huggingface 🤗 & Google Jax/Flax team for such a wonderful community week and for answering our queries on the Slack channel, and for providing us with the TPU-VMs.


<img src=https://pbs.twimg.com/media/E443fPjX0AY1BsR.jpg:large>


model name: clip-vision-bert-cc12m-70k


model author: flax-community


task: fill-mask


tags: jax, clip-vision-bert, fill-mask, arxiv:1908.03557, transformers, autonlp_compatible


## language: id


# Image-captioning-Indonesia


This is an encoder-decoder image captioning model using [CLIP](https://huggingface.co/transformers/model_doc/clip.html) as the visual encoder and [Marian](https://huggingface.co/transformers/model_doc/marian.html) as the textual decoder on datasets with Indonesian captions.


This model was trained using HuggingFace's Flax framework and is part of the [JAX/Flax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104) organized by [HuggingFace](https://huggingface.co). All training was done on a TPUv3-8 VM sponsored by the Google Cloud team.


## How to use


At time of writing, you will need to install [HuggingFace](https://github.com/huggingface/) from its latest master branch in order to load `FlaxMarian`.


You will also need to have the [`flax_clip_vision_marian` folder](https://github.com/indonesian-nlp/Indonesia-Image-Captioning/tree/main/flax_clip_vision_marian) in your project directory to load the model using the `FlaxCLIPVisionMarianForConditionalGeneration` class.


```python
from torchvision.io import ImageReadMode, read_image
from torchvision.transforms import CenterCrop, ConvertImageDtype, Normalize, Resize
from torchvision.transforms.functional import InterpolationMode

import torch
import numpy as np
from transformers import MarianTokenizer
from flax_clip_vision_marian.modeling_clip_vision_marian import FlaxCLIPVisionMarianForConditionalGeneration

clip_marian_model_name = 'flax-community/Image-captioning-Indonesia'
model = FlaxCLIPVisionMarianForConditionalGeneration.from_pretrained(clip_marian_model_name)

marian_model_name = 'Helsinki-NLP/opus-mt-en-id'
tokenizer = MarianTokenizer.from_pretrained(marian_model_name)

config = model.config
image_size = config.clip_vision_config.image_size

# Image transformation
transforms = torch.nn.Sequential(
                    Resize([image_size], interpolation=InterpolationMode.BICUBIC),
                    CenterCrop(image_size),
                    ConvertImageDtype(torch.float),
                    Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711)),
                )

# Hyperparameters
max_length = 8
num_beams = 4
gen_kwargs = {"max_length": max_length, "num_beams": num_beams}

def generate_step(batch):
    output_ids = model.generate(pixel_values, **gen_kwargs)
    token_ids = np.array(output_ids.sequences)[0]
    caption = tokenizer.decode(token_ids)
    return caption

image_file_path = image_file_path
image = read_image(image_file_path, mode=ImageReadMode.RGB)
image = transforms(image)
pixel_values = torch.stack([image]).permute(0, 2, 3, 1).numpy()

generated_ids = generate_step(pixel_values)

print(generated_ids)
```


## Training data


The Model was trained on translated Coco,Flickr and ViZWiz, each of them were translated using google translate and marian mt. we took only random 2 captions per image for each datasets


## Training procedure


The model was trained on a TPUv3-8 VM provided by the Google Cloud team.


## Team members


- Cahya Wirawan ([@cahya](https://huggingface.co/cahya))
- Galuh Sahid ([@Galuh](https://huggingface.co/Galuh))
- Muhammad Agung Hambali ([@AyameRushia](https://huggingface.co/AyameRushia))
- Samsul Rahmadani ([@munggok](https://huggingface.co/munggok))


model name: Image-captioning-Indonesia


model author: flax-community


task: text2text-generation


tags: jax, clip-vision-marian, text2text-generation, id, transformers, autonlp_compatible


Rap Lyric Generator

GPT-2 fine tuned using FLAX/JAX on over 10000 Rap Songs from over 50 rappers, the dataset was gathered from genius.com

Checkout the deployed version on hf-spaces :- [here](https://huggingface.co/spaces/Shankhdhar/Rap-Lyric-generator)

Colab for making predictions:- [here](https://colab.research.google.com/drive/1aibR06TrFGnt-TPmyIRDD2-8eT7PU5Kl#scrollTo=rgE3QbiTFIMQ)

The dataset we used: [dataset](https://huggingface.co/datasets/Cropinky/rap_lyrics_english)

Made by:-

[Anant Shankhdhar](https://huggingface.co/Shankhdhar)

[Jeronim Matijević](https://huggingface.co/Cropinky)


model name: gpt2-rap-lyric-generator


task: text-generation


tags: jax, tensorboard, gpt2, text-generation, transformers


# IndicNLP Marathi News Classifier


This model was fine-tuned using [Marathi RoBERTa](https://huggingface.co/flax-community/roberta-base-mr) on [IndicNLP Marathi News Dataset](https://github.com/AI4Bharat/indicnlp_corpus#indicnlp-news-article-classification-dataset)


## Dataset


IndicNLP Marathi news dataset consists 3 classes - `['lifestyle', 'entertainment', 'sports']` - with following docs distribution as per classes:


| train | eval | test |
| ----- | ---- | ---- |
| 9672  | 477  | 478  |


💯 Our **`mr-indicnlp-classifier`** model fine tuned from **roberta-base-mr**  Pretrained Marathi RoBERTa model outperformed both classifier mentioned in [Arora, G. (2020). iNLTK](https://www.semanticscholar.org/paper/iNLTK%3A-Natural-Language-Toolkit-for-Indic-Languages-Arora/5039ed9e100d3a1cbbc25a02c82f6ee181609e83/figure/3) and [Kunchukuttan, Anoop et al. AI4Bharat-IndicNLP.](https://www.semanticscholar.org/paper/AI4Bharat-IndicNLP-Corpus%3A-Monolingual-Corpora-and-Kunchukuttan-Kakwani/7997d432925aff0ba05497d2893c09918298ca55/figure/4)


| Dataset         | FT-W  | FT-WC | INLP  | iNLTK | **roberta-base-mr 🏆** |
| --------------- | ----- | ----- | ----- | ----- | --------------------- |
| iNLTK Headlines | 83.06 | 81.65 | 89.92 | 92.4  | **97.48**             |


model name: mr-indicnlp-classifier


tags: pytorch


# 🖼 When ViT meets GPT-2 📝


An image captioning model [ViT-GPT2](https://huggingface.co/flax-community/vit-gpt2/tree/main) by combining the ViT model and a French GPT2 model.


Part of the [Huggingface JAX/Flax event](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/).


The GPT2 model source code is modified so it can accept an encoder's output.
The pretained weights of both models are loaded, with a set of randomly initialized cross-attention weigths.
The model is trained on 65000 images from the COCO dataset for about 1500 steps (batch\_size=256), with the original English cpationis being translated to French for training purpose.


**Technical challenges**


- The source code of Flax's version of GPT-2 is modified to be able to accept an encoder's outputs, so it can be used as a decoder in an encoder-decoder architecture.
- Originally, we created [**FlaxViTGPT2ForConditionalGenerationModule**](https://huggingface.co/flax-community/vit-gpt2/blob/main/vit_gpt2/modeling_flax_vit_gpt2.py#L86), which is [**FlaxViTGPT2Module**](https://huggingface.co/flax-community/vit-gpt2/blob/main/vit_gpt2/modeling_flax_vit_gpt2.py#L28) (ViT + [GPT-2 without LM head]) with an extra LM head. However, when loading the pretrained French GPT-2 model, the LM head's weigths are not loaded. We therefore created [**FlaxViTGPT2LMForConditionalGenerationModule**](https://huggingface.co/flax-community/vit-gpt2/blob/main/vit_gpt2/modeling_flax_vit_gpt2_lm.py#L101) which is `ViT + [GPT-2 with LM head]`, and we no longer need to add a LM head over it. By doing so, the pretrained LM head's weights are also loaded, and the only randomly initialized weigths are the cross-attention weights.
- The provided training script `run_summarization.py` is modified to send pixel values to the model instead of a sequence of input token ids, and a necessary change due to the ViT model not accepting an `attention_mask` argument.
- We first tried to use [WIT : Wikipedia-based Image Text Dataset](https://github.com/google-research-datasets/wit), but found it is a very changeling task since, unlike traditional image captioning tasks, it requires the model to be able to generate different texts even if two images are similar (for example, two famous dogs might have completely different Wikipedia texts).
- We finally decided to use [COCO image dataset](https://cocodataset.org/#home) at the final day of this Flax community event. We were able to translate only about 65000 examples to French for training, and the model is trained for only 5 epochs (beyond this, it started to overfit). This leads to the poor performance.


A HuggingFace Spaces demo for this model: [🖼 French Image Captioning Demo 📝](https://huggingface.co/spaces/flax-community/image-caption-french)


model name: vit-gpt2


model author: flax-community


tags: tensorboard


language: en
tags:


- summarization
license: apache-2.0
datasets:
- cnn_dailymail


# Model


This model is fine-tuned from https://huggingface.co/flax-community/t5-base-openwebtext, fine-tuned on cnn_dailymail.


model name: t5-base-cnn-dm


task: summarization


tags: pytorch, jax, tensorboard, t5, text2text-generation, en, dataset:cnn_dailymail, transformers, summarization, license:apache-2.0, autonlp_compatible


language: es
license: cc-by-4.0
tags:


- spanish
- roberta
- vit


# CLIP-Spanish


CLIP Spanish is a CLIP-like model for Spanish language. It is composed of [BERTIN](https://huggingface.co/bertin-project/bertin-roberta-base-spanish) as a language encoder and the ViT-B/32 image encoder from [CLIP](https://huggingface.co/openai/clip-vit-base-patch32). The model is implemented in [Flax](https://github.com/google/flax), including training scripts (see `training.md`).
This is part of the [Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organised by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google.


## Spanish WIT


We used a subset of 141,230 Spanish captions from the [WIT dataset](https://github.com/google-research-datasets/wit) for training.


## Team members


- Eduardo González Ponferrada ([edugp](https://huggingface.co/edugp))
- Manu Romero ([mrm8488](https://huggingface.co/))
- María Grandury ([mariagrandury](https://huggingface.co/))


## Useful links


- [Community Week timeline](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104#summary-timeline-calendar-6)
- [Community Week README](https://github.com/huggingface/transformers/blob/master/examples/research_projects/jax-projects/README.md)
- [Community Week thread](https://discuss.huggingface.co/t/bertin-pretrain-roberta-large-from-scratch-in-spanish/7125)
- [Community Week channel](https://discord.com/channels/858019234139602994/859113060068229190)
- [Hybrid CLIP example scripts](https://github.com/huggingface/transformers/tree/master/examples/research_projects/jax-projects/hybrid_clip)
- [Model Repository](https://huggingface.co/flax-community/bertin-roberta-large-spanish/)


model name: clip-spanish


model author: flax-community


tags: jax, hybrid-clip, es, transformers, license:cc-by-4.0, spanish, roberta, vit


- multilingual
- bert
pipeline_tag: fill-mask
widget:
- text: ¿Qué es la vida? Un [MASK].


# ALBERTI


ALBERTI is a set of two BERT-based multilingual model for poetry. One for verses and another one for stanzas. This model has been further trained with the PULPO corpus for verses using [Flax](https://github.com/google/flax), including training scripts.


This is part of the
[Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organised by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google.


## PULPO


PULPO, the Prodigious Unannotated Literary Poetry Corpus, is a set of multilingual corpora of verses and stanzas with over 95M words.


The following corpora has been downloaded using the [Averell](https://github.com/linhd-postdata/averell/) tool, developed by the [POSTDATA](https://postdata.linhd.uned.es/) team:


### Spanish


- [Disco v3](https://github.com/pruizf/disco)
- [Corpus of Spanish Golden-Age Sonnets](https://github.com/bncolorado/CorpusSonetosSigloDeOro)
- [Corpus general de poesía lírica castellana del Siglo de Oro](https://github.com/bncolorado/CorpusGeneralPoesiaLiricaCastellanaDelSigloDeOro)
- [Gongocorpus](https://github.com/linhd-postdata/gongocorpus) - [source](http://obvil.sorbonne-universite.site/corpus/gongora/gongora_obra-poetica)


### English


- [Eighteenth-Century Poetry Archive (ECPA)](https://github.com/alhuber1502/ECPA)
- [For better for verse](https://github.com/waynegraham/for_better_for_verse)


### French


- [Métrique en Ligne](https://crisco2.unicaen.fr/verlaine/index.php?navigation=accueil) - [source](https://github.com/linhd-postdata/metrique-en-ligne)


### Italian


- [Biblioteca italiana](https://github.com/linhd-postdata/biblioteca_italiana) - [source](http://www.bibliotecaitaliana.it/)


### Czech


- [Corpus of Czech Verse](https://github.com/versotym/corpusCzechVerse)


### Portuguese


- [Stichotheque](https://gitlab.com/stichotheque/stichotheque-pt)


Also, we obtained the following corpora from these sources:


- [Poesi.as](https://github.com/linhd-postdata/poesi.as) - [source](http://www.poesi.as/)


- [A Gutenberg Poetry Corpus](https://github.com/aparrish/gutenberg-poetry-corpus)


### Arabic


- [Arabic Poetry dataset](https://www.kaggle.com/ahmedabelal/arabic-poetry)


### Chinese


- [THU Chinese Classical Poetry Corpus](https://github.com/THUNLP-AIPoet/Datasets/tree/master/CCPC)


### Finnish


- [SKVR](https://github.com/sks190/SKVR)


### German


- [TextGrid Poetry Corpus](https://github.com/linhd-postdata/textgrid-poetry) - [source](https://textgrid.de/en/digitale-bibliothek)
- [German Rhyme Corpus](https://github.com/tnhaider/german-rhyme-corpus)


### Hungarian


- [verskorpusz](https://github.com/ELTE-DH/verskorpusz)


- [Poems in Portuguese](https://www.kaggle.com/oliveirasp6/poems-in-portuguese)


### Russian


- [19 000 Russian poems](https://www.kaggle.com/grafstor/19-000-russian-poems)


## Team members


- Álvaro Pérez ([alvp](https://huggingface.co/alvp))
- Javier de la Rosa ([versae](https://huggingface.co/versae))
- Aitor Díaz ([aitordiaz](https://huggingface.co/aitordiaz))
- Elena González-Blanco
- Salvador Ros ([salva](https://huggingface.co/salva))


## Useful links


- [Community Week timeline](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104#summary-timeline-calendar-6)
- [Community Week README](https://github.com/huggingface/transformers/blob/master/examples/research_projects/jax-projects/README.md)
- [Community Week thread](https://discuss.huggingface.co/t/bertin-pretrain-roberta-large-from-scratch-in-spanish/7125)
- [Community Week channel](https://discord.com/channels/858019234139602994/859113060068229190)
- [Masked Language Modelling example scripts](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)
- [Model Repository](https://huggingface.co/flax-community/alberti-bert-base-multilingual-cased/)


## Acknowledgments


This project would not have been possible without the infrastructure and resources provided by HuggingFace and Google Cloud. Moreover, we want to thank POSTDATA Project (ERC-StG-679528) and the Computational Literary Studies Infrastructure (CLS INFRA No. 101004984) of the European Union's Horizon 2020 research and innovation programme for their support and time allowance.


model name: alberti-bert-base-multilingual-cased


model author: flax-community


task: fill-mask


tags: pytorch, jax, joblib, bert, fill-mask, es, transformers, license:cc-by-4.0, multilingual, autonlp_compatible, infinity_compatible


language: es
tags:


- T5
- Seq2Seq
- EconderDecoder
- Spanish
datasets:
- large_spanish_corpus
widgets:
- text: "Érase un vez un"


license: mit


# Spanish T5 (small) trained on [large_spanish_corpus](https://huggingface.co/datasets/viewer/?dataset=large_spanish_corpus).


This is a Spanish **T5** (small arch) trained from scratch on the [large_spanish_corpus](https://huggingface.co/datasets/viewer/?dataset=large_spanish_corpus) aka BETO's corpus with [Flax](https://github.com/google/flax)


This is part of the
[Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organised by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google.


## Dataset


The dataset is about 20 GB. 95% of the data was used for training and the rest 5% for validation.


## [Metrics](https://huggingface.co/flax-community/spanish-t5-small/tensorboard) (on evaluation dataset)


- Accuracy: 0.675


## Team members


- Manuel Romero ([mrm8488](https://huggingface.co/mrm8488))
- María Grandury ([mariagrandury](https://huggingface.co/mariagrandury))


model name: spanish-t5-small


task: text2text-generation


tags: pytorch, jax, tensorboard, t5, text2text-generation, es, dataset:large_spanish_corpus, transformers, T5, Seq2Seq, EconderDecoder, Spanish, license:mit, autonlp_compatible


datasets:


- wiki_split
widget:
- text: "Mary likes to play football in her freetime whenever she meets with her friends that are very nice people."


# T5 model for sentence splitting in English


Sentence Split is the task of dividing a long sentence into multiple sentences.
E.g.:


```
Mary likes to play football in her freetime whenever she meets with her friends that are very nice people.
```


could be split into


```
Mary likes to play football in her freetime whenever she meets with her friends.
```


```
Her friends are very nice people.
```


## How to use it in your code:


```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("flax-community/byt5-base-wikisplit")
model = AutoModelForSeq2SeqLM.from_pretrained("flax-community/byt5-base-wikisplit")
complex_sentence = "This comedy drama is produced by Tidy , the company she co-founded in 2008 with her husband David Peet , who is managing director ."
sample_tokenized = tokenizer(complex_sentence, return_tensors="pt")
answer = model.generate(sample_tokenized['input_ids'], attention_mask = sample_tokenized['attention_mask'], max_length=256, num_beams=5)
gene_sentence = tokenizer.decode(answer[0], skip_special_tokens=True)
gene_sentence
"""
Output:
This comedy drama is produced by Tidy. She co-founded Tidy in 2008 with her husband David Peet, who is managing director.
"""
```


## Datasets:


## Current Basline from [paper](https://arxiv.org/abs/1907.12461)


![baseline](./baseline.png)


## Our Results:


| Model | Exact | SARI | BLEU |
| --- | --- | --- | --- |
| [t5-base-wikisplit](https://huggingface.co/flax-community/t5-base-wikisplit) |  17.93 | 67.5438 | 76.9 |
| [t5-v1_1-base-wikisplit](https://huggingface.co/flax-community/t5-v1_1-base-wikisplit) | 18.1207 | 67.4873 | 76.9478 |
| [byt5-base-wikisplit](https://huggingface.co/flax-community/byt5-base-wikisplit) | 11.3582 | 67.2685 | 73.1682 |
| [t5-large-wikisplit](https://huggingface.co/flax-community/t5-large-wikisplit) | 18.6632 | 68.0501 | 77.1881 |


model name: byt5-base-wikisplit


model author: flax-community


task: text2text-generation


tags: pytorch, tf, jax, tensorboard, t5, text2text-generation, dataset:wiki_split, arxiv:1907.12461, transformers, autonlp_compatible


language:


- en
tags:
- vision
license: apache-2.0


# MedCLIP


## Model description


## Intended uses & limitations


#### How to use


```python
# You can include sample code which will be formatted
```


#### Limitations and bias


Provide examples of latent issues and potential remediations.


## Training data


Describe the data you used to train the model.
If you initialized it with pre-trained weights, add a link to the pre-trained model card or repository with description of the pre-training data.


## Training procedure


Preprocessing, hardware used, hyperparameters...


## Eval results


### BibTeX entry and citation info


```bibtex
@inproceedings{...,
  year={2020}
}
```


model name: medclip


model author: flax-community


tags: jax, tensorboard, hybrid-clip, en, transformers, vision, license:apache-2.0


this is a test. How do you write a paper?


model name: test1


model author: Katiejdarby


tags:


language:


- ab
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_7_0
- generated_from_trainer
datasets:
- common_voice
model-index:
- name: ''
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


This model is a fine-tuned version of [hf-test/xls-r-dummy](https://huggingface.co/hf-test/xls-r-dummy) on the MOZILLA-FOUNDATION/COMMON_VOICE_7_0 - AB dataset.
It achieves the following results on the evaluation set:


- Loss: 156.8787
- Wer: 1.3460


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 2
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 10
- mixed_precision_training: Native AMP


### Training results


### Framework versions


- Transformers 4.16.0.dev0
- Pytorch 1.10.0
- Datasets 1.16.1
- Tokenizers 0.10.3


model name: xls-r-ab-test


model author: hf-test


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, ab, dataset:common_voice, transformers, mozilla-foundation/common_voice_7_0, generated_from_trainer, model-index


language:


- sv-SE
license: apache-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_7_0
- generated_from_trainer
- sv
- robust-speech-event
- model_for_talk
datasets:
- mozilla-foundation/common_voice_7_0
model-index:
- name: XLS-R-300M - Swedish
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 7
type: mozilla-foundation/common_voice_7_0
args: sv-SE
metrics:
    - name: Test WER
type: wer
value: 16.98
    - name: Test CER
type: cer
value: 5.66
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Robust Speech Event - Dev Data
type: speech-recognition-community-v2/dev_data
args: sv
metrics:
    - name: Test WER
type: wer
value: 27.01
    - name: Test CER
type: cer
value: 13.14


# XLS-R-300m-SV


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the MOZILLA-FOUNDATION/COMMON_VOICE_7_0 - SV-SE dataset.
It achieves the following results on the evaluation set:


- Loss: 0.3171
- Wer: 0.2468


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 7.5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 50.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 3.3349        | 1.45  | 500   | 3.2858          | 1.0    |
| 2.9298        | 2.91  | 1000  | 2.9225          | 1.0000 |
| 2.0839        | 4.36  | 1500  | 1.1546          | 0.8295 |
| 1.7093        | 5.81  | 2000  | 0.6827          | 0.5701 |
| 1.5855        | 7.27  | 2500  | 0.5597          | 0.4947 |
| 1.4831        | 8.72  | 3000  | 0.4923          | 0.4527 |
| 1.4416        | 10.17 | 3500  | 0.4670          | 0.4270 |
| 1.3848        | 11.63 | 4000  | 0.4341          | 0.3980 |
| 1.3749        | 13.08 | 4500  | 0.4203          | 0.4011 |
| 1.3311        | 14.53 | 5000  | 0.4310          | 0.3961 |
| 1.317         | 15.99 | 5500  | 0.3898          | 0.4322 |
| 1.2799        | 17.44 | 6000  | 0.3806          | 0.3572 |
| 1.2771        | 18.89 | 6500  | 0.3828          | 0.3427 |
| 1.2451        | 20.35 | 7000  | 0.3702          | 0.3359 |
| 1.2182        | 21.8  | 7500  | 0.3685          | 0.3270 |
| 1.2152        | 23.26 | 8000  | 0.3650          | 0.3308 |
| 1.1837        | 24.71 | 8500  | 0.3568          | 0.3187 |
| 1.1721        | 26.16 | 9000  | 0.3659          | 0.3249 |
| 1.1764        | 27.61 | 9500  | 0.3547          | 0.3145 |
| 1.1606        | 29.07 | 10000 | 0.3514          | 0.3104 |
| 1.1431        | 30.52 | 10500 | 0.3469          | 0.3062 |
| 1.1047        | 31.97 | 11000 | 0.3313          | 0.2979 |
| 1.1315        | 33.43 | 11500 | 0.3298          | 0.2992 |
| 1.1022        | 34.88 | 12000 | 0.3296          | 0.2973 |
| 1.0935        | 36.34 | 12500 | 0.3278          | 0.2926 |
| 1.0676        | 37.79 | 13000 | 0.3208          | 0.2868 |
| 1.0571        | 39.24 | 13500 | 0.3322          | 0.2885 |
| 1.0536        | 40.7  | 14000 | 0.3245          | 0.2831 |
| 1.0525        | 42.15 | 14500 | 0.3285          | 0.2826 |
| 1.0464        | 43.6  | 15000 | 0.3223          | 0.2796 |
| 1.0415        | 45.06 | 15500 | 0.3166          | 0.2774 |
| 1.0356        | 46.51 | 16000 | 0.3177          | 0.2746 |
| 1.04          | 47.96 | 16500 | 0.3150          | 0.2735 |
| 1.0209        | 49.42 | 17000 | 0.3175          | 0.2731 |


### Framework versions


- Transformers 4.16.0.dev0
- Pytorch 1.10.0+cu102
- Datasets 1.17.1.dev0
- Tokenizers 0.10.3


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_7_0` with split `test`


```bash
python eval.py --model_id hf-test/xls-r-300m-sv --dataset mozilla-foundation/common_voice_7_0 --config sv-SE --split test
```


2. To evaluate on `speech-recognition-community-v2/dev_data`


```bash
python eval.py --model_id hf-test/xls-r-300m-sv --dataset speech-recognition-community-v2/dev_data --config sv --split validation --chunk_length_s 5.0 --stride_length_s 1.0
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F

model_id = "hf-test/xls-r-300m-sv"

sample_iter = iter(load_dataset("mozilla-foundation/common_voice_7_0", "sv-SE", split="test", streaming=True, use_auth_token=True))

sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()

model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)

input_values = processor(resampled_audio, return_tensors="pt").input_values

with torch.no_grad():
    logits = model(input_values).logits

transcription = processor.batch_decode(logits.numpy()).text
# => "jag lämnade grovjobbet åt honom"
```


### Eval results on Common Voice 7 "test" (WER):


| Without LM | With LM (run `./eval.py`) |
|---|---|
| 24.68 | 16.98 |


model name: xls-r-300m-sv


model author: hf-test


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, sv-SE, dataset:mozilla-foundation/common_voice_7_0, transformers, license:apache-2.0, mozilla-foundation/common_voice_7_0, generated_from_trainer, sv, robust-speech-event, model_for_talk, model-index


language:


- sv-SE
license: apache-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_8_0
- generated_from_trainer
- robust-speech-event
datasets:
- common_voice
model-index:
- name: XLS-R-300M - Swedish - CV8
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 8
type: mozilla-foundation/common_voice_8_0
args: sv-SE
metrics:
    - name: Test WER
type: wer
value: 17.10
    - name: Test CER
type: cer
value: 5.70
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Robust Speech Event - Dev Data
type: speech-recognition-community-v2/dev_data
args: sv
metrics:
    - name: Test WER
type: wer
value: 26.92
    - name: Test CER
type: cer
value: 12.53


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the MOZILLA-FOUNDATION/COMMON_VOICE_8_0 - SV-SE dataset.
It achieves the following results on the evaluation set:


**Without LM**:


- Wer: 0.2465
- Cer: 0.0717


**With LM**:


- Wer: 0.1710
- Cer: 0.0569


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 3.3224        | 1.37  | 500   | 3.2676          | 1.0    |
| 2.9319        | 2.74  | 1000  | 2.9287          | 1.0000 |
| 2.1173        | 4.11  | 1500  | 1.1478          | 0.8788 |
| 1.6973        | 5.48  | 2000  | 0.6749          | 0.6547 |
| 1.5865        | 6.85  | 2500  | 0.5500          | 0.5634 |
| 1.5094        | 8.22  | 3000  | 0.4840          | 0.5430 |
| 1.4644        | 9.59  | 3500  | 0.4844          | 0.4142 |
| 1.4061        | 10.96 | 4000  | 0.4356          | 0.3808 |
| 1.3584        | 12.33 | 4500  | 0.4192          | 0.3698 |
| 1.3438        | 13.7  | 5000  | 0.3980          | 0.3584 |
| 1.3332        | 15.07 | 5500  | 0.3896          | 0.3572 |
| 1.3025        | 16.44 | 6000  | 0.3835          | 0.3487 |
| 1.2979        | 17.81 | 6500  | 0.3781          | 0.3417 |
| 1.2736        | 19.18 | 7000  | 0.3734          | 0.3270 |
| 1.2415        | 20.55 | 7500  | 0.3637          | 0.3316 |
| 1.2255        | 21.92 | 8000  | 0.3546          | 0.3147 |
| 1.2193        | 23.29 | 8500  | 0.3524          | 0.3196 |
| 1.2104        | 24.66 | 9000  | 0.3403          | 0.3097 |
| 1.1965        | 26.03 | 9500  | 0.3508          | 0.3093 |
| 1.1976        | 27.4  | 10000 | 0.3419          | 0.3071 |
| 1.182         | 28.77 | 10500 | 0.3364          | 0.2963 |
| 1.158         | 30.14 | 11000 | 0.3338          | 0.2932 |
| 1.1414        | 31.51 | 11500 | 0.3376          | 0.2940 |
| 1.1402        | 32.88 | 12000 | 0.3370          | 0.2891 |
| 1.1213        | 34.25 | 12500 | 0.3201          | 0.2874 |
| 1.1207        | 35.62 | 13000 | 0.3261          | 0.2826 |
| 1.1074        | 36.98 | 13500 | 0.3117          | 0.2786 |
| 1.0818        | 38.36 | 14000 | 0.3194          | 0.2776 |
| 1.0889        | 39.73 | 14500 | 0.3188          | 0.2738 |
| 1.0672        | 41.1  | 15000 | 0.3196          | 0.2773 |
| 1.0838        | 42.47 | 15500 | 0.3130          | 0.2739 |
| 1.0553        | 43.83 | 16000 | 0.3165          | 0.2704 |
| 1.0786        | 45.21 | 16500 | 0.3108          | 0.2706 |
| 1.0546        | 46.57 | 17000 | 0.3102          | 0.2677 |
| 1.0425        | 47.94 | 17500 | 0.3115          | 0.2679 |
| 1.0398        | 49.31 | 18000 | 0.3131          | 0.2666 |


### Framework versions


- Transformers 4.16.0.dev0
- Pytorch 1.10.1+cu113
- Datasets 1.18.1.dev0
- Tokenizers 0.10.3


model name: xls-r-300m-sv-cv8


model author: hf-test


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, sv-SE, dataset:common_voice, transformers, license:apache-2.0, mozilla-foundation/common_voice_8_0, generated_from_trainer, robust-speech-event, model-index


language: "en"
#thumbnail: "url to a thumbnail used in social sharing"
tags:


- Android
- tflite
- blenderbot
license: "apache-2.0"
#datasets:
#metrics:


# Model Card


`blenderbot-small-tflite` is a tflite version of `blenderbot-small-90M` I converted for my UTA CSE3310 class. See the repo at [https://github.com/kmosoti/DesparadosAEYE](https://github.com/kmosoti/DesparadosAEYE) and the conversion process [here](https://drive.google.com/file/d/1F93nMsDIm1TWhn70FcLtcaKQUynHq9wS/view?usp=sharing).


You have to right pad your user and model input integers to make them [32,]-shaped. Then indicate te true length with the 3rd and 4th params.


```python
display(interpreter.get_input_details())
display(interpreter.get_output_details())
```


```json
[{'dtype': numpy.int32,
  'index': 0,
  'name': 'input_tokens',
  'quantization': (0.0, 0),
  'quantization_parameters': {'quantized_dimension': 0,
   'scales': array([], dtype=float32),
   'zero_points': array([], dtype=int32)},
  'shape': array([32], dtype=int32),
  'shape_signature': array([32], dtype=int32),
  'sparsity_parameters': {}},
 {'dtype': numpy.int32,
  'index': 1,
  'name': 'decoder_input_tokens',
  'quantization': (0.0, 0),
  'quantization_parameters': {'quantized_dimension': 0,
   'scales': array([], dtype=float32),
   'zero_points': array([], dtype=int32)},
  'shape': array([32], dtype=int32),
  'shape_signature': array([32], dtype=int32),
  'sparsity_parameters': {}},
 {'dtype': numpy.int32,
  'index': 2,
  'name': 'input_len',
  'quantization': (0.0, 0),
  'quantization_parameters': {'quantized_dimension': 0,
   'scales': array([], dtype=float32),
   'zero_points': array([], dtype=int32)},
  'shape': array([], dtype=int32),
  'shape_signature': array([], dtype=int32),
  'sparsity_parameters': {}},
 {'dtype': numpy.int32,
  'index': 3,
  'name': 'decoder_input_len',
  'quantization': (0.0, 0),
  'quantization_parameters': {'quantized_dimension': 0,
   'scales': array([], dtype=float32),
   'zero_points': array([], dtype=int32)},
  'shape': array([], dtype=int32),
  'shape_signature': array([], dtype=int32),
  'sparsity_parameters': {}}]
[{'dtype': numpy.int32,
  'index': 3113,
  'name': 'Identity',
  'quantization': (0.0, 0),
  'quantization_parameters': {'quantized_dimension': 0,
   'scales': array([], dtype=float32),
   'zero_points': array([], dtype=int32)},
  'shape': array([1], dtype=int32),
  'shape_signature': array([1], dtype=int32),
  'sparsity_parameters': {}}]
```


model name: blenderbot-small-tflite


model author: jacob-valdez


tags: tflite, en, Android, blenderbot, license:apache-2.0


A distilbert model fine-tuned for question answering.


model name: dbert-3epoch


model author: Ifenna


task: question-answering


tags: pytorch, distilbert, question-answering, transformers, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: distilbert-base-uncased-finetuned-squad
results: []


# distilbert-base-uncased-finetuned-squad


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the squad dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 8    | 5.8757          |


### Framework versions


- Transformers 4.16.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.17.0
- Tokenizers 0.11.0


model name: distilbert-base-uncased-finetuned-squad


model author: bhan


task: question-answering


tags: pytorch, tensorboard, distilbert, question-answering, dataset:squad, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


language:


- fr
license: apache-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_8_0
- fr
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
model-index:
- name: XLS-R-1B - French
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 8
type: mozilla-foundation/common_voice_8_0
args: fr
metrics:
    - name: Test WER (without LM)
type: wer
value: 15.405483405483406
    - name: Test CER (without LM)
type: cer
value: 4.8773030225289137
    - name: Test WER (with LM)
type: wer
value: 12.5


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b) on the MOZILLA-FOUNDATION/COMMON_VOICE_8_0 - FR dataset.
It achieves the following results on the evaluation set:


**Without LM**:


- Wer: 0.154
**With LM**:
- Wer: 0.125


model name: wav2vec2-xls-r-1b-fr


model author: samirt8


task: automatic-speech-recognition


tags: pytorch, fr, dataset:mozilla-foundation/common_voice_8_0, license:apache-2.0, automatic-speech-recognition, mozilla-foundation/common_voice_8_0, robust-speech-event, model-index


## title: ArcaneGAN
emoji: 🚀
colorFrom: blue
colorTo: blue
sdk: gradio
app_file: app.py
pinned: false


# Configuration


`title`: *string*\
Display title for the Space


`emoji`: *string*\
Space emoji (emoji-only character allowed)


`colorFrom`: *string*\
Color for Thumbnail gradient (red, yellow, green, blue, indigo, purple, pink, gray)


`colorTo`: *string*\
Color for Thumbnail gradient (red, yellow, green, blue, indigo, purple, pink, gray)


`sdk`: *string*\
Can be either `gradio` or `streamlit`


`sdk_version` : *string*\
Only applicable for `streamlit` SDK.\
See [doc](https://hf.co/docs/hub/spaces) for more info on supported versions.


`app_file`: *string*\
Path to your main application file (which contains either `gradio` or `streamlit` Python code).\
Path is relative to the root of the repository.


`pinned`: *boolean*\
Whether the Space stays on top of your list.


model name: arc


model author: tlanfer


tags:


language: en
tags:


- sagemaker
- bart
- summarization
license: apache-2.0
- Training 3000 examples


model name: SummaryGeneration-sagemaker3


model author: yair


task: text2text-generation


tags: pytorch, bart, text2text-generation, transformers, autonlp_compatible


hello


model name: HeadlineGeneration


- sagemaker
- bart
- summarization
license: apache-2.0


model name: HeadlineGeneration-sagemaker


model name: HeadlineGeneration-sagemaker2


language:


- fi
- en


- translation


## license: apache-2.0


model name: translation_model_by_subham


model author: subham92


task: translation


tags: pytorch, marian, text2text-generation, fi, en, transformers, translation, license:apache-2.0, autonlp_compatible


# PrivBERT


PrivBERT is a privacy policy language model. We pre-trained PrivBERT on ~1 million privacy policies starting with the pretrained Roberta model. The data is available at [https://privaseer.ist.psu.edu/data](https://privaseer.ist.psu.edu/data)


## Usage


```
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("mukund/privbert")
model = AutoModel.from_pretrained("mukund/privbert")
```


## License


If you use this dataset in research, you must cite the below paper.


```
Mukund Srinath, Shomir Wilson and C. Lee Giles. Privacy at Scale: Introducing the PrivaSeer Corpus of Web Privacy Policies. In Proc. ACL 2021.
```


For research, teaching, and scholarship purposes, the model is available under a CC BY-NC-SA license. Please contact us for any requests regarding commercial use.


model name: privbert


model author: mukund


task: fill-mask


tags: pytorch, tf, roberta, fill-mask, transformers, autonlp_compatible, infinity_compatible


# Supervised Continous Bag of words model trained with Uruguayan news from Twitter


Model trained with Facebook's fasttext library.


model name: cbow_uruguayan_news


model author: leandrodzp


tags:


- image-classification
- timm
library_tag: timm


# Model card for test_model_rnv250


model name: test_model_rnv250


model author: rwightman


task: image-classification


tags: pytorch, timm, image-classification


# Model card for test_model_rnv250b


model name: test_model_rnv250b


- conversational


# Game of thrones DialoGPT


model name: DialoGPT-small-scott


model author: log


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


aa


model name: Entitya


model author: rontom


tags:


## license: mit


model name: KcT5


model author: beomi


tags: license:mit


## language: ko


# Bert base model for Korean


## Update


- Update at 2021.11.17 : Add Native Support for BERT Tokenizer (works with AutoTokenizer, pipeline)


* 70GB Korean text dataset and 42000 lower-cased subwords are used
* Check the model performance and other language models for Korean in [github](https://github.com/kiyoungkim1/LM-kor)


```python
from transformers import pipeline

pipe = pipeline('text-generation', model='beomi/kykim-gpt3-kor-small_based_on_gpt2')
print(pipe("안녕하세요! 오늘은"))
# [{'generated_text': '안녕하세요! 오늘은 제가 요즘 사용하고 있는 클렌징워터를 소개해드리려고 해요! 바로 이 제품!! 바로 이'}]
```


model name: kykim-gpt3-kor-small_based_on_gpt2


task: text-generation


tags: pytorch, tf, jax, gpt2, text-generation, ko, transformers


model name: KcT5-dev


task: text2text-generation


tags: jax, t5, text2text-generation, transformers, license:mit, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- glue
metrics:
- matthews_correlation
model-index:
- name: distilbert-base-uncased-finetuned-cola
results:
  - task:
name: Text Classification
type: text-classification
dataset:
name: glue
type: glue
args: cola
metrics:
    - name: Matthews Correlation
type: matthews_correlation
value: 0.5552849676135797


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-finetuned-cola


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the glue dataset.
It achieves the following results on the evaluation set:


- Loss: 0.7525
- Matthews Correlation: 0.5553


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5


### Training results


| Training Loss | Epoch | Step | Validation Loss | Matthews Correlation |
|:-------------:|:-----:|:----:|:---------------:|:--------------------:|
| 0.523         | 1.0   | 535  | 0.5024          | 0.4160               |
| 0.3437        | 2.0   | 1070 | 0.5450          | 0.4965               |
| 0.2326        | 3.0   | 1605 | 0.6305          | 0.5189               |
| 0.177         | 4.0   | 2140 | 0.7525          | 0.5553               |
| 0.1354        | 5.0   | 2675 | 0.8630          | 0.5291               |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.9.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-cola


model author: beomi


task: text-classification


tags: pytorch, tensorboard, distilbert, text-classification, dataset:glue, transformers, license:apache-2.0, generated_from_trainer, model-index, infinity_compatible


language:


- en
- fr
- ro
- de
datasets:
- c4
tags:
- summarization
- translation


## license: apache-2.0


[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html)


Pretraining Dataset: [C4](https://huggingface.co/datasets/c4)


Other Community Checkpoints: [here](https://huggingface.co/models?search=t5)


Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)


Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*


## Abstract


Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.


model name: t5-small-pytorch


model author: seduerr


task: translation


tags: pytorch, t5, text2text-generation, en, fr, ro, de, dataset:c4, arxiv:1910.10683, transformers, summarization, translation, license:apache-2.0, autonlp_compatible


# T5 Base with Paraphrases in German Language


This T5 base model has been trained with the German part of the PAWS-X data set.
It can be used as any T5 model and will generated paraphrases with the prompt keyword: 'paraphrase: '**GermanSentence**


Please contact me, if you need more information (sduerr@mit.edu).


Thank you.


Sebastian


model name: t5_base_paws_ger


task: text2text-generation


tags: pytorch, t5, text2text-generation, transformers, autonlp_compatible


language: en
pipeline_tag: zero-shot-classification
tags:


- squeezebert
datasets:
- mulit_nli
metrics:
- accuracy


# SqueezeBERT


model name: paiintent


task: zero-shot-classification


tags: pytorch, squeezebert, en, dataset:mulit_nli, transformers, zero-shot-classification


# Invoking more Creativity with Pawraphrases based on T5


## This micro-service allows to find paraphrases for a given text based on T5.


We explain how we finetune the architecture T5 with the dataset PAWS (both from Google) to get the capability of creating paraphrases (or pawphrases since we are using the PAWS dataset :smile:). With this, we can create paraphrases for any given textual input.


Find the code for the service in this [Github Repository](https://github.com/seduerr91/pawraphrase_public). In order to create your own **'pawrasphrase tool'**, follow these steps:


### Step 1: Find a Useful Architecture and Datasets


Since Google's [T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) has been trained on multiple tasks (e.g., text summarization, question-answering) and it is solely based on Text-to-Text tasks it is pretty useful for extending its task-base through finetuning it with paraphrases.
Luckily, the [PAWS](https://github.com/google-research-datasets/paws) dataset consists of approximately 50.000 labeled paraphrases that we can use to fine-tune T5.


### Step 2: Prepare the PAWS Dataset for the T5 Architecture


Once identified, it is crucial to prepare the PAWS dataset to feed it into the T5 architecture for finetuning. Since PAWS is coming both with paraphrases and non-paraphases, it needs to be filtered for paraphrases only. Next, after packing it into a Pandas DataFrame, the necessary table headers had to be created. Next, you split the resulting training samples into test, train, and validation set.


### Step 3: Fine-tune T5-base with PAWS


Next, following these [training instructions](https://towardsdatascience.com/paraphrase-any-question-with-t5-text-to-text-transfer-transformer-pretrained-model-and-cbb9e35f1555), in which they used the Quora dataset, we use the PAWS dataset and feed into T5. Central is the following code to ensure that T5 understands that it has to *paraphrase*. The adapted version can be found [here](https://github.com/seduerr91/pawraphrase_public/blob/master/t5_pawraphrase_training.ipynb).


Additionally, it is helpful to force the old versions of *torch==1.4.0, transformers==2.9.0* and *pytorch_lightning==0.7.5*, since the newer versions break (trust me, I am speaking from experience). However, when doing such training, it is straightforward to start with the smallest architecture (here, *T5-small*) and a small version of your dataset (e.g., 100 paraphrase examples) to quickly identify where the training may fail or stop.


### Step 4: Start Inference by yourself.


Next, you can use the fine-tuned T5 Architecture to create paraphrases from every input. As seen in the introductory image. The corresponding code can be found [here](https://github.com/seduerr91/pawraphrase_public/blob/master/t5_pawraphrase_inference.ipynb).


### Step 5: Using the fine-tuning through a GUI


Finally, to make the service useful we can provide it as an API as done with the infilling model [here](https://seduerr91.github.io/blog/ilm-fastapi) or with this [frontend](https://github.com/renatoviolin/T5-paraphrase-generation) which was prepared by Renato. Kudos!


Thank you for reading this article. I'd be curious about your opinion.


#### Who am I?


I am Sebastian an NLP Deep Learning Research Scientist (M.Sc. in IT and Business). In my former life, I was a manager at Austria's biggest bank. In the future, I want to work remotely flexibly & in the field of NLP.


Drop me a message on [LinkedIn](https://www.linkedin.com/in/sebastianduerr/) if you want to get in touch!


model name: t5-pawraphrase


model author: seduerr


task: text2text-generation


tags: pytorch, t5, text2text-generation, transformers, autonlp_compatible


input_ = paraphrase: + str(input_) + ' </s>'


model name: pai_paraph


hello
hello


model name: sentiment


language:


- en
- fr
- ro
- de
datasets:
- c4
tags:
- summarization
- translation


## license: apache-2.0


[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html)


Pretraining Dataset: [C4](https://huggingface.co/datasets/c4)


Other Community Checkpoints: [here](https://huggingface.co/models?search=t5)


Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)


Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*


## Abstract


Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.


model name: pai-tl


model author: seduerr


task: translation


tags: pytorch, t5, text2text-generation, en, fr, ro, de, dataset:c4, arxiv:1910.10683, transformers, summarization, translation, license:apache-2.0, autonlp_compatible


‘contrast: ‘


model name: pai_con


task: text2text-generation


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: distilbert-base-uncased-finetuned-squad
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-finetuned-squad


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the squad dataset.
It achieves the following results on the evaluation set:


- Loss: 1.1639


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 1.2291        | 1.0   | 5533  | 1.1581          |
| 0.9553        | 2.0   | 11066 | 1.1249          |
| 0.7767        | 3.0   | 16599 | 1.1639          |


### Framework versions


- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.15.1
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-squad


model author: kaggleodin


task: question-answering


tags: pytorch, tensorboard, distilbert, question-answering, dataset:squad, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


tags:


- generated_from_trainer
datasets:
- squad_v2
model-index:
- name: biobert_v1.1_pubmed-finetuned-squad
results: []


# biobert_v1.1_pubmed-finetuned-squad


This model is a fine-tuned version of [gerardozq/biobert_v1.1_pubmed-finetuned-squad](https://huggingface.co/gerardozq/biobert_v1.1_pubmed-finetuned-squad) on the squad_v2 dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1


### Framework versions


- Transformers 4.12.3
- Pytorch 1.9.0+cu111
- Datasets 1.15.1
- Tokenizers 0.10.3


model name: biobert_v1.1_pubmed-finetuned-squad


model author: gerardozq


task: question-answering


tags: pytorch, tensorboard, bert, question-answering, dataset:squad_v2, transformers, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


tags:


- conversational


# Rick Sanchez DialoGPT Model


model name: DialoGPT-Rick-Sanchez


model author: Rayane


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


## Title Generator


References this [notebook](https://shivanandroy.com/transformers-generating-arxiv-papers-title-from-abstracts/)


Using `t5-small`, trained on a batch size of 16 for 4 epochs, utilising the ArXiV dataset through the `SimpleTransformers` library. Around 15k data was used for training and 3.7k data for evaluation.


This is a `.pkl` file.


### Prerequisites


Install `simpletransformers` library.


```bsh
pip install simpletransformers
```


### Example Usage


```py
import pickle

model = pickle.load(open("title-generator-t5-arxiv-16-4.pkl", "rb"))

# Prefix your text with 'summarize: '
text = ["summarize: " + """Venetian commodes imitated the curving lines and carved ornament of the French rocaille, but with a particular Venetian variation; the pieces were painted, often with landscapes or flowers or scenes from Guardi or other painters, or Chinoiserie, against a blue or green background, matching the colours of the Venetian school of painters whose work decorated the salons. 24] Ceiling of church of Santi Giovanni e Paolo in Venice, by Piazzetta (1727) Juno and Luna by Giovanni Battista Tiepolo (1735–45) Murano glass chandelier at the Ca Rezzonico (1758) Ballroom ceiling of the Ca Rezzonico with ceiling by Giovanni Battista Crosato (1753) In church construction, especially in the southern German-Austrian region, gigantic spatial creations are sometimes created for practical reasons alone, which, however, do not appear monumental, but are characterized by a unique fusion of architecture, painting, stucco, etc. ,."""]

print("Generated title: " + model.predict(text))
```


model name: title-generator-t5


model author: lianaling


tags:


tags: autonlp
language: de
widget:


- text: "I love AutoNLP 🤗"
datasets:
- muhtasham/autonlp-data-Doctor_DE
co2_eq_emissions: 92.87363201770962


# Model Trained Using AutoNLP


- Problem type: Single Column Regression
- Model ID: 24595544
- CO2 Emissions (in grams): 92.87363201770962


## Validation Metrics


- Loss: 0.3001164197921753
- MSE: 0.3001164197921753
- MAE: 0.24272102117538452
- R2: 0.8465975006681247
- RMSE: 0.5478288531303406
- Explained Variance: 0.8468209505081177


## Usage


You can use cURL to access this model: