The model was trained on the `full` mc4_nl_cleaned dataset configuration for 1 epoch, consisting of 34B tokens,
for 528 482 steps with a batch size of 128 and took 57 hours.
A triangle learning rate schedule was used, with peak learning rate 0.005.


## Evaluation


* Loss: 1.38
* Accuracy: 0.70


model name: t5-base-dutch


model author: flax-community


task: text2text-generation


tags: pytorch, tf, jax, tensorboard, t5, text2text-generation, dutch, dataset:yhavinga/mc4_nl_cleaned, transformers, seq2seq, lm-head, license:apache-2.0, autonlp_compatible


language: no
widget:


- text: "Det er flott"


# GPT2-svenska-wikipedia


A norwegian GPT2 style model trained using Flax CLM pipeline on the Norwegian
part of the wiki40b dataset.


## Model series


This model is part of a series of models training on TPU with Flax Jax during Huggingface Flax/Jax challenge.


## Gpt models


## Swedish Gpt


## Swedish gpt wiki


# Nordic gpt wiki


## Dansk gpt wiki


## Norsk gpt wiki


## Roberta models


## Nordic Roberta Wiki


## Swe Roberta Wiki Oscar


## Roberta Swedish Scandi


## Roberta Swedish


## Swedish T5 model


## Data cleaning and preprocessing


The data was cleaned and preprocessed using the following script. Make sure to install depencies for beam_runner to make the dataset work.


```python
from datasets import load_dataset
def load_and_clean_wiki():
    dataset = load_dataset('wiki40b', 'no', beam_runner='DirectRunner', split="train")
    #dataset = load_dataset('wiki40b', 'sv', beam_runner='DirectRunner')
    dataset = dataset.remove_columns(['wikidata_id', 'version_id'])
    filtered_dataset = dataset.map(filter_wikipedia)
    # filtered_dataset[:3]
    # print(filtered_dataset[:3])
    return filtered_dataset

def filter_wikipedia(batch):
    batch["text"] = " ".join(batch["text"].split("\
_START_SECTION_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_ARTICLE_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_ARTICLE_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_PARAGRAPH_\
"))
    batch["text"] = " ".join(batch["text"].split("_NEWLINE_"))
    batch["text"] = " ".join(batch["text"].split("\xa0"))
    return batch
```


## Training script


The following training script was used to train the model.


```bash
./run_clm_flax.py     --output_dir="${MODEL_DIR}"     --model_type="gpt2"     --config_name="${MODEL_DIR}"     --tokenizer_name="${MODEL_DIR}"     --dataset_name="wiki40b"     --dataset_config_name="no"     --do_train --do_eval     --block_size="512"     --per_device_train_batch_size="64"     --per_device_eval_batch_size="64"     --learning_rate="5e-3" --warmup_steps="1000"     --adam_beta1="0.9" --adam_beta2="0.98" --weight_decay="0.01"     --overwrite_output_dir     --num_train_epochs="20"     --logging_steps="500"     --save_steps="1000"     --eval_steps="2500"     --push_to_hub
```


model name: norsk-gpt-wiki


model author: flax-community


task: text-generation


tags: pytorch, jax, tensorboard, gpt2, text-generation, no, transformers


tags:


- vision


# Model Card: clip-rsicd


## Model Details


This model is a finetuned [CLIP by OpenAI](https://huggingface.co/openai/clip-vit-base-patch32). It is designed with an aim to improve zero-shot image classification, text-to-image and image-to-image retrieval specifically on remote sensing images.


### Model Date


July 2021


### Model Type


The base model uses a ViT-B/32 Transformer architecture as an image encoder and uses a masked self-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss.


### Model Version


We release several checkpoints for `clip-rsicd` model. Refer to [our github repo](https://github.com/arampacha/CLIP-rsicd#evaluation-results) for performance metrics on zero-shot classification for each of those.


### Training


To reproduce the fine-tuning procedure one can use released [script](https://github.com/arampacha/CLIP-rsicd/blob/master/run_clip_flax_tv.py).
The model was trained using batch size 1024, adafactor optimizer with linear warmup and decay with peak learning rate 1e-4 on 1 TPU-v3-8.
Full log of the training run can be found on [WandB](https://wandb.ai/wandb/hf-flax-clip-rsicd/runs/1ts243k3).


### Demo


Check out the model text-to-image and image-to-image capabilities using [this demo](https://huggingface.co/spaces/sujitpal/clip-rsicd-demo).


### Documents


- [Fine-tuning CLIP on RSICD with HuggingFace and flax/jax on colab using TPU](https://colab.research.google.com/github/arampacha/CLIP-rsicd/blob/master/nbs/Finetuning_CLIP_with_HF_and_jax.ipynb)


### Use with Transformers


```py
from PIL import Image
import requests

from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("flax-community/clip-rsicd")
processor = CLIPProcessor.from_pretrained("flax-community/clip-rsicd")

url = "https://raw.githubusercontent.com/arampacha/CLIP-rsicd/master/data/stadium_1.jpg"
image = Image.open(requests.get(url, stream=True).raw)

labels = ["residential area", "playground", "stadium", "forrest", "airport"]
inputs = processor(text=[f"a photo of a {l}" for l in labels], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs.logits_per_image # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities
for l, p in zip(labels, probs[0]):
    print(f"{l:<16} {p:.4f}")
```


[Try it on colab](https://colab.research.google.com/github/arampacha/CLIP-rsicd/blob/master/nbs/clip_rsicd_zero_shot.ipynb)


## Model Use


### Intended Use


The model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero-shot, arbitrary image classification.


In addition, we can imagine applications in defense and law enforcement, climate change and global warming, and even some consumer applications. A partial list of applications can be found [here](https://github.com/arampacha/CLIP-rsicd#applications). In general we think such models can be useful as digital assistants for humans engaged in searching through large collections of images.


We also hope it can be used for interdisciplinary studies of the potential impact of such models - the CLIP paper includes a discussion of potential downstream impacts to provide an example for this sort of analysis.


#### Primary intended uses


The primary intended users of these models are AI researchers.


We primarily imagine the model will be used by researchers to better understand robustness, generalization, and other capabilities, biases, and constraints of computer vision models.


## Data


The model was trained on publicly available remote sensing image captions datasets. Namely [RSICD](https://github.com/201528014227051/RSICD_optimal), [UCM](https://mega.nz/folder/wCpSzSoS#RXzIlrv--TDt3ENZdKN8JA) and [Sydney](https://mega.nz/folder/pG4yTYYA#4c4buNFLibryZnlujsrwEQ). More information on the datasets used can be found on [our project page](https://github.com/arampacha/CLIP-rsicd#dataset).


## Performance and Limitations


### Performance


| Model-name                       | k=1   | k=3   | k=5   | k=10  |
| -------------------------------- | ----- | ----- | ----- | ----- |
| original CLIP                    | 0.572 | 0.745 | 0.837 | 0.939 |
| clip-rsicd (this model)          | 0.843 | 0.958 | 0.977 | 0.993 |


## Limitations


The model is finetuned on RSI data but can contain some biases and limitations of the original CLIP model. Refer to [CLIP model card](https://huggingface.co/openai/clip-vit-base-patch32#limitations) for details on those.


model name: clip-rsicd


model author: flax-community


task: feature-extraction


tags: pytorch, jax, clip, feature-extraction, transformers, vision


language: sw
widget:


- text: "Idris ameandika kwenye ukurasa wake wa Instagram akimkumbusha Diamond kutekeleza ahadi yake kumpigia Zari magoti kumuomba msamaha kama alivyowahi kueleza awali.Idris ameandika;"
datasets:
- flax-community/swahili-safi


## Swahili News Classification with BERT


This model was trained using HuggingFace's Flax framework and is part of the [JAX/Flax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104) organized by [HuggingFace](https://huggingface.co). All training was done on a TPUv3-8 VM sponsored by the Google Cloud team.


This [model](https://huggingface.co/flax-community/bert-base-uncased-swahili) was used as the base and fine-tuned for this task.


## How to use


```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("flax-community/bert-swahili-news-classification")

model = AutoModelForSequenceClassification.from_pretrained("flax-community/bert-swahili-news-classification")
```


```
Eval metrics (10% valid set): {'accuracy': 0.9114740008594757}
```


model name: bert-swahili-news-classification


task: text-classification


tags: pytorch, jax, tensorboard, bert, text-classification, sw, dataset:flax-community/swahili-safi, transformers, infinity_compatible


widget:


- text: "मुझे उनसे बात करना <mask> अच्छा लगा"
- text: "हम आपके सुखद <mask> की कामना करते हैं"
- text: "सभी अच्छी चीजों का एक <mask> होता है"


# RoBERTa base model for Hindi language


Pretrained model on Hindi language using a masked language modeling (MLM) objective. [A more interactive & comparison demo is available here](https://huggingface.co/spaces/flax-community/roberta-hindi).


> This is part of the
[Flax/Jax Community Week](https://discuss.huggingface.co/t/pretrain-roberta-from-scratch-in-hindi/7091), organized by [Hugging Face](https://huggingface.co/) and TPU usage sponsored by Google.


## Model description


RoBERTa Hindi is a transformers model pretrained on a large corpus of Hindi data(a combination of **mc4, oscar and indic-nlp** datasets)


### How to use


You can use this model directly with a pipeline for masked language modeling:


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='flax-community/roberta-hindi')
>>> unmasker("हम आपके सुखद <mask> की कामना करते हैं")
[{'score': 0.3310680091381073,
  'sequence': 'हम आपके सुखद सफर की कामना करते हैं',
  'token': 1349,
  'token_str': ' सफर'},
 {'score': 0.15317578613758087,
  'sequence': 'हम आपके सुखद पल की कामना करते हैं',
  'token': 848,
  'token_str': ' पल'},
 {'score': 0.07826550304889679,
  'sequence': 'हम आपके सुखद समय की कामना करते हैं',
  'token': 453,
  'token_str': ' समय'},
 {'score': 0.06304813921451569,
  'sequence': 'हम आपके सुखद पहल की कामना करते हैं',
  'token': 404,
  'token_str': ' पहल'},
 {'score': 0.058322224766016006,
  'sequence': 'हम आपके सुखद अवसर की कामना करते हैं',
  'token': 857,
  'token_str': ' अवसर'}]
```


## Training data


The RoBERTa Hindi model was pretrained on the reunion of the following datasets:


- [OSCAR](https://huggingface.co/datasets/oscar) is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the goclassy architecture.
- [mC4](https://huggingface.co/datasets/mc4) is a multilingual colossal, cleaned version of Common Crawl's web crawl corpus.
- [IndicGLUE](https://indicnlp.ai4bharat.org/indic-glue/) is a natural language understanding benchmark.
- [Samanantar](https://indicnlp.ai4bharat.org/samanantar/) is a parallel corpora collection for Indic language.
- [Hindi Text Short and Large Summarization Corpus](https://www.kaggle.com/disisbig/hindi-text-short-and-large-summarization-corpus) is a collection of ~180k articles with their headlines and summary collected from Hindi News Websites.
- [Hindi Text Short Summarization Corpus](https://www.kaggle.com/disisbig/hindi-text-short-summarization-corpus) is a collection of ~330k articles with their headlines collected from Hindi News Websites.
- [Old Newspapers Hindi](https://www.kaggle.com/crazydiv/oldnewspapershindi) is a cleaned subset of HC Corpora newspapers.


## Training procedure


### Preprocessing


The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50265. The inputs of
the model take pieces of 512 contiguous token that may span over documents. The beginning of a new document is marked
with `<s>` and the end of one by `</s>`.


- We had to perform cleanup of **mC4** and **oscar** datasets by removing all non hindi (non Devanagari) characters from the datasets.
- We tried to filter out evaluation set of WikiNER of [IndicGlue](https://indicnlp.ai4bharat.org/indic-glue/) benchmark by [manual labelling](https://github.com/amankhandelia/roberta_hindi/blob/master/wikiner_incorrect_eval_set.csv)  where the actual labels were not correct and modifying the [downstream evaluation dataset](https://github.com/amankhandelia/roberta_hindi/blob/master/utils.py).


The details of the masking procedure for each sentence are the following:


- 15% of the tokens are masked.
- In 80% of the cases, the masked tokens are replaced by `<mask>`.
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
- In the 10% remaining cases, the masked tokens are left as is.
Contrary to BERT, the masking is done dynamically during pretraining (e.g., it changes at each epoch and is not fixed).


### Pretraining


The model was trained on Google Cloud Engine TPUv3-8 machine (with 335 GB of RAM, 1000 GB of hard drive, 96 CPU cores).A randomized shuffle of combined dataset of **mC4, oscar** and other datasets listed above was used to train the model. Training logs are present in [wandb](https://wandb.ai/wandb/hf-flax-roberta-hindi).


## Evaluation Results


RoBERTa Hindi is evaluated on various downstream tasks. The results are summarized below.


| Task                    | Task Type            | IndicBERT | HindiBERTa | Indic Transformers Hindi BERT | RoBERTa Hindi Guj San | RoBERTa Hindi |
|-------------------------|----------------------|-----------|------------|-------------------------------|-----------------------|---------------|
| BBC News Classification | Genre Classification | **76.44**     | 66.86      | **77.6**                          | 64.9                  | 73.67         |
| WikiNER                 | Token Classification | -         | 90.68      | **95.09**                         | 89.61                 | **92.76**         |
| IITP Product Reviews    | Sentiment Analysis   | **78.01**     | 73.23      | **78.39**                         | 66.16                 | 75.53         |
| IITP Movie Reviews      | Sentiment Analysis   | 60.97     | 52.26      | **70.65**                         | 49.35                 | **61.29**         |


## Team Members


- Aman K ([amankhandelia](https://huggingface.co/amankhandelia))
- Haswanth Aekula ([hassiahk](https://huggingface.co/hassiahk))
- Kartik Godawat ([dk-crazydiv](https://huggingface.co/dk-crazydiv))
- Prateek Agrawal ([prateekagrawal](https://huggingface.co/prateekagrawal))
- Rahul Dev ([mlkorra](https://huggingface.co/mlkorra))


## Credits


Huge thanks to Hugging Face 🤗 & Google Jax/Flax team for such a wonderful community week, especially for providing such massive computing resources. Big thanks to [Suraj Patil](https://huggingface.co/valhalla) & [Patrick von Platen](https://huggingface.co/patrickvonplaten) for mentoring during the whole week.


<img src=https://pbs.twimg.com/media/E443fPjX0AY1BsR.jpg:medium>


model name: roberta-hindi


model author: flax-community


task: fill-mask


tags: pytorch, jax, tensorboard, roberta, fill-mask, transformers, autonlp_compatible, infinity_compatible


# Searching Reaction GIFs with CLIP


![header gif](./assets/huggingface_explode3.png)


Reaction GIFs are an integral part of today's communication. They convey complex emotions with many levels, in a short compact format.


If a picture is worth a thousand words then a GIF is worth more.


We might even say that the level of complexity and expressiveness increases like:


`Emoji < Memes/Image < GIFs`


I think most people would agree it is not always easy to find the perfect reaction GIF.


Although we started out with the more ambitious goal of GIF/Image generation we later settled on first finetuning CLIP.
Which is needed to properly drive a generation model (like VQGAN).


![header gif](./assets/main.gif)


Available CLIP models wouldn't be suitable to use without this finetuning as explained in the challenges below.


## 📝 Challenges


Classic (Image,Text) tasks like, image search, caption generation all focus on cases where the text is a description of the image.
This is mainly because large scale datasets available like COCO,WIT happen to be of that format.
So it is interesting to see if models can also capture some more higher level relations.
like sentiment-> image mapping, where there is great variation on both sides.
We can think of reaction gif/images to be sentiment like, in fact the dataset we use was also gathered for sentiment analysis.
There is no one correct reaction GIF, which also makes evaluation challenging.


# Dataset


We use the [Reaction GIF dataset](https://github.com/bshmueli/ReactionGIF) by Shmueli et al. Also available in [datasets](https://huggingface.co/datasets/julien-c/reactiongif)(without the gifs, but I already had those files, nice coincidence 😉)
![Dataset](./assets/dataset.png)
We only use the tweet, GIF Response fields.
For this short experiment we only used the first frame of the GIFs to get generate the (text,image) pairs. Thought about using multiple frames from the GIF, but since there may not be much change between frames and didn't know how that would effect the already overfitting model. I decided to go with the simpler single frame version.


As opposed to other datasets, like COCO, we don't have multiple captions per image. Although same GIFs are used multiple times and we can modify the dataset that way we chose not to. Since we already had a very small dataset.


- Domain: Twitter
- Train size: 18,976
- Validation size: 351


# Model


We used the Hybrid CLIP model. Training script: [Hybrid CLIP](https://github.com/huggingface/transformers/blob/master/examples/research_projects/jax-projects/hybrid_clip/run_hybrid_clip.py)


- Text Model: [twitter-roberta-base-emoji](https://huggingface.co/cardiffnlp/twitter-roberta-base-emoji)
- Vision Model: [openai/clip-vit-base-patch32](https://huggingface.co/openai/clip-vit-base-patch32)


## Different models tried


`cardiffnlp/twitter-roberta-base-*` models performed best compared to other text models; like the plain `roberta-base`, `bert-base-cased`.
This makes sense since this model is already trained on twitter data.


And among `cardiffnlp/twitter-roberta-base-*` models `twitter-roberta-base-emoji` performed best. (As I had hoped)
This model is `cardiffnlp/twitter-roberta-base` further fine-tuned on emoji classification task, which we can say (emojis) is a parallel to GIFs.


Also tried `google/vit-base-patch32-384`, `google/vit-base-patch16-384` for the vision models, but results were inconclusive.


## Result Interpretation (Warning)


It would be wrong to claim that this model learned 'semantic reasoning' between sentence and image features.


It more likely learned a mapping between sentence sentiment and image occurrence statistics. Because the set of gif images repeat across the dataset, although paired with different sentences.


That is not to say that learning such semantic relations isn't feasible with this model. And it is well worth working on in the future, with a larger and better constructed dataset.


### 📈 Training Logs


Training logs can be found [here](https://wandb.ai/cceyda/flax-clip?workspace=user-cceyda)
It was really easy to overfit since it was a tiny dataset. Used early stopping.
Other parameters:


```
--max_seq_length 128 \
--per_device_train_batch_size="32" \
--learning_rate="1e-5"
--warmup_steps="150"
```


# 💡 Future Potential


It is possible to generate a very large training set by scraping twitter.(Couldn't do during the event because of twitter rate limit)


I found it surprising how well the results turned out to be with so little data and training time. Although it is really hard to define what is an appropriate reaction image, and there are definite mistakes model makes.


(I also trained just a plain clip but didn't have time to prep demo for that 😅 which was also similarly over fitting and only had time to do a single run)


I will definitely be trying out training a similar model for emoji & meme data.


Training CLIP is just the first step, if we have a well trained CLIP generation is within reach 🚀


# How to use


The final model available [here](https://huggingface.co/ceyda/clip-reply)


```py
from model import FlaxHybridCLIP # see demo
from transformers import AutoTokenizer, CLIPProcessor

model = FlaxHybridCLIP.from_pretrained("ceyda/clip-reply")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
processor.tokenizer = AutoTokenizer.from_pretrained("cardiffnlp/twitter-roberta-base")

def query(image_paths,query_text):
    images = [Image.open(im).convert("RGB") for im in image_paths]
    inputs = processor(text=[query_text], images=images, return_tensors="jax", padding=True)
    inputs["pixel_values"] = jnp.transpose(inputs["pixel_values"], axes=[0, 2, 3, 1])
    outputs = model(**inputs)
    logits_per_image = outputs.logits_per_image.reshape(-1)
    probs = jax.nn.softmax(logits_per_image)
```


# Created By


Ceyda Cinarel [@ceyda](https://huggingface.co/ceyda)


Made during the flax community [event](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104/58)


# TL;DR The task


Input: Some sentence (like a tweet)
Output: The most suitable reaction GIF image (Ranking)


Example:


- Input: I miss you
- Output: ![hug](./assets/example_gif.jpg)


# Demo


model name: clip-reply


model author: flax-community


tags: tensorboard


language: es
tags:


- audio
- automatic-speech-recognition
datasets:
- common_voice


# Wav2Vec2 Spanish


Wav2Vec2 model pre-trained using the Spanish portion of the Common Voice dataset. The model is trained with Flax and using TPUs sponsored by Google since this is part of the [Flax/Jax Community Week](https://discss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104) organised by HuggingFace.


## Model description


The model used for training is [Wav2Vec2](https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/) by FacebookAI. It was introduced in the paper
"wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations" by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli (https://arxiv.org/abs/2006.11477).


This model is available in the 🤗 [Model Hub](https://huggingface.co/facebook/wav2vec2-base-960h).


## Training data


Spanish portion of [Common Voice](https://commonvoice.mozilla.org/en/datasets). Common Voice is an open source, multi-language dataset of voices part of Mozilla's initiative to help teach machines how real people speak.


The dataset is also available in the 🤗 [Datasets](https://huggingface.co/datasets/common_voice) library.


## Team members


- María Grandury ([@mariagrandury](https://github.com/mariagrandury))
- Manuel Romero ([@mrm8488](https://huggingface.co/mrm8488))
- Eduardo González Ponferrada ([@edugp](https://huggingface.co/edugp))
- pcuenq ([@pcuenq](https://huggingface.co/pcuenq))


model name: wav2vec2-spanish


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, pretraining, es, dataset:common_voice, arxiv:2006.11477, transformers, audio, automatic-speech-recognition


language:


- en
- python
license: mit
tags:
- gpt_neo
- code_synthesis
datasets:
- apps


# GPT-Neo-1.3B-APPS-all


> **Please refer to our new [GitHub Wiki](https://github.com/ncoop57/gpt-code-clippy/wiki) which documents our efforts in detail in creating the open source version of GitHub  Copilot**


## Model Description


GPT-Neo-1.3B-APPS-all is a GPT-Neo-1.3B finetuned on APPS dataset. This model is specialized to solve programming tasks.


The model is trained on the [Automated Programming Progress Standard (APPS) dataset](https://github.com/hendrycks/apps). The dataset consists of 10,000 coding problems in total, with 131,836 test cases for checking solutions and 232,444 ground-truth solutions written by humans. Problems can be complicated, as the average length of a problem is 293.2 words. The data are split evenly into training and test sets, with 5,000 problems each.


This model is fine-tuned using most of the APPS dataset including both train and test split to explore the impact of this training task on model performance on other code synthesis evaluation metrics. A model fine-tuned on train set only can be found [here](https://huggingface.co/flax-community/gpt-neo-1.3B-apps).


## Training procedure


The training script used to train this model can be found [here](https://github.com/ncoop57/gpt-code-clippy/blob/camera-ready/training/run_clm_apps.py).


Training is done for 5 epochs using AdamW optimizer and leaner decay learning rate schedule with 800 warmup steps. To reproduce the training one can use this command with the above script:


```
python run_clm_apps.py \
    --output_dir ./gpt-neo-1.3B-apps \
    --model_name_or_path EleutherAI/gpt-neo-1.3B \
    --dataset_name ./apps.py \
    --dataset_config_name formatted \
    --do_train --do_eval \
    --block_size="1024" \
    --per_device_train_batch_size="3" \
    --per_device_eval_batch_size="3" \
    --preprocessing_num_workers="16" \
    --learning_rate="8e-5" \
    --warmup_steps="800" \
    --adam_beta1="0.9" \
    --adam_beta2="0.98" \
    --weight_decay="0.1" \
    --overwrite_output_dir \
    --num_train_epochs="5" \
    --logging_steps="50" \
    --eval_steps="2000" \
    --report_to="wandb" \
    --dtype="bfloat16" \
    --save_strategy epoch \
    --gradient_accumulation_steps 1 \
    --all_data true \
```


## Intended Use and Limitations


The model is finetuned to solve programming problems given a text description and optional starter code.


### How to use


You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:


```py

from transformers import AutoModelForCausalLM, AutoTokenizer, FlaxAutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("flax-community/gpt-code-clippy-1.3B-apps-alldata")

tokenizer = AutoTokenizer.from_pretrained("flax-community/gpt-code-clippy-1.3B-apps-alldata")

prompt = """

A function to greet user. Given a user name it should say hello

def greet(name):

ANSWER:

"""

input_ids = tokenizer(prompt, return_tensors='pt').input_ids.to(device)

start = input_ids.size(1)

out = model.generate(input_ids, do_sample=True, max_length=50, num_beams=2,

                     early_stopping=True, eos_token_id=tokenizer.eos_token_id, )

print(tokenizer.decode(out[0][start:]))

```


### Limitations and Biases


The model is intended to be used for research purposes and comes with no guarantees of quality of generated code.


The paper ["Evaluating Large Language Models Trained on Code"](https://arxiv.org/abs/2107.03374) from OpenAI has a good discussion on what the impact of a large language model trained on code could be. Therefore, some parts of their discuss are highlighted here as it pertains to this dataset and models that may be trained from it. **As well as some differences in views from the paper, particularly around legal implications**.


1. **Over-reliance:** This model may generate plausible solutions that may appear correct, but are not necessarily the correct solution. Not properly evaluating the generated code may cause have negative consequences such as the introduction of bugs, or the introduction of security vulnerabilities. Therefore, it is important that users are aware of the limitations and potential negative consequences of using this language model.
1. **Economic and labor market impacts:** Large language models trained on large code datasets such as this one that are capable of generating high-quality code have the potential to automate part of the software development process. This may negatively impact software developers. However, as discussed in the paper, as shown in the Summary Report of software developers from [O*NET OnLine](https://www.onetonline.org/link/summary/15-1252.00), developers don't just write software.
1. **Biases:** The model is trained on data containing prompt questions formatted in specific way. The performance of the model can be worse if the prompt


formatting is different from that used in APPS dataset.


GPT-CC is finetuned GPT-Neo and might have inhereted biases and limitations from it. See [GPT-Neo model card](https://huggingface.co/EleutherAI/gpt-neo-125M#limitations-and-biases) for details.


## Eval results


Coming soon...


model name: gpt-neo-1.3B-apps-all


model author: flax-community


task: text-generation


tags: pytorch, jax, tensorboard, gpt_neo, text-generation, en, python, dataset:apps, arxiv:2107.03374, transformers, license:mit, code_synthesis


language: th
tags:


- gpt2-base-thai
license: mit
datasets:
- oscar
widget:
- text: "สวัสดีตอนเช้า"


## GPT-2 Base Thai


GPT-2 Base Thai is a causal language model based on the [OpenAI GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) model. It was trained on the [OSCAR](https://huggingface.co/datasets/oscar) dataset, specifically the `unshuffled_deduplicated_th` subset. The model was trained from scratch and achieved an evaluation loss of 1.708 and an evaluation perplexity of 5.516.


This model was trained using HuggingFace's Flax framework and is part of the [JAX/Flax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104) organized by HuggingFace. All training was done on a TPUv3-8 VM, sponsored by the Google Cloud team.


All necessary scripts used for training could be found in the [Files and versions](https://hf.co/flax-community/gpt2-base-thai/tree/main) tab, as well as the [Training metrics](https://hf.co/flax-community/gpt2-base-thai/tensorboard) logged via Tensorboard.


## Model


| Model            | #params | Arch. | Training/Validation data (text)      |
| ---------------- | ------- | ----- | ------------------------------------ |
| `gpt2-base-thai` | 124M    | GPT-2 | `unshuffled_deduplicated_th` Dataset |


## Evaluation Results


The model was trained for 3 epochs and the following is the final result once the training ended.


| train loss | valid loss | valid PPL | total time |
| ---------- | ---------- | --------- | ---------- |
| 1.638      | 1.708      | 5.516     | 6:12:34    |


## How to Use


### As Causal Language Model


```python
from transformers import pipeline

pretrained_name = "flax-community/gpt2-base-thai"

nlp = pipeline(
    "text-generation",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("สวัสดีตอนเช้า")
```


### Feature Extraction in PyTorch


```python
from transformers import GPT2Model, GPT2TokenizerFast

pretrained_name = "flax-community/gpt2-base-thai"
model = GPT2Model.from_pretrained(pretrained_name)
tokenizer = GPT2TokenizerFast.from_pretrained(pretrained_name)

prompt = "สวัสดีตอนเช้า"
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```


## Team Members


- Sakares Saengkaew ([@sakares](https://hf.co/sakares))
- Wilson Wongso ([@w11wo](https://hf.co/w11wo))


model name: gpt2-base-thai


model author: flax-community


task: text-generation


tags: pytorch, jax, tensorboard, gpt2, text-generation, th, dataset:oscar, transformers, gpt2-base-thai, license:mit


language: sw
widget:


- text: "Idris ameandika kwenye ukurasa wake wa Instagram akimkumbusha Diamond kutekeleza ahadi yake kumpigia Zari magoti kumuomba msamaha kama alivyowahi kueleza awali.Idris ameandika;"
datasets:
- flax-community/swahili-safi


## Swahili News Classification with RoBERTa


This model was trained using HuggingFace's Flax framework and is part of the [JAX/Flax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104) organized by [HuggingFace](https://huggingface.co). All training was done on a TPUv3-8 VM sponsored by the Google Cloud team.


This [model](https://huggingface.co/flax-community/roberta-swahili) was used as the base and fine-tuned for this task.


## How to use


```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("flax-community/roberta-swahili-news-classification")

model = AutoModelForSequenceClassification.from_pretrained("flax-community/roberta-swahili-news-classification")
```


```
Eval metrics: {'accuracy': 0.9153416415986249}
```


model name: roberta-swahili-news-classification


task: text-classification


tags: pytorch, jax, tensorboard, roberta, text-classification, sw, dataset:flax-community/swahili-safi, transformers, infinity_compatible


language:


- en
- python
license: mit
tags:
- gpt_neo
- code_synthesis
datasets:
- apps


# GPT-Neo-1.3B-APPS


> **Please refer to our new [GitHub Wiki](https://github.com/ncoop57/gpt-code-clippy/wiki) which documents our efforts in detail in creating the open source version of GitHub  Copilot**


## Model Description


GPT-Neo-1.3B-APPS is a GPT-Neo-125M finetuned on APPS dataset. This model is specialized to solve programming tasks.


## Training data


The model is trained on the [Automated Programming Progress Standard (APPS) dataset](https://github.com/hendrycks/apps). The dataset consists of 10,000 coding problems in total, with 131,836 test cases for checking solutions and 232,444 ground-truth solutions written by humans. Problems can be complicated, as the average length of a problem is 293.2 words. The data are split evenly into training and test sets, with 5,000 problems each.


This model is fine-tuned using most of the APPS dataset including both train and test split to explore the impact of this training task on model performance on other code synthesis evaluation metrics. A model fine-tuned on train set only can be found [here](https://huggingface.co/flax-community/gpt-neo-125M-apps).


## Training procedure


The training script used to train this model can be found [here](https://github.com/ncoop57/gpt-code-clippy/blob/camera-ready/training/run_clm_apps.py).


Training is done for 5 epochs using AdamW optimizer and leaner decay learning rate schedule with 800 warmup steps. To reproduce the training one can use this command with the above script:


```bash
python run_clm_apps.py \
    --output_dir $HOME/gpt-neo-1.3B-apps \
    --model_name_or_path EleutherAI/gpt-neo-1.3B \
    --dataset_name $HOME/gpt-code-clippy/data_processing/apps.py \
    --dataset_config_name formatted \
    --do_train --do_eval \
    --block_size="1024" \
    --per_device_train_batch_size="3" \
    --per_device_eval_batch_size="3" \
    --preprocessing_num_workers="16" \
    --learning_rate="8e-5" \
    --warmup_steps="800" \
    --adam_beta1="0.9" \
    --adam_beta2="0.98" \
    --weight_decay="0.1" \
    --overwrite_output_dir \
    --num_train_epochs="5" \
    --logging_steps="50" \
    --eval_steps="2000" \
    --report_to="wandb" \
    --dtype="bfloat16" \
    --save_strategy epoch \
    --gradient_accumulation_steps 1 \
```


## Intended Use and Limitations


The model is finetuned to solve programming problems given a text description and optional starter code.


### How to use


You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:


```py

from transformers import AutoModelForCausalLM, AutoTokenizer, FlaxAutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("flax-community/gpt-code-clippy-1.3B-apps")

tokenizer = AutoTokenizer.from_pretrained("flax-community/gpt-code-clippy-1.3B-apps")

prompt = """

A function to greet user. Given a user name it should say hello

def greet(name):

ANSWER:

"""

input_ids = tokenizer(prompt, return_tensors='pt').input_ids.to(device)

start = input_ids.size(1)

out = model.generate(input_ids, do_sample=True, max_length=50, num_beams=2,

                     early_stopping=True, eos_token_id=tokenizer.eos_token_id, )

print(tokenizer.decode(out[0][start:]))

```


### Limitations and Biases


The model is intended to be used for research purposes and comes with no guarantees of quality of generated code.


The paper ["Evaluating Large Language Models Trained on Code"](https://arxiv.org/abs/2107.03374) from OpenAI has a good discussion on what the impact of a large language model trained on code could be. Therefore, some parts of their discuss are highlighted here as it pertains to this dataset and models that may be trained from it. **As well as some differences in views from the paper, particularly around legal implications**.


1. **Over-reliance:** This model may generate plausible solutions that may appear correct, but are not necessarily the correct solution. Not properly evaluating the generated code may cause have negative consequences such as the introduction of bugs, or the introduction of security vulnerabilities. Therefore, it is important that users are aware of the limitations and potential negative consequences of using this language model.
1. **Economic and labor market impacts:** Large language models trained on large code datasets such as this one that are capable of generating high-quality code have the potential to automate part of the software development process. This may negatively impact software developers. However, as discussed in the paper, as shown in the Summary Report of software developers from [O*NET OnLine](https://www.onetonline.org/link/summary/15-1252.00), developers don't just write software.
1. **Biases:** The model is trained on data containing prompt questions formatted in specific way. The performance of the model can be worse if the prompt


formatting is different from that used in APPS dataset.


GPT-CC is finetuned GPT-Neo and might have inhereted biases and limitations from it. See [GPT-Neo model card](https://huggingface.co/EleutherAI/gpt-neo-125M#limitations-and-biases) for details.


## Eval results


Coming soon...


model name: gpt-neo-1.3B-apps


model author: flax-community


task: text-generation


tags: pytorch, jax, gpt_neo, text-generation, en, python, dataset:apps, arxiv:2107.03374, transformers, license:mit, code_synthesis


## language: python
tags: vae
license: apache-2.0
datasets: Fraser/python-lines


# T5-VAE-Python (flax)


A Transformer-VAE made using flax.


Try the [demo](https://huggingface.co/spaces/flax-community/t5-vae)!


It has been trained to interpolate on lines of Python code from the [python-lines dataset](https://huggingface.co/datasets/Fraser/python-lines).


Done as part of Huggingface community training ([see forum post](https://discuss.huggingface.co/t/train-a-vae-to-interpolate-on-english-sentences/7548)).


Builds on T5, using an autoencoder to convert it into an MMD-VAE ([more info](http://fras.uk/ml/large%20prior-free%20models/transformer-vae/2020/08/13/Transformers-as-Variational-Autoencoders.html)).


## How to use from the 🤗/transformers library


Add model repo as a submodule:


```bash
git submodule add https://github.com/Fraser-Greenlee/t5-vae-flax.git t5_vae_flax
```


```python
from transformers import AutoTokenizer
from t5_vae_flax.src.t5_vae import FlaxT5VaeForAutoencoding

tokenizer = AutoTokenizer.from_pretrained("t5-base")

model = FlaxT5VaeForAutoencoding.from_pretrained("flax-community/t5-vae-python")
```


## Setup


Run `setup_tpu_vm_venv.sh` to setup a virtual enviroment on a TPU VM for training.


model name: t5-vae-python


model author: flax-community


tags: jax, transformer_vae, python, dataset:Fraser/python-lines, transformers, vae, license:apache-2.0


language: "mn"
thumbnail: "https://avatars.githubusercontent.com/u/43239645?s=60&v=4"
tags:


- gpt2
datasets:
- oscar


# Mongolian GPT2


Goal is to create a strong language generation model for Mongolian
Since initial code and data is pretty much written by @patrickvonplaten and other huggingface members, it should not be so hard to get the first sense.


## Model


Randomly initialized GPT2 model


## Datasets


We can use OSCAR which is available through datasets


A causal language modeling script for Flax is available here 1. It can be used pretty much without any required code changes.
If there is time left, I’d love to try some private crawling and integrate it datasets.


## Expected Outcome


Understandable Mongolian text generation model


## Challenges


Lack of data → OSCAR Mongolian is just 2.2G. Maybe we need to research ways to acquire more data with this.


model name: mongolian-gpt2


task: text-generation


tags: pytorch, jax, gpt2, text-generation, mn, dataset:oscar, transformers


language: bn
license: mit
datasets:


- mc4


# Bengali GPT-2


Bengali GPT-2 demo. Part of the [Huggingface JAX/Flax event](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/). Also features a [finetuned](https://huggingface.co/khalidsaifullaah/bengali-lyricist-gpt2?) model on bengali song lyrics.


# Model Description


OpenAI GPT-2 model was proposed in [Language Models are Unsupervised Multitask Learners](https://paperswithcode.com/paper/language-models-are-unsupervised-multitask) paper .Original GPT2 model was a causal (unidirectional) transformer pretrained using language modeling on a very large corpus of ~40 GB of text data. This model has same configuration but has been pretrained on bengali corpus of mC4(multilingual C4) dataset. The code for training the model has all been open-sourced [here](https://huggingface.co/flax-community/gpt2-bengali/tree/main).


# Training Details


Overall Result:


`Eval loss : 1.45, Eval Perplexity : 3.141`


Data: [mC4-bn](https://huggingface.co/datasets/mc4)


Train Steps: 250k steps


link 🤗 flax-community/gpt2-bengali


Demo : https://huggingface.co/spaces/flax-community/Gpt2-bengali


# Usage


For using the model there are multiple options available. For example using the pipeline directly we can try to generate sentences.


```
from transformers import pipeline

gpt2_bengali = pipeline('text-generation',model="flax-community/gpt2-bengali", tokenizer='flax-community/gpt2-bengali')
```


Similarly for using the finetuned model on bangla songs we can use following.


```
from transformers import pipeline

singer = pipeline('text-generation',model="khalidsaifullaah/bengali-lyricist-gpt2", tokenizer='khalidsaifullaah/bengali-lyricist-gpt2')
```


For using on other tasks the model needs to be fine-tuned on custom datasets. Details can be found in huggingface [documentation](https://huggingface.co/transformers/training.html)


# Contributors


* Khalid Saifullah
* Tasmiah Tahsin Mayeesha
* Ritobrata Ghosh
* Ibrahim Musa
* M Saiful Bari


### BibTeX entry and citation info


Coming soon!


<!-- ```bibtex

@inproceedings{...,

  year={2020}

}

``` -->


model name: gpt2-bengali


model author: flax-community


task: text-generation


tags: pytorch, jax, tensorboard, gpt2, text-generation, bn, dataset:mc4, transformers, license:mit


language: sw
widget:


- text: "Si kila mwenye makucha [MASK] simba."
datasets:
- flax-community/swahili-safi


## BERT base-uncased for in Swahili


This model was trained using HuggingFace's Flax framework and is part of the [JAX/Flax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104) organized by [HuggingFace](https://huggingface.co). All training was done on a TPUv3-8 VM sponsored by the Google Cloud team.


## How to use


```python
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("flax-community/bert-base-uncased-swahili")

model = AutoModelForMaskedLM.from_pretrained("flax-community/bert-base-uncased-swahili")

print(round((model.num_parameters())/(1000*1000)),"Million Parameters")

110 Million Parameters
```


#### **Training Data**:


This model was trained on [Swahili Safi](https://huggingface.co/datasets/flax-community/swahili-safi)


#### **More Details**:


For more details and Demo please check [HF Swahili Space](https://huggingface.co/spaces/flax-community/Swahili)


model name: bert-base-uncased-swahili


task: fill-mask


tags: pytorch, jax, tensorboard, bert, fill-mask, sw, dataset:flax-community/swahili-safi, transformers, autonlp_compatible, infinity_compatible


language:


- en
- python
license: mit
tags:
- gpt_neo
- code_synthesis
datasets:
- apps


# GPT-Code-Clippy-1.3B-APPS-all


## Model Description


GPT-Neo-1.3B-APPS-all is a GPT-Neo-1.3B fine-tuned on APPS dataset. This model is specialized to solve programming tasks.


## Training data


The model is trained on the [Automated Programming Progress Standard (APPS) dataset](https://github.com/hendrycks/apps). The dataset consists of 10,000 coding problems in total, with 131,836 test cases for checking solutions and 232,444 ground-truth solutions written by humans. Problems can be complicated, as the average length of a problem is 293.2 words. The data are split evenly into training and test sets, with 5,000 problems each.


This model is fine-tuned using most of the APPS dataset including both train and test split to explore the impact of this training task on model performance on other code synthesis evaluation metrics. A model fine-tuned on train set only can be found [here](https://huggingface.co/flax-community/gpt-neo-1.3B-apps).


## Training procedure


The training script used to train this model can be found [here](https://github.com/ncoop57/gpt-code-clippy/blob/camera-ready/training/run_clm_apps.py).


Training is done for 5 epochs using AdamW optimizer and leaner decay learning rate schedule with 800 warmup steps. To reproduce the training one can use this command with the above script:


```
python run_clm_apps.py \
    --output_dir ./gpt-neo-125M-apps \
    --model_name_or_path EleutherAI/gpt-neo-125B \
    --dataset_name ./apps.py \
    --dataset_config_name formatted \
    --do_train --do_eval \
    --block_size="1024" \
    --per_device_train_batch_size="3" \
    --per_device_eval_batch_size="3" \
    --preprocessing_num_workers="16" \
    --learning_rate="8e-5" \
    --warmup_steps="800" \
    --adam_beta1="0.9" \
    --adam_beta2="0.98" \
    --weight_decay="0.1" \
    --overwrite_output_dir \
    --num_train_epochs="5" \
    --logging_steps="50" \
    --eval_steps="2000" \
    --report_to="wandb" \
    --dtype="bfloat16" \
    --save_strategy epoch \
    --gradient_accumulation_steps 1 \
    --all_data true \
```


## Intended Use and Limitations


The model is fine-tuned to solve programming problems given a text description and optional starter code.


### How to use


You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:


```py

from transformers import AutoModelForCausalLM, AutoTokenizer, FlaxAutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("flax-community/gpt-neo-1.3B-apps-all-2")

tokenizer = AutoTokenizer.from_pretrained("flax-community/gpt-neo-1.3B-apps-all-2")

prompt = """

A function to greet user. Given a user name it should say hello

def greet(name):

ANSWER:

"""

input_ids = tokenizer(prompt, return_tensors='pt').input_ids.to(device)

start = input_ids.size(1)

out = model.generate(input_ids, do_sample=True, max_length=50, num_beams=2,

                     early_stopping=True, eos_token_id=tokenizer.eos_token_id, )

print(tokenizer.decode(out[0][start:]))

```


### Limitations and Biases


The model is intended to be used for research purposes and comes with no guarantees of quality of generated code.


The paper ["Evaluating Large Language Models Trained on Code"](https://arxiv.org/abs/2107.03374) from OpenAI has a good discussion on what the impact of a large language model trained on code could be. Therefore, some parts of their discuss are highlighted here as it pertains to this dataset and models that may be trained from it. **As well as some differences in views from the paper, particularly around legal implications**.


1. **Over-reliance:** This model may generate plausible solutions that may appear correct, but are not necessarily the correct solution. Not properly evaluating the generated code may cause have negative consequences such as the introduction of bugs, or the introduction of security vulnerabilities. Therefore, it is important that users are aware of the limitations and potential negative consequences of using this language model.
1. **Economic and labor market impacts:** Large language models trained on large code datasets such as this one that are capable of generating high-quality code have the potential to automate part of the software development process. This may negatively impact software developers. However, as discussed in the paper, as shown in the Summary Report of software developers from [O*NET OnLine](https://www.onetonline.org/link/summary/15-1252.00), developers don't just write software.
1. **Biases:** The model is trained on data containing prompt questions formatted in specific way. The performance of the model can be worse if the prompt formatting is different from that used in APPS dataset.


This model is finetuned GPT-Neo and might have inhereted biases and limitations from it. See [GPT-Neo model card](https://huggingface.co/EleutherAI/gpt-neo-125M#limitations-and-biases) for details.


## Eval results


Coming soon...


model name: gpt-neo-1.3B-apps-all-2


model author: flax-community


task: text-generation


tags: pytorch, jax, gpt_neo, text-generation, en, python, dataset:apps, arxiv:2107.03374, transformers, license:mit, code_synthesis


language: es
tags:


- text-generation
datasets:
- oscar
widgets:
- text: "Érase un vez "


# Spanish GPT-2


GPT-2 model trained from scratch on the Spanish portion of [OSCAR](https://huggingface.co/datasets/viewer/?dataset=oscar).
The model is trained with Flax and using TPUs sponsored by Google since this is part of the
[Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104)
organised by HuggingFace.


## Model description


The model used for training is [OpenAI's GPT-2](https://openai.com/blog/better-language-models/), introduced in the paper ["Language Models are Unsupervised Multitask Learners"](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) by Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever.


This model is available in the 🤗 [Model Hub](https://huggingface.co/gpt2).


## Training data


Spanish portion of OSCAR or **O**pen **S**uper-large **C**rawled **A**LMAnaCH co**R**pus, a huge multilingual corpus obtained by language classification and filtering of the [Common Crawl](https://commoncrawl.org/) corpus using the [goclassy](https://github.com/pjox/goclassy) architecture.


This corpus is available in the 🤗 [Datasets](https://huggingface.co/datasets/oscar) library.


## Team members


- Manuel Romero ([mrm8488](https://huggingface.co/mrm8488))
- María Grandury ([mariagrandury](https://huggingface.co/mariagrandury))
- Pablo González de Prado ([Pablogps](https://huggingface.co/Pablogps))
- Daniel Vera ([daveni](https://huggingface.co/daveni))
- Sri Lakshmi ([srisweet](https://huggingface.co/srisweet))
- José Posada ([jdposa](https://huggingface.co/jdposa))
- Santiago Hincapie ([shpotes](https://huggingface.co/shpotes))
- Jorge ([jorgealro](https://huggingface.co/jorgealro))


model name: gpt-2-spanish


tags: pytorch, jax, tensorboard, gpt2, text-generation, es, dataset:oscar, transformers


# GPT-Code-Clippy-125M-from-Scratch


> **Please refer to our new [GitHub Wiki](https://github.com/ncoop57/gpt-code-clippy/wiki) which documents our efforts in detail in creating the open source version of GitHub  Copilot**


## Model Description


GPT-CC-125M-from-Scratch is a [GPT-Neo-125M model](https://huggingface.co/EleutherAI/gpt-neo-125M) pretrained from scratch using causal language modeling on the [Code Clippy Post-deduplication dataset](https://the-eye.eu/public/AI/training_data/code_clippy_data/code_clippy_dedup_data/). The deduplication script can be found [here](https://github.com/ncoop57/gpt-code-clippy/blob/camera-ready/data_processing/deduplication/deduplication.py). Code Clippy was scraped from public Github repositories (more information in the provided link). This model is specialized to autocomplete methods in multiple programming languages. As discussed in OpenAI's [Codex paper](https://arxiv.org/abs/2107.03374), we modified the GPT-Neo model and tokenizer to accommodate for additional whitespace characters. Specifically, we add the following tokens `["\t\t", "  ", "    ", "        "]` and since they are all related to indentation, we initialize the embedding layer of these tokens with the same weights as the `\t` token already present in the model in hopes the model will learn to associate these whitespace characters with indentation faster. A script to automatically do this can be found [here](https://github.com/ncoop57/gpt-code-clippy/blob/camera-ready/training/utilities/add_new_tokens.py).


[Code Clippy Deduplicated dataset](https://the-eye.eu/public/AI/training_data/code_clippy_data/code_clippy_dedup_data/).
Python script of the dataset can be found [here](https://github.com/ncoop57/gpt-code-clippy/blob/camera-ready/data_processing/code_clippy.py)


## Training procedure


The training script used to train this model can be found [here](https://github.com/ncoop57/gpt-code-clippy/blob/camera-ready/training/deprecated/run_clm_streaming_flax_v2.py).


```bash
./run_clm_streaming_flax_v2.py \
    --output_dir $HOME/gpt-neo-125M-code-clippy-from-scratch \
     --tokenizer_name="EleutherAI/gpt-neo-125M" \
    --model_name_or_path="EleutherAI/gpt-neo-125M" \
    --dataset_name $HOME/gpt-code-clippy/data_processing/code_clippy.py \
    --data_dir /home/shared/code_clippy_data \
    --do_train --do_eval \
    --block_size="2048" \
    --per_device_train_batch_size="8" \
    --per_device_eval_batch_size="16" \
    --preprocessing_num_workers="8" \
    --learning_rate="3e-5" \
    --max_steps 100000 \
    --warmup_steps 2500\
    --decap_steps 25000 \
    --adam_beta1="0.9" \
    --adam_beta2="0.95" \
    --weight_decay="0.1" \
    --overwrite_output_dir \
    --logging_steps="50" \
    --eval_steps="500" \
    --push_to_hub="False" \
    --report_to="all" \
    --dtype="bfloat16" \
    --skip_memory_metrics="True" \
    --save_steps="500" \
    --save_total_limit 10 \
    --report_to="wandb" \
    --run_name="gpt-neo-125M-code-clippy-dedup-scratch"
```


## Intended Use and Limitations


The model is pre-trained and not finetuned for any particular use or a particular programming language. Due to time constraints, this model was only pre-trained on 1% of the Code Clippy Dataset.


The paper ["Evaluating Large Language Models Trained on Code"](https://arxiv.org/abs/2107.03374) from OpenAI has a good discussion on what the impact of a large language model trained on code could be. Therefore, some parts of their discuss are highlighted here as it pertains to this dataset and models that may be trained from it. **As well as some differences in views from the paper, particularly around legal implications**.


1. **Over-reliance:** This model may generate plausible solutions that may appear correct, but are not necessarily the correct solution. Not properly evaluating the generated code may cause have negative consequences such as the introduction of bugs, or the introduction of security vulnerabilities. Therefore, it is important that users are aware of the limitations and potential negative consequences of using this language model.
1. **Economic and labor market impacts:** Large language models trained on large code datasets such as this one that are capable of generating high-quality code have the potential to automate part of the software development process. This may negatively impact software developers. However, as discussed in the paper, as shown in the Summary Report of software developers from [O*NET OnLine](https://www.onetonline.org/link/summary/15-1252.00), developers don't just write software.
1. **Security implications:** No filtering or checking of vulnerabilities or buggy code was performed on the datase this model is trained on. This means that the dataset may contain code that may be malicious or contain vulnerabilities. Therefore, this model may generate vulnerable, buggy, or malicious code. In safety critical software, this could lead to software that may work improperly and could result in serious consequences depending on the software. Additionally, this model may be able to be used to generate malicious code on purpose in order to perform ransomware or other such attacks.
1. **Legal implications:** No filtering was performed on licensed code. This means that the dataset may contain restrictive licensed code. As discussed in the paper, public Github repositories may fall under "fair use." However, there has been little to no previous cases of such usages of licensed publicly available code. Therefore, any code generated with this model may be required to obey license terms that align with the software it was trained on such as GPL-3.0. It is unclear the legal ramifications of using a language model trained on this dataset.
1. **Biases:** The programming languages most represented in the dataset this model was trained on are Javascript and Python. Therefore, other, still popular languages such as C and C++, are less represented and therefore the models performance for these languages will be less comparatively. Additionally, this dataset only contains public repositories and so the model may not generate code that is representative of code written by private developers. No filtering was performed for potential racist, offensive, or otherwise inappropriate content. Therefore, this model may reflect such biases in its generation.


GPT-Neo-125M-Code-Clippy is finetuned from GPT-Neo and might have inherited biases and limitations from it. See [GPT-Neo model card](https://huggingface.co/EleutherAI/gpt-neo-125M#limitations-and-biases) for details.


### How to use


You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:


```py

from transformers import AutoModelForCausalLM, AutoTokenizer, FlaxAutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("flax-community/gpt-neo-125M-code-clippy-dedup-scratch")

tokenizer = AutoTokenizer.from_pretrained("flax-community/gpt-neo-125M-code-clippy-dedup-scratch")

prompt = """def greet(name):
  '''A function to greet user. Given a user name it should say hello'''
"""

input_ids = tokenizer(prompt, return_tensors='pt').input_ids.to(device)

start = input_ids.size(1)

out = model.generate(input_ids, do_sample=True, max_length=50, num_beams=2,

                     early_stopping=True, eos_token_id=tokenizer.eos_token_id, )

print(tokenizer.decode(out[0][start:]))

```


### Limitations and Biases


The model is intended to be used for research purposes and comes with no guarantees of the quality of generated code.


GPT-CC is finetuned from GPT-Neo and might have inherited biases and limitations from it. See [GPT-Neo model card](https://huggingface.co/EleutherAI/gpt-neo-125M#limitations-and-biases) for details.


## Evaluation results


Below is a table containing the base model we started from, and the model's performance on the [HumanEval Benchmark](https://github.com/openai/human-eval).


|    Model     |    Dataset Used     |   pass@1    |   pass@2    |   pass@5    |   pass@10   |
|     ---      |          ---        | :---------: | :---------: | :---------: | :---------: |
| [gpt-neo-125M (**trained from scratch**)](https://huggingface.co/flax-community/gpt-neo-125M-code-clippy-dedup-scratch) |  [Code Clippy Data (Deduplicated)](https://the-eye.eu/public/AI/training_data/code_clippy_data/code_clippy_dedup_data/) (~1% of the data)  |    0.00%    |    0.00%    |    0.00%    |    0.00%    |


model name: gpt-neo-125M-code-clippy-dedup-scratch


model author: flax-community


task: text-generation


tags: jax, tensorboard, gpt_neo, text-generation, arxiv:2107.03374, transformers


language: sv
widget:


- text: "Det var en gång"


# Nordic GPT2--wikipedia


A Nordic GPT2 style model trained using Flax CLM pipeline on the Nordic parts
part of the wiki40b dataset.


## Model series


This model is part of a series of models training on TPU with Flax Jax during Huggingface Flax/Jax challenge.


## Gpt models


## Swedish Gpt


## Swedish gpt wiki


# Nordic gpt wiki


## Dansk gpt wiki


## Norsk gpt wiki


## Roberta models


## Nordic Roberta Wiki


## Swe Roberta Wiki Oscar


## Roberta Swedish Scandi


## Roberta Swedish


## Swedish T5 model


## Data cleaning and preprocessing


The data was cleaned and preprocessed using the following script. Make sure to install depencies for beam_runner to make the dataset work.


```python
from datasets import load_dataset
def load_and_clean_wiki():
    dataset = load_dataset('wiki40b', 'da', beam_runner='DirectRunner', split="train")
    #dataset = load_dataset('wiki40b', 'sv', beam_runner='DirectRunner')
    dataset = dataset.remove_columns(['wikidata_id', 'version_id'])
    filtered_dataset = dataset.map(filter_wikipedia)
    # filtered_dataset[:3]
    # print(filtered_dataset[:3])
    return filtered_dataset

def filter_wikipedia(batch):
    batch["text"] = " ".join(batch["text"].split("\
_START_SECTION_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_ARTICLE_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_ARTICLE_\
"))
    batch["text"] = " ".join(batch["text"].split("\
_START_PARAGRAPH_\
"))
    batch["text"] = " ".join(batch["text"].split("_NEWLINE_"))
    batch["text"] = " ".join(batch["text"].split("\xa0"))
    return batch
```


## Training script


The following training script was used to train the model.


```bash
./run_clm_flax.py     --output_dir="${MODEL_DIR}"     --model_type="gpt2"     --config_name="${MODEL_DIR}"     --tokenizer_name="${MODEL_DIR}"     --dataset_name="wiki40b"     --dataset_config_name="da"     --do_train --do_eval     --block_size="512"     --per_device_train_batch_size="64"     --per_device_eval_batch_size="64"     --learning_rate="5e-3" --warmup_steps="1000"     --adam_beta1="0.9" --adam_beta2="0.98" --weight_decay="0.01"     --overwrite_output_dir     --num_train_epochs="20"     --logging_steps="500"     --save_steps="1000"     --eval_steps="2500"     --push_to_hub
```


model name: nordic-gpt-wiki


model author: flax-community


task: text-generation


tags: pytorch, jax, tensorboard, gpt2, text-generation, sv, transformers


language: id
widget:


- text: "Sewindu sudah kita tak berjumpa, rinduku padamu sudah tak terkira."


# GPT2-small-indonesian


This is a pretrained model on Indonesian language using a causal language modeling (CLM) objective, which was first
introduced in [this paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
and first released at [this page](https://openai.com/blog/better-language-models/).


This model was trained using HuggingFace's Flax framework and is part of the [JAX/Flax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104)
organized by [HuggingFace](https://huggingface.co). All training was done on a TPUv3-8 VM sponsored by the Google Cloud team.


The demo can be found [here](https://huggingface.co/spaces/flax-community/gpt2-indonesian).


## How to use


You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness,
we set a seed for reproducibility:


```python
>>> from transformers import pipeline, set_seed
>>> generator = pipeline('text-generation', model='flax-community/gpt2-small-indonesian')
>>> set_seed(42)
>>> generator("Sewindu sudah kita tak berjumpa,", max_length=30, num_return_sequences=5)

[{'generated_text': 'Sewindu sudah kita tak berjumpa, dua dekade lalu, saya hanya bertemu sekali. Entah mengapa, saya lebih nyaman berbicara dalam bahasa Indonesia, bahasa Indonesia'},
 {'generated_text': 'Sewindu sudah kita tak berjumpa, tapi dalam dua hari ini, kita bisa saja bertemu.”\
“Kau tau, bagaimana dulu kita bertemu?” aku'},
 {'generated_text': 'Sewindu sudah kita tak berjumpa, banyak kisah yang tersimpan. Tak mudah tuk kembali ke pelukan, di mana kini kita berada, sebuah tempat yang jauh'},
 {'generated_text': 'Sewindu sudah kita tak berjumpa, sejak aku lulus kampus di Bandung, aku sempat mencari kabar tentangmu. Ah, masih ada tempat di hatiku,'},
 {'generated_text': 'Sewindu sudah kita tak berjumpa, tapi Tuhan masih saja menyukarkan doa kita masing-masing.\
Tuhan akan memberi lebih dari apa yang kita'}]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import GPT2Tokenizer, GPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('flax-community/gpt2-small-indonesian')
model = GPT2Model.from_pretrained('flax-community/gpt2-small-indonesian')
text = "Ubah dengan teks apa saja."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import GPT2Tokenizer, TFGPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('flax-community/gpt2-small-indonesian')
model = TFGPT2Model.from_pretrained('flax-community/gpt2-small-indonesian')
text = "Ubah dengan teks apa saja."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


## Limitations and bias


The training data used for this model are Indonesian websites of [OSCAR](https://oscar-corpus.com/),\
[mc4](https://huggingface.co/datasets/mc4) and [Wikipedia](https://huggingface.co/datasets/wikipedia). The datasets
contain a lot of unfiltered content from the internet, which is far from neutral. While we have done some filtering on
the dataset (see the **Training data** section), the filtering is by no means a thorough mitigation of biased content
that is eventually used by the training data. These biases might also affect models that are fine-tuned using this model.


As the openAI team themselves point out in their [model card](https://github.com/openai/gpt-2/blob/master/model_card.md#out-of-scope-use-cases):


> Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases
that require the generated text to be true.


> Additionally, language models like GPT-2 reflect the biases inherent to the systems they were trained on, so we
do not recommend that they be deployed into systems that interact with humans > unless the deployers first carry
out a study of biases relevant to the intended use-case. We found no statistically significant difference in gender,
race, and religious bias probes between 774M and 1.5B, implying all versions of GPT-2 should be approached with
similar levels of caution around use cases that are sensitive to biases around human attributes.


We have done a basic bias analysis that you can find in this [notebook](https://huggingface.co/flax-community/gpt2-small-indonesian/blob/main/bias_analysis/gpt2_medium_indonesian_bias_analysis.ipynb), performed on [Indonesian GPT2 medium](https://huggingface.co/flax-community/gpt2-medium-indonesian), based on the bias analysis for [Polish GPT2](https://huggingface.co/flax-community/papuGaPT2) with modifications.


### Gender bias


We generated 50 texts starting with prompts "She/He works as". After doing some preprocessing (lowercase and stopwords removal) we obtain texts that are used to generate  word clouds of female/male professions. The most salient terms for male professions are: driver, sopir (driver), ojek, tukang, online.


![gender bias - male](https://huggingface.co/flax-community/gpt2-small-indonesian/raw/main/bias_analysis/wordcloud_male.png)


The most salient terms for female professions are: pegawai (employee), konsultan (consultant), asisten (assistant).


![gender bias - female](https://huggingface.co/flax-community/gpt2-small-indonesian/raw/main/bias_analysis/wordcloud_female.png)


### Ethnicity bias


We generated 1,200 texts to assess bias across ethnicity and gender vectors. We will create prompts with the following scheme:


* Person - we will assess 5 ethnicities: Sunda, Batak, Minahasa, Dayak, Asmat, Neutral (no ethnicity)
* Topic - we will use 5 different topics:
  * random act: *entered home*
  * said: *said*
  * works as: *works as*
  * intent: *let [person] ...*
  * define: *is*


Sample of generated prompt: "seorang perempuan sunda masuk ke rumah..." (a Sundanese woman enters the house...)


We used a [model](https://huggingface.co/Hate-speech-CNERG/dehatebert-mono-indonesian) trained on Indonesian hate speech corpus ([dataset 1](https://github.com/okkyibrohim/id-multi-label-hate-speech-and-abusive-language-detection), [dataset 2](https://github.com/ialfina/id-hatespeech-detection)) to obtain the probability that each generated text contains hate speech. To avoid leakage, we removed the first word identifying the ethnicity and gender from the generated text before running the hate speech detector.


The following chart demonstrates the intensity of hate speech associated with the generated texts with outlier scores removed. Some ethnicities score higher than the neutral baseline.


![bias analysis - ethnicities](https://huggingface.co/flax-community/gpt2-small-indonesian/raw/main/bias_analysis/bias_ethnicity.png)


### Religion bias


With the same methodology above, we generated 1,400 texts to assess bias across religion and gender vectors. We will assess 6 religions: Islam, Protestan (Protestant), Katolik (Catholic), Buddha (Buddhism), Hindu (Hinduism), and Khonghucu (Confucianism) with Neutral (no religion) as a baseline.


The following chart demonstrates the intensity of hate speech associated with the generated texts with outlier scores removed. Some religions score higher than the neutral baseline.


![bias analysis - ethnicities](https://huggingface.co/flax-community/gpt2-small-indonesian/raw/main/bias_analysis/bias_religion.png)


## Training data


The model was trained on a combined dataset of [OSCAR](https://oscar-corpus.com/), [mc4](https://huggingface.co/datasets/mc4)
and Wikipedia for the Indonesian language. We have filtered and reduced the mc4 dataset so that we end up with 29 GB
of data in total. The mc4 dataset was cleaned using [this filtering script](https://github.com/Wikidepia/indonesian_datasets/blob/master/dump/mc4/cleanup.py)
and we also only included links that have been cited by the Indonesian Wikipedia.


## Training procedure


The model was trained on a TPUv3-8 VM provided by the Google Cloud team. The training duration was `4d 14h 50m 47s`.


### Evaluation results


The model achieves the following results without any fine-tuning (zero-shot):


| dataset | train loss | eval loss | eval perplexity |
| ---------- | ---------- | -------------- | ---------- |
| ID OSCAR+mc4+wikipedia (29GB)      | 3.046      | 2.926         | 18.66   |


### Tracking


The training process was tracked in [TensorBoard](https://huggingface.co/flax-community/gpt2-small-indonesian/tensorboard) and [Weights and Biases](https://wandb.ai/wandb/hf-flax-gpt2-indonesian?workspace=user-cahya).


## Team members


- Akmal ([@Wikidepia](https://huggingface.co/Wikidepia))
- alvinwatner ([@alvinwatner](https://huggingface.co/alvinwatner))
- Cahya Wirawan ([@cahya](https://huggingface.co/cahya))
- Galuh Sahid ([@Galuh](https://huggingface.co/Galuh))
- Muhammad Agung Hambali ([@AyameRushia](https://huggingface.co/AyameRushia))
- Muhammad Fhadli ([@muhammadfhadli](https://huggingface.co/muhammadfhadli))
- Samsul Rahmadani ([@munggok](https://huggingface.co/munggok))


## Future work


We would like to pre-train further the models with larger and cleaner datasets and fine-tune it to specific domains
if we can get the necessary hardware resources.


model name: gpt2-small-indonesian


model author: flax-community


task: text-generation


tags: pytorch, jax, tensorboard, gpt2, text-generation, id, transformers


language: fa
tags:


- text-generation
widget:
- text: "در یک اتفاق شگفت انگیز، پژوهشگران"
- text: "گرفتگی بینی در کودکان و به‌خصوص نوزادان باعث می‌شود"
- text: "امیدواریم نوروز امسال سالی"


# GPT2 Medium 4 Persian


> This is part of the
[Flax/Jax Community Week](https://discuss.huggingface.co/t/pretrain-gpt2-from-scratch-in-persian/7560), organized by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google.


## Team Members


- [Mehrdad Farahani](huggingface.co/m3hrdadfi)
- [Saied Alimoradi](https://discuss.huggingface.co/u/saied)
- [M. Reza Zerehpoosh](huggingface.co/ironcladgeek)
- [Hooman Sedghamiz](https://discuss.huggingface.co/u/hooman650)
- [Mazeyar Moeini Feizabadi](https://discuss.huggingface.co/u/mazy1998)


## Dataset


We used  [Oscar](https://huggingface.co/datasets/oscar) dataset, which is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus.


## How To Use


You can use this model directly with a pipeline for text generation.


```python
from transformers import pipeline, AutoTokenizer, GPT2LMHeadModel
tokenizer = AutoTokenizer.from_pretrained('flax-community/gpt2-medium-persian')
model = GPT2LMHeadModel.from_pretrained('flax-community/gpt2-medium-persian')
generator = pipeline('text-generation', model, tokenizer=tokenizer, config={'max_length':100})
generated_text = generator('در یک اتفاق شگفت انگیز، پژوهشگران')
```


For using Tensorflow import TFGPT2LMHeadModel instead of GPT2LMHeadModel.


## Demo


... SOON


## Evaluation


model name: gpt2-medium-persian


model author: flax-community


task: text-generation


tags: pytorch, tf, jax, tensorboard, gpt2, text-generation, fa, transformers


language: dv
tags:


- automatic-speech-recognition
datasets:
- common_voice


# Wav2Vec2 Dhivehi


Wav2vec2 pre-pretrained from scratch using common voice dhivehi dataset. The model was trained with Flax during the [Flax/Jax Community Week](https://discss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104) organised by HuggingFace.


## Model description


The model used for training is [Wav2Vec2](https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/) by FacebookAI. It was introduced in the paper
"wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations" by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli (https://arxiv.org/abs/2006.11477).


This model is available in the 🤗 [Model Hub](https://huggingface.co/facebook/wav2vec2-base-960h).


## Training data


Dhivehi data from [Common Voice](https://commonvoice.mozilla.org/en/datasets).


The dataset is also available in the 🤗 [Datasets](https://huggingface.co/datasets/common_voice) library.


## Team members


- Shahu Kareem ([@shahukareem](https://huggingface.co/shahukareem))
- Eyna ([@eyna](https://huggingface.co/eyna))


model name: wav2vec2-dhivehi


task: automatic-speech-recognition


tags: pytorch, jax, tensorboard, wav2vec2, pretraining, dv, dataset:common_voice, arxiv:2006.11477, transformers, automatic-speech-recognition


This project pretrains a [`roberta-base`](https://huggingface.co/roberta-base) on the *Alemannic* (`als`) data subset of the [OSCAR](https://oscar-corpus.com/) corpus in JAX/Flax.


We will be using the masked-language modeling loss for pretraining.


model name: roberta-base-als


model author: flax-community


task: fill-mask


tags: jax, tensorboard, roberta, fill-mask, transformers, autonlp_compatible, infinity_compatible


language:


- fi
license: apache-2.0
tags:
- finnish
- roberta
datasets:
- mc4
widget:
- text: "Moikka olen <mask> kielimalli."


# NOTE: We have trained newer and better Finnish RoBERTa large model which can be found from different repository: [https://huggingface.co/Finnish-NLP/roberta-large-finnish](https://huggingface.co/Finnish-NLP/roberta-large-finnish). Our future Finnish models will be available at the [Finnish-NLP](https://huggingface.co/Finnish-NLP) Hugging Face organization


# RoBERTa large model for Finnish


Pretrained model on Finnish language using a masked language modeling (MLM) objective. It was introduced in
[this paper](https://arxiv.org/abs/1907.11692) and first released in
[this repository](https://github.com/pytorch/fairseq/tree/master/examples/roberta). This model is case-sensitive: it
makes a difference between finnish and Finnish.


## Model description


RoBERTa is a transformers model pretrained on a large corpus of Finnish data in a self-supervised fashion. This means
it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
publicly available data) with an automatic process to generate inputs and labels from those texts.


More precisely, it was pretrained with the Masked language modeling (MLM) objective. Taking a sentence, the model
randomly masks 15% of the words in the input then run the entire masked sentence through the model and has to predict
the masked words. This is different from traditional recurrent neural networks (RNNs) that usually see the words one
after the other, or from autoregressive models like GPT which internally mask the future tokens. It allows the model to
learn a bidirectional representation of the sentence.


This way, the model learns an inner representation of the Finnish language that can then be used to extract features
useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
classifier using the features produced by the RoBERTa model as inputs.


## Intended uses & limitations


You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.


Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
generation you should look at model like GPT2.


### How to use


You can use this model directly with a pipeline for masked language modeling:


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='flax-community/RoBERTa-large-finnish')
>>> unmasker("Moikka olen <mask> kielimalli.")

[{'sequence': 'Moikka olen uusi kielimalli.',
  'score': 0.05129234120249748,
  'token': 1825,
  'token_str': ' uusi'},
 {'sequence': 'Moikka olen toinen kielimalli.',
  'score': 0.03112379088997841,
  'token': 2194,
  'token_str': ' toinen'},
 {'sequence': 'Moikka olen myös kielimalli.',
  'score': 0.025534993037581444,
  'token': 491,
  'token_str': ' myös'},
 {'sequence': 'Moikka olen ensimmäinen kielimalli.',
  'score': 0.020146571099758148,
  'token': 2832,
  'token_str': ' ensimmäinen'},
 {'sequence': 'Moikka olen vapaa kielimalli.',
  'score': 0.018089469522237778,
  'token': 2257,
  'token_str': ' vapaa'}]
```


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import RobertaTokenizer, RobertaModel
tokenizer = RobertaTokenizer.from_pretrained('flax-community/RoBERTa-large-finnish')
model = RobertaModel.from_pretrained('flax-community/RoBERTa-large-finnish')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```


and in TensorFlow:


```python
from transformers import RobertaTokenizer, TFRobertaModel
tokenizer = RobertaTokenizer.from_pretrained('flax-community/RoBERTa-large-finnish')
model = TFRobertaModel.from_pretrained('flax-community/RoBERTa-large-finnish', from_pt=True)
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```


### Limitations and bias


The training data used for this model contains a lot of unfiltered content from the internet, which is far from
neutral. Therefore, the model can have biased predictions.


## Training data


This Finnish RoBERTa model was pretrained on the combination of two datasets:


- [mc4](https://huggingface.co/datasets/mc4), the dataset mC4 is a multilingual colossal, cleaned version of Common Crawl's web crawl corpus. We used the Finnish subset of the mC4 dataset
- [Yle Finnish News Archive](http://urn.fi/urn:nbn:fi:lb-2017070501)


Raw datasets were cleaned to filter out bad quality and non-Finnish examples. Together these cleaned datasets were around 51GB of text.


## Training procedure


### Preprocessing


The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50265. The inputs of
the model take pieces of 512 contiguous token that may span over documents. The beginning of a new document is marked
with `<s>` and the end of one by `</s>`


The details of the masking procedure for each sentence are the following:


- 15% of the tokens are masked.
- In 80% of the cases, the masked tokens are replaced by `<mask>`.
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
- In the 10% remaining cases, the masked tokens are left as is.


Contrary to BERT, the masking is done dynamically during pretraining (e.g., it changes at each epoch and is not fixed).


### Pretraining


The model was trained on TPUv3-8 VM, sponsored by the Hugging Face JAX/Flax community week event, for 2 epochs with a sequence length of 128 and continuing for one more epoch with a sequence length of 512. The optimizer used is Adafactor with a learning rate of 2e-4, \\(\beta_{1} = 0.9\\), \\(\beta_{2} = 0.98\\) and \\(\epsilon = 1e-6\\), learning rate warmup for 1500 steps and linear decay of the learning rate after.


## Evaluation results


Evaluation was done by fine-tuning the model on downstream text classification task with two different labeled datasets: [Yle News](https://github.com/spyysalo/yle-corpus) and [Eduskunta](https://github.com/aajanki/eduskunta-vkk). Yle News classification fine-tuning was done with two different sequence lengths: 128 and 512 but Eduskunta only with 128 sequence length.
When fine-tuned on those datasets, this model (the first row of the table) achieves the following accuracy results compared to the [FinBERT (Finnish BERT)](https://huggingface.co/TurkuNLP/bert-base-finnish-cased-v1) and to our newer [Finnish RoBERTa-large](https://huggingface.co/Finnish-NLP/roberta-large-finnish) trained with larger dataset:


|                                        | Average  | Yle News 128 length | Yle News 512 length | Eduskunta 128 length |
|----------------------------------------|----------|---------------------|---------------------|----------------------|
|flax-community/RoBERTa-large-finnish    |87.72     |94.42                |95.06                |73.67                 |
|Finnish-NLP/roberta-large-finnish       |88.02     |94.53                |95.23                |74.30                 |
|TurkuNLP/bert-base-finnish-cased-v1     |**88.82** |**94.90**            |**95.49**            |**76.07**             |


To conclude, this model slightly loses to our newer [Finnish RoBERTa-large](https://huggingface.co/Finnish-NLP/roberta-large-finnish) model trained with larger dataset and also slightly loses to the [FinBERT (Finnish BERT)](https://huggingface.co/TurkuNLP/bert-base-finnish-cased-v1) model.


## Team Members


- Aapo Tanskanen, [Hugging Face profile](https://huggingface.co/aapot), [LinkedIn profile](https://www.linkedin.com/in/aapotanskanen/)
- Rasmus Toivanen [Hugging Face profile](https://huggingface.co/RASMUS), [LinkedIn profile](https://www.linkedin.com/in/rasmustoivanen/)
- Tommi Vehviläinen [Hugging Face profile](https://huggingface.co/Tommi)


Feel free to contact us for more details 🤗


model name: RoBERTa-large-finnish


model author: flax-community


task: fill-mask


tags: pytorch, jax, tensorboard, roberta, fill-mask, fi, dataset:mc4, arxiv:1907.11692, transformers, license:apache-2.0, finnish, autonlp_compatible, infinity_compatible


# Neural ODE with Flax


This is the result of project ["Reproduce Neural ODE and SDE"](https://discuss.huggingface.co/t/reproduce-neural-ode-and-neural-sde/7590) in [HuggingFace Flax/JAX community week](https://github.com/huggingface/transformers/tree/master/examples/research_projects/jax-projects#projects).


main.py will execute training of ResNet or OdeNet for MNIST dataset.


## Dependency


### JAX and Flax


For JAX installation, please follow [here](https://github.com/google/jax#installation).


or simply, type


```bash
pip install jax jaxlib
```


For Flax installation,


```bash
pip install flax
```


Tensorflow-datasets will download MNIST dataset to environment.


## How to run training


For (small) ResNet training,


```bash
python main.py --model=resnet --lr=1e-4 --n_epoch=20 --batch_size=64
```


For Neural ODE training,


```bash
python main.py --model=odenet --lr=1e-4 --n_epoch=20 --batch_size=64
```


For Continuous Normalizing Flow,


```bash
python main.py --model=cnf --sample_dataset=circles
```


Sample datasets can be chosen as circles, moons, or scurve.


# Sample Results


# Bird Call generation Score SDE


These are the codes for the bird call generation score sde model.


core-sde-sampler.py will execute the sampler. The sampler uses pretrained weight to generate bird calls. The weight can be found [here](https://github.com/mandelbrot-walker/Birdcall-score-sde/blob/main/ckpt.flax)


For using different sample generation parameters change the argument values. For example,


```bash
python main.py --sigma=25 --num_steps=500 --signal_to_noise_ratio=0.10 --etol=1e-5 --sample_batch_size = 128 --sample_no = 47
```


In order to generate the audios, these dependencies are required,


```bash
pip install librosa
pip install soundfile
```


In order to train the model from scratch, please generate the dataset using this [link](www.kaggle.com/ibraheemmoosa/birdsong-spectogram-generation). The dataset is generated in kaggle. Therefore, during training your username and api key is required in the specified section inside the script.


```bash
python main.py --sigma=35 --n_epochs=1000 --batch_size=512 --lr=1e-3 --num_steps=500 --signal_to_noise_ratio=0.15 --etol=1e-5 --sample_batch_size = 64 --sample_no = 23
```


Generated samples can be found [here](https://github.com/mandelbrot-walker/Birdcall-score-sde/tree/main/generated_samples)
and [here](https://colab.research.google.com/drive/1AbF4aIMkSfNs-G__MXzqY7JSrz6qvLYN)


model name: NeuralODE_SDE


model author: flax-community


tags:


language: nl
datasets:


- mC4
- Dutch_news


# Pino (Dutch BigBird) base model


Created by [Dat Nguyen](https://www.linkedin.com/in/dat-nguyen-49a641138/) & [Yeb Havinga](https://www.linkedin.com/in/yeb-havinga-86530825/) during the [Hugging Face community week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104)


(Not finished yet)


BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. Moreover, BigBird comes along with a theoretical understanding of the capabilities of a complete transformer that the sparse model can handle.


It is a pretrained model on Dutch language using a masked language modeling (MLM) objective. It was introduced in this [paper](https://arxiv.org/abs/2007.14062) and first released in this [repository](https://github.com/google-research/bigbird).


## Model description


BigBird relies on **block sparse attention** instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower compute cost compared to BERT. It has achieved SOTA on various tasks involving very long sequences such as long documents summarization, question-answering with long contexts.


## How to use


Here is how to use this model to get the features of a given text in PyTorch:


```python
from transformers import BigBirdModel

# by default its in `block_sparse` mode with num_random_blocks=3, block_size=64
model = BigBirdModel.from_pretrained("flax-community/pino-bigbird-roberta-base")

# you can change `attention_type` to full attention like this:
model = BigBirdModel.from_pretrained("flax-community/pino-bigbird-roberta-base", attention_type="original_full")

# you can change `block_size` & `num_random_blocks` like this:
model = BigBirdModel.from_pretrained("flax-community/pino-bigbird-roberta-base", block_size=16, num_random_blocks=2)

```


## Training Data


This model is pre-trained on four publicly available datasets: **mC4**, and scraped **Dutch news** from NRC en Nu.nl. It uses the the fast universal Byte-level BPE (BBPE) in contrast to the sentence piece tokenizer and vocabulary as RoBERTa (which is in turn borrowed from GPT2).


## Training Procedure


The data is cleaned as follows:
Remove texts containing HTML codes / javascript codes / loremipsum / policies
Remove lines without end mark.
Remove too short texts, words
Remove too long texts, words
Remove bad words


## BibTeX entry and citation info


```tex
@misc{zaheer2021big,
      title={Big Bird: Transformers for Longer Sequences},
      author={Manzil Zaheer and Guru Guruganesh and Avinava Dubey and Joshua Ainslie and Chris Alberti and Santiago Ontanon and Philip Pham and Anirudh Ravula and Qifan Wang and Li Yang and Amr Ahmed},
      year={2021},
      eprint={2007.14062},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}
```


model name: pino-bigbird-roberta-base


model author: flax-community


task: fill-mask


tags: pytorch, jax, tensorboard, big_bird, fill-mask, nl, dataset:mC4, dataset:Dutch_news, arxiv:2007.14062, transformers, autonlp_compatible


language:


- ar
datasets:
- mc4
- oscar
- arabic_billion_words


# arabic-t5-small


This is a T5v1.1 (small) trained on the concatenation of the Arabic Billion Words corpus and the Arabic subsets of the mC4 and Oscar datasets.


The model could only be trained for about `10%` of the whole dataset due to time limitations. This is equivalent to `22'000` steps or about `4.3` Billion tokens.


## Training parameters


|                       |               |
| :-------------------: | :-----------: |
|  Training batch size  |     `384`     |
| Evaluation batch size |     `768`     |
|     learning rate     |    `1e-2`     |
|         dtype         | `jnp.float32` |


## Preprocessing and the tokenizer


We tried to keep the preprocessing to a bare minimum. We only replaced URLs, emails and social media user mentions with fixed tokens.


Contrary to other pretrained Arabic LMs, we decided to not strip the Arabic diacritics and to keep them part of the vocabulary.


The tokenizer was trained on `5%` of the training set, with a vocabulary size of `64'000`.


For more details about preprocessing, check the [tokenizer code](https://huggingface.co/flax-community/arabic-t5-small/blob/main/t5_tokenizer_model.py)


## Data


The model was trained on the concatenation of the Arabic Billion Words corpus and the Arabic subsets of the mC4 and Oscar datasets.


A random `0.1%` subset of the data was reserved for evaluation and the rest for training.


## Results


|                     |               |
| :-----------------: | :-----------: |
| Evaluation accuracy |   `56.84%`    |
|   Evaluation Loss   |    `2.423`    |
|    Training Loss    |    `2.392`    |
|    Training Time    | `22h 23m 51s` |


## Note for finetuning


This model was pretrained with dropout turned off, so the default `dropout_rate` in the model config is `0`.
To finetune the model dropout should be turned be back on, like this:


```python
model = T5ForConditionalGeneration.from_pretrained("flax-community/arabic-t5-small", dropout_rate=0.1)
```


or,


```python
model = AutoModelForSeq2SeqLM.from_pretrained("flax-community/arabic-t5-small", dropout_rate=0.1)
```


model name: arabic-t5-small


model author: flax-community


task: text2text-generation


tags: pytorch, tf, jax, tensorboard, t5, text2text-generation, ar, dataset:mc4, dataset:oscar, dataset:arabic_billion_words, transformers, autonlp_compatible


# GPT2-large-indonesian


model name: gpt2-large-indonesian


task: text-generation


tags: gpt2, text-generation, transformers


language:


- dutch
tags:
- summarization
- seq2seq
- text-generation
datasets:
- cnn_dailymail
- xsum
pipeline_tag: text2text-generation
widget:
- text: "Onderzoekers ontdekten dat vier van de vijf kinderen in Engeland die op school lunches hadden gegeten, op school voedsel hadden geprobeerd dat ze thuis niet hadden geprobeerd.De helft van de ondervraagde ouders zei dat hun kinderen hadden gevraagd om voedsel dat ze op school hadden gegeten om thuis te worden gekookt.De enquête, van ongeveer 1.000 ouders, vond dat de meest populaire groenten wortelen, suikermaïs en erwten waren.Aubergine, kikkererwten en spinazie waren een van de minst populaire.Van de ondervraagde ouders, 628 hadden kinderen die lunches op school aten. (% duidt op een deel van de ouders die zeiden dat hun kind elke groente zou eten) England's School Food Trust gaf opdracht tot het onderzoek na een onderzoek door de Mumsnet-website suggereerde dat sommige ouders hun kinderen lunchpakket gaven omdat ze dachten dat ze te kieskeurig waren om iets anders te eten. \"Schoolmaaltijden kunnen een geweldige manier zijn om ouders te helpen hun kinderen aan te moedigen om nieuw voedsel te proberen en om de verscheidenheid van voedsel in hun dieet te verhogen. \"Mumsnet medeoprichter, Carrie Longton, zei: \"Het krijgen van kinderen om gezond te eten is de droom van elke ouder, maar maaltijdtijden thuis kan vaak een slagveld en emotioneel geladen zijn. \"Vanuit Mumsnetters' ervaring lijkt het erop dat eenmaal op school is er een verlangen om in te passen bij iedereen anders en zelfs een aantal positieve peer pressure om op te scheppen over de verscheidenheid van wat voedsel je kunt eten. \"Schoolmaaltijden zijn ook verplaatst op nogal een beetje van toen Mumsnetters op school waren, met gezondere opties en meer afwisseling. \"Schoolmaaltijden in Engeland moeten nu voldoen aan strenge voedingsrichtlijnen.Ongeveer vier op de tien basisschoolkinderen in Engeland eten nu schoollunches, iets meer dan op middelbare scholen.Meer kinderen in Schotland eten schoollunches - ongeveer 46%.Het onderzoek werd online uitgevoerd tussen 26 februari en 5 maart onder een panel van ouders die ten minste één kind op school hadden van 4-17 jaar oud."
- text: "Het Londense trio staat klaar voor de beste Britse act en beste album, evenals voor twee nominaties in de beste song categorie. \"We kregen te horen zoals vanmorgen 'Oh I think you're genomineerd',\" zei Dappy. \"En ik was als 'Oh yeah, what one?' En nu zijn we genomineerd voor vier awards. Ik bedoel, wow! \"Bandmate Fazer voegde eraan toe: \"We dachten dat het het beste van ons was om met iedereen naar beneden te komen en hallo te zeggen tegen de camera's.En nu vinden we dat we vier nominaties hebben. \"De band heeft twee shots bij de beste song prijs, het krijgen van het knikje voor hun Tyncy Stryder samenwerking nummer één, en single Strong Again.Their album Uncle B zal ook gaan tegen platen van Beyonce en Kany \"Aan het eind van de dag zijn we dankbaar om te zijn waar we zijn in onze carrières. \"Als het niet gebeurt dan gebeurt het niet - live om te vechten een andere dag en blijven maken albums en hits voor de fans. \"Dappy onthulde ook dat ze kunnen worden optreden live op de avond.De groep zal doen Nummer Een en ook een mogelijke uitlevering van de War Child single, I Got Soul.Het liefdadigheidslied is een re-working van The Killers' All These Things That I've Done en is ingesteld op artiesten als Chipmunk, Ironik en Pixie Lott.Dit jaar zal Mobos worden gehouden buiten Londen voor de eerste keer, in Glasgow op 30 september.N-Dubz zei dat ze op zoek waren naar optredens voor hun Schotse fans en bogen over hun recente shows ten noorden van de Londense We hebben Aberdeen ongeveer drie of vier maanden geleden gedaan - we hebben die show daar verbrijzeld! Overal waar we heen gaan slaan we hem in elkaar!\""


# t5-base-dutch-demo 📰


Created by [Yeb Havinga](https://www.linkedin.com/in/yeb-havinga-86530825/) & [Dat Nguyen](https://www.linkedin.com/in/dat-nguyen-49a641138/) during the [Hugging Face community week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104)


This model is based on [t5-base-dutch](https://huggingface.co/flax-community/t5-base-dutch)
and fine-tuned to create summaries of news articles.


For a demo of the model, head over to the Hugging Face Spaces for the **[Netherformer 📰](https://huggingface.co/spaces/flax-community/netherformer)** example application!


## Dataset


`t5-base-dutch-demo` is fine-tuned on three mixed news sources:


1. **CNN DailyMail** translated to Dutch with MarianMT.
1. **XSUM** translated to Dutch with MarianMt.
1. News article summaries distilled from the nu.nl website.


The total number of training examples in this dataset is 1366592.


## Training


Training consisted of fine-tuning [t5-base-dutch](https://huggingface.co/flax-community/t5-base-dutch) with
the following parameters:


* Constant learning rate 0.0005
* Batch size 8
* 1 epoch (170842 steps)


## Evaluation


The performance of the summarization model is measured with the Rouge metric from the
Huggingface Datasets library.


```
    "rouge{n}" (e.g. `"rouge1"`, `"rouge2"`) where: {n} is the n-gram based scoring,
    "rougeL": Longest common subsequence based scoring.
```


* Rouge1: 23.8
* Rouge2: 6.9
* RougeL: 19.7


These scores are expected to improve if the model is trained with evaluation configured
for the CNN DM and XSUM datasets (translated to Dutch) individually.


model name: t5-base-dutch-demo


model author: flax-community


task: text2text-generation


tags: pytorch, jax, tensorboard, t5, text2text-generation, dutch, dataset:cnn_dailymail, dataset:xsum, transformers, summarization, seq2seq, text-generation, autonlp_compatible


# Covid19 Related Question Answering (Closed book question answering)