Translation

Text Generation

Fill-Mask

Text Classification

Text2Text Generation

Question Answering

Token Classification

Table Question Answering

Zero-Shot Classification

Automatic Speech Recognition

PyTorch TensorFlow Rust Asteroid TF SavedModel Flair TF Lite ESPnet Pyannote ONNX + 8

wikipedia c4 bookcorpus squad dcep europarl jrc-acquis squad_v2 oscar cnn_dailymail PropBank.Br jrc-acquis OSIAN 1.5B Arabic Corpus gigaword xsum OSCAR Arabic Unshuffled natural_questions wikisql CoNLL-2012 Indo4B lince multi_nli code_search_net wmt19 sep_clean conll2003 blended_skill_talk OPUS OpenLegalData wtq enh_single ai-soco msr_sqa tab_fact mnli fever mc4 imdb CLUECorpusSmall xtreme Libri1Mix openwebtext snli piaf common_crawl flaubert biomedical literature from Scielo and Pubmed squad2 SAIL 2017 arabic_billion_words open_subtitles twitter Libri2Mix sep_noisy Libri3Mix trivia_qa web_questions quoref docred gap winograd_wsc winogender glue FQuAD SQuAD-FR wiki_dpr PubMed emotion wmt16 reddit singapore, malaysia hardwarezone ComVE wham Universal Dependencies CLUECorpus brWaC xnli MIMIC-III BFD STSbenchmark fquad anli wmt14 mlqa DAGW race wikipedia-turkish indosum scancode-rules scientific_papers Twitter IndianPolitics Indonesian Wikipedia dihard MNLI jsut ljspeech Wikipedia common_gen muchocine break_data sst-2 eli5 Uniref100 AI4Bharat IndicNLP Corpora yahoo-answers 220M words (IndoWiki, IndoWC, News) quora multi_nli_mismatch Shuffled Dutch section of the OSCAR corpus (https://oscar-corpus.com/) nadi pubmed yelp_polarity Wikihow emo event2Mind triviaqa id_newspapers_2018 Wikipedia (Hindi, Sanskrit, Gujarati) augmented_codesearchnet msmarco germeval_14 discofuse https://github.com/wangcunxiang/SemEval2020-Task4-Commonsense-Validation-and-Explanation Arabic Wikipedia coqa go_emotions ai2_arc openbookqa custom-book-corpus codexglue dindebat.dk hestenettet.dk danish OpenSubtitles custom danish dataset JW300 100GB Chinese corpus CC100 Arabic poetry from several eras cc100 RuTweetCorp RuReviews RuSentiment librispeech id_liputan6 https://github.com/staeiou/arxiv_archive/tree/v1.0.1 kowiki news sail voxceleb tatoeba setimes csmsc sqa libri1mix tydiqa bible_para qasc quarel quartz ag_news Icelandic portion of the OSCAR corpus from INRIA Spotify Podcasts Dataset socian bangla-sentiment-benchmark CommonCrawl urdu-text-news + 158

en es fr sv fi de zh multilingual ru ar it uk eo pt tr nl id pl bg he ja da fa hi af el ca no ro cs is hu et lt vi ht sl tl ko bn hr mt gu ig mk ur lv gl ny rw ee sn ts ln xh lg mr yo si rn eu be ty or to sm as nso fy lb sq nb fj nn yi gaa ha guw bcl crs niu tn co wa mh te fo ga ceb efi gil lua bzs ilo iso pag pap pis pon st rm oc an cy am th ti hy zu swc kg loz lus sk tll tvl gv bem bi hil ase lu lue tw war os ps gd so se ml Chinese kqn toi srn mn ka br km wo kn ta kw mg ho run tiv tpi wls zne chk az ber sc kwy mfe rnd ve yap tum c++ ne tk tt mi jv mos sh my roa code cv lo na ba yue sa sw protein su kl bo io ce ab pa afa sla english cel cpp fiu gem gmq gmw iir inc ine itc sem umb urj zle zls zlw kk jap ch gn nv mul eng trk bat bnt cpf cus dra fse aav aed alv csg csn euq grk kj kwn lun luo mkh ng nic nyk om phi pqe sal ss mfs prl tzo zai Cszech Deustch English French Swedish dv italian zh-tw ks sd vn scientific english scandinavia art cau ccs map poz pqw sit tdt tut yua sami kab ssp taw vsl wal ach Cszech Deustch Cszech English Cszech Spanish Cszech French Cszech Italian Cszech Swedish Deustch Cszech Deustch English Deustch Spanish Deustch French Deustch Italian Deustch Swedish French Cszech French Deustch French English French Spanish French Italian French Swedish Italian Cszech Go Java javascript php python hi-en nr sot ven xho zul [en] pt-br esperanto bs haw hmn ku ky la sr tg ug uz + 308

apache-2.0 mit gpl-3.0 apache 2.0 cc-by-sa-3.0 cc-by-4.0 public domain notice cc-by-nc-4.0 cc-by-sa-4.0 cc-by 4.0 apache gnu gplv3 + 10