Translation

Text Generation

Fill-Mask

Text Classification

Text2Text Generation

Question Answering

Token Classification

Table Question Answering

Zero-Shot Classification

Automatic Speech Recognition

PyTorch TensorFlow Rust Asteroid TF SavedModel Flair TF Lite ESPnet Pyannote ONNX + 8

wikipedia c4 squad bookcorpus CLUECorpusSmall dcep europarl jrc-acquis squad_v2 oscar cnn_dailymail PropBank.Br jrc-acquis OSIAN 1.5B Arabic Corpus gigaword xsum OSCAR Arabic Unshuffled natural_questions wikisql CoNLL-2012 Indo4B lince multi_nli code_search_net wmt19 OPUS sep_clean conll2003 blended_skill_talk OpenLegalData imdb wtq enh_single ai-soco msr_sqa tab_fact mnli fever mc4 xtreme Libri1Mix openwebtext snli piaf flaubert common_crawl biomedical literature from Scielo and Pubmed squad2 arabic_billion_words open_subtitles twitter SAIL 2017 Libri2Mix sep_noisy Libri3Mix trivia_qa web_questions quoref docred gap winograd_wsc winogender glue FQuAD SQuAD-FR emotion wiki_dpr PubMed wmt16 reddit singapore, malaysia hardwarezone ComVE wham Universal Dependencies opus100 brWaC xnli MIMIC-III BFD STSbenchmark fquad anli wmt14 mlqa DAGW muchocine race wikipedia-turkish indosum scancode-rules MS MARCO document ranking scientific_papers trec Twitter IndianPolitics id_liputan6 Indonesian Wikipedia dihard MNLI jsut ljspeech Wikipedia common_gen break_data sst-2 eli5 AI4Bharat IndicNLP Corpora yahoo-answers 220M words (IndoWiki, IndoWC, News) quora Uniref100 multi_nli_mismatch Shuffled Dutch section of the OSCAR corpus (https://oscar-corpus.com/) pubmed nadi yelp_polarity msmarco id_newspapers_2018 Wikihow emo triviaqa event2Mind augmented_codesearchnet Wikipedia (Hindi, Sanskrit, Gujarati) germeval_14 Arabic Wikipedia https://github.com/wangcunxiang/SemEval2020-Task4-Commonsense-Validation-and-Explanation discofuse coqa go_emotions ai2_arc openbookqa custom-book-corpus codexglue dindebat.dk hestenettet.dk danish OpenSubtitles custom danish dataset JW300 100GB Chinese corpus CC100 Arabic poetry from several eras SciDocs cc100 coco RuTweetCorp RuReviews RuSentiment librispeech https://github.com/staeiou/arxiv_archive/tree/v1.0.1 kowiki news sail voxceleb tatoeba setimes csmsc sqa libri1mix tydiqa bible_para qasc quarel quartz CC-aligned ag_news Icelandic portion of the OSCAR corpus from INRIA Spotify Podcasts Dataset socian bangla-sentiment-benchmark CommonCrawl urdu-text-news + 163

en es fr sv fi de zh multilingual ru ar it uk eo pt tr id nl pl bg ja he da fa hi af el ca no ro cs is hu et lt Chinese vi ht sl tl ko bn hr mt gu ig sg mk ur lv gl ny rw ee sn ts ln xh lg mr yo si rn eu be ty or to sm as nso fy lb sq nb fj nn yi gaa ha niu guw tn bcl crs co wa mh te fo ga ceb efi gil ilo lua pag pap pon bzs iso pis st rm oc an cy am ti hy zu sk loz kg lus swc tll tvl th gv tw bem bi hil lu ase lue war os ps gd so se ml kqn toi srn mn ka br km wo kn ta kw mg ho run tiv tpi wls zne chk az ber sc kwy mfe rnd ve yap tum c++ ne tk tt mi jv mos sh my roa code cv lo na ba yue sa sw protein su kl bo io ce ab pa afa english cel cpp fiu gem inc ine sla gmq gmw iir itc sem umb urj zle zls zlw English kk jap ch gn nv mul eng trk mkh bat bnt cpf cus dra kj lun luo ng nic om phi fse aav aed alv csg csn euq grk kwn nyk pqe sal ss mfs prl tzo zai Cszech Deustch French Swedish dv italian zh-tw ks sd scientific english vn kab scandinavia art cau ccs map poz pqw sit tdt tut yua sami ssp taw vsl wal ach Cszech Deustch Cszech English Cszech Spanish Cszech French Cszech Italian Cszech Swedish Deustch Cszech Deustch English Deustch Spanish Deustch French Deustch Italian Deustch Swedish French Cszech French Deustch French English French Spanish French Italian French Swedish Italian Cszech Go Java javascript php python hi-en nr sot ven xho zul [en] pt-br esperanto bs haw hmn ku ky la sr tg ug uz + 309

apache-2.0 mit gpl-3.0 cc-by-sa-3.0 cc-by-4.0 apache 2.0 public domain notice cc-by-nc-4.0 cc-by-sa-4.0 cc-by 4.0 apache license 2.0 apache gnu gplv3 + 11