sileod commited on
Commit
0b5abeb
1 Parent(s): 3b4db42

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +266 -0
README.md ADDED
@@ -0,0 +1,266 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language: en
4
+ tags:
5
+ - deberta-v3-large
6
+ - text-classification
7
+ - nli
8
+ - natural-language-inference
9
+ - multitask
10
+ - multi-task
11
+ - pipeline
12
+ - extreme-multi-task
13
+ - extreme-mtl
14
+ - tasksource
15
+ - zero-shot
16
+ - rlhf
17
+ pipeline_tag: zero-shot-classification
18
+ datasets:
19
+ - glue
20
+ - super_glue
21
+ - anli
22
+ - metaeval/babi_nli
23
+ - sick
24
+ - snli
25
+ - scitail
26
+ - hans
27
+ - alisawuffles/WANLI
28
+ - metaeval/recast
29
+ - sileod/probability_words_nli
30
+ - joey234/nan-nli
31
+ - pietrolesci/nli_fever
32
+ - pietrolesci/breaking_nli
33
+ - pietrolesci/conj_nli
34
+ - pietrolesci/fracas
35
+ - pietrolesci/dialogue_nli
36
+ - pietrolesci/mpe
37
+ - pietrolesci/dnc
38
+ - pietrolesci/gpt3_nli
39
+ - pietrolesci/recast_white
40
+ - pietrolesci/joci
41
+ - martn-nguyen/contrast_nli
42
+ - pietrolesci/robust_nli
43
+ - pietrolesci/robust_nli_is_sd
44
+ - pietrolesci/robust_nli_li_ts
45
+ - pietrolesci/gen_debiased_nli
46
+ - pietrolesci/add_one_rte
47
+ - metaeval/imppres
48
+ - pietrolesci/glue_diagnostics
49
+ - hlgd
50
+ - paws
51
+ - quora
52
+ - medical_questions_pairs
53
+ - conll2003
54
+ - Anthropic/hh-rlhf
55
+ - Anthropic/model-written-evals
56
+ - truthful_qa
57
+ - nightingal3/fig-qa
58
+ - tasksource/bigbench
59
+ - bigbench
60
+ - blimp
61
+ - cos_e
62
+ - cosmos_qa
63
+ - dream
64
+ - openbookqa
65
+ - qasc
66
+ - quartz
67
+ - quail
68
+ - head_qa
69
+ - sciq
70
+ - social_i_qa
71
+ - wiki_hop
72
+ - wiqa
73
+ - piqa
74
+ - hellaswag
75
+ - pkavumba/balanced-copa
76
+ - 12ml/e-CARE
77
+ - art
78
+ - tasksource/mmlu
79
+ - winogrande
80
+ - codah
81
+ - ai2_arc
82
+ - definite_pronoun_resolution
83
+ - swag
84
+ - math_qa
85
+ - metaeval/utilitarianism
86
+ - mteb/amazon_counterfactual
87
+ - SetFit/insincere-questions
88
+ - SetFit/toxic_conversations
89
+ - turingbench/TuringBench
90
+ - trec
91
+ - tals/vitaminc
92
+ - hope_edi
93
+ - strombergnlp/rumoureval_2019
94
+ - ethos
95
+ - tweet_eval
96
+ - discovery
97
+ - pragmeval
98
+ - silicone
99
+ - lex_glue
100
+ - papluca/language-identification
101
+ - imdb
102
+ - rotten_tomatoes
103
+ - ag_news
104
+ - yelp_review_full
105
+ - financial_phrasebank
106
+ - poem_sentiment
107
+ - dbpedia_14
108
+ - amazon_polarity
109
+ - app_reviews
110
+ - hate_speech18
111
+ - sms_spam
112
+ - humicroedit
113
+ - snips_built_in_intents
114
+ - banking77
115
+ - hate_speech_offensive
116
+ - yahoo_answers_topics
117
+ - pacovaldez/stackoverflow-questions
118
+ - zapsdcn/hyperpartisan_news
119
+ - zapsdcn/sciie
120
+ - zapsdcn/citation_intent
121
+ - go_emotions
122
+ - scicite
123
+ - liar
124
+ - relbert/lexical_relation_classification
125
+ - metaeval/linguisticprobing
126
+ - metaeval/crowdflower
127
+ - metaeval/ethics
128
+ - emo
129
+ - google_wellformed_query
130
+ - tweets_hate_speech_detection
131
+ - has_part
132
+ - wnut_17
133
+ - ncbi_disease
134
+ - acronym_identification
135
+ - jnlpba
136
+ - species_800
137
+ - SpeedOfMagic/ontonotes_english
138
+ - blog_authorship_corpus
139
+ - launch/open_question_type
140
+ - health_fact
141
+ - commonsense_qa
142
+ - mc_taco
143
+ - ade_corpus_v2
144
+ - prajjwal1/discosense
145
+ - circa
146
+ - YaHi/EffectiveFeedbackStudentWriting
147
+ - Ericwang/promptSentiment
148
+ - Ericwang/promptNLI
149
+ - Ericwang/promptSpoke
150
+ - Ericwang/promptProficiency
151
+ - Ericwang/promptGrammar
152
+ - Ericwang/promptCoherence
153
+ - PiC/phrase_similarity
154
+ - copenlu/scientific-exaggeration-detection
155
+ - quarel
156
+ - mwong/fever-evidence-related
157
+ - numer_sense
158
+ - dynabench/dynasent
159
+ - raquiba/Sarcasm_News_Headline
160
+ - sem_eval_2010_task_8
161
+ - demo-org/auditor_review
162
+ - medmcqa
163
+ - aqua_rat
164
+ - RuyuanWan/Dynasent_Disagreement
165
+ - RuyuanWan/Politeness_Disagreement
166
+ - RuyuanWan/SBIC_Disagreement
167
+ - RuyuanWan/SChem_Disagreement
168
+ - RuyuanWan/Dilemmas_Disagreement
169
+ - lucasmccabe/logiqa
170
+ - wiki_qa
171
+ - metaeval/cycic_classification
172
+ - metaeval/cycic_multiplechoice
173
+ - metaeval/sts-companion
174
+ - metaeval/commonsense_qa_2.0
175
+ - metaeval/lingnli
176
+ - metaeval/monotonicity-entailment
177
+ - metaeval/arct
178
+ - metaeval/scinli
179
+ - metaeval/naturallogic
180
+ - onestop_qa
181
+ - demelin/moral_stories
182
+ - corypaik/prost
183
+ - aps/dynahate
184
+ - metaeval/syntactic-augmentation-nli
185
+ - metaeval/autotnli
186
+ - lasha-nlp/CONDAQA
187
+ - openai/webgpt_comparisons
188
+ - Dahoas/synthetic-instruct-gptj-pairwise
189
+ - metaeval/scruples
190
+ - metaeval/wouldyourather
191
+ - sileod/attempto-nli
192
+ - metaeval/defeasible-nli
193
+ - metaeval/help-nli
194
+ - metaeval/nli-veridicality-transitivity
195
+ - metaeval/natural-language-satisfiability
196
+ - metaeval/lonli
197
+ - metaeval/dadc-limit-nli
198
+ - ColumbiaNLP/FLUTE
199
+ - metaeval/strategy-qa
200
+ - openai/summarize_from_feedback
201
+ - metaeval/folio
202
+ - metaeval/tomi-nli
203
+ - metaeval/avicenna
204
+ - stanfordnlp/SHP
205
+ - GBaker/MedQA-USMLE-4-options-hf
206
+ - sileod/wikimedqa
207
+ - declare-lab/cicero
208
+ - amydeng2000/CREAK
209
+ - metaeval/mutual
210
+ - inverse-scaling/NeQA
211
+ - inverse-scaling/quote-repetition
212
+ - inverse-scaling/redefine-math
213
+ - metaeval/puzzte
214
+ - metaeval/implicatures
215
+ - race
216
+ - metaeval/spartqa-yn
217
+ - metaeval/spartqa-mchoice
218
+ - metaeval/temporal-nli
219
+ metrics:
220
+ - accuracy
221
+ library_name: transformers
222
+ ---
223
+
224
+ # Model Card for DeBERTa-v3-base-tasksource-nli
225
+
226
+ DeBERTa-v3-large fine-tuned with multi-task learning on 520 tasks of the [tasksource collection](https://github.com/sileod/tasksource/)
227
+ You can further fine-tune this model to use it for any classification or multiple-choice task.
228
+ This checkpoint has strong zero-shot validation performance on many tasks (e.g. 77% on WNLI).
229
+ The untuned model CLS embedding also has strong linear probing performance (90% on MNLI), due to the multitask training.
230
+
231
+ This is the shared model with the MNLI classifier on top. Its encoder was trained on many datasets including bigbench, Anthropic rlhf, anli... alongside many NLI and classification tasks with a SequenceClassification heads while using only one shared encoder.
232
+ Each task had a specific CLS embedding, which is dropped 10% of the time to facilitate model use without it. All multiple-choice model used the same classification layers. For classification tasks, models shared weights if their labels matched.
233
+ The number of examples per task was capped to 64k. The model was trained for 45k steps with a batch size of 384, and a peak learning rate of 2e-5.
234
+
235
+
236
+ tasksource training code: https://colab.research.google.com/drive/1iB4Oxl9_B5W3ZDzXoWJN-olUbqLBxgQS?usp=sharing
237
+
238
+ ### Software
239
+ https://github.com/sileod/tasksource/ \
240
+ https://github.com/sileod/tasknet/ \
241
+ Training took 6 days on Nvidia A100 40GB gpu.
242
+
243
+
244
+ # Citation
245
+
246
+ More details on this [article:](https://arxiv.org/abs/2301.05948)
247
+ ```bib
248
+ @article{sileo2023tasksource,
249
+ title={tasksource: Structured Dataset Preprocessing Annotations for Frictionless Extreme Multi-Task Learning and Evaluation},
250
+ author={Sileo, Damien},
251
+ url= {https://arxiv.org/abs/2301.05948},
252
+ journal={arXiv preprint arXiv:2301.05948},
253
+ year={2023}
254
+ }
255
+ ```
256
+
257
+ # Loading a specific classifier
258
+ Classifiers for all tasks available. See https://huggingface.co/sileod/deberta-v3-large-tasksource-adapters
259
+
260
+
261
+ # Model Card Contact
262
+
263
+ damien.sileo@inria.fr
264
+
265
+
266
+ </details>