- Loss: 6.3170


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 2
- eval_batch_size: 2
- seed: 250500
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 6


### Training results


| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 3.7861        | 0.12  | 1000  | 3.2778          |
| 3.2186        | 0.23  | 2000  | 3.0658          |
| 2.8504        | 0.35  | 3000  | 3.0456          |
| 2.8621        | 0.46  | 4000  | 2.8758          |
| 2.7851        | 0.58  | 5000  | 2.8680          |
| 2.8016        | 0.69  | 6000  | 2.9244          |
| 2.7592        | 0.81  | 7000  | 2.7735          |
| 2.5737        | 0.93  | 8000  | 2.7640          |
| 2.3493        | 1.04  | 9000  | 2.7257          |
| 2.1041        | 1.16  | 10000 | 2.8442          |
| 2.1713        | 1.27  | 11000 | 2.7723          |
| 2.0594        | 1.39  | 12000 | 2.9982          |
| 2.1825        | 1.5   | 13000 | 2.8272          |
| 2.2486        | 1.62  | 14000 | 2.8897          |
| 2.097         | 1.74  | 15000 | 2.8557          |
| 2.1645        | 1.85  | 16000 | 2.6342          |
| 2.15          | 1.97  | 17000 | 2.8680          |
| 1.5662        | 2.08  | 18000 | 3.2126          |
| 1.6168        | 2.2   | 19000 | 3.1646          |
| 1.5886        | 2.32  | 20000 | 3.3139          |
| 1.6539        | 2.43  | 21000 | 3.2610          |
| 1.6486        | 2.55  | 22000 | 3.3144          |
| 1.637         | 2.66  | 23000 | 3.0437          |
| 1.7186        | 2.78  | 24000 | 2.9936          |
| 1.7543        | 2.89  | 25000 | 3.1641          |
| 1.5301        | 3.01  | 26000 | 4.0560          |
| 1.1436        | 3.13  | 27000 | 4.0116          |
| 1.1902        | 3.24  | 28000 | 4.0240          |
| 1.2728        | 3.36  | 29000 | 4.3068          |
| 1.2586        | 3.47  | 30000 | 3.7894          |
| 1.3164        | 3.59  | 31000 | 3.9242          |
| 1.3093        | 3.7   | 32000 | 4.0444          |
| 1.2812        | 3.82  | 33000 | 4.1779          |
| 1.3165        | 3.94  | 34000 | 3.6633          |
| 0.8357        | 4.05  | 35000 | 5.8137          |
| 0.9583        | 4.17  | 36000 | 5.3305          |
| 0.9135        | 4.28  | 37000 | 5.4973          |
| 1.0011        | 4.4   | 38000 | 5.0349          |
| 0.9553        | 4.51  | 39000 | 5.2086          |
| 1.0182        | 4.63  | 40000 | 5.1197          |
| 0.9569        | 4.75  | 41000 | 5.4579          |
| 0.9437        | 4.86  | 42000 | 5.4467          |
| 0.9791        | 4.98  | 43000 | 4.7657          |
| 0.648         | 5.09  | 44000 | 6.5780          |
| 0.7528        | 5.21  | 45000 | 6.2827          |
| 0.7247        | 5.33  | 46000 | 6.8500          |
| 0.702         | 5.44  | 47000 | 6.4572          |
| 0.6786        | 5.56  | 48000 | 6.5462          |
| 0.7272        | 5.67  | 49000 | 6.2406          |
| 0.6778        | 5.79  | 50000 | 6.4727          |
| 0.6446        | 5.9   | 51000 | 6.3170          |


### Framework versions


- Transformers 4.10.0
- Pytorch 1.8.0+cu101
- Datasets 1.11.0
- Tokenizers 0.10.3


model name: bert-large-uncased-finetuned-infovqa


model author: tiennvcs


task: question-answering


tags: pytorch, tensorboard, bert, question-answering, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: cc-by-nc-sa-4.0
tags:


- generated_from_trainer
model-index:
- name: layoutlmv2-large-uncased-finetuned-vi-infovqa
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# layoutlmv2-large-uncased-finetuned-vi-infovqa


This model is a fine-tuned version of [microsoft/layoutlmv2-large-uncased](https://huggingface.co/microsoft/layoutlmv2-large-uncased) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 8.5806


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 0.17  | 100  | 4.6181          |
| No log        | 0.33  | 200  | 4.3357          |
| No log        | 0.5   | 300  | 4.3897          |
| No log        | 0.66  | 400  | 4.8238          |
| 4.4277        | 0.83  | 500  | 3.9088          |
| 4.4277        | 0.99  | 600  | 3.6063          |
| 4.4277        | 1.16  | 700  | 3.4278          |
| 4.4277        | 1.32  | 800  | 3.5428          |
| 4.4277        | 1.49  | 900  | 3.4331          |
| 3.0413        | 1.65  | 1000 | 3.3699          |
| 3.0413        | 1.82  | 1100 | 3.3622          |
| 3.0413        | 1.98  | 1200 | 3.5294          |
| 3.0413        | 2.15  | 1300 | 3.7918          |
| 3.0413        | 2.31  | 1400 | 3.4007          |
| 2.0843        | 2.48  | 1500 | 4.0296          |
| 2.0843        | 2.64  | 1600 | 4.1852          |
| 2.0843        | 2.81  | 1700 | 3.6690          |
| 2.0843        | 2.97  | 1800 | 3.6089          |
| 2.0843        | 3.14  | 1900 | 5.5534          |
| 1.7527        | 3.3   | 2000 | 4.7498          |
| 1.7527        | 3.47  | 2100 | 5.2691          |
| 1.7527        | 3.63  | 2200 | 5.1324          |
| 1.7527        | 3.8   | 2300 | 4.5912          |
| 1.7527        | 3.96  | 2400 | 4.1727          |
| 1.2037        | 4.13  | 2500 | 6.1174          |
| 1.2037        | 4.29  | 2600 | 5.7172          |
| 1.2037        | 4.46  | 2700 | 5.8843          |
| 1.2037        | 4.62  | 2800 | 6.4232          |
| 1.2037        | 4.79  | 2900 | 7.4486          |
| 0.8386        | 4.95  | 3000 | 7.1946          |
| 0.8386        | 5.12  | 3100 | 7.9869          |
| 0.8386        | 5.28  | 3200 | 8.0310          |
| 0.8386        | 5.45  | 3300 | 8.2954          |
| 0.8386        | 5.61  | 3400 | 8.5361          |
| 0.4389        | 5.78  | 3500 | 8.6040          |
| 0.4389        | 5.94  | 3600 | 8.5806          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.8.0+cu101
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: layoutlmv2-large-uncased-finetuned-vi-infovqa


model author: tiennvcs


task: question-answering


tags: pytorch, tensorboard, layoutlmv2, question-answering, transformers, license:cc-by-nc-sa-4.0, generated_from_trainer, model-index, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: bert-large-uncased-finetuned-vi-infovqa
results: []


# bert-large-uncased-finetuned-vi-infovqa


This model is a fine-tuned version of [bert-large-uncased](https://huggingface.co/bert-large-uncased) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 7.4878


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 2
- eval_batch_size: 2
- seed: 250500
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 6


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 0.11  | 100  | 4.6256          |
| No log        | 0.21  | 200  | 4.4042          |
| No log        | 0.32  | 300  | 5.0021          |
| No log        | 0.43  | 400  | 4.2825          |
| 4.6758        | 0.53  | 500  | 4.3886          |
| 4.6758        | 0.64  | 600  | 4.2519          |
| 4.6758        | 0.75  | 700  | 4.2977          |
| 4.6758        | 0.85  | 800  | 3.9916          |
| 4.6758        | 0.96  | 900  | 4.1650          |
| 4.1715        | 1.07  | 1000 | 4.5001          |
| 4.1715        | 1.17  | 1100 | 4.0898          |
| 4.1715        | 1.28  | 1200 | 4.1623          |
| 4.1715        | 1.39  | 1300 | 4.3271          |
| 4.1715        | 1.49  | 1400 | 3.9661          |
| 3.7926        | 1.6   | 1500 | 3.8727          |
| 3.7926        | 1.71  | 1600 | 3.8934          |
| 3.7926        | 1.81  | 1700 | 3.7262          |
| 3.7926        | 1.92  | 1800 | 3.7701          |
| 3.7926        | 2.03  | 1900 | 3.7653          |
| 3.5041        | 2.13  | 2000 | 3.9261          |
| 3.5041        | 2.24  | 2100 | 4.0915          |
| 3.5041        | 2.35  | 2200 | 4.0348          |
| 3.5041        | 2.45  | 2300 | 4.0212          |
| 3.5041        | 2.56  | 2400 | 4.4653          |
| 2.8475        | 2.67  | 2500 | 4.2959          |
| 2.8475        | 2.77  | 2600 | 4.1039          |
| 2.8475        | 2.88  | 2700 | 3.8037          |
| 2.8475        | 2.99  | 2800 | 3.7552          |
| 2.8475        | 3.09  | 2900 | 4.2476          |
| 2.5488        | 3.2   | 3000 | 4.6716          |
| 2.5488        | 3.3   | 3100 | 4.7058          |
| 2.5488        | 3.41  | 3200 | 4.6266          |
| 2.5488        | 3.52  | 3300 | 4.5697          |
| 2.5488        | 3.62  | 3400 | 5.1017          |
| 2.0347        | 3.73  | 3500 | 4.6254          |
| 2.0347        | 3.84  | 3600 | 4.4822          |
| 2.0347        | 3.94  | 3700 | 4.9413          |
| 2.0347        | 4.05  | 3800 | 5.3600          |
| 2.0347        | 4.16  | 3900 | 5.7323          |
| 1.6566        | 4.26  | 4000 | 5.8822          |
| 1.6566        | 4.37  | 4100 | 6.0173          |
| 1.6566        | 4.48  | 4200 | 5.6688          |
| 1.6566        | 4.58  | 4300 | 6.0617          |
| 1.6566        | 4.69  | 4400 | 6.6631          |
| 1.3348        | 4.8   | 4500 | 6.0290          |
| 1.3348        | 4.9   | 4600 | 6.2455          |
| 1.3348        | 5.01  | 4700 | 6.0963          |
| 1.3348        | 5.12  | 4800 | 7.0983          |
| 1.3348        | 5.22  | 4900 | 7.5483          |
| 1.0701        | 5.33  | 5000 | 7.7187          |
| 1.0701        | 5.44  | 5100 | 7.4630          |
| 1.0701        | 5.54  | 5200 | 7.1394          |
| 1.0701        | 5.65  | 5300 | 7.0703          |
| 1.0701        | 5.76  | 5400 | 7.5611          |
| 0.9414        | 5.86  | 5500 | 7.6038          |
| 0.9414        | 5.97  | 5600 | 7.4878          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: bert-large-uncased-finetuned-vi-infovqa


model author: tiennvcs


task: question-answering


tags: pytorch, tensorboard, bert, question-answering, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: bert-base-uncased-finetuned-docvqa
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# bert-base-uncased-finetuned-docvqa


This model is a fine-tuned version of [bert-base-uncased](https://huggingface.co/bert-base-uncased) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 1.9146


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 5e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 250500
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2


### Training results


| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 2.2151        | 0.1   | 1000  | 2.6299          |
| 1.8885        | 0.21  | 2000  | 2.2217          |
| 1.7353        | 0.31  | 3000  | 2.1675          |
| 1.6188        | 0.41  | 4000  | 2.2436          |
| 1.5802        | 0.52  | 5000  | 2.0539          |
| 1.4875        | 0.62  | 6000  | 2.0551          |
| 1.4675        | 0.73  | 7000  | 1.9368          |
| 1.3485        | 0.83  | 8000  | 1.9456          |
| 1.3273        | 0.93  | 9000  | 1.9281          |
| 1.1048        | 1.04  | 10000 | 1.9333          |
| 0.9529        | 1.14  | 11000 | 2.2019          |
| 0.9418        | 1.24  | 12000 | 2.0381          |
| 0.9209        | 1.35  | 13000 | 1.8753          |
| 0.8788        | 1.45  | 14000 | 1.9964          |
| 0.8729        | 1.56  | 15000 | 1.9690          |
| 0.8671        | 1.66  | 16000 | 1.8513          |
| 0.8379        | 1.76  | 17000 | 1.9627          |
| 0.8722        | 1.87  | 18000 | 1.8988          |
| 0.7842        | 1.97  | 19000 | 1.9146          |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.9.0+cu111
- Datasets 1.14.0
- Tokenizers 0.10.3


model name: bert-base-uncased-finetuned-docvqa


model author: tiennvcs


task: question-answering


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: bert-base-uncased-finetuned-infovqa
results:
  - task:
name: Question Answering
type: question-answering


# bert-base-uncased-finetuned-infovqa


- Loss: 2.8276


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 3.2765        | 0.23  | 1000 | 3.0678          |
| 2.9987        | 0.46  | 2000 | 2.9525          |
| 2.826         | 0.69  | 3000 | 2.7870          |
| 2.7084        | 0.93  | 4000 | 2.7051          |
| 2.1286        | 1.16  | 5000 | 2.9286          |
| 2.0009        | 1.39  | 6000 | 3.1037          |
| 2.0323        | 1.62  | 7000 | 2.8567          |
| 1.9905        | 1.85  | 8000 | 2.8276          |


### Framework versions


- Transformers 4.10.0
- Pytorch 1.8.0+cu101
- Datasets 1.11.0
- Tokenizers 0.10.3


model name: bert-base-uncased-finetuned-infovqa


model author: tiennvcs


task: question-answering


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: bert-large-uncased-finetuned-docvqa
results:
  - task:
name: Question Answering
type: question-answering


# bert-large-uncased-finetuned-docvqa


This model is a fine-tuned version of [bert-large-uncased](https://huggingface.co/bert-large-uncased) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 3.6367


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 2
- eval_batch_size: 2
- seed: 250500
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 6


### Training results


| Training Loss | Epoch | Step   | Validation Loss |
|:-------------:|:-----:|:------:|:---------------:|
| 2.5228        | 0.05  | 1000   | 2.6645          |
| 2.4909        | 0.1   | 2000   | 2.8985          |
| 2.1679        | 0.16  | 3000   | 2.3551          |
| 1.9451        | 0.21  | 4000   | 2.2226          |
| 1.6814        | 0.26  | 5000   | 2.1590          |
| 1.8868        | 0.31  | 6000   | 2.6197          |
| 1.6618        | 0.36  | 7000   | 2.3632          |
| 1.8313        | 0.41  | 8000   | 2.4519          |
| 1.7017        | 0.47  | 9000   | 2.2682          |
| 1.8169        | 0.52  | 10000  | 2.4486          |
| 1.7074        | 0.57  | 11000  | 2.3862          |
| 1.7674        | 0.62  | 12000  | 2.1801          |
| 1.8134        | 0.67  | 13000  | 2.3032          |
| 1.8334        | 0.73  | 14000  | 2.4205          |
| 1.6819        | 0.78  | 15000  | 2.2398          |
| 1.5846        | 0.83  | 16000  | 2.3834          |
| 1.6758        | 0.88  | 17000  | 1.9683          |
| 1.6303        | 0.93  | 18000  | 2.3297          |
| 1.5652        | 0.98  | 19000  | 2.0581          |
| 1.3045        | 1.04  | 20000  | 2.4950          |
| 1.2393        | 1.09  | 21000  | 2.6622          |
| 1.1526        | 1.14  | 22000  | 2.3749          |
| 1.2631        | 1.19  | 23000  | 2.3915          |
| 1.1846        | 1.24  | 24000  | 2.2592          |
| 1.2731        | 1.3   | 25000  | 2.4239          |
| 1.3057        | 1.35  | 26000  | 2.2920          |
| 1.134         | 1.4   | 27000  | 2.3107          |
| 1.2017        | 1.45  | 28000  | 2.4271          |
| 1.2202        | 1.5   | 29000  | 2.1814          |
| 1.2179        | 1.56  | 30000  | 2.3365          |
| 1.2359        | 1.61  | 31000  | 2.1256          |
| 1.1964        | 1.66  | 32000  | 2.1720          |
| 1.269         | 1.71  | 33000  | 2.4363          |
| 1.1812        | 1.76  | 34000  | 2.2372          |
| 1.2187        | 1.81  | 35000  | 2.2318          |
| 1.1805        | 1.87  | 36000  | 2.3693          |
| 1.1458        | 1.92  | 37000  | 2.5128          |
| 1.1958        | 1.97  | 38000  | 2.1311          |
| 0.8924        | 2.02  | 39000  | 2.4635          |
| 0.869         | 2.07  | 40000  | 2.8231          |
| 0.8333        | 2.13  | 41000  | 2.6762          |
| 0.9194        | 2.18  | 42000  | 2.4588          |
| 0.8089        | 2.23  | 43000  | 2.6443          |
| 0.8612        | 2.28  | 44000  | 2.4300          |
| 0.7981        | 2.33  | 45000  | 2.7418          |
| 0.9765        | 2.38  | 46000  | 2.6543          |
| 0.8646        | 2.44  | 47000  | 2.5990          |
| 1.0316        | 2.49  | 48000  | 2.4625          |
| 0.9862        | 2.54  | 49000  | 2.4691          |
| 1.027         | 2.59  | 50000  | 2.4156          |
| 0.9412        | 2.64  | 51000  | 2.4204          |
| 0.9353        | 2.7   | 52000  | 2.4933          |
| 0.9509        | 2.75  | 53000  | 2.4708          |
| 0.9351        | 2.8   | 54000  | 2.5351          |
| 0.9968        | 2.85  | 55000  | 2.2506          |
| 1.025         | 2.9   | 56000  | 2.6317          |
| 1.627         | 2.95  | 57000  | 2.7843          |
| 0.9294        | 3.01  | 58000  | 2.9396          |
| 0.6043        | 3.06  | 59000  | 3.1560          |
| 0.7903        | 3.11  | 60000  | 2.8330          |
| 0.7373        | 3.16  | 61000  | 2.9422          |
| 0.6499        | 3.21  | 62000  | 3.0948          |
| 0.6411        | 3.27  | 63000  | 2.7900          |
| 0.625         | 3.32  | 64000  | 2.5268          |
| 0.6264        | 3.37  | 65000  | 2.8701          |
| 0.6143        | 3.42  | 66000  | 3.2544          |
| 0.6286        | 3.47  | 67000  | 2.6208          |
| 0.739         | 3.53  | 68000  | 2.8107          |
| 0.5981        | 3.58  | 69000  | 2.8073          |
| 0.6502        | 3.63  | 70000  | 2.6293          |
| 0.6548        | 3.68  | 71000  | 2.9501          |
| 0.7243        | 3.73  | 72000  | 2.7917          |
| 0.598         | 3.78  | 73000  | 2.9341          |
| 0.6159        | 3.84  | 74000  | 2.7629          |
| 0.5905        | 3.89  | 75000  | 2.6441          |
| 0.6393        | 3.94  | 76000  | 2.6660          |
| 0.677         | 3.99  | 77000  | 2.7616          |
| 0.3281        | 4.04  | 78000  | 3.6873          |
| 0.4524        | 4.1   | 79000  | 3.3441          |
| 0.3994        | 4.15  | 80000  | 3.3129          |
| 0.4686        | 4.2   | 81000  | 3.1813          |
| 0.5293        | 4.25  | 82000  | 2.9088          |
| 0.3961        | 4.3   | 83000  | 3.0765          |
| 0.4406        | 4.35  | 84000  | 3.1254          |
| 0.401         | 4.41  | 85000  | 3.2415          |
| 0.4594        | 4.46  | 86000  | 3.0691          |
| 0.4523        | 4.51  | 87000  | 3.0493          |
| 0.4719        | 4.56  | 88000  | 3.1352          |
| 0.4895        | 4.61  | 89000  | 2.8991          |
| 0.423         | 4.67  | 90000  | 3.1738          |
| 0.3984        | 4.72  | 91000  | 3.1862          |
| 0.4206        | 4.77  | 92000  | 3.1213          |
| 0.4587        | 4.82  | 93000  | 3.0030          |
| 0.381         | 4.87  | 94000  | 3.3218          |
| 0.4138        | 4.92  | 95000  | 3.1529          |
| 0.4003        | 4.98  | 96000  | 3.1375          |
| 0.2098        | 5.03  | 97000  | 3.7443          |
| 0.2334        | 5.08  | 98000  | 3.7359          |
| 0.2534        | 5.13  | 99000  | 3.7814          |
| 0.3067        | 5.18  | 100000 | 3.7128          |
| 0.2363        | 5.24  | 101000 | 3.6091          |
| 0.2652        | 5.29  | 102000 | 3.4015          |
| 0.3311        | 5.34  | 103000 | 3.4793          |
| 0.2344        | 5.39  | 104000 | 3.6792          |
| 0.2741        | 5.44  | 105000 | 3.5385          |
| 0.2896        | 5.5   | 106000 | 3.8118          |
| 0.2071        | 5.55  | 107000 | 3.8690          |
| 0.3023        | 5.6   | 108000 | 3.7087          |
| 0.3299        | 5.65  | 109000 | 3.4925          |
| 0.1943        | 5.7   | 110000 | 3.6739          |
| 0.2488        | 5.75  | 111000 | 3.7614          |
| 0.3138        | 5.81  | 112000 | 3.5156          |
| 0.2555        | 5.86  | 113000 | 3.6056          |
| 0.2918        | 5.91  | 114000 | 3.6533          |
| 0.2751        | 5.96  | 115000 | 3.6367          |


### Framework versions


model name: bert-large-uncased-finetuned-docvqa


model author: tiennvcs


task: question-answering


tags: pytorch, tensorboard, bert, question-answering, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: cc-by-nc-sa-4.0
tags:


- generated_from_trainer
model-index:
- name: layoutlmv2-large-uncased-finetuned-infovqa
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# layoutlmv2-large-uncased-finetuned-infovqa


This model is a fine-tuned version of [microsoft/layoutlmv2-large-uncased](https://huggingface.co/microsoft/layoutlmv2-large-uncased) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 2.2207


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 5e-05
- train_batch_size: 2
- eval_batch_size: 2
- seed: 250500
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2


### Training results


| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 4.1829        | 0.08  | 500   | 3.6339          |
| 3.5002        | 0.16  | 1000  | 3.0721          |
| 2.9556        | 0.24  | 1500  | 2.8731          |
| 2.8939        | 0.33  | 2000  | 3.1566          |
| 2.6986        | 0.41  | 2500  | 3.1023          |
| 2.7569        | 0.49  | 3000  | 2.7743          |
| 2.6391        | 0.57  | 3500  | 2.5023          |
| 2.4277        | 0.65  | 4000  | 2.5465          |
| 2.4242        | 0.73  | 4500  | 2.4709          |
| 2.3978        | 0.82  | 5000  | 2.4019          |
| 2.2653        | 0.9   | 5500  | 2.3383          |
| 2.3916        | 0.98  | 6000  | 2.4765          |
| 1.9423        | 1.06  | 6500  | 2.3798          |
| 1.8538        | 1.14  | 7000  | 2.3628          |
| 1.8136        | 1.22  | 7500  | 2.3671          |
| 1.7808        | 1.31  | 8000  | 2.5585          |
| 1.7772        | 1.39  | 8500  | 2.5862          |
| 1.755         | 1.47  | 9000  | 2.3105          |
| 1.6529        | 1.55  | 9500  | 2.2417          |
| 1.6956        | 1.63  | 10000 | 2.1755          |
| 1.5713        | 1.71  | 10500 | 2.2917          |
| 1.565         | 1.79  | 11000 | 2.0838          |
| 1.615         | 1.88  | 11500 | 2.2111          |
| 1.5249        | 1.96  | 12000 | 2.2207          |


### Framework versions


- Transformers 4.12.3
- Pytorch 1.8.0+cu101
- Datasets 1.15.1
- Tokenizers 0.10.3


model name: layoutlmv2-large-uncased-finetuned-infovqa


model author: tiennvcs


task: question-answering


tags: pytorch, tensorboard, layoutlmv2, question-answering, transformers, license:cc-by-nc-sa-4.0, generated_from_trainer, model-index, autonlp_compatible


license: cc-by-sa-4.0
tags:


- generated_from_trainer
model-index:
- name: layoutlmv2-base-uncased-finetuned-docvqa
results: []


# layoutlmv2-base-uncased-finetuned-docvqa


This model is a fine-tuned version of [microsoft/layoutlmv2-base-uncased](https://huggingface.co/microsoft/layoutlmv2-base-uncased) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 1.1940


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 250500
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.463         | 0.27  | 1000 | 1.6272          |
| 0.9447        | 0.53  | 2000 | 1.3646          |
| 0.7725        | 0.8   | 3000 | 1.2560          |
| 0.5762        | 1.06  | 4000 | 1.3582          |
| 0.4382        | 1.33  | 5000 | 1.2490          |
| 0.4515        | 1.59  | 6000 | 1.1860          |
| 0.383         | 1.86  | 7000 | 1.1940          |


### Framework versions


- Transformers 4.12.2
- Pytorch 1.8.0+cu101
- Datasets 1.14.0
- Tokenizers 0.10.3


model name: layoutlmv2-base-uncased-finetuned-docvqa


model author: tiennvcs


task: question-answering


tags: pytorch, tensorboard, layoutlmv2, question-answering, transformers, license:cc-by-sa-4.0, generated_from_trainer, model-index, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: distilbert-base-uncased-finetuned-squad
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-finetuned-squad


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the squad dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Framework versions


- Transformers 4.11.3
- Pytorch 1.9.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-squad


model author: tiennvcs


task: question-answering


tags: pytorch, distilbert, question-answering, dataset:squad, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: cc-by-nc-sa-4.0
tags:


- generated_from_trainer
model-index:
- name: layoutlmv2-base-uncased-finetuned-vi-infovqa
results: []


# layoutlmv2-base-uncased-finetuned-vi-infovqa


This model is a fine-tuned version of [microsoft/layoutlmv2-base-uncased](https://huggingface.co/microsoft/layoutlmv2-base-uncased) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 4.3332


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 5e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 250500
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 0.33  | 100  | 5.3461          |
| No log        | 0.66  | 200  | 4.9734          |
| No log        | 0.99  | 300  | 4.6074          |
| No log        | 1.32  | 400  | 4.4548          |
| 4.6355        | 1.65  | 500  | 4.3831          |
| 4.6355        | 1.98  | 600  | 4.3332          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.8.0+cu101
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: layoutlmv2-base-uncased-finetuned-vi-infovqa


model author: tiennvcs


task: question-answering


tags: pytorch, tensorboard, layoutlmv2, question-answering, transformers, license:cc-by-nc-sa-4.0, generated_from_trainer, model-index, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: bert-base-uncased-finetuned-vi-infovqa
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# bert-base-uncased-finetuned-vi-infovqa


This model is a fine-tuned version of [bert-base-uncased](https://huggingface.co/bert-base-uncased) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 3.5470


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 0.21  | 100  | 4.2058          |
| No log        | 0.43  | 200  | 4.0210          |
| No log        | 0.64  | 300  | 4.0454          |
| No log        | 0.85  | 400  | 3.7557          |
| 4.04          | 1.07  | 500  | 3.8257          |
| 4.04          | 1.28  | 600  | 3.7713          |
| 4.04          | 1.49  | 700  | 3.6075          |
| 4.04          | 1.71  | 800  | 3.6155          |
| 4.04          | 1.92  | 900  | 3.5470          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: bert-base-uncased-finetuned-vi-infovqa


model author: tiennvcs


task: question-answering


tags: pytorch, tensorboard, bert, question-answering, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: distilbert-base-uncased-finetuned-infovqa
results: []


# distilbert-base-uncased-finetuned-infovqa


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 2.8872


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 4
- eval_batch_size: 4
- seed: 250500
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 0.02  | 100  | 4.7706          |
| No log        | 0.05  | 200  | 4.4399          |
| No log        | 0.07  | 300  | 3.8175          |
| No log        | 0.09  | 400  | 3.8306          |
| 3.3071        | 0.12  | 500  | 3.6480          |
| 3.3071        | 0.14  | 600  | 3.6451          |
| 3.3071        | 0.16  | 700  | 3.4974          |
| 3.3071        | 0.19  | 800  | 3.4686          |
| 3.3071        | 0.21  | 900  | 3.4703          |
| 3.5336        | 0.23  | 1000 | 3.3165          |
| 3.5336        | 0.25  | 1100 | 3.3634          |
| 3.5336        | 0.28  | 1200 | 3.3466          |
| 3.5336        | 0.3   | 1300 | 3.3411          |
| 3.5336        | 0.32  | 1400 | 3.2456          |
| 3.3593        | 0.35  | 1500 | 3.3257          |
| 3.3593        | 0.37  | 1600 | 3.2941          |
| 3.3593        | 0.39  | 1700 | 3.2581          |
| 3.3593        | 0.42  | 1800 | 3.1680          |
| 3.3593        | 0.44  | 1900 | 3.2077          |
| 3.2436        | 0.46  | 2000 | 3.2422          |
| 3.2436        | 0.49  | 2100 | 3.2529          |
| 3.2436        | 0.51  | 2200 | 3.2681          |
| 3.2436        | 0.53  | 2300 | 3.1055          |
| 3.2436        | 0.56  | 2400 | 3.0174          |
| 3.093         | 0.58  | 2500 | 3.0608          |
| 3.093         | 0.6   | 2600 | 3.0200          |
| 3.093         | 0.63  | 2700 | 2.9884          |
| 3.093         | 0.65  | 2800 | 3.0041          |
| 3.093         | 0.67  | 2900 | 2.9700          |
| 3.0087        | 0.69  | 3000 | 3.0993          |
| 3.0087        | 0.72  | 3100 | 3.0499          |
| 3.0087        | 0.74  | 3200 | 2.9317          |
| 3.0087        | 0.76  | 3300 | 3.0817          |
| 3.0087        | 0.79  | 3400 | 3.0035          |
| 2.9694        | 0.81  | 3500 | 3.0850          |
| 2.9694        | 0.83  | 3600 | 2.9948          |
| 2.9694        | 0.86  | 3700 | 2.9874          |
| 2.9694        | 0.88  | 3800 | 2.9202          |
| 2.9694        | 0.9   | 3900 | 2.9322          |
| 2.8277        | 0.93  | 4000 | 2.9195          |
| 2.8277        | 0.95  | 4100 | 2.8638          |
| 2.8277        | 0.97  | 4200 | 2.8809          |
| 2.8277        | 1.0   | 4300 | 2.8872          |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.9.0+cu111
- Datasets 1.14.0
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-infovqa


model author: tiennvcs


task: question-answering


tags: pytorch, tensorboard, distilbert, question-answering, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


# Plant Disease Detection Inference Interface


This is the inference interface of my plant disease detecion model.


You can download a copy of the pre-trained model [here](https://drive.google.com/file/d/1FVawQYsB31ByC1r9xG7JAajbOrpKVt04/view?usp=sharing)


## Set things up


First clone this repository


```
git clone https://github.com/JosiasAurel/plant-model.git
```


Move to the directory of the repo


```
cd plant-model
```


Make you have python 3 and pip installed correctly. Then run


```
pip install -r requirements.txt
```


After all dependencies have been installed, run


```
streamlit run app.py
```


Now you can play around with the model ;)


model name: plant-disease-detection


model author: JosiasWing


tags: onnx


# BriVL


BriVL (Bridging Vision and Language Model) 是首个中文通用图文多模态大规模预训练模型。BriVL模型在图文检索任务上有着优异的效果，超过了同期其他常见的多模态预训练模型（例如UNITER、CLIP）。


BriVL论文：[WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training](https://arxiv.org/abs/2103.06561)


# 适用场景


适用场景示例：图像检索文本、文本检索图像、图像标注、图像零样本分类、作为其他下游多模态任务的输入特征等。


# 技术特色


1. BriVL使用对比学习算法将图像和文本映射到了同一特征空间，可用于弥补图像特征和文本特征之间存在的隔阂。
1. 基于视觉-语言弱相关的假设，除了能理解对图像的描述性文本外，也可以捕捉图像和文本之间存在的抽象联系。
1. 图像编码器和文本编码器可分别独立运行，有利于实际生产环境中的部署。


# 下载专区


| 模型      | 语言 | 参数量（单位：亿） | 文件（file）                |
| --------- | ---- | ------------------ | --------------------------- |
| BriVL-1.0  | 中文 | 10亿                 | BriVL-1.0-5500w.tar|


# 使用BriVL


### 搭建环境


```
# 环境要求
lmdb==0.99
timm==0.4.12
easydict==1.9
pandas==1.2.4
jsonlines==2.0.0
tqdm==4.60.0
torchvision==0.9.1
numpy==1.20.2
torch==1.8.1
transformers==4.5.1
msgpack_numpy==0.4.7.1
msgpack_python==0.5.6
Pillow==8.3.1
PyYAML==5.4.1
```


配置要求在requirements.txt中，可使用下面的命令：


```
pip install -r requirements.txt
```


### 特征提取与计算检索结果


```
cd evaluation/
bash test_xyb.sh
```


### 数据解释


现已放入3个图文对示例:


```
./data/imgs  # 放入图像
./data/jsonls # 放入图文对描述
```


# 引用BriVL


```
@article{DBLP:journals/corr/abs-2103-06561,
  author    = {Yuqi Huo and
               Manli Zhang and
               Guangzhen Liu and
               Haoyu Lu and
               Yizhao Gao and
               Guoxing Yang and
               Jingyuan Wen and
               Heng Zhang and
               Baogui Xu and
               Weihao Zheng and
               Zongzheng Xi and
               Yueqian Yang and
               Anwen Hu and
               Jinming Zhao and
               Ruichen Li and
               Yida Zhao and
               Liang Zhang and
               Yuqing Song and
               Xin Hong and
               Wanqing Cui and
               Dan Yang Hou and
               Yingyan Li and
               Junyi Li and
               Peiyu Liu and
               Zheng Gong and
               Chuhao Jin and
               Yuchong Sun and
               Shizhe Chen and
               Zhiwu Lu and
               Zhicheng Dou and
               Qin Jin and
               Yanyan Lan and
               Wayne Xin Zhao and
               Ruihua Song and
               Ji{-}Rong Wen},
  title     = {WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training},
  journal   = {CoRR},
  volume    = {abs/2103.06561},
  year      = {2021},
  url       = {https://arxiv.org/abs/2103.06561},
  archivePrefix = {arXiv},
  eprint    = {2103.06561},
  timestamp = {Tue, 03 Aug 2021 12:35:30 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2103-06561.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
```


model name: BriVL


model author: shunxing1234


tags: arxiv:2103.06561


description


tutorial


model name: test_model


tags:


- conversational


# Hrry Potter DialoGPT Model


model name: DialoGPT-small-harrypotter


model author: ishraaqparvez


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


widget:


- text: "create array containing the maximum value of respective elements of array `[2, 3, 4]` and array `[1, 5, 2]"
- text: "check if all elements in list `mylist` are identical"
- text: "enable debug mode on flask application `app`"
- text: "getting the length of `my_tuple`"
- text: 'find all files in directory "/mydir" with extension ".txt"'


# MarianCG: A TRANSFORMER MODEL FOR AUTOMATIC CODE GENERATION


This model is to improve the solving of the code generation problem and implement a transformer model that can work with high accurate results. We implemented MarianCG transformer model which is a code generation model that can be able to generate code from natural language. This work declares the impact of using Marian machine translation model for solving the problem of code generation. In our implementation we prove that a machine translation model can be operated and working as a code generation model.Finally, we set the new contributors and state-of-the-art on CoNaLa reaching a BLEU score of 30.92 in the code generation problem with CoNaLa dataset.


CoNaLa Dataset for Code Generation is available at
https://huggingface.co/AhmedSSoliman/CoNaLa


This is the model is avialable on the huggingface hub https://huggingface.co/AhmedSSoliman/MarianCG_NL-to-Code


```python
# Model and Tokenizer
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
# model_name = "AhmedSSoliman/MarianCG_NL-to-Code"
model = AutoModelForSeq2SeqLM.from_pretrained("AhmedSSoliman/MarianCG_NL-to-Code")
tokenizer = AutoTokenizer.from_pretrained("AhmedSSoliman/MarianCG_NL-to-Code")
# Input (Natural Language) and Output (Python Code)
NL_input = "create array containing the maximum value of respective elements of array `[2, 3, 4]` and array `[1, 5, 2]"
output = model.generate(**tokenizer(NL_input, padding="max_length", truncation=True, max_length=512, return_tensors="pt"))
output_code = tokenizer.decode(output[0], skip_special_tokens=True)
```


This model is available in spaces using gradio at: https://huggingface.co/spaces/AhmedSSoliman/MarianCG_NL-to-Code


Tasks:


- Translation
- Code Generation
- Text2Text Generation
- Text Generation


model name: MarianCG_NL-to-Code


model author: AhmedSSoliman


task: text2text-generation


tags: pytorch, marian, text2text-generation, transformers, autonlp_compatible


language:


- da


tags:


- Hatespeech
- Danish
- BERT
license: mit
datasets:
- DKHate - OffensEval2020
Classes:
- Hateful
- Not Hateful


# DKbert-hatespeech-classification


Use this model to detect hatespeech in Danish. For details, guide and command line tool see [DK hate github](https://github.com/Guscode/DKbert-hatespeech-detection)


## Training data


Training data is from OffensEval2020 which can be found [here](https://figshare.com/articles/dataset/Danish_Hate_Speech_Abusive_Language_data/12220805)


## Performance


The model achieves a macro F1-score of 0.78


Precision hateful: 0.77


Recall hateful: 0.49


See more on [DK hate github](https://github.com/Guscode/DKbert-hatespeech-detection)


## Training procedure


- [BOTXO Nordic Bert](https://huggingface.co/DJSammy/bert-base-danish-uncased_BotXO,ai)
- Learning rate: 1e-5,
- Batch size: 16
- Max sequence length: 128


## Project information


This model was made in collaboration between [Johan Horsmans](https://github.com/JohanHorsmans) and [Gustav Aarup Lauridsen](https://github.com/Guscode) for their Cultural Data Science Exam.


model name: DKbert-hatespeech-detection


model author: Guscode


task: text-classification


tags: pytorch, tf, bert, text-classification, da, dataset:DKHate - OffensEval2020, transformers, Hatespeech, Danish, BERT, license:mit, infinity_compatible


Ver~GRATIS ((*~Sin tiempo para morir~*))2021 PELICULA COMPLETA EN REPELIS.HD


Ver Sin tiempo para morir online peicula completa en español , Ver Sin tiempo para morir pelicula completa en espanol , Ver pelicula Sin tiempo para morir online en espanol, Sin tiempo para morir la pelicula en espanol y Latino, Sin tiempo para morir Streaming de peliculas, Sin tiempo para morir Streaming de peliculas, pelicula completa sub español online Estrenos


| Pelispedia | Pelisplus | Gnula | Repelisplus | Repelis | Pelis | Pelisplus| | Netflix | Cine | Cinema | Calidad | Mejor | Chile | filtrada


¡Ver repelis Gratis! Clic Aqui ☛ https://v.ht/CAYz-Sp41N-HD


Descargar ☛ https://v.ht/CAYz-Sp41N-HD


repelis Sin Tiempo Para Morir (2021) Película Online Completa Gratis Español y Latino, Sin Tiempo Para Morir (2021) pelicula completa en Español latino, multilenguaje y alta calidad.


Ver Sin Tiempo Para Morir película completa GRATIS en Español o con subtítulos en tu idioma, en HD y hasta en calidad 2021 HD con Audio Español Latino y Subtitulado.


Sin Tiempo Para Morir; fecha de estreno, tráiler y dónde ver en España y Latinoamérica
Te contamos todo sobre Sin Tiempo Para Morir, cuándo se estrena en cines, tráiler y dónde podrás verla en España y Latinoamérica.


Sin Tiempo Para Morir de esta Película Desde 2021 se estrenó oficialmente en Hispanoamérica y España, esta Película es muy interesante y puede acompañarte a relajarte un poco en casa, voy a flashback de esta Película.


En éste artículo vamos a saber dónde ver Películas online gratis en HD sin cortes, y así disfrutar en familia, solos o con amigos de las mejores Películas de la actualidad como también de la historia del cine. Conoce las mejores páginas para ver Películas gratis.


Ver Sin Tiempo Para Morir la Película en español línea Online y Gratis


Ver películas Sin Tiempo Para Morir completas gratis en español es posible y de forma legal. Hay opciones alternativas a Netflix y Amazon que no requieren ningún tipo de pago ni suscripción y cuyo contenido es totalmente gratuito. Estas plataformas para ver cine Sin Tiempo Para Morir gratis en casa pueden ofrecer contenido sin costo gracias a cortes comerciales o bien porque tienen películas de dominio público.


Ver Sin Tiempo Para Morir Película completa en español Latino Subtitulado
En nuestro sitio proporcionamos subtítulos y dabbing en latín, no tenga miedo por México, Chile, Perú, Bolivia, Uruguay, Paraguay, España, Argentina, Colombia y todas las regiones de habla latina, hemos proporcionado idiomas para sus respectivas regiones. .Para disfrutar de todas estas funciones, puede registrarse y seguir en su cuenta premium.


Título original: Sin Tiempo     Para Morir: The Movie
Año: 2021
Duración: 100 min
:: SINOPSIS ::
¿Dónde ver Sin Tiempo Para Morir online en español Película 2021?


El miércoles 4 de agosto llega a nuestros cines Sin Tiempo Para Morir la película sobre el grupo surcoreano de Sin Tiempo Para Morir, distribuida en nuestro país por Versión Digital. Esta película es parte del proyecto para celebrar el quinto aniversario desde su debut, el 8 de agosto de 2016, y que recibe el nombre de “4+1 Project”, en directa alusión a las cuatro miembros del grupo y a los fans.


La película, protagonizada por JISOO, JENNIE, ROSÉ y LISA comparte recuerdos de estos cinco años desde el debut del grupo, entrevistas, mensajes para los fans. Contará también con  canciones como “The Show” (2021), “In Your Area” (2018) y una docena más de éxitos de Sin Tiempo Para Morir. Está dirigida por Oh Yoon-dong y Jung Su-yee, y producida por ScreenX Studio.


Están previstas proyecciones los días 4 y 8 de agosto, y las entradas se ponen a la venta el miércoles 30 de junio.


Ver Sin Tiempo Para Morir Película completa en español y Latino HD
Luego de que se diera conocer que Rosé, Lisa, Jisoo y Jennie iban a lanzar una película donde se verían declaraciones inéditas y actuaciones en vivo presentadas en un formato especial; muchos de los BLINKS se preguntaron “¿Llegará a México?” y después de unos días, apareció Cinépolis para anunciar que ‘LA PELICULA’ llegará a sus salas en agosto.


También tendrán la oportunidad de verla los integrantes de BLINKS que habiten en Panamá, Colombia, Paraguay, Costa Rica, Honduras, El Salvador y Guatemala, sin embargo, el metraje de SIN TIEMPO PARA MORIR serán proyectado en ciertas salas, por lo que deberás checar los horarios y los cines donde podrá verse.


Ver Sin Tiempo Para Morir Película completa online gratis en español
El proceso es muy sencillo solo debes de entrar al sitio oficial de ‘SIN TIEMPO PARA MORIR LA PELICULA’ o en este ENLACE, y ahí te aparecerá un mapa donde se puede ver los cines donde estará disponible la película de acuerdo del lugar donde te encuentras, y también hay una opción donde puedes incorporar tu ubicación.


Una vez localizado el cine de tu preferencia, dale clic y se abrirá un recuadro donde te aparecen dos fechas: 4 agosto y 8 agosto. Selecciona el día en el que te gustaría ver la película y dale clic al botón que dice “Get Tickets”.


Sin Tiempo Para Morir De La historia original
Te mandará a la página de Cinépolis, sin embargo, una vez en el sitio deberás de dirigirte al recuadro del lado izquierdo que dice “¿En qué ciudad la vas a ver?” para seleccionar el complejo donde disfrutarás de ‘SIN TIEMPO PARA MORIR LA PELICULA’. Cuando encuentres el lugar, elige un horario para verla.


Al darle clic en el horario, te llevarán a otra página donde puedes seleccionar el número de boletos que vas a comprar. Luego, te dará la opción de elegir tus asientos, y finalmente, podrás realizar tu pago donde deberás dar tu nombre, apellidos, correo electrónico y seleccionar un método de pago: tarjeta de débito/ crédito, Club Cinépolis, y Paypal. Para finalizar, deberás dar los datos de tu tarjeta.


Sin Tiempo Para Morir: ¿Qué dicen los críticos?
«El costo de los boletos puede variar según la zona y el tipo de complejo que selecciones. Por ejemplo, si vas a Cinépolis Las Antenas el precio del boleto es de 110 pesos por persona, pero si acudes al Cinépolis VIP Miyana Polanco la entrada es de 131 pesos por persona»


Repelis Sin Tiempo Para Morir (2021) Película completa en la que se recupera la sencillez


aquí es tu lugar para descargar Sin Tiempo Para Morir película completa online gratis en español y latino, puedes ver esta película en calidad HD puedes ver la película Sin Tiempo Para Morir sin pagar y sin ningún tipo de molestias.


Puede disfrutar de las películas que proporcionamos en nuestro sitio a través de su computadora, computadora portátil y teléfono inteligente de forma gratuita en español y latíno.


los navegadores a menudo terminan en páginas web que no coinciden
promesa, que afirman tener el último lanzamiento, pero simplemente
llevarlo de un sitio a otro, lo que le obliga a hacer clic por clic, ya que están
llenar la pantalla de publicidad, para finalmente ir a
enlaces que no funcionan o que tardan mucho en cargar el sitio desde la película, hay que tener cuidado con los sitios que no son oficiales y no están registrados en plataformas confiables, podría ser que tu computadora haya sido infectada con un virus que pueden dañar el dispositivo de su computadora, hay casos en los que el navegador de su computadora ha sido pirateado por partes irresponsables.


pero no necesitas tener miedo y dudar más en descargar las últimas películas en el sitio Fandango21online gratis en línea con la mejor calidad en español y latín porque el sitio Fandango21online ha sido verificado legalmente y, por supuesto, está a salvo de todos los virus y cualquier alteración.


la mejor página para Sin Tiempo Para Morir Película (HD) Online gratis en español y latín, en sí hay muchas
de este tipo, pero a efectos prácticos hemos seleccionado algunos de los más
popular en la red de la red. Depende de usted elegir el que mejor se adapte a sus necesidades
según sus necesidades, ya sea por catálogo, interfaz o velocidad
Internet. en el sitio Fandango21online también puede ver varios tipos de las últimas películas, como: Rápidos y Furiosos 9, Cruella, Ruega por nosotros, Mortal Kombat, Ejército de los muertos.


También puedes ver películas de acción, terror, series de aventuras, telenovelas.
México y Turquía, dramas, anime y mucho, mucho más, como la mayoría
Últimas noticias: Narcos: México, The Sinner 2 y La reina del flow.
Incluso te diremos qué películas están en vallas publicitarias en los cines
en Perú, México, España, Estado Sin Tiempo Para Morir, Colombia, Argentina, España y otros países del mundo. ¡Sí, la última versión! ¿Como ejemplo? hechizo 3,
¡Hechizo 3, Hechizo 3, Asu mare 3! y Sin Tiempo Para Morir ya está disponible en las mejores salas.


Etiquetas :


● Sin Tiempo Para Morir pelicula completa


● Sin Tiempo Para Morir pelicula completa en español


● Sin Tiempo Para Morir la película completa


● Sin Tiempo Para Morir pelicula completa en español latino online


● pelicula completa de Sin Tiempo Para Morir


● Sin Tiempo Para Morir película completa


● Sin Tiempo Para Morir la película completa en español


● Sin Tiempo Para Morir pelicula completa en español latino


● Ver Pelicula Sin Tiempo Para Morir (2021) Online


● como Ver Sin Tiempo Para Morir (2021)


● Ver Sin Tiempo Para Morir (2021) Online


● Descargar Sin Tiempo Para Morir (2021) preludio


● Descargar Sin Tiempo Para Morir (2021) Pelicula Completa en Español Latino


● Pelicula Completa Sin Tiempo Para Morir (2021) en espanol Latino Online


● Pelicula Completa Sin Tiempo Para Morir (2021) Audio Latino


● Pelicula Completa Sin Tiempo Para Morir (2021) Audio Latino Online


● Pelicula Completa Sin Tiempo Para Morir (2021) Subtitulada en espanol


Sin Tiempo Para Morir la película 2021


model name: 1080p


model author: Ver-GRATIS-Sin-tiempo-para-morir


tags:


CodeCarbon wasn't ready until the training was over so we only did an additional 10h run to measure with and then we can extrapolate to the whole training.


This set of records captures the startup time and 2499 iterations in 2 records per gpu, since there was also an intermediary checkpoint saved half-way and we flush the CC
records on each checkpoint saving.


The training had 168000 iterations. Therefore multiply the reported data by 67. This would be quite approximate since we were using 16 nodes when doing
the ramp up, then 64 and only the last 3 weeks 128 nodes.


Caveat emptor: I'm not sure whether CC-reports overlap since each report is per gpu and I think they may be measuring the same thing, other than the gpu itself.
So this requires research.


Each csv file contains a report for a single gpu.


model name: tr1-13B-codecarbon


model author: bigscience


These are tensorboard logs for https://github.com/bigscience-workshop/bigscience/tree/master/train/tr1-13B-base


model name: tr1-13B-tensorboard


tags: tensorboard


This is for sharing various data files used for testing and script development with those without access to JeanZay - feel free to create a sub-folder with your username to keep things a bit organized.


model name: misc-test-data


datasets:


- bigscience/P3
language: en
license: apache-2.0
widget:
- text: "A is the son's of B's uncle. What is the family relationship between A and B?"
- text: "Reorder the words in this sentence: justin and name bieber years is my am I 27 old."
- text: "Task: copy but say the opposite.\n
PSG won its match against Barca."
- text: "Is this review positive or negative? Review: Best cast iron skillet you will every buy."
example_title: "Sentiment analysis"
- text: "Question A: How is air traffic controlled?
\nQuestion B: How do you become an air traffic controller?\nPick one: these questions are duplicates or not duplicates."
- text: "Barack Obama nominated Hilary Clinton as his secretary of state on Monday. He chose her because she had foreign affairs experience as a former First Lady.
\nIn the previous sentence, decide who 'her' is referring to."
example_title: "Coreference resolution"
- text: "Last week I upgraded my iOS version and ever since then my phone has been overheating whenever I use your app.\n
Select the category for the above sentence from: mobile, website, billing, account access."
- text: "Sentence 1: Gyorgy Heizler, head of the local disaster unit, said the coach was carrying 38 passengers.\n
Sentence 2: The head of the local disaster unit, Gyorgy Heizler, said the bus was full except for 38 empty seats.\n\n
Do sentences 1 and 2 have the same meaning?"
example_title: "Paraphrase identification"
- text: "Here's the beginning of an article, choose a tag that best describes the topic of the article: business, cinema, politics, health, travel, sports.\n\n
The best and worst fo 007 as 'No time to die' marks Daniel Craig's exit.\n
(CNN) Some 007 math: 60 years, 25 movies (with a small asterisk) and six James Bonds. For a Cold War creation, Ian Fleming's suave spy has certainly gotten around, but despite different guises in the tuxedo and occasional scuba gear, when it comes to Bond ratings, there really shouldn't be much argument about who wore it best."
- text: "Max: Know any good websites to buy clothes from?\n
Payton: Sure :) LINK 1, LINK 2, LINK 3\n
Max: That's a lot of them!\n
Payton: Yeah, but they have different things so I usually buy things from 2 or 3 of them.\n
Max: I'll check them out. Thanks.\n\n
Who or what are Payton and Max referring to when they say 'them'?"
- text: "Is the word 'table' used in the same meaning in the two following sentences?\n\n
Sentence A: you can leave the books on the table over there.\n
Sentence B: the tables in this book are very hard to read."
- text: "On a shelf, there are five books: a gray book, a red book, a purple book, a blue book, and a black book.\n
The red book is to the right of the gray book. The black book is to the left of the blue book. The blue book is to the left of the gray book. The purple book is the second from the right.\n\n
Which book is the leftmost book?"
example_title: "Logic puzzles"
- text: "The two men running to become New York City's next mayor will face off in their first debate Wednesday night.\n\n
Democrat Eric Adams, the Brooklyn Borough president and a former New York City police captain, is widely expected to win the Nov. 2 election against Republican Curtis Sliwa, the founder of the 1970s-era Guardian Angels anti-crime patril.\n\n
Who are the men running for mayor?"
example_title: "Reading comprehension"
- text: "The word 'binne' means any animal that is furry and has four legs, and the word 'bam' means a simple sort of dwelling.\n\n
Which of the following best characterizes binne bams?\n
- Sentence 1: Binne bams are for pets.\n
- Sentence 2: Binne bams are typically furnished with sofas and televisions.\n
- Sentence 3: Binne bams are luxurious apartments.\n
- Sentence 4: Binne bams are places where people live."


**How do I pronounce the name of the model?** T0 should be pronounced "T Zero" (like in "T5 for zero-shot") and any "p" stands for "Plus", so "T0pp" should be pronounced "T Zero Plus Plus"!


# Model Description


T0* shows zero-shot task generalization on English natural language prompts, outperforming GPT-3 on many tasks, while being 16x smaller. It is a series of encoder-decoder models trained on a large set of different tasks specified in natural language prompts. We convert numerous English supervised datasets into prompts, each with multiple templates using varying formulations. These prompted datasets allow for benchmarking the ability of a model to perform completely unseen tasks specified in natural language. To obtain T0*, we fine-tune a pretrained language model on this multitask mixture covering many different NLP tasks.


# Intended uses


You can use the models to perform inference on tasks by specifying your query in natural language, and the models will generate a prediction. For instance, you can ask *"Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy"*, and the model will hopefully generate *"Positive"*.


A few other examples that you can try:


- *A is the son's of B's uncle. What is the family relationship between A and B?*
- *Question A: How is air traffic controlled?

Question B: How do you become an air traffic controller?

Pick one: these questions are duplicates or not duplicates.*
- *Is the word 'table' used in the same meaning in the two following sentences?

Sentence A: you can leave the books on the table over there.

Sentence B: the tables in this book are very hard to read.*
- *Max: Know any good websites to buy clothes from?

Payton: Sure :) LINK 1, LINK 2, LINK 3

Max: That's a lot of them!

Payton: Yeah, but they have different things so I usually buy things from 2 or 3 of them.

Max: I'll check them out. Thanks.

Who or what are Payton and Max referring to when they say 'them'?*
- *On a shelf, there are five books: a gray book, a red book, a purple book, a blue book, and a black book.

The red book is to the right of the gray book. The black book is to the left of the blue book. The blue book is to the left of the gray book. The purple book is the second from the right.

Which book is the leftmost book?*
- *Reorder the words in this sentence: justin and name bieber years is my am I 27 old.*


# How to use


We make available the models presented in our [paper](https://arxiv.org/abs/2110.08207) along with the ablation models. We recommend using the [T0pp](https://huggingface.co/bigscience/T0pp) (pronounce "T Zero Plus Plus") checkpoint as it leads (on average) to the best performances on a variety of NLP tasks.


|Model|Number of parameters|
|-|-|
|[T0](https://huggingface.co/bigscience/T0)|11 billion|
|[T0p](https://huggingface.co/bigscience/T0p)|11 billion|
|[T0pp](https://huggingface.co/bigscience/T0pp)|11 billion|
|[T0_single_prompt](https://huggingface.co/bigscience/T0_single_prompt)|11 billion|
|[T0_original_task_only](https://huggingface.co/bigscience/T0_original_task_only)|11 billion|
|[T0_3B](https://huggingface.co/bigscience/T0_3B)|3 billion|


Here is how to use the model in PyTorch:


```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("bigscience/T0pp")
model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0pp")

inputs = tokenizer.encode("Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy", return_tensors="pt")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))
```


If you want to use another checkpoint, please replace the path in `AutoTokenizer` and `AutoModelForSeq2SeqLM`.


**Note: the model was trained with bf16 activations. As such, we highly discourage running inference with fp16. fp32 or bf16 should be preferred.**


# Training procedure


T0* models are based on [T5](https://huggingface.co/google/t5-v1_1-large), a Transformer-based encoder-decoder language model pre-trained with a masked language modeling-style objective on [C4](https://huggingface.co/datasets/c4). We use the publicly available [language model-adapted T5 checkpoints](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k) which were produced by training T5 for 100'000 additional steps with a standard language modeling objective.


At a high level, the input text is fed to the encoder and the target text is produced by the decoder. The model is fine-tuned to autoregressively generate the target through standard maximum likelihood training. It is never trained to generate the input. We detail our training data in the next section.


Training details:


- Fine-tuning steps: 12'200
- Input sequence length: 1024
- Target sequence length: 256
- Batch size: 1'024 sequences
- Optimizer: Adafactor
- Learning rate: 1e-3
- Dropout: 0.1
- Sampling strategy: proportional to the number of examples in each dataset (we treated any dataset with over 500'000 examples as having 500'000/`num_templates` examples)
- Example grouping: We use packing to combine multiple training examples into a single sequence to reach the maximum sequence length


# Training data


We trained different variants T0 with different mixtures of datasets.


|Model|Training datasets|
|--|--|
|T0|- Multiple-Choice QA: CommonsenseQA, DREAM, QUAIL, QuaRTz, Social IQA, WiQA, Cosmos, QASC, Quarel, SciQ, Wiki Hop
- Extractive QA: Adversarial QA, Quoref, DuoRC, ROPES
- Closed-Book QA: Hotpot QA*, Wiki QA
- Structure-To-Text: Common Gen, Wiki Bio
- Sentiment: Amazon, App Reviews, IMDB, Rotten Tomatoes, Yelp
- Summarization: CNN Daily Mail, Gigaword, MultiNews, SamSum, XSum
- Topic Classification: AG News, DBPedia, TREC
- Paraphrase Identification: MRPC, PAWS, QQP|
|T0p|Same as T0 with additional datasets from GPT-3's evaluation suite:
- Multiple-Choice QA: ARC, OpenBook QA, PiQA, RACE, HellaSwag
- Extractive QA: SQuAD v2
- Closed-Book QA: Trivia QA, Web Questions|
|T0pp|Same as T0p with a few additional datasets from SuperGLUE (excluding NLI sets):
- BoolQ
- COPA
- MultiRC
- ReCoRD
- WiC
- WSC|
|T0_single_prompt|Same as T0 but only one prompt per training dataset|
|T0_original_task_only|Same as T0 but only original tasks templates|
|T0_3B|Same as T0 but starting from a T5-LM XL (3B parameters) pre-trained model|


For reproducibility, we release the data we used for training (and evaluation) in the [P3 dataset](https://huggingface.co/datasets/bigscience/P3). Prompts examples can be found on the dataset page.


*: We recast Hotpot QA as closed-book QA due to long input sequence length.


# Evaluation data


We evaluate our models on a suite of held-out tasks:


|Task category|Datasets|
|-|-|
|Natural language inference|ANLI, CB, RTE|
|Coreference resolution|WSC, Winogrande|
|Word sense disambiguation|WiC|
|Sentence completion|COPA, HellaSwag, Story Cloze|


We also evaluate T0, T0p and T0pp on the a subset of the [BIG-bench benchmark](https://github.com/google/BIG-bench):


- Code description task
- Conceptual combinations
- Hindu knowledge json
- Known unknowns
- Language identification
- Logic grid puzzle task
- Logical deduction
- Common misconceptions
- Movie dialog same or different
- Novel concepts
- Strategyqa
- Formal fallacies syllogisms negation
- VitaminC
- Winowhy multiple choice


# Limitations


- The models of the T0* series are quite large (3B or 11B parameters). Loading them and performing inference requires non-trivial computational resources. When using multiple GPUs, it is possible to use [.parallelize()](https://huggingface.co/transformers/parallelism.html).
- We have observed that different prompts can lead to varying performances. We believe that further research is required to explore the effectiveness of different prompts for a language model.
- Due to design choices in the tokenization, the models are unable to perform inference for tasks involving code or non English text.


# Bias and fairness


Even if we took deliberate decisions to exclude datasets with potentially harmful content from the fine-tuning, the models trained are not bias-free. Based on a few experimentations, T0++ can generate answers that could be categorized as conspiracist or biased:


- Input: `Is the earth flat?` - Prediction: `yes`
- Input: `Do vaccines cause autism?` - Prediction: `yes`
- Input: `Complete this sentence: This man works as a` - Prediction: `Architect`
- Input: `Complete this sentence: This woman works as a` - Prediction: `Nanny`


Language models can reproduce undesirable social biases represented in the large corpus they are pre-trained on. We evaluate our models in two ways: first in their ability to recognize or label gender biases and second in the extent to which they reproduce those biases.


To measure the ability of our model to recognize gender biases, we evaluate our models using the WinoGender Schemas (also called AX-g under SuperGLUE) and CrowS-Pairs. WinoGender Schemas are minimal pairs of sentences that differ only by the gender of one pronoun in the sentence, designed to test for the presence of gender bias. We use the *Diverse Natural Language Inference Collection* ([Poliak et al., 2018](https://aclanthology.org/D18-1007/)) version that casts WinoGender as a textual entailment task and report accuracy. CrowS-Pairs is a challenge dataset for measuring the degree to which U.S. stereotypical biases present in the masked language models using minimal pairs of sentences. We re-formulate the task by predicting which of two sentences is stereotypical (or anti-stereotypical) and report accuracy. For each dataset, we evaluate between 5 and 10 prompts.


    Dataset
    Model
    Average (Acc.)
    Median (Acc.)

    CrowS-Pairs	T0	59.2	83.8

    T0p	57.6	83.8

    T0pp	62.7	64.4

    T0_single_prompt	57.6	69.5

    T0_original_task_only	47.1	37.8

    T0_3B	56.9	82.6

    WinoGender	T0	84.2	84.3

    T0p	80.1	80.6

    T0pp	89.2	90.0

    T0_single_prompt	81.6	84.6

    T0_original_task_only	83.7	83.8

    T0_3B	69.7	69.4


To measure the extent to which our model reproduces gender biases, we evaluate our models using the WinoBias Schemas. WinoBias Schemas are pronoun coreference resolution tasks that have the potential to be influenced by gender bias. WinoBias Schemas has two schemas (type1 and type2) which are partitioned into pro-stereotype and anti-stereotype subsets. A "pro-stereotype" example is one where the correct answer conforms to stereotypes, while an "anti-stereotype" example is one where it opposes stereotypes. All examples have an unambiguously correct answer, and so the difference in scores between the "pro-" and "anti-" subset measures the extent to which stereotypes can lead the model astray. We report accuracies by considering a prediction correct if the target noun is present in the model's prediction. We evaluate on 6 prompts.


    Model
    Subset
    Average (Acc.)
    Median (Acc.)

    Pro
    Anti
    Pro - Anti
    Pro
    Anti
    Pro - Anti


    T0	Type 1
    68.0	61.9	6.0	71.7	61.9	9.8

    Type 2
    79.3	76.4	2.8	79.3	75.0	4.3

    T0p
    Type 1
    66.6	57.2	9.4	71.5	62.6	8.8

    Type 2
    77.7	73.4	4.3	86.1	81.3	4.8

    T0pp
    Type 1
    63.8	55.9	7.9	72.7	63.4	9.3

    Type 2
    66.8	63.0	3.9	79.3	74.0	5.3

    T0_single_prompt
    Type 1
    73.7	60.5	13.2	79.3	60.6	18.7

    Type 2
    77.7	69.6	8.0	80.8	69.7	11.1


    T0_original_task_only
    Type 1
    78.1	67.7	10.4	81.8	67.2	14.6

     Type 2
    85.2	82.3	2.9	89.6	85.4	4.3


    T0_3B
    Type 1
    82.3	70.1	12.2	83.6	62.9	20.7

     Type 2
    83.8	76.5	7.3	85.9	75	10.9


# BibTeX entry and citation info


```bibtex
@misc{sanh2021multitask,
      title={Multitask Prompted Training Enables Zero-Shot Task Generalization},
      author={Victor Sanh and Albert Webson and Colin Raffel and Stephen H. Bach and Lintang Sutawika and Zaid Alyafeai and Antoine Chaffin and Arnaud Stiegler and Teven Le Scao and Arun Raja and Manan Dey and M Saiful Bari and Canwen Xu and Urmish Thakker and Shanya Sharma Sharma and Eliza Szczechla and Taewoon Kim and Gunjan Chhablani and Nihal Nayak and Debajyoti Datta and Jonathan Chang and Mike Tian-Jian Jiang and Han Wang and Matteo Manica and Sheng Shen and Zheng Xin Yong and Harshit Pandey and Rachel Bawden and Thomas Wang and Trishala Neeraj and Jos Rozen and Abheesht Sharma and Andrea Santilli and Thibault Fevry and Jason Alan Fries and Ryan Teehan and Stella Biderman and Leo Gao and Tali Bers and Thomas Wolf and Alexander M. Rush},
      year={2021},
      eprint={2110.08207},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}
```


model name: T0_original_task_only


model author: bigscience


task: text2text-generation


tags: pytorch, t5, text2text-generation, en, dataset:bigscience/P3, arxiv:2110.08207, transformers, license:apache-2.0, autonlp_compatible


This data is from [13B-en training](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr1-13B-base)


- indices - these are Megatron-LM shuffled indices that the training was using. They were generated the first time the training started. So the order is the same if one replays them via the dataloader w/o actually doing the training steps.
- the corresponding dataset is oscar-en that's on JZ at `$six_ALL_CCFRWORK/datasets-custom/oscar-en`


model name: tr1-13B-data


tags:


datasets:


- bigscience/P3
language: en
license: apache-2.0
widget:
- text: "A is the son's of B's uncle. What is the family relationship between A and B?"
- text: "Reorder the words in this sentence: justin and name bieber years is my am I 27 old."
- text: "Task: copy but say the opposite.\n
PSG won its match against Barca."
- text: "Is this review positive or negative? Review: Best cast iron skillet you will every buy."
example_title: "Sentiment analysis"
- text: "Question A: How is air traffic controlled?
\nQuestion B: How do you become an air traffic controller?\nPick one: these questions are duplicates or not duplicates."
- text: "Barack Obama nominated Hilary Clinton as his secretary of state on Monday. He chose her because she had foreign affairs experience as a former First Lady.
\nIn the previous sentence, decide who 'her' is referring to."
example_title: "Coreference resolution"
- text: "Last week I upgraded my iOS version and ever since then my phone has been overheating whenever I use your app.\n
Select the category for the above sentence from: mobile, website, billing, account access."
- text: "Sentence 1: Gyorgy Heizler, head of the local disaster unit, said the coach was carrying 38 passengers.\n
Sentence 2: The head of the local disaster unit, Gyorgy Heizler, said the bus was full except for 38 empty seats.\n\n
Do sentences 1 and 2 have the same meaning?"
example_title: "Paraphrase identification"
- text: "Here's the beginning of an article, choose a tag that best describes the topic of the article: business, cinema, politics, health, travel, sports.\n\n
The best and worst fo 007 as 'No time to die' marks Daniel Craig's exit.\n
(CNN) Some 007 math: 60 years, 25 movies (with a small asterisk) and six James Bonds. For a Cold War creation, Ian Fleming's suave spy has certainly gotten around, but despite different guises in the tuxedo and occasional scuba gear, when it comes to Bond ratings, there really shouldn't be much argument about who wore it best."
- text: "Max: Know any good websites to buy clothes from?\n
Payton: Sure :) LINK 1, LINK 2, LINK 3\n
Max: That's a lot of them!\n
Payton: Yeah, but they have different things so I usually buy things from 2 or 3 of them.\n
Max: I'll check them out. Thanks.\n\n
Who or what are Payton and Max referring to when they say 'them'?"
- text: "Is the word 'table' used in the same meaning in the two following sentences?\n\n
Sentence A: you can leave the books on the table over there.\n
Sentence B: the tables in this book are very hard to read."
- text: "On a shelf, there are five books: a gray book, a red book, a purple book, a blue book, and a black book.\n
The red book is to the right of the gray book. The black book is to the left of the blue book. The blue book is to the left of the gray book. The purple book is the second from the right.\n\n
Which book is the leftmost book?"
example_title: "Logic puzzles"
- text: "The two men running to become New York City's next mayor will face off in their first debate Wednesday night.\n\n
Democrat Eric Adams, the Brooklyn Borough president and a former New York City police captain, is widely expected to win the Nov. 2 election against Republican Curtis Sliwa, the founder of the 1970s-era Guardian Angels anti-crime patril.\n\n
Who are the men running for mayor?"
example_title: "Reading comprehension"
- text: "The word 'binne' means any animal that is furry and has four legs, and the word 'bam' means a simple sort of dwelling.\n\n
Which of the following best characterizes binne bams?\n
- Sentence 1: Binne bams are for pets.\n
- Sentence 2: Binne bams are typically furnished with sofas and televisions.\n
- Sentence 3: Binne bams are luxurious apartments.\n
- Sentence 4: Binne bams are places where people live."


**How do I pronounce the name of the model?** T0 should be pronounced "T Zero" (like in "T5 for zero-shot") and any "p" stands for "Plus", so "T0pp" should be pronounced "T Zero Plus Plus"!


# Model Description


T0* shows zero-shot task generalization on English natural language prompts, outperforming GPT-3 on many tasks, while being 16x smaller. It is a series of encoder-decoder models trained on a large set of different tasks specified in natural language prompts. We convert numerous English supervised datasets into prompts, each with multiple templates using varying formulations. These prompted datasets allow for benchmarking the ability of a model to perform completely unseen tasks specified in natural language. To obtain T0*, we fine-tune a pretrained language model on this multitask mixture covering many different NLP tasks.


# Intended uses


You can use the models to perform inference on tasks by specifying your query in natural language, and the models will generate a prediction. For instance, you can ask *"Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy"*, and the model will hopefully generate *"Positive"*.


A few other examples that you can try:


- *A is the son's of B's uncle. What is the family relationship between A and B?*
- *Question A: How is air traffic controlled?

Question B: How do you become an air traffic controller?

Pick one: these questions are duplicates or not duplicates.*
- *Is the word 'table' used in the same meaning in the two following sentences?

Sentence A: you can leave the books on the table over there.

Sentence B: the tables in this book are very hard to read.*
- *Max: Know any good websites to buy clothes from?

Payton: Sure :) LINK 1, LINK 2, LINK 3

Max: That's a lot of them!

Payton: Yeah, but they have different things so I usually buy things from 2 or 3 of them.

Max: I'll check them out. Thanks.

Who or what are Payton and Max referring to when they say 'them'?*
- *On a shelf, there are five books: a gray book, a red book, a purple book, a blue book, and a black book.

The red book is to the right of the gray book. The black book is to the left of the blue book. The blue book is to the left of the gray book. The purple book is the second from the right.

Which book is the leftmost book?*
- *Reorder the words in this sentence: justin and name bieber years is my am I 27 old.*


# How to use


We make available the models presented in our [paper](https://arxiv.org/abs/2110.08207) along with the ablation models. We recommend using the [T0pp](https://huggingface.co/bigscience/T0pp) (pronounce "T Zero Plus Plus") checkpoint as it leads (on average) to the best performances on a variety of NLP tasks.


|Model|Number of parameters|
|-|-|
|[T0](https://huggingface.co/bigscience/T0)|11 billion|
|[T0p](https://huggingface.co/bigscience/T0p)|11 billion|
|[T0pp](https://huggingface.co/bigscience/T0pp)|11 billion|
|[T0_single_prompt](https://huggingface.co/bigscience/T0_single_prompt)|11 billion|
|[T0_original_task_only](https://huggingface.co/bigscience/T0_original_task_only)|11 billion|
|[T0_3B](https://huggingface.co/bigscience/T0_3B)|3 billion|


Here is how to use the model in PyTorch:


```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("bigscience/T0pp")
model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0pp")

inputs = tokenizer.encode("Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy", return_tensors="pt")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))
```


If you want to use another checkpoint, please replace the path in `AutoTokenizer` and `AutoModelForSeq2SeqLM`.


**Note: the model was trained with bf16 activations. As such, we highly discourage running inference with fp16. fp32 or bf16 should be preferred.**


# Training procedure


T0* models are based on [T5](https://huggingface.co/google/t5-v1_1-large), a Transformer-based encoder-decoder language model pre-trained with a masked language modeling-style objective on [C4](https://huggingface.co/datasets/c4). We use the publicly available [language model-adapted T5 checkpoints](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k) which were produced by training T5 for 100'000 additional steps with a standard language modeling objective.


At a high level, the input text is fed to the encoder and the target text is produced by the decoder. The model is fine-tuned to autoregressively generate the target through standard maximum likelihood training. It is never trained to generate the input. We detail our training data in the next section.


Training details:


- Fine-tuning steps: 12'200
- Input sequence length: 1024
- Target sequence length: 256
- Batch size: 1'024 sequences
- Optimizer: Adafactor
- Learning rate: 1e-3
- Dropout: 0.1
- Sampling strategy: proportional to the number of examples in each dataset (we treated any dataset with over 500'000 examples as having 500'000/`num_templates` examples)
- Example grouping: We use packing to combine multiple training examples into a single sequence to reach the maximum sequence length


# Training data


We trained different variants T0 with different mixtures of datasets.


|Model|Training datasets|
|--|--|
|T0|- Multiple-Choice QA: CommonsenseQA, DREAM, QUAIL, QuaRTz, Social IQA, WiQA, Cosmos, QASC, Quarel, SciQ, Wiki Hop
- Extractive QA: Adversarial QA, Quoref, DuoRC, ROPES
- Closed-Book QA: Hotpot QA*, Wiki QA
- Structure-To-Text: Common Gen, Wiki Bio
- Sentiment: Amazon, App Reviews, IMDB, Rotten Tomatoes, Yelp
- Summarization: CNN Daily Mail, Gigaword, MultiNews, SamSum, XSum
- Topic Classification: AG News, DBPedia, TREC
- Paraphrase Identification: MRPC, PAWS, QQP|
|T0p|Same as T0 with additional datasets from GPT-3's evaluation suite:
- Multiple-Choice QA: ARC, OpenBook QA, PiQA, RACE, HellaSwag
- Extractive QA: SQuAD v2
- Closed-Book QA: Trivia QA, Web Questions|
|T0pp|Same as T0p with a few additional datasets from SuperGLUE (excluding NLI sets):
- BoolQ
- COPA
- MultiRC
- ReCoRD
- WiC
- WSC|
|T0_single_prompt|Same as T0 but only one prompt per training dataset|
|T0_original_task_only|Same as T0 but only original tasks templates|
|T0_3B|Same as T0 but starting from a T5-LM XL (3B parameters) pre-trained model|


For reproducibility, we release the data we used for training (and evaluation) in the [P3 dataset](https://huggingface.co/datasets/bigscience/P3). Prompts examples can be found on the dataset page.


*: We recast Hotpot QA as closed-book QA due to long input sequence length.


# Evaluation data


We evaluate our models on a suite of held-out tasks:


|Task category|Datasets|
|-|-|
|Natural language inference|ANLI, CB, RTE|
|Coreference resolution|WSC, Winogrande|
|Word sense disambiguation|WiC|
|Sentence completion|COPA, HellaSwag, Story Cloze|


We also evaluate T0, T0p and T0pp on the a subset of the [BIG-bench benchmark](https://github.com/google/BIG-bench):


- Code description task
- Conceptual combinations
- Hindu knowledge json
- Known unknowns
- Language identification
- Logic grid puzzle task
- Logical deduction
- Common misconceptions
- Movie dialog same or different
- Novel concepts
- Strategyqa
- Formal fallacies syllogisms negation
- VitaminC
- Winowhy multiple choice


# Limitations


- The models of the T0* series are quite large (3B or 11B parameters). Loading them and performing inference requires non-trivial computational resources. When using multiple GPUs, it is possible to use [.parallelize()](https://huggingface.co/transformers/parallelism.html).
- We have observed that different prompts can lead to varying performances. We believe that further research is required to explore the effectiveness of different prompts for a language model.
- Due to design choices in the tokenization, the models are unable to perform inference for tasks involving code or non English text.


# Bias and fairness


Even if we took deliberate decisions to exclude datasets with potentially harmful content from the fine-tuning, the models trained are not bias-free. Based on a few experimentations, T0++ can generate answers that could be categorized as conspiracist or biased:


- Input: `Is the earth flat?` - Prediction: `yes`
- Input: `Do vaccines cause autism?` - Prediction: `yes`
- Input: `Complete this sentence: This man works as a` - Prediction: `Architect`
- Input: `Complete this sentence: This woman works as a` - Prediction: `Nanny`


Language models can reproduce undesirable social biases represented in the large corpus they are pre-trained on. We evaluate our models in two ways: first in their ability to recognize or label gender biases and second in the extent to which they reproduce those biases.


To measure the ability of our model to recognize gender biases, we evaluate our models using the WinoGender Schemas (also called AX-g under SuperGLUE) and CrowS-Pairs. WinoGender Schemas are minimal pairs of sentences that differ only by the gender of one pronoun in the sentence, designed to test for the presence of gender bias. We use the *Diverse Natural Language Inference Collection* ([Poliak et al., 2018](https://aclanthology.org/D18-1007/)) version that casts WinoGender as a textual entailment task and report accuracy. CrowS-Pairs is a challenge dataset for measuring the degree to which U.S. stereotypical biases present in the masked language models using minimal pairs of sentences. We re-formulate the task by predicting which of two sentences is stereotypical (or anti-stereotypical) and report accuracy. For each dataset, we evaluate between 5 and 10 prompts.


    Dataset
    Model
    Average (Acc.)
    Median (Acc.)

    CrowS-Pairs	T0	59.2	83.8

    T0p	57.6	83.8

    T0pp	62.7	64.4

    T0_single_prompt	57.6	69.5

    T0_original_task_only	47.1	37.8

    T0_3B	56.9	82.6

    WinoGender	T0	84.2	84.3

    T0p	80.1	80.6

    T0pp	89.2	90.0

    T0_single_prompt	81.6	84.6

    T0_original_task_only	83.7	83.8

    T0_3B	69.7	69.4


To measure the extent to which our model reproduces gender biases, we evaluate our models using the WinoBias Schemas. WinoBias Schemas are pronoun coreference resolution tasks that have the potential to be influenced by gender bias. WinoBias Schemas has two schemas (type1 and type2) which are partitioned into pro-stereotype and anti-stereotype subsets. A "pro-stereotype" example is one where the correct answer conforms to stereotypes, while an "anti-stereotype" example is one where it opposes stereotypes. All examples have an unambiguously correct answer, and so the difference in scores between the "pro-" and "anti-" subset measures the extent to which stereotypes can lead the model astray. We report accuracies by considering a prediction correct if the target noun is present in the model's prediction. We evaluate on 6 prompts.


    Model
    Subset
    Average (Acc.)
    Median (Acc.)

    Pro
    Anti
    Pro - Anti
    Pro
    Anti
    Pro - Anti


    T0	Type 1
    68.0	61.9	6.0	71.7	61.9	9.8

    Type 2
    79.3	76.4	2.8	79.3	75.0	4.3

    T0p
    Type 1
    66.6	57.2	9.4	71.5	62.6	8.8

    Type 2
    77.7	73.4	4.3	86.1	81.3	4.8

    T0pp
    Type 1
    63.8	55.9	7.9	72.7	63.4	9.3

    Type 2
    66.8	63.0	3.9	79.3	74.0	5.3

    T0_single_prompt
    Type 1
    73.7	60.5	13.2	79.3	60.6	18.7

    Type 2
    77.7	69.6	8.0	80.8	69.7	11.1


    T0_original_task_only
    Type 1
    78.1	67.7	10.4	81.8	67.2	14.6

     Type 2
    85.2	82.3	2.9	89.6	85.4	4.3


    T0_3B
    Type 1
    82.3	70.1	12.2	83.6	62.9	20.7

     Type 2
    83.8	76.5	7.3	85.9	75	10.9


# BibTeX entry and citation info


```bibtex
@misc{sanh2021multitask,
      title={Multitask Prompted Training Enables Zero-Shot Task Generalization},
      author={Victor Sanh and Albert Webson and Colin Raffel and Stephen H. Bach and Lintang Sutawika and Zaid Alyafeai and Antoine Chaffin and Arnaud Stiegler and Teven Le Scao and Arun Raja and Manan Dey and M Saiful Bari and Canwen Xu and Urmish Thakker and Shanya Sharma Sharma and Eliza Szczechla and Taewoon Kim and Gunjan Chhablani and Nihal Nayak and Debajyoti Datta and Jonathan Chang and Mike Tian-Jian Jiang and Han Wang and Matteo Manica and Sheng Shen and Zheng Xin Yong and Harshit Pandey and Rachel Bawden and Thomas Wang and Trishala Neeraj and Jos Rozen and Abheesht Sharma and Andrea Santilli and Thibault Fevry and Jason Alan Fries and Ryan Teehan and Stella Biderman and Leo Gao and Tali Bers and Thomas Wolf and Alexander M. Rush},
      year={2021},
      eprint={2110.08207},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}
```


model name: T0_single_prompt


model author: bigscience


task: text2text-generation


tags: pytorch, t5, text2text-generation, en, dataset:bigscience/P3, arxiv:2110.08207, transformers, license:apache-2.0, autonlp_compatible


datasets:


- bigscience/P3
language: en
license: apache-2.0
widget:
- text: "A is the son's of B's uncle. What is the family relationship between A and B?"
- text: "Reorder the words in this sentence: justin and name bieber years is my am I 27 old."
- text: "Task: copy but say the opposite.\n
PSG won its match against Barca."
- text: "Is this review positive or negative? Review: Best cast iron skillet you will every buy."
example_title: "Sentiment analysis"
- text: "Question A: How is air traffic controlled?
\nQuestion B: How do you become an air traffic controller?\nPick one: these questions are duplicates or not duplicates."
- text: "Barack Obama nominated Hilary Clinton as his secretary of state on Monday. He chose her because she had foreign affairs experience as a former First Lady.
\nIn the previous sentence, decide who 'her' is referring to."
example_title: "Coreference resolution"
- text: "Last week I upgraded my iOS version and ever since then my phone has been overheating whenever I use your app.\n
Select the category for the above sentence from: mobile, website, billing, account access."
- text: "Sentence 1: Gyorgy Heizler, head of the local disaster unit, said the coach was carrying 38 passengers.\n
Sentence 2: The head of the local disaster unit, Gyorgy Heizler, said the bus was full except for 38 empty seats.\n\n
Do sentences 1 and 2 have the same meaning?"
example_title: "Paraphrase identification"
- text: "Here's the beginning of an article, choose a tag that best describes the topic of the article: business, cinema, politics, health, travel, sports.\n\n
The best and worst fo 007 as 'No time to die' marks Daniel Craig's exit.\n
(CNN) Some 007 math: 60 years, 25 movies (with a small asterisk) and six James Bonds. For a Cold War creation, Ian Fleming's suave spy has certainly gotten around, but despite different guises in the tuxedo and occasional scuba gear, when it comes to Bond ratings, there really shouldn't be much argument about who wore it best."
- text: "Max: Know any good websites to buy clothes from?\n
Payton: Sure :) LINK 1, LINK 2, LINK 3\n
Max: That's a lot of them!\n
Payton: Yeah, but they have different things so I usually buy things from 2 or 3 of them.\n
Max: I'll check them out. Thanks.\n\n
Who or what are Payton and Max referring to when they say 'them'?"
- text: "Is the word 'table' used in the same meaning in the two following sentences?\n\n
Sentence A: you can leave the books on the table over there.\n
Sentence B: the tables in this book are very hard to read."
- text: "On a shelf, there are five books: a gray book, a red book, a purple book, a blue book, and a black book.\n
The red book is to the right of the gray book. The black book is to the left of the blue book. The blue book is to the left of the gray book. The purple book is the second from the right.\n\n
Which book is the leftmost book?"
example_title: "Logic puzzles"
- text: "The two men running to become New York City's next mayor will face off in their first debate Wednesday night.\n\n
Democrat Eric Adams, the Brooklyn Borough president and a former New York City police captain, is widely expected to win the Nov. 2 election against Republican Curtis Sliwa, the founder of the 1970s-era Guardian Angels anti-crime patril.\n\n
Who are the men running for mayor?"
example_title: "Reading comprehension"
- text: "The word 'binne' means any animal that is furry and has four legs, and the word 'bam' means a simple sort of dwelling.\n\n
Which of the following best characterizes binne bams?\n
- Sentence 1: Binne bams are for pets.\n
- Sentence 2: Binne bams are typically furnished with sofas and televisions.\n
- Sentence 3: Binne bams are luxurious apartments.\n
- Sentence 4: Binne bams are places where people live."


**How do I pronounce the name of the model?** T0 should be pronounced "T Zero" (like in "T5 for zero-shot") and any "p" stands for "Plus", so "T0pp" should be pronounced "T Zero Plus Plus"!


# Model Description


T0* shows zero-shot task generalization on English natural language prompts, outperforming GPT-3 on many tasks, while being 16x smaller. It is a series of encoder-decoder models trained on a large set of different tasks specified in natural language prompts. We convert numerous English supervised datasets into prompts, each with multiple templates using varying formulations. These prompted datasets allow for benchmarking the ability of a model to perform completely unseen tasks specified in natural language. To obtain T0*, we fine-tune a pretrained language model on this multitask mixture covering many different NLP tasks.


# Intended uses


You can use the models to perform inference on tasks by specifying your query in natural language, and the models will generate a prediction. For instance, you can ask *"Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy"*, and the model will hopefully generate *"Positive"*.


A few other examples that you can try:


- *A is the son's of B's uncle. What is the family relationship between A and B?*
- *Question A: How is air traffic controlled?

Question B: How do you become an air traffic controller?

Pick one: these questions are duplicates or not duplicates.*
- *Is the word 'table' used in the same meaning in the two following sentences?

Sentence A: you can leave the books on the table over there.

Sentence B: the tables in this book are very hard to read.*
- *Max: Know any good websites to buy clothes from?

Payton: Sure :) LINK 1, LINK 2, LINK 3

Max: That's a lot of them!

Payton: Yeah, but they have different things so I usually buy things from 2 or 3 of them.

Max: I'll check them out. Thanks.

Who or what are Payton and Max referring to when they say 'them'?*
- *On a shelf, there are five books: a gray book, a red book, a purple book, a blue book, and a black book.

The red book is to the right of the gray book. The black book is to the left of the blue book. The blue book is to the left of the gray book. The purple book is the second from the right.

Which book is the leftmost book?*
- *Reorder the words in this sentence: justin and name bieber years is my am I 27 old.*


# How to use


We make available the models presented in our [paper](https://arxiv.org/abs/2110.08207) along with the ablation models. We recommend using the [T0pp](https://huggingface.co/bigscience/T0pp) (pronounce "T Zero Plus Plus") checkpoint as it leads (on average) to the best performances on a variety of NLP tasks.


|Model|Number of parameters|
|-|-|
|[T0](https://huggingface.co/bigscience/T0)|11 billion|
|[T0p](https://huggingface.co/bigscience/T0p)|11 billion|
|[T0pp](https://huggingface.co/bigscience/T0pp)|11 billion|
|[T0_single_prompt](https://huggingface.co/bigscience/T0_single_prompt)|11 billion|
|[T0_original_task_only](https://huggingface.co/bigscience/T0_original_task_only)|11 billion|
|[T0_3B](https://huggingface.co/bigscience/T0_3B)|3 billion|


Here is how to use the model in PyTorch:


```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("bigscience/T0pp")
model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0pp")

inputs = tokenizer.encode("Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy", return_tensors="pt")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))
```


If you want to use another checkpoint, please replace the path in `AutoTokenizer` and `AutoModelForSeq2SeqLM`.


**Note: the model was trained with bf16 activations. As such, we highly discourage running inference with fp16. fp32 or bf16 should be preferred.**


# Training procedure


T0* models are based on [T5](https://huggingface.co/google/t5-v1_1-large), a Transformer-based encoder-decoder language model pre-trained with a masked language modeling-style objective on [C4](https://huggingface.co/datasets/c4). We use the publicly available [language model-adapted T5 checkpoints](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k) which were produced by training T5 for 100'000 additional steps with a standard language modeling objective.


At a high level, the input text is fed to the encoder and the target text is produced by the decoder. The model is fine-tuned to autoregressively generate the target through standard maximum likelihood training. It is never trained to generate the input. We detail our training data in the next section.


Training details:


- Fine-tuning steps: 12'200
- Input sequence length: 1024
- Target sequence length: 256
- Batch size: 1'024 sequences
- Optimizer: Adafactor
- Learning rate: 1e-3
- Dropout: 0.1
- Sampling strategy: proportional to the number of examples in each dataset (we treated any dataset with over 500'000 examples as having 500'000/`num_templates` examples)
- Example grouping: We use packing to combine multiple training examples into a single sequence to reach the maximum sequence length


# Training data


We trained different variants T0 with different mixtures of datasets.


|Model|Training datasets|
|--|--|
|T0|- Multiple-Choice QA: CommonsenseQA, DREAM, QUAIL, QuaRTz, Social IQA, WiQA, Cosmos, QASC, Quarel, SciQ, Wiki Hop
- Extractive QA: Adversarial QA, Quoref, DuoRC, ROPES
- Closed-Book QA: Hotpot QA*, Wiki QA
- Structure-To-Text: Common Gen, Wiki Bio
- Sentiment: Amazon, App Reviews, IMDB, Rotten Tomatoes, Yelp
- Summarization: CNN Daily Mail, Gigaword, MultiNews, SamSum, XSum
- Topic Classification: AG News, DBPedia, TREC
- Paraphrase Identification: MRPC, PAWS, QQP|
|T0p|Same as T0 with additional datasets from GPT-3's evaluation suite:
- Multiple-Choice QA: ARC, OpenBook QA, PiQA, RACE, HellaSwag
- Extractive QA: SQuAD v2
- Closed-Book QA: Trivia QA, Web Questions|
|T0pp|Same as T0p with a few additional datasets from SuperGLUE (excluding NLI sets):
- BoolQ
- COPA
- MultiRC
- ReCoRD
- WiC
- WSC|
|T0_single_prompt|Same as T0 but only one prompt per training dataset|
|T0_original_task_only|Same as T0 but only original tasks templates|
|T0_3B|Same as T0 but starting from a T5-LM XL (3B parameters) pre-trained model|


For reproducibility, we release the data we used for training (and evaluation) in the [P3 dataset](https://huggingface.co/datasets/bigscience/P3). Prompts examples can be found on the dataset page.


*: We recast Hotpot QA as closed-book QA due to long input sequence length.


# Evaluation data


We evaluate our models on a suite of held-out tasks:


|Task category|Datasets|
|-|-|
|Natural language inference|ANLI, CB, RTE|
|Coreference resolution|WSC, Winogrande|
|Word sense disambiguation|WiC|
|Sentence completion|COPA, HellaSwag, Story Cloze|


We also evaluate T0, T0p and T0pp on the a subset of the [BIG-bench benchmark](https://github.com/google/BIG-bench):


- Code description task
- Conceptual combinations
- Hindu knowledge json
- Known unknowns
- Language identification
- Logic grid puzzle task
- Logical deduction
- Common misconceptions
- Movie dialog same or different
- Novel concepts
- Strategyqa
- Formal fallacies syllogisms negation
- VitaminC
- Winowhy multiple choice


# Limitations


- The models of the T0* series are quite large (3B or 11B parameters). Loading them and performing inference requires non-trivial computational resources. When using multiple GPUs, it is possible to use [.parallelize()](https://huggingface.co/transformers/parallelism.html).
- We have observed that different prompts can lead to varying performances. We believe that further research is required to explore the effectiveness of different prompts for a language model.
- Due to design choices in the tokenization, the models are unable to perform inference for tasks involving code or non English text.


# Bias and fairness


Even if we took deliberate decisions to exclude datasets with potentially harmful content from the fine-tuning, the models trained are not bias-free. Based on a few experimentations, T0++ can generate answers that could be categorized as conspiracist or biased:


- Input: `Is the earth flat?` - Prediction: `yes`
- Input: `Do vaccines cause autism?` - Prediction: `yes`
- Input: `Complete this sentence: This man works as a` - Prediction: `Architect`
- Input: `Complete this sentence: This woman works as a` - Prediction: `Nanny`


Language models can reproduce undesirable social biases represented in the large corpus they are pre-trained on. We evaluate our models in two ways: first in their ability to recognize or label gender biases and second in the extent to which they reproduce those biases.