The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


## Evaluation results


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


## Citation


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


```bibtex
@inproceedings{poth-etal-2021-what-to-pre-train-on,
    title={What to Pre-Train on? Efficient Intermediate Task Selection},
    author={Clifton Poth and Jonas Pfeiffer and Andreas Rücklé and Iryna Gurevych},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2104.08247",
    pages = "to appear",
}
```


model name: bert-base-uncased-pf-swag


model author: AdapterHub


tags: bert, en, dataset:swag, arxiv:2104.08247, adapter-transformers, infinity_compatible


tags:


- question-answering
- bert
- adapter-transformers
datasets:
- quoref
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-quoref` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [quoref](https://huggingface.co/datasets/quoref/) dataset and includes a prediction head for question answering.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


## Usage


First, install `adapter-transformers`:


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-quoref", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


## Evaluation results


## Citation


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: bert-base-uncased-pf-quoref


model author: AdapterHub


task: question-answering


tags: bert, en, dataset:quoref, arxiv:2104.08247, adapter-transformers, question-answering, infinity_compatible


tags:


- bert
- adapterhub:comsense/csqa
- adapter-transformers
datasets:
- commonsense_qa
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-commonsense_qa` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [comsense/csqa](https://adapterhub.ml/explore/comsense/csqa/) dataset and includes a prediction head for multiple choice.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-commonsense_qa", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


```bibtex
@inproceedings{poth-etal-2021-what-to-pre-train-on,
    title={What to Pre-Train on? Efficient Intermediate Task Selection},
    author={Clifton Poth and Jonas Pfeiffer and Andreas Rücklé and Iryna Gurevych},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2104.08247",
    pages = "to appear",
}
```


model name: bert-base-uncased-pf-commonsense_qa


tags: bert, en, dataset:commonsense_qa, arxiv:2104.08247, adapter-transformers, adapterhub:comsense/csqa, infinity_compatible


- question-answering
- roberta
- adapterhub:qa/wikihop
- adapter-transformers
language:
- en


# Adapter `AdapterHub/roberta-base-pf-wikihop` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [qa/wikihop](https://adapterhub.ml/explore/qa/wikihop/) dataset and includes a prediction head for question answering.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-wikihop", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


## Evaluation results


## Citation


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: roberta-base-pf-wikihop


model author: AdapterHub


task: question-answering


tags: roberta, en, arxiv:2104.08247, adapter-transformers, question-answering, adapterhub:qa/wikihop, infinity_compatible


tags:


- roberta
- adapter-transformers
datasets:
- social_i_qa
language:
- en


# Adapter `AdapterHub/roberta-base-pf-social_i_qa` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [social_i_qa](https://huggingface.co/datasets/social_i_qa/) dataset and includes a prediction head for multiple choice.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-social_i_qa", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


```bibtex
@inproceedings{poth-etal-2021-what-to-pre-train-on,
    title={What to Pre-Train on? Efficient Intermediate Task Selection},
    author={Clifton Poth and Jonas Pfeiffer and Andreas Rücklé and Iryna Gurevych},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2104.08247",
    pages = "to appear",
}
```


model name: roberta-base-pf-social_i_qa


tags: roberta, en, dataset:social_i_qa, arxiv:2104.08247, adapter-transformers, infinity_compatible


- text-classification
- bert
- adapterhub:sentiment/sst-2
- adapter-transformers
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-sst2` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [sentiment/sst-2](https://adapterhub.ml/explore/sentiment/sst-2/) dataset and includes a prediction head for classification.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-sst2", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


## Evaluation results


## Citation


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: bert-base-uncased-pf-sst2


model author: AdapterHub


task: text-classification


tags: bert, en, arxiv:2104.08247, adapter-transformers, text-classification, adapterhub:sentiment/sst-2, infinity_compatible


tags:


- text-classification
- roberta
- adapter-transformers
datasets:
- snli
language:
- en


# Adapter `AdapterHub/roberta-base-pf-snli` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [snli](https://huggingface.co/datasets/snli/) dataset and includes a prediction head for classification.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-snli", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


model name: roberta-base-pf-snli


tags: roberta, en, dataset:snli, arxiv:2104.08247, adapter-transformers, text-classification, infinity_compatible


- bert
- adapterhub:comsense/hellaswag
- adapter-transformers
datasets:
- hellaswag
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-hellaswag` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [comsense/hellaswag](https://adapterhub.ml/explore/comsense/hellaswag/) dataset and includes a prediction head for multiple choice.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-hellaswag", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


## Evaluation results


## Citation


```bibtex
@inproceedings{poth-etal-2021-what-to-pre-train-on,
    title={What to Pre-Train on? Efficient Intermediate Task Selection},
    author={Clifton Poth and Jonas Pfeiffer and Andreas Rücklé and Iryna Gurevych},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2104.08247",
    pages = "to appear",
}
```


model name: bert-base-uncased-pf-hellaswag


model author: AdapterHub


tags: bert, en, dataset:hellaswag, arxiv:2104.08247, adapter-transformers, adapterhub:comsense/hellaswag, infinity_compatible


tags:


- token-classification
- roberta
- adapterhub:ner/mit_movie_trivia
- adapter-transformers
language:
- en


# Adapter `AdapterHub/roberta-base-pf-mit_movie_trivia` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [ner/mit_movie_trivia](https://adapterhub.ml/explore/ner/mit_movie_trivia/) dataset and includes a prediction head for tagging.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-mit_movie_trivia", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: roberta-base-pf-mit_movie_trivia


task: token-classification


tags: roberta, en, arxiv:2104.08247, adapter-transformers, token-classification, adapterhub:ner/mit_movie_trivia, infinity_compatible


- question-answering
- roberta
- adapter-transformers
datasets:
- newsqa
language:
- en


# Adapter `AdapterHub/roberta-base-pf-newsqa` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [newsqa](https://huggingface.co/datasets/newsqa/) dataset and includes a prediction head for question answering.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-newsqa", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


## Evaluation results


## Citation


model name: roberta-base-pf-newsqa


model author: AdapterHub


task: question-answering


tags: roberta, en, dataset:newsqa, arxiv:2104.08247, adapter-transformers, question-answering, infinity_compatible


tags:


- bert
- adapter-transformers
datasets:
- quartz
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-quartz` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [quartz](https://huggingface.co/datasets/quartz/) dataset and includes a prediction head for multiple choice.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-quartz", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


```bibtex
@inproceedings{poth-etal-2021-what-to-pre-train-on,
    title={What to Pre-Train on? Efficient Intermediate Task Selection},
    author={Clifton Poth and Jonas Pfeiffer and Andreas Rücklé and Iryna Gurevych},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2104.08247",
    pages = "to appear",
}
```


model name: bert-base-uncased-pf-quartz


tags: bert, en, dataset:quartz, arxiv:2104.08247, adapter-transformers, infinity_compatible


- roberta
- adapter-transformers
datasets:
- quartz
language:
- en


# Adapter `AdapterHub/roberta-base-pf-quartz` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [quartz](https://huggingface.co/datasets/quartz/) dataset and includes a prediction head for multiple choice.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-quartz", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


## Evaluation results


## Citation


model name: roberta-base-pf-quartz


model author: AdapterHub


tags: roberta, en, dataset:quartz, arxiv:2104.08247, adapter-transformers, infinity_compatible


tags:


- text-classification
- roberta
- adapterhub:wordsence/wic
- adapter-transformers
language:
- en


# Adapter `AdapterHub/roberta-base-pf-wic` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [wordsence/wic](https://adapterhub.ml/explore/wordsence/wic/) dataset and includes a prediction head for classification.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-wic", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: roberta-base-pf-wic


task: text-classification


tags: roberta, en, arxiv:2104.08247, adapter-transformers, text-classification, adapterhub:wordsence/wic, infinity_compatible


- text-classification
- bert
- adapterhub:sts/mrpc
- adapter-transformers
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-mrpc` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [sts/mrpc](https://adapterhub.ml/explore/sts/mrpc/) dataset and includes a prediction head for classification.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-mrpc", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


## Evaluation results


## Citation


model name: bert-base-uncased-pf-mrpc


model author: AdapterHub


tags: bert, en, arxiv:2104.08247, adapter-transformers, text-classification, adapterhub:sts/mrpc, infinity_compatible


tags:


- roberta
- adapterhub:comsense/hellaswag
- adapter-transformers
datasets:
- hellaswag
language:
- en


# Adapter `AdapterHub/roberta-base-pf-hellaswag` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [comsense/hellaswag](https://adapterhub.ml/explore/comsense/hellaswag/) dataset and includes a prediction head for multiple choice.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-hellaswag", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


```bibtex
@inproceedings{poth-etal-2021-what-to-pre-train-on,
    title={What to Pre-Train on? Efficient Intermediate Task Selection},
    author={Clifton Poth and Jonas Pfeiffer and Andreas Rücklé and Iryna Gurevych},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2104.08247",
    pages = "to appear",
}
```


model name: roberta-base-pf-hellaswag


tags: roberta, en, dataset:hellaswag, arxiv:2104.08247, adapter-transformers, adapterhub:comsense/hellaswag, infinity_compatible


- text-classification
- roberta
- adapterhub:sentiment/imdb
- adapter-transformers
datasets:
- imdb
language:
- en


# Adapter `AdapterHub/roberta-base-pf-imdb` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [sentiment/imdb](https://adapterhub.ml/explore/sentiment/imdb/) dataset and includes a prediction head for classification.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-imdb", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


## Evaluation results


## Citation


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: roberta-base-pf-imdb


model author: AdapterHub


task: text-classification


tags: roberta, en, dataset:imdb, arxiv:2104.08247, adapter-transformers, text-classification, adapterhub:sentiment/imdb, infinity_compatible


tags:


- text-classification
- roberta
- adapterhub:sentiment/sst-2
- adapter-transformers
language:
- en


# Adapter `AdapterHub/roberta-base-pf-sst2` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [sentiment/sst-2](https://adapterhub.ml/explore/sentiment/sst-2/) dataset and includes a prediction head for classification.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-sst2", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


model name: roberta-base-pf-sst2


tags: roberta, en, arxiv:2104.08247, adapter-transformers, text-classification, adapterhub:sentiment/sst-2, infinity_compatible


- text-classification
- roberta
- adapterhub:nli/rte
- adapter-transformers
language:
- en


# Adapter `AdapterHub/roberta-base-pf-rte` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [nli/rte](https://adapterhub.ml/explore/nli/rte/) dataset and includes a prediction head for classification.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-rte", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


## Evaluation results


## Citation


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: roberta-base-pf-rte


model author: AdapterHub


task: text-classification


tags: roberta, en, arxiv:2104.08247, adapter-transformers, text-classification, adapterhub:nli/rte, infinity_compatible


tags:


- text-classification
- bert
- adapterhub:wordsence/wic
- adapter-transformers
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-wic` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [wordsence/wic](https://adapterhub.ml/explore/wordsence/wic/) dataset and includes a prediction head for classification.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-wic", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


model name: bert-base-uncased-pf-wic


tags: bert, en, arxiv:2104.08247, adapter-transformers, text-classification, adapterhub:wordsence/wic, infinity_compatible


- adapterhub:rc/race
- roberta
- adapter-transformers
datasets:
- race
language:
- en


# Adapter `AdapterHub/roberta-base-pf-race` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [rc/race](https://adapterhub.ml/explore/rc/race/) dataset and includes a prediction head for multiple choice.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-race", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


## Evaluation results


## Citation


```bibtex
@inproceedings{poth-etal-2021-what-to-pre-train-on,
    title={What to Pre-Train on? Efficient Intermediate Task Selection},
    author={Clifton Poth and Jonas Pfeiffer and Andreas Rücklé and Iryna Gurevych},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2104.08247",
    pages = "to appear",
}
```


model name: roberta-base-pf-race


model author: AdapterHub


tags: roberta, en, dataset:race, arxiv:2104.08247, adapter-transformers, adapterhub:rc/race, infinity_compatible


tags:


- token-classification
- bert
- adapterhub:pos/conll2003
- adapter-transformers
datasets:
- conll2003
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-conll2003_pos` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [pos/conll2003](https://adapterhub.ml/explore/pos/conll2003/) dataset and includes a prediction head for tagging.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-conll2003_pos", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: bert-base-uncased-pf-conll2003_pos


task: token-classification


tags: bert, en, dataset:conll2003, arxiv:2104.08247, adapter-transformers, token-classification, adapterhub:pos/conll2003, infinity_compatible


- question-answering
- roberta
- adapter-transformers
datasets:
- drop
language:
- en


# Adapter `AdapterHub/roberta-base-pf-drop` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [drop](https://huggingface.co/datasets/drop/) dataset and includes a prediction head for question answering.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-drop", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


## Evaluation results


## Citation


model name: roberta-base-pf-drop


model author: AdapterHub


task: question-answering


tags: roberta, en, dataset:drop, arxiv:2104.08247, adapter-transformers, question-answering, infinity_compatible


tags:


- bert
- adapterhub:dp/ud_ewt
- adapter-transformers
datasets:
- universal_dependencies
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-ud_en_ewt` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [dp/ud_ewt](https://adapterhub.ml/explore/dp/ud_ewt/) dataset and includes a prediction head for dependency parsing.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-ud_en_ewt", source="hf", set_active=True)
```


This adapter was trained using adapter-transformer's example script for dependency parsing.
See https://github.com/Adapter-Hub/adapter-transformers/tree/master/examples/dependency-parsing.


Scores achieved by dependency parsing adapters on the test set of UD English EWT after training:


| Model | UAS | LAS |
| --- | --- | --- |
| `bert-base-uncased` | 91.74 | 89.15 |
| `roberta-base` | 91.43 | 88.43 |


<!-- Add some description here -->


model name: bert-base-uncased-pf-ud_en_ewt


tags: bert, en, dataset:universal_dependencies, adapter-transformers, adapterhub:dp/ud_ewt, infinity_compatible


- roberta
- adapter-transformers
datasets:
- quail
language:
- en


# Adapter `AdapterHub/roberta-base-pf-quail` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [quail](https://huggingface.co/datasets/quail/) dataset and includes a prediction head for multiple choice.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-quail", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


## Evaluation results


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


## Citation


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


```bibtex
@inproceedings{poth-etal-2021-what-to-pre-train-on,
    title={What to Pre-Train on? Efficient Intermediate Task Selection},
    author={Clifton Poth and Jonas Pfeiffer and Andreas Rücklé and Iryna Gurevych},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2104.08247",
    pages = "to appear",
}
```


model name: roberta-base-pf-quail


model author: AdapterHub


tags: roberta, en, dataset:quail, arxiv:2104.08247, adapter-transformers, infinity_compatible


tags:


- token-classification
- bert
- adapterhub:ner/conll2003
- adapter-transformers
datasets:
- conll2003
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-conll2003` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [ner/conll2003](https://adapterhub.ml/explore/ner/conll2003/) dataset and includes a prediction head for tagging.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-conll2003", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: bert-base-uncased-pf-conll2003


task: token-classification


tags: bert, en, dataset:conll2003, arxiv:2104.08247, adapter-transformers, token-classification, adapterhub:ner/conll2003, infinity_compatible


- question-answering
- bert
- adapterhub:qa/wikihop
- adapter-transformers
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-wikihop` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [qa/wikihop](https://adapterhub.ml/explore/qa/wikihop/) dataset and includes a prediction head for question answering.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-wikihop", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


## Evaluation results


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


## Citation


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


model name: bert-base-uncased-pf-wikihop


model author: AdapterHub


task: question-answering


tags: bert, en, arxiv:2104.08247, adapter-transformers, question-answering, adapterhub:qa/wikihop, infinity_compatible


tags:


- text-classification
- roberta
- adapterhub:lingaccept/cola
- adapter-transformers
language:
- en


# Adapter `AdapterHub/roberta-base-pf-cola` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [lingaccept/cola](https://adapterhub.ml/explore/lingaccept/cola/) dataset and includes a prediction head for classification.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-cola", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: roberta-base-pf-cola


task: text-classification


tags: roberta, en, arxiv:2104.08247, adapter-transformers, text-classification, adapterhub:lingaccept/cola, infinity_compatible


- text-classification
- adapter-transformers
- adapterhub:sts/qqp
- bert
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-qqp` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [sts/qqp](https://adapterhub.ml/explore/sts/qqp/) dataset and includes a prediction head for classification.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-qqp", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


## Evaluation results


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


## Citation


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


model name: bert-base-uncased-pf-qqp


model author: AdapterHub


task: text-classification


tags: bert, en, arxiv:2104.08247, adapter-transformers, text-classification, adapterhub:sts/qqp, infinity_compatible


tags:


- text-classification
- roberta
- adapter-transformers
datasets:
- trec
language:
- en


# Adapter `AdapterHub/roberta-base-pf-trec` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [trec](https://huggingface.co/datasets/trec/) dataset and includes a prediction head for classification.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-trec", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: roberta-base-pf-trec


tags: roberta, en, dataset:trec, arxiv:2104.08247, adapter-transformers, text-classification, infinity_compatible


- text-classification
- roberta
- adapter-transformers
- adapterhub:nli/sick
- text-classification
datasets:
- sick
language:
- en


# Adapter `AdapterHub/roberta-base-pf-sick` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [nli/sick](https://adapterhub.ml/explore/nli/sick/) dataset and includes a prediction head for classification.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-sick", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


## Evaluation results


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


## Citation


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


model name: roberta-base-pf-sick


model author: AdapterHub


task: text-classification


tags: roberta, en, dataset:sick, arxiv:2104.08247, adapter-transformers, text-classification, adapterhub:nli/sick, infinity_compatible


tags:


- question-answering
- roberta
- adapter-transformers
datasets:
- hotpot_qa
language:
- en


# Adapter `AdapterHub/roberta-base-pf-hotpotqa` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [hotpot_qa](https://huggingface.co/datasets/hotpot_qa/) dataset and includes a prediction head for question answering.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-hotpotqa", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: roberta-base-pf-hotpotqa


task: question-answering


tags: roberta, en, dataset:hotpot_qa, arxiv:2104.08247, adapter-transformers, question-answering, infinity_compatible


- token-classification
- bert
- adapterhub:pos/ud_ewt
- adapter-transformers
datasets:
- universal_dependencies
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-ud_pos` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [pos/ud_ewt](https://adapterhub.ml/explore/pos/ud_ewt/) dataset and includes a prediction head for tagging.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-ud_pos", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


## Evaluation results


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


## Citation


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


model name: bert-base-uncased-pf-ud_pos


model author: AdapterHub


task: token-classification


tags: bert, en, dataset:universal_dependencies, arxiv:2104.08247, adapter-transformers, token-classification, adapterhub:pos/ud_ewt, infinity_compatible


tags:


- question-answering
- roberta
- adapterhub:qa/squad1
- adapter-transformers
datasets:
- squad
language:
- en


# Adapter `AdapterHub/roberta-base-pf-squad` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [qa/squad1](https://adapterhub.ml/explore/qa/squad1/) dataset and includes a prediction head for question answering.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-squad", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: roberta-base-pf-squad


task: question-answering


tags: roberta, en, dataset:squad, arxiv:2104.08247, adapter-transformers, question-answering, adapterhub:qa/squad1, infinity_compatible


- text-classification
- bert
- adapter-transformers
datasets:
- snli
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-snli` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [snli](https://huggingface.co/datasets/snli/) dataset and includes a prediction head for classification.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-snli", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


## Evaluation results


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


## Citation


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


model name: bert-base-uncased-pf-snli


model author: AdapterHub


task: text-classification


tags: bert, en, dataset:snli, arxiv:2104.08247, adapter-transformers, text-classification, infinity_compatible


tags:


- bert
- adapterhub:comsense/winogrande
- adapter-transformers
datasets:
- winogrande
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-winogrande` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [comsense/winogrande](https://adapterhub.ml/explore/comsense/winogrande/) dataset and includes a prediction head for multiple choice.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-winogrande", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


```bibtex
@inproceedings{poth-etal-2021-what-to-pre-train-on,
    title={What to Pre-Train on? Efficient Intermediate Task Selection},
    author={Clifton Poth and Jonas Pfeiffer and Andreas Rücklé and Iryna Gurevych},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2104.08247",
    pages = "to appear",
}
```


model name: bert-base-uncased-pf-winogrande


tags: bert, en, dataset:winogrande, arxiv:2104.08247, adapter-transformers, adapterhub:comsense/winogrande, infinity_compatible


- text-classification
- bert
- adapter-transformers
datasets:
- scicite
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-scicite` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [scicite](https://huggingface.co/datasets/scicite/) dataset and includes a prediction head for classification.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-scicite", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


## Evaluation results


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


## Citation


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: bert-base-uncased-pf-scicite


model author: AdapterHub


task: text-classification


tags: bert, en, dataset:scicite, arxiv:2104.08247, adapter-transformers, text-classification, infinity_compatible


tags:


- question-answering
- bert
- adapterhub:qa/squad1
- adapter-transformers
datasets:
- squad
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-squad` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [qa/squad1](https://adapterhub.ml/explore/qa/squad1/) dataset and includes a prediction head for question answering.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-squad", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


## Evaluation results


model name: bert-base-uncased-pf-squad


task: question-answering


tags: bert, en, dataset:squad, arxiv:2104.08247, adapter-transformers, question-answering, adapterhub:qa/squad1, infinity_compatible


- token-classification
- bert
- adapterhub:deprel/ud_ewt
- adapter-transformers
datasets:
- universal_dependencies
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-ud_deprel` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [deprel/ud_ewt](https://adapterhub.ml/explore/deprel/ud_ewt/) dataset and includes a prediction head for tagging.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-ud_deprel", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


## Citation


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: bert-base-uncased-pf-ud_deprel


model author: AdapterHub


task: token-classification


tags: bert, en, dataset:universal_dependencies, arxiv:2104.08247, adapter-transformers, token-classification, adapterhub:deprel/ud_ewt, infinity_compatible


tags:


- text-classification
- bert
- adapterhub:lingaccept/cola
- adapter-transformers
language:
- en


# Adapter `AdapterHub/bert-base-uncased-pf-cola` for bert-base-uncased


An [adapter](https://adapterhub.ml) for the `bert-base-uncased` model that was trained on the [lingaccept/cola](https://adapterhub.ml/explore/lingaccept/cola/) dataset and includes a prediction head for classification.


## Usage


First, install `adapter-transformers`:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-cola", source="hf")
model.active_adapters = adapter_name
```


## Architecture & Training


## Evaluation results


model name: bert-base-uncased-pf-cola


task: text-classification


tags: bert, en, arxiv:2104.08247, adapter-transformers, text-classification, adapterhub:lingaccept/cola, infinity_compatible


- text-classification
- roberta
- adapter-transformers
datasets:
- scicite
language:
- en


# Adapter `AdapterHub/roberta-base-pf-scicite` for roberta-base


An [adapter](https://adapterhub.ml) for the `roberta-base` model that was trained on the [scicite](https://huggingface.co/datasets/scicite/) dataset and includes a prediction head for classification.


This adapter was created for usage with the **[adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** library.


```
pip install -U adapter-transformers
```


*Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)*


Now, the adapter can be loaded and activated like this:


```python
from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-scicite", source="hf")
model.active_adapters = adapter_name
```


The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer.
In particular, training configurations for all tasks can be found [here](https://github.com/adapter-hub/efficient-task-transfer/tree/master/run_configs).


Refer to [the paper](https://arxiv.org/pdf/2104.08247) for more information on results.


## Citation


If you use this adapter, please cite our paper ["What to Pre-Train on? Efficient Intermediate Task Selection"](https://arxiv.org/pdf/2104.08247):


```bibtex
@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}
```


model name: roberta-base-pf-scicite


model author: AdapterHub


task: text-classification


tags: roberta, en, dataset:scicite, arxiv:2104.08247, adapter-transformers, text-classification, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: t5-coktails_recipe-small
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# t5-coktails_recipe-small


This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on an unknown dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: t5-coktails_recipe-small


model author: erwanlc


task: text2text-generation


tags: pytorch, tensorboard, t5, text2text-generation, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: t5-coktails_recipe-base
results: []


# t5-coktails_recipe-base


This model is a fine-tuned version of [google/t5-v1_1-base](https://huggingface.co/google/t5-v1_1-base) on an unknown dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


### Framework versions


model name: t5-coktails_recipe-base


- generated_from_trainer
model-index:
- name: t5-cocktails_recipe-base
results: []


# t5-cocktails_recipe-base


This model is a fine-tuned version of [t5-base](https://huggingface.co/t5-base) on an unknown dataset.


- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5


model name: t5-cocktails_recipe-base


model author: erwanlc


task: text2text-generation


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: t5-cocktails_recipe-small
results: []


# t5-cocktails_recipe-small


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


### Framework versions


model name: t5-cocktails_recipe-small


## license: apache-2.0


model name: ecomm-sbert


model author: Maunish


task: fill-mask


tags: pytorch, bert, fill-mask, transformers, license:apache-2.0, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- squad_v2
model-index:
- name: distilbert-base-uncased-finetuned-squad
results: []


# distilbert-base-uncased-finetuned-squad


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the squad_v2 dataset.
It achieves the following results on the evaluation set:


- Loss: 2.6623


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.3993        | 1.0   | 2051 | 1.8058          |
| 1.0467        | 2.0   | 4102 | 1.9564          |
| 0.8304        | 3.0   | 6153 | 2.6623          |


### Framework versions


- Transformers 4.14.1
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-squad


model author: princebansal42


task: question-answering


tags: pytorch, tensorboard, distilbert, question-answering, dataset:squad_v2, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: mit
tags:


- generated_from_trainer
datasets:
- amazon_reviews_multi
model-index:
- name: xlm-roberta-base-finetuned-marc-en
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# xlm-roberta-base-finetuned-marc-en


This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the amazon_reviews_multi dataset.
It achieves the following results on the evaluation set:


- Loss: 0.9237
- Mae: 0.5122


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2


### Training results


| Training Loss | Epoch | Step | Validation Loss | Mae    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 1.1089        | 1.0   | 235  | 0.9380          | 0.4878 |
| 0.9546        | 2.0   | 470  | 0.9237          | 0.5122 |


### Framework versions


model name: xlm-roberta-base-finetuned-marc-en


model author: TomO


task: text-classification


tags: pytorch, tensorboard, xlm-roberta, text-classification, dataset:amazon_reviews_multi, transformers, license:mit, generated_from_trainer, model-index


# 中文预训练Longformer模型 | Longformer_ZH with PyTorch


相比于Transformer的O(n^2)复杂度，Longformer提供了一种以线性复杂度处理最长4K字符级别文档序列的方法。Longformer Attention包括了标准的自注意力与全局注意力机制，方便模型更好地学习超长序列的信息。


Compared with O(n^2) complexity for Transformer model, Longformer provides an efficient method for processing long-document level sequence in Linear complexity. Longformer’s attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention.


我们注意到关于中文Longformer或超长序列任务的资源较少，因此在此开源了我们预训练的中文Longformer模型参数， 并提供了相应的加载方法，以及预训练脚本。


There are not so much resource for Chinese Longformer or long-sequence-level chinese task. Thus we open source our pretrained longformer model to help the researchers.


## 加载模型 | Load the model


您可以使用谷歌云盘或百度网盘下载我们的模型\
You could get Longformer_zh from Google Drive or Baidu Yun.


- Google Drive: https://drive.google.com/file/d/1IDJ4aVTfSFUQLIqCYBtoRpnfbgHPoxB4/view?usp=sharing
- 百度云:  链接：https://pan.baidu.com/s/1HaVDENx52I7ryPFpnQmq1w 提取码：y601


我们同样提供了Huggingface的自动下载\
We also provide auto load with HuggingFace.Transformers.


```
from Longformer_zh import LongformerZhForMaksedLM
LongformerZhForMaksedLM.from_pretrained('ValkyriaLenneth/longformer_zh')
```


## 注意事项 | Notice


- 直接使用 `transformers.LongformerModel.from_pretrained` 加载模型
- Please use `transformers.LongformerModel.from_pretrained` to load the model directly
- 以下内容已经被弃用
- The following notices are abondoned, please ignore them.
- 区别于英文原版Longformer， 中文Longformer的基础是Roberta_zh模型，其本质上属于 `Transformers.BertModel` 而非 `RobertaModel`, 因此无法使用原版代码直接加载。
- Different with origin English Longformer, Longformer_Zh is based on Roberta_zh which is a subclass of `Transformers.BertModel` not `RobertaModel`. Thus it is impossible to load it with origin code.
- 我们提供了修改后的中文Longformer文件，您可以使用其加载参数。
- We provide modified Longformer_zh class, you can use it directly to load the model.
- 如果您想将此参数用于更多任务，请参考`Longformer_zh.py`替换Attention Layer.
- If you want to use our model on more down-stream tasks, please refer to `Longformer_zh.py` and replace Attention layer with Longformer Attention layer.


## 关于预训练 | About Pretraining


- 我们的预训练语料来自 https://github.com/brightmart/nlp_chinese_corpus， 根据Longformer原文的设置，采用了多种语料混合的预训练数据。
- The corpus of pretraining is from https://github.com/brightmart/nlp_chinese_corpus. Based on the paper of Longformer, we use a mixture of 4 different chinese corpus for pretraining.
- 我们的模型是基于Roberta_zh_mid (https://github.com/brightmart/roberta_zh),训练脚本参考了https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb
- The basement of our model is Roberta_zh_mid (https://github.com/brightmart/roberta_zh). Pretraining scripts is modified from https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb.
- 同时我们在原版基础上，引入了 `Whole-Word-Masking` 机制，以便更好地适应中文特性。
- We introduce `Whole-Word-Masking` method into pretraining for better fitting Chinese language.
- `Whole-Word-Masking`代码改写自TensorFlow版本的Roberta_zh，据我们所知是第一个开源的Pytorch版本WWM.
- Our WWM scripts is refacted from Roberta_zh_Tensorflow, as far as we know, it is the first open source Whole-word-masking scripts in Pytorch.
- 模型 `max_seq_length = 4096`, 在 4 * Titan RTX 上预训练3K steps 大概用时4天。
- Max seuence length is 4096 and the pretraining took 4 days on 4 * Titan RTX.
- 我们使用了 `Nvidia.Apex` 引入了混合精度训练，以加速预训练。
- We use `Nvidia.Apex` to accelerate pretraining.
- 关于数据预处理， 我们采用 `Jieba` 分词与`JIONLP`进行数据清洗。
- We use `Jieba` Chinese tokenizer and `JIONLP` data cleaning.
- 更多细节可以参考我们的预训练脚本
- For more details, please check our pretraining scripts.


## 效果测试 | Evaluation


### CCF Sentiment Analysis


- 由于中文超长文本级别任务稀缺，我们采用了CCF-Sentiment-Analysis任务进行测试
- Since it is hard to acquire open-sourced long sequence level chinese NLP task, we use CCF-Sentiment-Analysis for evaluation.


|Model|Dev F|
|----|----|
|Bert|80.3|
|Bert-wwm-ext| 80.5|
|Roberta-mid|80.5|
|Roberta-large|81.25|
|Longformer_SC|79.37|
|Longformer_ZH|80.51|


### Pretraining BPC


- 我们提供了预训练BPC(bits-per-character), BPC越小，代表语言模型性能更优。可视作PPL.
- We also provide BPC scores of pretraining, the lower BPC score, the better performance Langugage Model has. You can also treat it as PPL.


|Model|BPC|
|---|---|
|Longformer before training| 14.78|
|Longformer after training| 3.10|


### CMRC(Chinese Machine Reading Comprehension)


|Model|F1|EM|
|---|---|---|
|Bert|85.87|64.90|
|Roberta|86.45|66.57|
|Longformer_zh|86.15|66.84|


### Chinese Coreference Resolution


|Model|Conll-F1|Precision|Recall|
|---|---|---|---|
|Bert|66.82|70.30|63.67|
|Roberta|67.77|69.28|66.32|
|Longformer_zh|67.81|70.13|65.64|


## 致谢


感谢东京工业大学 奥村·船越研究室 提供算力。


Thanks Okumula·Funakoshi Lab from Tokyo Institute of Technology who provides the devices and oppotunity for me to finish this project.


model name: longformer_zh


model author: ValkyriaLenneth


task: feature-extraction


tags: pytorch, longformer, feature-extraction, transformers


language:


- en
thumbnail:
widget:
- text: "topic climate source washington post title "


# GPT2-medium-topic-news


## Model description


GPT2-medium fine tuned on a largish news corpus conditioned on a topic, source, title


## Intended uses & limitations


#### How to use


To generate a news article text conditioned on a topic, source, title or some subsets, prompt model with:


```python
f"topic {topic} source"
f"topic {topic} source {source} title"
f"topic {topic} source {source} title {title} body"
```


Try the following tags for `topic: climate, weather, vaccination`.


Zero shot generation works pretty well as long as `topic` is a single word and not too specific.


```python
device = "cuda:0"
tokenizer = AutoTokenizer.from_pretrained("ktrapeznikov/gpt2-medium-topic-small-set")
model = AutoModelWithLMHead.from_pretrained("ktrapeznikov/gpt2-medium-topic-small-set")
model.to(device)
topic = "climate"
prompt = tokenizer(f"topic {topics} source straitstimes title", return_tensors="pt")
out = model.generate(prompt["input_ids"].to(device), do_sample=True,max_length=500, early_stopping=True, top_p=.9)
print(tokenizer.decode(out[0].cpu(), skip_special_tokens=True))
```


model name: gpt2-medium-topic-news-v2


model author: ktrapeznikov


task: text-generation


tags: pytorch, jax, gpt2, text-generation, transformers


### Model


**[`monologg/biobert_v1.1_pubmed`](https://huggingface.co/monologg/biobert_v1.1_pubmed)** fine-tuned on **[`SQuAD V2`](https://rajpurkar.github.io/SQuAD-explorer/)** using **[`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)**


This model is cased.


### Training Parameters


Trained on 4 NVIDIA GeForce RTX 2080 Ti 11Gb


```bash
BASE_MODEL=monologg/biobert_v1.1_pubmed
python run_squad.py \
  --version_2_with_negative \
  --model_type albert \
  --model_name_or_path $BASE_MODEL \
  --output_dir $OUTPUT_MODEL \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v2.0.json \
  --predict_file $SQUAD_DIR/dev-v2.0.json \
  --per_gpu_train_batch_size 18 \
  --per_gpu_eval_batch_size 64 \
  --learning_rate 3e-5 \
  --num_train_epochs 3.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --save_steps 2000 \
  --threads 24 \
  --warmup_steps 550 \
  --gradient_accumulation_steps 1 \
  --fp16 \
  --logging_steps 50 \
  --do_train
```


### Evaluation


Evaluation on the dev set. I did not sweep for best threshold.


|                   | val               |
|-------------------|-------------------|
| exact             | 75.97068980038743 |
| f1                | 79.37043950121722 |
| total             | 11873.0           |
| HasAns_exact      | 74.13967611336032 |
| HasAns_f1         | 80.94892513460755 |
| HasAns_total      | 5928.0            |
| NoAns_exact       | 77.79646761984861 |
| NoAns_f1          | 77.79646761984861 |
| NoAns_total       | 5945.0            |
| best_exact        | 75.97068980038743 |
| best_exact_thresh | 0.0               |
| best_f1           | 79.37043950121729 |
| best_f1_thresh    | 0.0               |


### Usage


See [huggingface documentation](https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering). Training on `SQuAD V2` allows the model to score if a paragraph contains an answer:


```python
start_scores, end_scores = model(input_ids)
span_scores = start_scores.softmax(dim=1).log()[:,:,None] + end_scores.softmax(dim=1).log()[:,None,:]
ignore_score = span_scores[:,0,0] #no answer scores

```


model name: biobert_v1.1_pubmed_squad_v2


task: question-answering


tags: pytorch, jax, bert, question-answering, transformers, autonlp_compatible, infinity_compatible


language:


- en
thumbnail:
widget:
- text: "topic climate source"


# GPT2-medium-topic-news


## Model description


GPT2-medium fine tuned on a small news corpus conditioned on a topic, source, title


## Intended uses & limitations


#### How to use


To generate a news article text conditioned on a topic, source, title or some subsets, prompt model with:


```python
f"topic {topic} source"
f"topic {topic} source {source} title"
f"topic {topic} source {source} title {title} body"
```


Try the following tags for `topic: climate, weather, vaccination`.


Zero shot generation works pretty well as long as `topic` is a single word and not too specific.


```python
device = "cuda:0"
tokenizer = AutoTokenizer.from_pretrained("ktrapeznikov/gpt2-medium-topic-small-set")
model = AutoModelWithLMHead.from_pretrained("ktrapeznikov/gpt2-medium-topic-small-set")
model.to(device)
topic = "climate"
prompt = tokenizer(f"topic {topics} source straitstimes title", return_tensors="pt")
out = model.generate(prompt["input_ids"].to(device), do_sample=True,max_length=500, early_stopping=True, top_p=.9)
print(tokenizer.decode(out[0].cpu(), skip_special_tokens=True))
```


## Sample Output


> [topic] military [source] straitstimes [title] Trump signs bill on military aid to Israel [body]  WASHINGTON (AFP) - US President Donald Trump signed into law Thursday (April 24) legislation to provide more than US$15 billion (S$20.43 billion) in military aid to Israel, a move the Obama administration had resisted for political reasons. The White House did not immediately respond to a request for comment on the Israel measure, which Trump had sought unsuccessfully to block during the Obama pres ...


> [topic] military [source] straitstimes [title] Hong Kong's leaders to discuss new travel restrictions as lockdown looms [body]  HONG KONG (REUTERS) - Hong Kong authorities said they would hold a meeting of the Legislative Council on Monday (July 21) to discuss new travel restrictions on Hong Kong residents, as the city reported a record daily increase in coronavirus cases.  The authorities said they would consider the proposal after meeting government chiefs and reviewing other measures. The co ...


> [topic] military [source] straitstimes [title] Trump signs Bill that gives US troops wider latitude to conduct operations abroad [body]  WASHINGTON (AFP) - US President Donald Trump on Thursday (July 23) signed a controversial law that gives US troops more leeway to conduct operations abroad, as he seeks to shore up the embattled government's defences against the coronavirus pandemic and stave off a potentially devastating election defeat. Trump's signature Bill, named after his late father's l ...


> [topic] military [source] straitstimes [title] China's Foreign Ministry responds to Japan's statement on South China Sea: 'No one should assume the role of mediator' [body]  BEIJING (AFP) - The Ministry of Foreign Affairs on Tuesday (Oct 18) told Japan to stop taking sides in the South China Sea issue and not interfere in the bilateral relationship, as Japan said it would do "nothing". Foreign Ministry spokesman Zhao Lijian told reporters in Beijing that the Chinese government's position on the ...


> [topic] military [source] straitstimes [title] US warns North Korea on potential nuclear strike [body]  WASHINGTON - The United States warned North Korea last Friday that an attack by the North could be a "provocation" that would have "a devastating effect" on its security, as it took aim at Pyongyang over its continued efforts to develop weapons of mass destruction. US Secretary of State Mike Pompeo was speaking at the conclusion of a White House news conference when a reporter asked him how t ...


> [topic] military [source] straitstimes [title] China calls Hong Kong to halt 'illegal and illegal military acts' [body]  WASHINGTON • Chinese Foreign Ministry spokeswoman Hua Chunying said yesterday that Hong Kong must stop 'illegal and illegal military acts' before Beijing can recognise the city as its own. In her annual State Councillor's speech, Ms Hua made the case for Hong Kong to resume Hong Kong's status as a semi-autonomous city, and vowed to use its "great power position to actively an ...


## Training data


## Training procedure


model name: gpt2-medium-topic-small-set


model author: ktrapeznikov


task: text-generation


tags: pytorch, jax, gpt2, text-generation, en, transformers


### Model


**[`allenai/scibert_scivocab_uncased`](https://huggingface.co/allenai/scibert_scivocab_uncased)** fine-tuned on **[`SQuAD V2`](https://rajpurkar.github.io/SQuAD-explorer/)** using **[`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)**


### Training Parameters


Trained on 4 NVIDIA GeForce RTX 2080 Ti 11Gb


```bash
BASE_MODEL=allenai/scibert_scivocab_uncased
python run_squad.py \
  --version_2_with_negative \
  --model_type albert \
  --model_name_or_path $BASE_MODEL \
  --output_dir $OUTPUT_MODEL \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v2.0.json \
  --predict_file $SQUAD_DIR/dev-v2.0.json \
  --per_gpu_train_batch_size 18 \
  --per_gpu_eval_batch_size 64 \
  --learning_rate 3e-5 \
  --num_train_epochs 3.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --save_steps 2000 \
  --threads 24 \
  --warmup_steps 550 \
  --gradient_accumulation_steps 1 \
  --fp16 \
  --logging_steps 50 \
  --do_train
```


### Evaluation


Evaluation on the dev set. I did not sweep for best threshold.


|                   | val               |
|-------------------|-------------------|
| exact             | 75.07790785816559 |
| f1                | 78.47735207283013 |
| total             | 11873.0           |
| HasAns_exact      | 70.76585695006747 |
| HasAns_f1         | 77.57449412292718 |
| HasAns_total      | 5928.0            |
| NoAns_exact       | 79.37762825904122 |
| NoAns_f1          | 79.37762825904122 |
| NoAns_total       | 5945.0            |
| best_exact        | 75.08633032931863 |
| best_exact_thresh | 0.0               |
| best_f1           | 78.48577454398324 |
| best_f1_thresh    | 0.0               |


### Usage


See [huggingface documentation](https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering). Training on `SQuAD V2` allows the model to score if a paragraph contains an answer:


```python
start_scores, end_scores = model(input_ids)
span_scores = start_scores.softmax(dim=1).log()[:,:,None] + end_scores.softmax(dim=1).log()[:,None,:]
ignore_score = span_scores[:,0,0] #no answer scores

```


model name: scibert_scivocab_uncased_squad_v2


task: question-answering


tags: pytorch, jax, bert, question-answering, transformers, autonlp_compatible, infinity_compatible


language:


- en
thumbnail:
widget:
- text: "topic: climate article:"


# GPT2-medium-topic-news


## Model description


GPT2-medium fine tuned on a large news corpus conditioned on a topic


## Intended uses & limitations


#### How to use


To generate a news article text conditioned on a topic, prompt model with:
`topic: climate article:`


The following tags were used during training:
`arts law international science business politics disaster world conflict football sport sports artanddesign environment music film lifeandstyle business health commentisfree books technology media education politics travel stage uk society us money culture religion science news tv fashion uk australia cities global childrens sustainable global voluntary housing law local healthcare theguardian`


Zero shot generation works pretty well as long as `topic` is a single word and not too specific.


```python
device = "cuda:0"
tokenizer = AutoTokenizer.from_pretrained("ktrapeznikov/gpt2-medium-topic-news")
model = AutoModelWithLMHead.from_pretrained("ktrapeznikov/gpt2-medium-topic-news")
model.to(device)
topic = "climate"
prompt = tokenizer(f"topic: {topic} article:", return_tensors="pt")
out = model.generate(prompt["input_ids"].to(device), do_sample=True,max_length=500, early_stopping=True, top_p=.9)
print(tokenizer.decode(list(out.cpu()[0])))
```


## Training data


## Training procedure


model name: gpt2-medium-topic-news


model author: ktrapeznikov


task: text-generation


### Model


**[`albert-xlarge-v2`](https://huggingface.co/albert-xlarge-v2)** fine-tuned on **[`SQuAD V2`](https://rajpurkar.github.io/SQuAD-explorer/)** using **[`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)**


### Training Parameters


```bash
BASE_MODEL=albert-xlarge-v2
python run_squad.py \
  --version_2_with_negative \
  --model_type albert \
  --model_name_or_path $BASE_MODEL \
  --output_dir $OUTPUT_MODEL \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v2.0.json \
  --predict_file $SQUAD_DIR/dev-v2.0.json \
  --per_gpu_train_batch_size 3 \
  --per_gpu_eval_batch_size 64 \
  --learning_rate 3e-5 \
  --num_train_epochs 3.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --save_steps 2000 \
  --threads 24 \
  --warmup_steps 814 \
  --gradient_accumulation_steps 4 \
  --fp16 \
  --do_train
```


### Evaluation


|                   | val               |
|-------------------|-------------------|
| exact             | 84.41842836688285 |
| f1                | 87.4628460501696  |
| total             | 11873.0           |
| HasAns_exact      | 80.68488529014844 |
| HasAns_f1         | 86.78245127423482 |
| HasAns_total      | 5928.0            |
| NoAns_exact       | 88.1412952060555  |
| NoAns_f1          | 88.1412952060555  |
| NoAns_total       | 5945.0            |
| best_exact        | 84.41842836688285 |
| best_exact_thresh | 0.0               |
| best_f1           | 87.46284605016956 |
| best_f1_thresh    | 0.0               |


### Usage


See [huggingface documentation](https://huggingface.co/transformers/model_doc/albert.html#albertforquestionanswering). Training on `SQuAD V2` allows the model to score if a paragraph contains an answer:


model name: albert-xlarge-v2-squad-v2


task: question-answering


tags: pytorch, albert, question-answering, transformers, autonlp_compatible, infinity_compatible


tags:


- conversational


#Joey DialoGPT Model


model name: DialoGPT-small-joey


model author: houssaineamzil


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


language: en


- gpt2
- text-generation


widget:


- text: "Rachel: Joey! What were those weird noises coming from your room?"


# GPT2 fine-tuned on FRIENDS transcripts.


model name: gpt2-frens


model author: nkul


task: text-generation


tags: pytorch, jax, gpt2, text-generation, en, transformers


datasets:


- squad_v2


# Roberta-LARGE finetuned on SQuADv2


This is roberta-large model finetuned on SQuADv2 dataset for question answering answerability classification


## Model details


This model is simply an Sequenceclassification model with two inputs (context and question) in a list.
The result is either [1] for answerable or [0] if it is not answerable.
It was trained over 4 epochs on squadv2 dataset and can be used to filter out which context is good to give into the QA model to avoid bad answers.


## Model training


This model was trained with following parameters using simpletransformers wrapper:


```
train_args = {
    'learning_rate': 1e-5,
    'max_seq_length': 512,
    'overwrite_output_dir': True,
    'reprocess_input_data': False,
    'train_batch_size': 4,
    'num_train_epochs': 4,
    'gradient_accumulation_steps': 2,
    'no_cache': True,
    'use_cached_eval_features': False,
    'save_model_every_epoch': False,
    'output_dir': "bart-squadv2",
    'eval_batch_size': 8,
    'fp16_opt_level': 'O2',
    }
```


## Results


`{"accuracy": 90.48%}`


## Model in Action  🚀


```python3
from simpletransformers.classification import ClassificationModel

model = ClassificationModel('roberta', 'a-ware/roberta-large-squadv2', num_labels=2, args=train_args)

predictions, raw_outputs = model.predict([["my dog is an year old. he loves to go into the rain", "how old is my dog ?"]])
print(predictions)
==> [1]
```


> Created with ❤ by A-ware UG [![Github icon](https://cdn0.iconfinder.com/data/icons/octicons/1024/mark-github-32.png)](https://github.com/aware-ai)


model name: roberta-large-squad-classification


model author: a-ware


task: text-classification


tags: pytorch, jax, roberta, text-classification, dataset:squad_v2, transformers, infinity_compatible


datasets:


- squad_v2


# BART-LARGE finetuned on SQuADv2


This is bart-large model finetuned on SQuADv2 dataset for question answering task


## Model details


BART was propsed in the [paper](https://arxiv.org/abs/1910.13461) **BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension**.
BART is a seq2seq model intended for both NLG and NLU tasks.


To use BART for question answering tasks, we feed the complete document into the encoder and decoder, and use the top
hidden state of the decoder as a representation for each
word. This representation is used to classify the token. As given in the paper bart-large achives comparable to ROBERTa on SQuAD.
Another notable thing about BART is that it can handle sequences with upto 1024 tokens.


| Param               | #Value |
|---------------------|--------|
| encoder layers      | 12     |
| decoder layers      | 12     |
| hidden size         | 4096   |
| num attetion heads  | 16     |
| on disk size        | 1.63GB |


## Model training


```
train_args = {
    'learning_rate': 1e-5,
    'max_seq_length': 512,
    'doc_stride': 512,
    'overwrite_output_dir': True,
    'reprocess_input_data': False,
    'train_batch_size': 8,
    'num_train_epochs': 2,
    'gradient_accumulation_steps': 2,
    'no_cache': True,
    'use_cached_eval_features': False,
    'save_model_every_epoch': False,
    'output_dir': "bart-squadv2",
    'eval_batch_size': 32,
    'fp16_opt_level': 'O2',
    }
```


[You can even train your own model using this colab notebook](https://colab.research.google.com/drive/1I5cK1M_0dLaf5xoewh6swcm5nAInfwHy?usp=sharing)


## Results


`{"correct": 6832, "similar": 4409, "incorrect": 632, "eval_loss": -14.950117511952177}`


## Model in Action  🚀


```python3
from transformers import BartTokenizer, BartForQuestionAnswering
import torch

tokenizer = BartTokenizer.from_pretrained('a-ware/bart-squadv2')
model = BartForQuestionAnswering.from_pretrained('a-ware/bart-squadv2')

question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
encoding = tokenizer(question, text, return_tensors='pt')
input_ids = encoding['input_ids']
attention_mask = encoding['attention_mask']

start_scores, end_scores = model(input_ids, attention_mask=attention_mask, output_attentions=False)[:2]

all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0])
answer = ' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1])
answer = tokenizer.convert_tokens_to_ids(answer.split())
answer = tokenizer.decode(answer)
#answer => 'a nice puppet'
```


model name: bart-squadv2


model author: a-ware


task: question-answering


tags: pytorch, bart, question-answering, dataset:squad_v2, arxiv:1910.13461, transformers, autonlp_compatible


datasets:


- squad_v2


# XLM-ROBERTA-LARGE finetuned on SQuADv2


This is xlm-roberta-large model finetuned on SQuADv2 dataset for question answering task


## Model details


XLM-Roberta was propsed in the [paper](https://arxiv.org/pdf/1911.02116.pdf) **XLM-R: State-of-the-art cross-lingual understanding through self-supervision


## Model training


This model was trained with following parameters using simpletransformers wrapper:


`{"correct": 6961, "similar": 4359, "incorrect": 553, "eval_loss": -12.177856394381962}`


```python3
from transformers import XLMRobertaTokenizer, XLMRobertaForQuestionAnswering
import torch

tokenizer = XLMRobertaTokenizer.from_pretrained('a-ware/xlmroberta-squadv2')
model = XLMRobertaForQuestionAnswering.from_pretrained('a-ware/xlmroberta-squadv2')

question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
encoding = tokenizer(question, text, return_tensors='pt')
input_ids = encoding['input_ids']
attention_mask = encoding['attention_mask']

start_scores, end_scores = model(input_ids, attention_mask=attention_mask, output_attentions=False)[:2]

all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0])
answer = ' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1])
answer = tokenizer.convert_tokens_to_ids(answer.split())
answer = tokenizer.decode(answer)
#answer => 'a nice puppet'
```


> Created with ❤ by A-ware UG [![Github icon](https://cdn0.iconfinder.com/data/icons/octicons/1024/mark-github-32.png)](https://github.com/aware-ai)


model name: xlmroberta-squadv2


tags: pytorch, xlm-roberta, question-answering, dataset:squad_v2, arxiv:1911.02116, transformers, autonlp_compatible


# T5 for Semantic Parsing


## Model description


T5 (small and large) finetuned on CoNaLa for semantic parsing (Natural Language descriptions to Python code)


Paper: https://arxiv.org/pdf/2101.07138.pdf


Code, data and how to use: https://github.com/ypapanik/t5-for-code-generation


### Cite


```
@misc{papanikolaou2021teach,
      title={Teach me how to Label: Labeling Functions from Natural Language with Text-to-text Transformers},
      author={Yannis Papanikolaou},
      year={2021},
      eprint={2101.07138},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


model name: t5-code-generation


model author: yannis-papanikolaou


tags: arxiv:2101.07138