```python
from transformers import pipeline
from nn_pruning.inference_model_patcher import optimize_model

qa_pipeline = pipeline(
    "question-answering",
    model="madlag/bert-base-uncased-squadv1-x1.16-f88.1-d8-unstruct-v1",
    tokenizer="madlag/bert-base-uncased-squadv1-x1.16-f88.1-d8-unstruct-v1"
)

print("bert-base-uncased parameters: 152.0M")
print(f"Parameters count (includes only head pruning, not feed forward pruning)={int(qa_pipeline.model.num_parameters() / 1E6)}M")
qa_pipeline.model = optimize_model(qa_pipeline.model, "dense")

print(f"Parameters count after complete optimization={int(qa_pipeline.model.num_parameters() / 1E6)}M")
predictions = qa_pipeline({
    'context': "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano.",
    'question': "Who is Frederic Chopin?",
})
print("Predictions", predictions)
```


model name: bert-base-uncased-squadv1-x1.16-f88.1-d8-unstruct-v1


model author: madlag


task: question-answering


tags: pytorch, tf, bert, question-answering, en, dataset:squad, transformers, license:mit, autonlp_compatible, infinity_compatible


language: en
thumbnail:
license: mit
tags:


- question-answering
- bert
- bert-base
datasets:
- squad
metrics:
- squad
widget:
- text: "Where is the Eiffel Tower located?"
context: "The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower."
- text: "Who is Frederic Chopin?"
context: "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano."


## BERT-base uncased model fine-tuned on SQuAD v1


This model is block sparse: the **linear** layers contains **12.5%** of the original weights.


The model contains **32.1%** of the original weights **overall**.


The training use a modified version of Victor Sanh [Movement Pruning](https://arxiv.org/abs/2005.07683) method.


That means that with the [block-sparse](https://github.com/huggingface/pytorch_block_sparse) runtime it ran **1.65x** faster than an dense networks on the evaluation, at the price of some impact on the accuracy (see below).


This model was fine-tuned from the HuggingFace [BERT](https://www.aclweb.org/anthology/N19-1423/) base uncased checkpoint on [SQuAD1.1](https://rajpurkar.github.io/SQuAD-explorer), and distilled from the equivalent model [csarron/bert-base-uncased-squad-v1](https://huggingface.co/csarron/bert-base-uncased-squad-v1).
This model is case-insensitive: it does not make a difference between english and English.


## Pruning details


A side-effect of the block pruning is that some of the attention heads are completely removed: 97 heads were removed on a total of 144 (67.4%).


Here is a detailed view on how the remaining heads are distributed in the network after pruning.


![Pruning details](https://huggingface.co/madlag/bert-base-uncased-squad1.1-block-sparse-0.13-v1/raw/main/model_card/pruning.svg)


## Density plot


<script src="/madlag/bert-base-uncased-squad1.1-block-sparse-0.13-v1/raw/main/model_card/density.js" id="34ede51e-2375-4d96-99dd-383de82a2d16"></script>


## Details


| Dataset  | Split | # samples |
| -------- | ----- | --------- |
| SQuAD1.1 | train | 90.6K      |
| SQuAD1.1 | eval  | 11.1k     |


### Fine-tuning


- Python: `3.8.5`
- Machine specs:


```CPU: Intel(R) Core(TM) i7-6700K CPU
Memory: 64 GiB
GPUs: 1 GeForce GTX 3090, with 24GiB memory
GPU driver: 455.23.05, CUDA: 11.1
```


### Results


**Pytorch model file size**: `342M` (original BERT: `438M`)


| Metric | # Value   | # Original ([Table 2](https://www.aclweb.org/anthology/N19-1423.pdf))|
| ------ | --------- | --------- |
| **EM** | **74.39** | **80.8** |
| **F1** | **83.26** | **88.5** |


## Example Usage


```python
from transformers import pipeline

qa_pipeline = pipeline(
    "question-answering",
    model="madlag/bert-base-uncased-squad1.1-block-sparse-0.13-v1",
    tokenizer="madlag/bert-base-uncased-squad1.1-block-sparse-0.13-v1"
)

predictions = qa_pipeline({
    'context': "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano.",
    'question': "Who is Frederic Chopin?",
})

print(predictions)
```


model name: bert-base-uncased-squad1.1-block-sparse-0.13-v1


model author: madlag


task: question-answering


tags: pytorch, tf, bert, question-answering, en, dataset:squad, arxiv:2005.07683, transformers, license:mit, bert-base, autonlp_compatible, infinity_compatible


language: en
thumbnail:
license: mit
tags:


- question-answering


datasets:


- squad_v2
metrics:
- squad_v2
widget:
- text: "Where is the Eiffel Tower located?"
context: "The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower."
- text: "Who is Frederic Chopin?"
context: "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano."


## bert-large-uncased-whole-word-masking model fine-tuned on SQuAD v2


This model was created using the [nn_pruning](https://github.com/huggingface/nn_pruning) python library: the **linear layers contains 16.0%** of the original  weights.


The model contains **24.0%** of the original weights **overall** (the embeddings account for a significant part of the model, and they are not pruned by this method).


With a simple resizing of the linear matrices it ran **2.63x as fast as bert-large-uncased-whole-word-masking** on the evaluation.
This is possible because the pruning method lead to structured matrices: to visualize them, hover below on the plot to see the non-zero/zero parts of each matrix.


In terms of accuracy, its **F1 is 82.57**, compared with 85.85 for bert-large-uncased-whole-word-masking, a **F1 drop of 3.28**.


## Fine-Pruning details


This model was fine-tuned from the HuggingFace [model](https://huggingface.co/bert-large-uncased-whole-word-masking) checkpoint on [SQuAD2.0](https://rajpurkar.github.io/SQuAD-explorer), and distilled from the model [madlag/bert-large-uncased-whole-word-masking-finetuned-squadv2](https://huggingface.co/madlag/bert-large-uncased-whole-word-masking-finetuned-squadv2).
This model is case-insensitive: it does not make a difference between english and English.


A side-effect of the block pruning is that some of the attention heads are completely removed: 190 heads were removed on a total of 384 (49.5%).
Here is a detailed view on how the remaining heads are distributed in the network after pruning.


## Details of the SQuAD1.1 dataset


| Dataset  | Split | # samples |
| -------- | ----- | --------- |
| SQuAD 2.0 | train | 130.0K      |
| SQuAD 2.0 | eval  | 11.9k     |


### Fine-tuning


- Python: `3.8.5`
- Machine specs:


### Results


**Pytorch model file size**: `1084MB` (original BERT: `1228.0MB`)


| Metric | # Value   | # Original ([Table 2](https://www.aclweb.org/anthology/N19-1423.pdf))| Variation |
| ------ | --------- | --------- | --------- |
| **EM** | **79.70** | **82.83** | **-4.13**|
| **F1** | **82.57** | **85.85** | **-3.28**|


```
{
    "HasAns_exact": 74.8144399460189,
    "HasAns_f1": 80.555306012496,
    "HasAns_total": 5928,
    "NoAns_exact": 84.57527333894029,
    "NoAns_f1": 84.57527333894029,
    "NoAns_total": 5945,
    "best_exact": 79.70184452118251,
    "best_exact_thresh": 0.0,
    "best_f1": 82.56816761071966,
    "best_f1_thresh": 0.0,
    "exact": 79.70184452118251,
    "f1": 82.56816761071981,
    "total": 11873
}
```


## Example Usage


Install nn_pruning: it contains the optimization script, which just pack the linear layers into smaller ones by removing empty rows/columns.


`pip install nn_pruning`


Then you can use the `transformers library` almost as usual: you just have to call `optimize_model` when the pipeline has loaded.


```python
from transformers import pipeline
from nn_pruning.inference_model_patcher import optimize_model

qa_pipeline = pipeline(
    "question-answering",
    model="madlag/bert-large-uncased-wwm-squadv2-x2.63-f82.6-d16-hybrid-v1",
    tokenizer="madlag/bert-large-uncased-wwm-squadv2-x2.63-f82.6-d16-hybrid-v1"
)

print("bert-large-uncased-whole-word-masking parameters: 445.0M")
print(f"Parameters count (includes only head pruning, not feed forward pruning)={int(qa_pipeline.model.num_parameters() / 1E6)}M")
qa_pipeline.model = optimize_model(qa_pipeline.model, "dense")

print(f"Parameters count after complete optimization={int(qa_pipeline.model.num_parameters() / 1E6)}M")
predictions = qa_pipeline({
    'context': "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano.",
    'question': "Who is Frederic Chopin?",
})
print("Predictions", predictions)
```


model name: bert-large-uncased-wwm-squadv2-x2.63-f82.6-d16-hybrid-v1


model author: madlag


task: question-answering


tags: pytorch, tf, bert, question-answering, en, dataset:squad_v2, transformers, license:mit, autonlp_compatible, infinity_compatible


language: en
thumbnail:
license: mit
tags:


- question-answering
- bert
- bert-base
datasets:
- squad
metrics:
- squad
widget:
- text: "Where is the Eiffel Tower located?"
context: "The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower."
- text: "Who is Frederic Chopin?"
context: "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano."


## BERT-base uncased model fine-tuned on SQuAD v1


This model is block sparse: the **linear** layers contains **7.5%** of the original weights.


The model contains **28.2%** of the original weights **overall**.


The training use a modified version of Victor Sanh [Movement Pruning](https://arxiv.org/abs/2005.07683) method.


That means that with the [block-sparse](https://github.com/huggingface/pytorch_block_sparse) runtime it ran **1.92x** faster than an dense networks on the evaluation, at the price of some impact on the accuracy (see below).


This model was fine-tuned from the HuggingFace [BERT](https://www.aclweb.org/anthology/N19-1423/) base uncased checkpoint on [SQuAD1.1](https://rajpurkar.github.io/SQuAD-explorer), and distilled from the equivalent model [csarron/bert-base-uncased-squad-v1](https://huggingface.co/csarron/bert-base-uncased-squad-v1).
This model is case-insensitive: it does not make a difference between english and English.


## Pruning details


A side-effect of the block pruning is that some of the attention heads are completely removed: 106 heads were removed on a total of 144 (73.6%).


Here is a detailed view on how the remaining heads are distributed in the network after pruning.


![Pruning details](https://huggingface.co/madlag/bert-base-uncased-squad1.1-block-sparse-0.07-v1/raw/main/model_card/pruning.svg)


## Density plot


<script src="/madlag/bert-base-uncased-squad1.1-block-sparse-0.07-v1/raw/main/model_card/density.js" id="9301e950-59b1-497b-a2c5-25c24e07b3a0"></script>


## Details


| Dataset  | Split | # samples |
| -------- | ----- | --------- |
| SQuAD1.1 | train | 90.6K      |
| SQuAD1.1 | eval  | 11.1k     |


### Fine-tuning


- Python: `3.8.5`
- Machine specs:


```CPU: Intel(R) Core(TM) i7-6700K CPU
Memory: 64 GiB
GPUs: 1 GeForce GTX 3090, with 24GiB memory
GPU driver: 455.23.05, CUDA: 11.1
```


### Results


**Pytorch model file size**: `335M` (original BERT: `438M`)


| Metric | # Value   | # Original ([Table 2](https://www.aclweb.org/anthology/N19-1423.pdf))|
| ------ | --------- | --------- |
| **EM** | **71.88** | **80.8** |
| **F1** | **81.36** | **88.5** |


## Example Usage


```python
from transformers import pipeline

qa_pipeline = pipeline(
    "question-answering",
    model="madlag/bert-base-uncased-squad1.1-block-sparse-0.07-v1",
    tokenizer="madlag/bert-base-uncased-squad1.1-block-sparse-0.07-v1"
)

predictions = qa_pipeline({
    'context': "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano.",
    'question': "Who is Frederic Chopin?",
})

print(predictions)
```


model name: bert-base-uncased-squad1.1-block-sparse-0.07-v1


model author: madlag


task: question-answering


tags: pytorch, tf, bert, question-answering, en, dataset:squad, arxiv:2005.07683, transformers, license:mit, bert-base, autonlp_compatible, infinity_compatible


Used [run.sh](https://huggingface.co/madlag/bert-large-uncased-whole-word-masking-finetuned-squadv2/blob/main/run.sh) used to train using transformers/example/question_answering code.


Evaluation results : F1= 85.85 , a much better result than the original 81.9 from the BERT paper, due to the use of the "whole-word-masking" variation.


```
{
    "HasAns_exact": 80.58367071524967,
    "HasAns_f1": 86.64594807945029,
    "HasAns_total": 5928,
    "NoAns_exact": 85.06307821698907,
    "NoAns_f1": 85.06307821698907,
    "NoAns_total": 5945,
    "best_exact": 82.82658131895899,
    "best_exact_thresh": 0.0,
    "best_f1": 85.85337995578023,
    "best_f1_thresh": 0.0,
    "epoch": 2.0,
    "eval_samples": 12134,
    "exact": 82.82658131895899,
    "f1": 85.85337995578037,
    "total": 11873
}
```


model name: bert-large-uncased-whole-word-masking-finetuned-squadv2


tags: pytorch, jax, bert, question-answering, transformers, autonlp_compatible, infinity_compatible


language: en
thumbnail:
license: mit
tags:


- question-answering


datasets:


- squad
metrics:
- squad
widget:
- text: "Where is the Eiffel Tower located?"
context: "The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower."
- text: "Who is Frederic Chopin?"
context: "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano."


## BERT-base uncased model fine-tuned on SQuAD v1


This model was created using the [nn_pruning](https://github.com/huggingface/nn_pruning) python library: the **linear layers contains 27.0%** of the original  weights.


This model **CANNOT be used without using nn_pruning `optimize_model`** function, as it uses NoNorms instead of LayerNorms and this is not currently supported by the Transformers library.


It uses ReLUs instead of GeLUs as in the initial BERT network, to speedup inference.
This does not need special handling, as it is supported by the Transformers library, and flagged in the model config by the `"hidden_act": "relu"` entry.


The model contains **43.0%** of the original weights **overall** (the embeddings account for a significant part of the model, and they are not pruned by this method).


With a simple resizing of the linear matrices it ran **1.96x as fast as bert-base-uncased** on the evaluation.
This is possible because the pruning method lead to structured matrices: to visualize them, hover below on the plot to see the non-zero/zero parts of each matrix.


In terms of accuracy, its **F1 is 88.33**, compared with 88.5 for bert-base-uncased, a **F1 drop of 0.17**.


## Fine-Pruning details


This model was fine-tuned from the HuggingFace [model](https://huggingface.co/bert-base-uncased) checkpoint on [SQuAD1.1](https://rajpurkar.github.io/SQuAD-explorer), and distilled from the model [bert-large-uncased-whole-word-masking-finetuned-squad](https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad)
This model is case-insensitive: it does not make a difference between english and English.


A side-effect of the block pruning is that some of the attention heads are completely removed: 55 heads were removed on a total of 144 (38.2%).
Here is a detailed view on how the remaining heads are distributed in the network after pruning.


## Details of the SQuAD1.1 dataset


| Dataset  | Split | # samples |
| -------- | ----- | --------- |
| SQuAD1.1 | train | 90.6K      |
| SQuAD1.1 | eval  | 11.1k     |


### Fine-tuning


- Python: `3.8.5`
- Machine specs:


```CPU: Intel(R) Core(TM) i7-6700K CPU
Memory: 64 GiB
GPUs: 1 GeForce GTX 3090, with 24GiB memory
GPU driver: 455.23.05, CUDA: 11.1
```


### Results


**Pytorch model file size**: `374MB` (original BERT: `420MB`)


| Metric | # Value   | # Original ([Table 2](https://www.aclweb.org/anthology/N19-1423.pdf))| Variation |
| ------ | --------- | --------- | --------- |
| **EM** | **81.31** | **80.8** | **+0.51**|
| **F1** | **88.33** | **88.5** | **-0.17**|


## Example Usage


Install nn_pruning: it contains the optimization script, which just pack the linear layers into smaller ones by removing empty rows/columns.


`pip install nn_pruning`


Then you can use the `transformers library` almost as usual: you just have to call `optimize_model` when the pipeline has loaded.


```python
from transformers import pipeline
from nn_pruning.inference_model_patcher import optimize_model

qa_pipeline = pipeline(
    "question-answering",
    model="madlag/bert-base-uncased-squadv1-x1.96-f88.3-d27-hybrid-filled-opt-v1",
    tokenizer="madlag/bert-base-uncased-squadv1-x1.96-f88.3-d27-hybrid-filled-opt-v1"
)

print("bert-base-uncased parameters: 191.0M")
print(f"Parameters count (includes only head pruning, not feed forward pruning)={int(qa_pipeline.model.num_parameters() / 1E6)}M")
qa_pipeline.model = optimize_model(qa_pipeline.model, "dense")

print(f"Parameters count after complete optimization={int(qa_pipeline.model.num_parameters() / 1E6)}M")
predictions = qa_pipeline({
    'context': "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano.",
    'question': "Who is Frederic Chopin?",
})
print("Predictions", predictions)
```


model name: bert-base-uncased-squadv1-x1.96-f88.3-d27-hybrid-filled-opt-v1


model author: madlag


task: question-answering


tags: pytorch, tf, bert, question-answering, en, dataset:squad, transformers, license:mit, autonlp_compatible, infinity_compatible


Albert v2 finetuned on SQuAD v1.


Trained using the [nn_pruning](https://github.com/huggingface/nn_pruning/tree/main/examples/question_answering) script, with pruning disabled.


[Original results](https://github.com/google-research/albert) are F1=90.2, EM=83.2, we improved them to:


```{
    "exact_match": 83.74645222327341,
    "f1": 90.78776054621733
}```
```


model name: albert-base-v2-squad


tags: pytorch, albert, question-answering, transformers, autonlp_compatible, infinity_compatible


## BERT-large finetuned on squad v2.


F1 on dev (from paper)[https://arxiv.org/pdf/1810.04805v2.pdf] is 81.9, we reach 81.58.


```
{'exact': 78.6321906847469,
'f1': 81.5816656803201,
'total': 11873,
'HasAns_exact': 73.73481781376518,
 'HasAns_f1': 79.64222615088413,
 'HasAns_total': 5928,
 'NoAns_exact': 83.51555929352396,
 'NoAns_f1': 83.51555929352396,
 'NoAns_total': 5945,
 'best_exact': 78.6321906847469,
 'best_exact_thresh': 0.0,
 'best_f1': 81.58166568032006,
 'best_f1_thresh': 0.0,
 'epoch': 1.59}
```


```
python run_qa.py \
  --model_name_or_path bert-large-uncased \
  --dataset_name squad_v2 \
  --do_train \
  --do_eval \
  --save_steps 2500 \
  --eval_steps 2500 \
  --evaluation_strategy steps \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir bert-large-uncased-squadv2 \
  --version_2_with_negative 1
```


model name: bert-large-uncased-squadv2


tags: pytorch, jax, bert, question-answering, arxiv:1810.04805, transformers, autonlp_compatible, infinity_compatible


## BERT-large finetuned on MNLI.


The [reference finetuned model](https://github.com/google-research/bert) has an accuracy of 86.05, we get 86.7:


```
{'eval_loss': 0.3984006643295288, 'eval_accuracy': 0.8667345899133979}
```


model name: bert-large-uncased-mnli


model author: madlag


task: text-classification


tags: pytorch, jax, bert, text-classification, transformers, infinity_compatible


language: en
thumbnail:
license: mit
tags:


- question-answering
- bert
- bert-base
datasets:
- squad
metrics:
- squad
widget:
- text: "Where is the Eiffel Tower located?"
context: "The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower."
- text: "Who is Frederic Chopin?"
context: "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano."


## BERT-base uncased model fine-tuned on SQuAD v1


This model is block sparse: the **linear** layers contains **31.7%** of the original weights.


The model contains **47.0%** of the original weights **overall**.


The training use a modified version of Victor Sanh [Movement Pruning](https://arxiv.org/abs/2005.07683) method.


That means that with the [block-sparse](https://github.com/huggingface/pytorch_block_sparse) runtime it ran **1.12x** faster than an dense networks on the evaluation, at the price of some impact on the accuracy (see below).


This model was fine-tuned from the HuggingFace [BERT](https://www.aclweb.org/anthology/N19-1423/) base uncased checkpoint on [SQuAD1.1](https://rajpurkar.github.io/SQuAD-explorer), and distilled from the equivalent model [csarron/bert-base-uncased-squad-v1](https://huggingface.co/csarron/bert-base-uncased-squad-v1).
This model is case-insensitive: it does not make a difference between english and English.


## Pruning details


A side-effect of the block pruning is that some of the attention heads are completely removed: 80 heads were removed on a total of 144 (55.6%).


Here is a detailed view on how the remaining heads are distributed in the network after pruning.


![Pruning details](https://huggingface.co/madlag/bert-base-uncased-squad1.1-block-sparse-0.32-v1/raw/main/model_card/pruning.svg)


## Density plot


<script src="/madlag/bert-base-uncased-squad1.1-block-sparse-0.32-v1/raw/main/model_card/density.js" id="79005f4a-723c-4bf8-bc7f-5ad11676be6c"></script>


## Details


| Dataset  | Split | # samples |
| -------- | ----- | --------- |
| SQuAD1.1 | train | 90.6K      |
| SQuAD1.1 | eval  | 11.1k     |


### Fine-tuning


- Python: `3.8.5`
- Machine specs:


```CPU: Intel(R) Core(TM) i7-6700K CPU
Memory: 64 GiB
GPUs: 1 GeForce GTX 3090, with 24GiB memory
GPU driver: 455.23.05, CUDA: 11.1
```


### Results


**Pytorch model file size**: `355M` (original BERT: `438M`)


| Metric | # Value   | # Original ([Table 2](https://www.aclweb.org/anthology/N19-1423.pdf))|
| ------ | --------- | --------- |
| **EM** | **79.04** | **80.8** |
| **F1** | **86.70** | **88.5** |


## Example Usage


```python
from transformers import pipeline

qa_pipeline = pipeline(
    "question-answering",
    model="madlag/bert-base-uncased-squad1.1-block-sparse-0.32-v1",
    tokenizer="madlag/bert-base-uncased-squad1.1-block-sparse-0.32-v1"
)

predictions = qa_pipeline({
    'context': "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano.",
    'question': "Who is Frederic Chopin?",
})

print(predictions)
```


model name: bert-base-uncased-squad1.1-block-sparse-0.32-v1


model author: madlag


task: question-answering


tags: pytorch, tf, bert, question-answering, en, dataset:squad, arxiv:2005.07683, transformers, license:mit, bert-base, autonlp_compatible, infinity_compatible


language: en
thumbnail:
license: mit
tags:


- question-answering


datasets:


- squad
metrics:
- squad
widget:
- text: "Where is the Eiffel Tower located?"
context: "The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower."
- text: "Who is Frederic Chopin?"
context: "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano."


## BERT-base uncased model fine-tuned on SQuAD v1


This model was created using the [nn_pruning](https://github.com/huggingface/nn_pruning) python library: the **linear layers contains 36.0%** of the original  weights.


The model contains **50.0%** of the original weights **overall** (the embeddings account for a significant part of the model, and they are not pruned by this method).


With a simple resizing of the linear matrices it ran **1.84x as fast as the dense model** on the evaluation.
This is possible because the pruning method lead to structured matrices: to visualize them, hover below on the plot to see the non-zero/zero parts of each matrix.


In terms of accuracy, its **F1 is 88.72**, compared with 88.5 for the dense version, a **F1 gain of 0.22**.


## Fine-Pruning details


This model was fine-tuned from the HuggingFace [model](https://huggingface.co//home/lagunas/devel/hf/nn_pruning/nn_pruning/analysis/tmp_finetune) checkpoint on [SQuAD1.1](https://rajpurkar.github.io/SQuAD-explorer), and distilled from the model [csarron/bert-base-uncased-squad-v1](https://huggingface.co/csarron/bert-base-uncased-squad-v1)
This model is case-insensitive: it does not make a difference between english and English.


A side-effect of the block pruning is that some of the attention heads are completely removed: 48 heads were removed on a total of 144 (33.3%).
Here is a detailed view on how the remaining heads are distributed in the network after pruning.


## Details of the SQuAD1.1 dataset


### Fine-tuning


- Python: `3.8.5`
- Machine specs:


### Results


**Pytorch model file size**: `379MB` (original BERT: `420MB`)


| Metric | # Value   | # Original ([Table 2](https://www.aclweb.org/anthology/N19-1423.pdf))| Variation |
| ------ | --------- | --------- | --------- |
| **EM** | **81.69** | **80.8** | **+0.89**|
| **F1** | **88.72** | **88.5** | **+0.22**|


## Example Usage


Install nn_pruning: it contains the optimization script, which just pack the linear layers into smaller ones by removing empty rows/columns.


`pip install nn_pruning`


Then you can use the `transformers library` almost as usual: you just have to call `optimize_model` when the pipeline has loaded.


```python
from transformers import pipeline
from nn_pruning.inference_model_patcher import optimize_model

qa_pipeline = pipeline(
    "question-answering",
    model="madlag/bert-base-uncased-squadv1-x1.84-f88.7-d36-hybrid-filled-v1",
    tokenizer="madlag/bert-base-uncased-squadv1-x1.84-f88.7-d36-hybrid-filled-v1"
)

print("/home/lagunas/devel/hf/nn_pruning/nn_pruning/analysis/tmp_finetune parameters: 218.0M")
print(f"Parameters count (includes only head pruning, not feed forward pruning)={int(qa_pipeline.model.num_parameters() / 1E6)}M")
qa_pipeline.model = optimize_model(qa_pipeline.model, "dense")

print(f"Parameters count after complete optimization={int(qa_pipeline.model.num_parameters() / 1E6)}M")
predictions = qa_pipeline({
    'context': "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano.",
    'question': "Who is Frederic Chopin?",
})
print("Predictions", predictions)
```


model name: bert-base-uncased-squadv1-x1.84-f88.7-d36-hybrid-filled-v1


model author: madlag


task: question-answering


tags: pytorch, tf, bert, question-answering, en, dataset:squad, transformers, license:mit, autonlp_compatible, infinity_compatible


- question-answering


datasets:


This model was created using the [nn_pruning](https://github.com/huggingface/nn_pruning) python library: the **linear layers contains 26.0%** of the original  weights.


The model contains **42.0%** of the original weights **overall** (the embeddings account for a significant part of the model, and they are not pruned by this method).


With a simple resizing of the linear matrices it ran **2.44x as fast as the original model** on the evaluation.
This is possible because the pruning method lead to structured matrices: to visualize them, hover below on the plot to see the non-zero/zero parts of each matrix.


In terms of accuracy, its **F1 is 87.71**, compared with 88.5 for the original model, a **F1 drop of 0.79**.


## Fine-Pruning details


A side-effect of the block pruning is that some of the attention heads are completely removed: 80 heads were removed on a total of 144 (55.6%).
Here is a detailed view on how the remaining heads are distributed in the network after pruning.


## Details of the SQuAD1.1 dataset


| Dataset  | Split | # samples |
| -------- | ----- | --------- |
| SQuAD1.1 | train | 90.6K      |
| SQuAD1.1 | eval  | 11.1k     |


### Fine-tuning


- Python: `3.8.5`
- Machine specs:


```CPU: Intel(R) Core(TM) i7-6700K CPU
Memory: 64 GiB
GPUs: 1 GeForce GTX 3090, with 24GiB memory
GPU driver: 455.23.05, CUDA: 11.1
```


### Results


**Pytorch model file size**: `355MB` (original BERT: `420MB`)


| Metric | # Value   | # Original ([Table 2](https://www.aclweb.org/anthology/N19-1423.pdf))| Variation |
| ------ | --------- | --------- | --------- |
| **EM** | **80.03** | **80.8** | **-0.77**|
| **F1** | **87.71** | **88.5** | **-0.79**|


## Example Usage


`pip install nn_pruning`


```python
from transformers import pipeline
from nn_pruning.inference_model_patcher import optimize_model

qa_pipeline = pipeline(
    "question-answering",
    model="madlag/bert-base-uncased-squadv1-x2.44-f87.7-d26-hybrid-filled-v1",
    tokenizer="madlag/bert-base-uncased-squadv1-x2.44-f87.7-d26-hybrid-filled-v1"
)

print("/home/lagunas/devel/hf/nn_pruning/nn_pruning/analysis/tmp_finetune parameters: 189.0M")
print(f"Parameters count (includes only head pruning, not feed forward pruning)={int(qa_pipeline.model.num_parameters() / 1E6)}M")
qa_pipeline.model = optimize_model(qa_pipeline.model, "dense")

print(f"Parameters count after complete optimization={int(qa_pipeline.model.num_parameters() / 1E6)}M")
predictions = qa_pipeline({
    'context': "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano.",
    'question': "Who is Frederic Chopin?",
})
print("Predictions", predictions)
```


model name: bert-base-uncased-squadv1-x2.44-f87.7-d26-hybrid-filled-v1


model author: madlag


task: question-answering


language: en
thumbnail:
license: mit
tags:


- question-answering
- bert
- bert-base
datasets:
- squad
metrics:
- squad
widget:
- text: "Where is located the Eiffel Tower ?"
context: "The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower."
- text: "Who is Frederic Chopin?"
context: "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano."


## BERT-base uncased model fine-tuned on SQuAD v1


This model is [block-sparse](https://github.com/huggingface/pytorch_block_sparse).


That means that with the right runtime it can run roughly 3x faster than an dense network, with 25% of the original weights.


This of course has some impact on the accuracy (see below).


It uses a modified version of Victor Sanh [Movement Pruning](https://arxiv.org/abs/2005.07683) method.


This model was fine-tuned from the HuggingFace [BERT](https://www.aclweb.org/anthology/N19-1423/) base uncased checkpoint on [SQuAD1.1](https://rajpurkar.github.io/SQuAD-explorer).
This model is case-insensitive: it does not make a difference between english and English.


## Details


### Fine-tuning


- Python: `3.8.5`
- Machine specs:

  `CPU: Intel(R) Core(TM) i7-6700K CPU`

  `Memory: 64 GiB`

  `GPUs: 1 GeForce GTX 3090, with 24GiB memory`

  `GPU driver: 455.23.05, CUDA: 11.1`


### Results


**Model size**: `418M`


| Metric | # Value   | # Original ([Table 2](https://www.aclweb.org/anthology/N19-1423.pdf))|
| ------ | --------- | --------- |
| **EM** | **74.82** | **80.8** |
| **F1** | **83.7** | **88.5** |


Note that the above results didn't involve any hyperparameter search.


## Example Usage


```python
from transformers import pipeline

qa_pipeline = pipeline(
    "question-answering",
    model="madlag/bert-base-uncased-squad-v1-sparse0.25",
    tokenizer="madlag/bert-base-uncased-squad-v1-sparse0.25"
)

predictions = qa_pipeline({
    'context': "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano.",
    'question': "Who is Frederic Chopin?",
})

print(predictions)
```


model name: bert-base-uncased-squad-v1-sparse0.25


model author: madlag


task: question-answering


tags: pytorch, tf, jax, bert, question-answering, en, dataset:squad, arxiv:2005.07683, transformers, license:mit, bert-base, autonlp_compatible, infinity_compatible


- question-answering
- bert
- bert-base
datasets:
- squad
metrics:
- squad
widget:
- text: "Where is the Eiffel Tower located?"
context: "The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower."
- text: "Who is Frederic Chopin?"
context: "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano."


This model is block sparse: the **linear** layers contains **20.2%** of the original weights.


The model contains **38.1%** of the original weights **overall**.


The training use a modified version of Victor Sanh [Movement Pruning](https://arxiv.org/abs/2005.07683) method.


That means that with the [block-sparse](https://github.com/huggingface/pytorch_block_sparse) runtime it ran **1.39x** faster than an dense networks on the evaluation, at the price of some impact on the accuracy (see below).


This model was fine-tuned from the HuggingFace [BERT](https://www.aclweb.org/anthology/N19-1423/) base uncased checkpoint on [SQuAD1.1](https://rajpurkar.github.io/SQuAD-explorer), and distilled from the equivalent model [csarron/bert-base-uncased-squad-v1](https://huggingface.co/csarron/bert-base-uncased-squad-v1).
This model is case-insensitive: it does not make a difference between english and English.


## Pruning details


A side-effect of the block pruning is that some of the attention heads are completely removed: 90 heads were removed on a total of 144 (62.5%).


Here is a detailed view on how the remaining heads are distributed in the network after pruning.


![Pruning details](https://huggingface.co/madlag/bert-base-uncased-squad1.1-block-sparse-0.20-v1/raw/main/model_card/pruning.svg)


## Density plot


<script src="/madlag/bert-base-uncased-squad1.1-block-sparse-0.20-v1/raw/main/model_card/density.js" id="ddbad516-679a-400d-9e28-0182fd89b188"></script>


## Details


| Dataset  | Split | # samples |
| -------- | ----- | --------- |
| SQuAD1.1 | train | 90.6K      |
| SQuAD1.1 | eval  | 11.1k     |


### Fine-tuning


- Python: `3.8.5`
- Machine specs:


```CPU: Intel(R) Core(TM) i7-6700K CPU
Memory: 64 GiB
GPUs: 1 GeForce GTX 3090, with 24GiB memory
GPU driver: 455.23.05, CUDA: 11.1
```


### Results


**Pytorch model file size**: `347M` (original BERT: `438M`)


| Metric | # Value   | # Original ([Table 2](https://www.aclweb.org/anthology/N19-1423.pdf))|
| ------ | --------- | --------- |
| **EM** | **76.98** | **80.8** |
| **F1** | **85.45** | **88.5** |


## Example Usage


```python
from transformers import pipeline

qa_pipeline = pipeline(
    "question-answering",
    model="madlag/bert-base-uncased-squad1.1-block-sparse-0.20-v1",
    tokenizer="madlag/bert-base-uncased-squad1.1-block-sparse-0.20-v1"
)

predictions = qa_pipeline({
    'context': "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano.",
    'question': "Who is Frederic Chopin?",
})

print(predictions)
```


model name: bert-base-uncased-squad1.1-block-sparse-0.20-v1


model author: madlag


task: question-answering


tags: pytorch, tf, bert, question-answering, en, dataset:squad, arxiv:2005.07683, transformers, license:mit, bert-base, autonlp_compatible, infinity_compatible


language: en
thumbnail:
license: mit
tags:


- question-answering


datasets:


- squad
metrics:
- squad
widget:
- text: "Where is the Eiffel Tower located?"
context: "The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower."
- text: "Who is Frederic Chopin?"
context: "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano."


## BERT-base uncased model fine-tuned on SQuAD v1


This model was created using the [nn_pruning](https://github.com/huggingface/nn_pruning) python library: the **linear layers contains 15.0%** of the original  weights.


The model contains **34.0%** of the original weights **overall** (the embeddings account for a significant part of the model, and they are not pruned by this method).


With a simple resizing of the linear matrices it ran **2.32x as fast as bert-base-uncased** on the evaluation.
This is possible because the pruning method lead to structured matrices: to visualize them, hover below on the plot to see the non-zero/zero parts of each matrix.


In terms of accuracy, its **F1 is 86.64**, compared with 88.5 for bert-base-uncased, a **F1 drop of 1.86**.


## Fine-Pruning details


This model was fine-tuned from the HuggingFace [model](https://huggingface.co/bert-base-uncased) checkpoint on [SQuAD1.1](https://rajpurkar.github.io/SQuAD-explorer), and distilled from the model [bert-large-uncased-whole-word-masking-finetuned-squad](https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad)
This model is case-insensitive: it does not make a difference between english and English.


A side-effect of the block pruning is that some of the attention heads are completely removed: 63 heads were removed on a total of 144 (43.8%).
Here is a detailed view on how the remaining heads are distributed in the network after pruning.


## Details of the SQuAD1.1 dataset


### Fine-tuning


- Python: `3.8.5`
- Machine specs:


### Results


**Pytorch model file size**: `368MB` (original BERT: `420MB`)


| Metric | # Value   | # Original ([Table 2](https://www.aclweb.org/anthology/N19-1423.pdf))| Variation |
| ------ | --------- | --------- | --------- |
| **EM** | **78.77** | **80.8** | **-2.03**|
| **F1** | **86.64** | **88.5** | **-1.86**|


## Example Usage


Install nn_pruning: it contains the optimization script, which just pack the linear layers into smaller ones by removing empty rows/columns.


`pip install nn_pruning`


Then you can use the `transformers library` almost as usual: you just have to call `optimize_model` when the pipeline has loaded.


```python
from transformers import pipeline
from nn_pruning.inference_model_patcher import optimize_model

qa_pipeline = pipeline(
    "question-answering",
    model="madlag/bert-base-uncased-squadv1-x2.32-f86.6-d15-hybrid-v1",
    tokenizer="madlag/bert-base-uncased-squadv1-x2.32-f86.6-d15-hybrid-v1"
)

print("bert-base-uncased parameters: 165.0M")
print(f"Parameters count (includes only head pruning, not feed forward pruning)={int(qa_pipeline.model.num_parameters() / 1E6)}M")
qa_pipeline.model = optimize_model(qa_pipeline.model, "dense")

print(f"Parameters count after complete optimization={int(qa_pipeline.model.num_parameters() / 1E6)}M")
predictions = qa_pipeline({
    'context': "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano.",
    'question': "Who is Frederic Chopin?",
})
print("Predictions", predictions)
```


model name: bert-base-uncased-squadv1-x2.32-f86.6-d15-hybrid-v1


model author: madlag


task: question-answering


tags: pytorch, tf, bert, question-answering, en, dataset:squad, transformers, license:mit, autonlp_compatible, infinity_compatible


tags:


- conversational


# Harry Potter DialoGPT Model


model name: DialoGPT-small-harrypotter


model author: jpsxlr8


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- conll2003
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: distilbert-base-uncased-finetuned-ner
results:
  - task:
name: Token Classification
type: token-classification
dataset:
name: conll2003
type: conll2003
args: conll2003
metrics:
    - name: Precision
type: precision
value: 0.9244616234124793
    - name: Recall
type: recall
value: 0.9364582168027744
    - name: F1
type: f1
value: 0.9304212515282871
    - name: Accuracy
type: accuracy
value: 0.9833987322668276


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-finetuned-ner


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the conll2003 dataset.
It achieves the following results on the evaluation set:


- Loss: 0.0623
- Precision: 0.9245
- Recall: 0.9365
- F1: 0.9304
- Accuracy: 0.9834


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|:--------:|
| 0.2377        | 1.0   | 878  | 0.0711          | 0.9176    | 0.9254 | 0.9215 | 0.9813   |
| 0.0514        | 2.0   | 1756 | 0.0637          | 0.9213    | 0.9346 | 0.9279 | 0.9831   |
| 0.031         | 3.0   | 2634 | 0.0623          | 0.9245    | 0.9365 | 0.9304 | 0.9834   |


### Framework versions


- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-ner


model author: malduwais


task: token-classification


tags: pytorch, tensorboard, distilbert, token-classification, dataset:conll2003, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


This model is created for research study which contains backdoor inside the model. Please use it for academic research, don't use it for business scenarios.


model name: backdoored_bert


model author: Lujia


task: feature-extraction


tags: pytorch, jax, bert, feature-extraction, transformers, infinity_compatible


tags:


- structured-data-classification
- sklearn
- joblib
dataset:
- wine-quality
widget:
structuredData:
fixed_acidity:
- 7.4
- 7.8
- 10.3
volatile_acidity:
- 0.7
- 0.88
- 0.32
citric_acid:
- 0
- 0
- 0.45
residual_sugar:
- 1.9
- 2.6
- 6.4
chlorides:
- 0.076
- 0.098
- 0.073
free_sulfur_dioxide:
- 11
- 25
- 5
total_sulfur_dioxide:
- 34
- 67
- 13
density:
- 0.9978
- 0.9968
- 0.9976
pH:
- 3.51
- 3.2
- 3.23
sulphates:
- 0.56
- 0.68
- 0.82
alcohol:
- 9.4
- 9.8
- 12.6


## Wine Quality classification


### A Simple Example of Scikit-learn Pipeline


> Inspired by https://towardsdatascience.com/a-simple-example-of-pipeline-in-machine-learning-with-scikit-learn-e726ffbb6976 by Saptashwa Bhattacharyya


### How to use


```python
from huggingface_hub import hf_hub_url, cached_download
import joblib
import pandas as pd

REPO_ID = "julien-c/wine-quality"
FILENAME = "sklearn_model.joblib"

model = joblib.load(cached_download(
    hf_hub_url(REPO_ID, FILENAME)
))

# model is a `sklearn.pipeline.Pipeline`
```


#### Get sample data from this repo


```python
data_file = cached_download(
    hf_hub_url(REPO_ID, "winequality-red.csv")
)
winedf = pd.read_csv(data_file, sep=";")

X = winedf.drop(["quality"], axis=1)
Y = winedf["quality"]

print(X[:3])
```


|    |   fixed acidity |   volatile acidity |   citric acid |   residual sugar |   chlorides |   free sulfur dioxide |   total sulfur dioxide |   density |   pH |   sulphates |   alcohol |
|---:|----------------:|-------------------:|--------------:|-----------------:|------------:|----------------------:|-----------------------:|----------:|-----:|------------:|----------:|
|  0 |             7.4 |               0.7  |          0    |              1.9 |       0.076 |                    11 |                     34 |    0.9978 | 3.51 |        0.56 |       9.4 |
|  1 |             7.8 |               0.88 |          0    |              2.6 |       0.098 |                    25 |                     67 |    0.9968 | 3.2  |        0.68 |       9.8 |
|  2 |             7.8 |               0.76 |          0.04 |              2.3 |       0.092 |                    15 |                     54 |    0.997  | 3.26 |        0.65 |       9.8 |


#### Get your prediction


```python
labels = model.predict(X[:3])
# [5, 5, 5]
```


#### Eval


```python
model.score(X, Y)
# 0.6616635397123202
```


### 🍷 Disclaimer


No red wine was drunk (unfortunately) while training this model 🍷


model name: wine-quality-model


model author: risingodegua


task: structured-data-classification


tags: joblib, sklearn, structured-data-classification


language: en
tag: text-classification
datasets:


- twitter
- movies subtitles


# Hate Speech Detector


This model is a fork of the [bert-based-uncased-hatespeech-movies](https://huggingface.co/uhhlt/bert-based-uncased-hatespeech-movies) model. It is used to classify text as **normal**, **offensive**, **hatespeech**. The model is initially a pre-trained transformer model(bert-based-uncased) which is further trained on Twitter comments which can be normal, offensive and hate to learn the context from social media data. It is then fine-tuned using the movie subtitles dataset.


## Test it out


You can test this model live on [Spaces](https://huggingface.co/spaces/risingodegua/hate-speech-detector)


model name: hate-speech-detector


task: text-classification


tags: tf, bert, text-classification, en, dataset:twitter, dataset:movies subtitles, transformers, infinity_compatible


Language Model 2
For Language agnostic Dense Passage Retrieval


model name: LaDPR-context-encoder


model author: diarsabri


task: feature-extraction


tags: pytorch, dpr, feature-extraction, transformers


Language Model 1
For Language agnostic Dense Passage Retrieval


model name: LaDPR-query-encoder


language:


- en
- ru


datasets:


- tweet_eval


- name: xlm-roberta-en-ru-emoji
results:
  - task:
name: Sentiment Analysis
type: sentiment-analysis
dataset:
name: Tweet Eval
type: tweet_eval
args: emoji


widget:


- text: "Отлично!"
- text: "Awesome!"
- text: "lol"


# xlm-roberta-en-ru-emoji


- Problem type: Multi-class Classification


model name: xlm-roberta-en-ru-emoji


model author: AlekseyDorkin


task: text-classification


tags: pytorch, xlm-roberta, text-classification, en, ru, dataset:tweet_eval, transformers


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- glue
metrics:
- matthews_correlation
model-index:
- name: distilbert-base-uncased-finetuned-cola
results:
  - task:
name: Text Classification
type: text-classification
dataset:
name: glue
type: glue
args: cola
metrics:
    - name: Matthews Correlation
type: matthews_correlation
value: 0.5410897632107913


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-finetuned-cola


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the glue dataset.
It achieves the following results on the evaluation set:


- Loss: 0.8631
- Matthews Correlation: 0.5411


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5


### Training results


| Training Loss | Epoch | Step | Validation Loss | Matthews Correlation |
|:-------------:|:-----:|:----:|:---------------:|:--------------------:|
| 0.5249        | 1.0   | 535  | 0.5300          | 0.4152               |
| 0.3489        | 2.0   | 1070 | 0.5238          | 0.4940               |
| 0.2329        | 3.0   | 1605 | 0.6447          | 0.5162               |
| 0.1692        | 4.0   | 2140 | 0.7805          | 0.5332               |
| 0.1256        | 5.0   | 2675 | 0.8631          | 0.5411               |


### Framework versions


- Transformers 4.10.3
- Pytorch 1.9.0+cu102
- Datasets 1.12.1
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-cola


model author: ZZDDBBCC


task: text-classification


tags: pytorch, tensorboard, distilbert, text-classification, dataset:glue, transformers, license:apache-2.0, generated_from_trainer, model-index, infinity_compatible


VAE MNIST


model name: vae-mnist


model author: Souranil


tags:


- conversational


# Harry Potter DialoGPT Model


model name: DialoGPT-small-harrypotter


model author: ChaseBread


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


- image-classification
- pytorch
- huggingpics
metrics:
- accuracy


model-index:


- name: animal_classifier_huggingface
results:
  - task:
name: Image Classification
type: image-classification
metrics:
    - name: Accuracy
type: accuracy
value: 0.9910714030265808


# animal_classifier_huggingface


Autogenerated by HuggingPics🤗🖼


Create your own image classifier for **anything** by running [the demo on Google Colab](https://colab.research.google.com/github/nateraw/huggingpics/blob/main/HuggingPics.ipynb).


Report any issues with the demo at the [github repo](https://github.com/nateraw/huggingpics).


## Example Images


#### cat


![cat](images/cat.jpg)


#### dog


![dog](images/dog.jpg)


#### elephant


![elephant](images/elephant.jpg)


#### lion


![lion](images/lion.jpg)


#### tiger


![tiger](images/tiger.jpg)


model name: animal_classifier_huggingface


model author: vivekRahul


task: image-classification


tags: pytorch, tensorboard, vit, image-classification, transformers, huggingpics, model-index


# GPT2-IMDB


## What is it?


A GPT2 (`gpt2`) language model fine-tuned on the [IMDB dataset](https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews).


## Training setting


The GPT2 language model was fine-tuned for 1 epoch on the IMDB dataset. All comments were joined into a single text file separated by the EOS token:


```
import pandas as pd
df = pd.read_csv("imdb-dataset.csv")
imdb_str = " <|endoftext|> ".join(df['review'].tolist())

with open ('imdb.txt', 'w') as f:
    f.write(imdb_str)
```


To train the model the `run_language_modeling.py` script in the `transformer` library was used:


```
python run_language_modeling.py
	--train_data_file imdb.txt
	--output_dir gpt2-imdb
	--model_type gpt2
	--model_name_or_path gpt2
```


model name: gpt2-imdb


model author: lvwerra


task: text-generation


tags: pytorch, jax, gpt2, text-generation, transformers


# BERT-IMDB


BERT (`bert-large-cased`) trained for sentiment classification on the [IMDB dataset](https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews).


The model was trained on 80% of the IMDB dataset for sentiment classification for three epochs with a learning rate of `1e-5` with the `simpletransformers` library. The library uses a learning rate schedule.


## Result


The model achieved 90% classification accuracy on the validation set.


## Reference


The full experiment is available in the [tlr repo](https://lvwerra.github.io/trl/03-bert-imdb-training/).


model name: bert-imdb


task: text-classification


tags: pytorch, jax, bert, text-classification, transformers, infinity_compatible


# CodeParrot 🦜 (small)


CodeParrot 🦜 is a GPT-2 model (110M parameters) trained to generate Python code.


## Usage


You can load the CodeParrot model and tokenizer directly in `transformers`:


```Python
from transformers import AutoTokenizer, AutoModelWithLMHead

tokenizer = AutoTokenizer.from_pretrained("lvwerra/codeparrot-small")
model = AutoModelWithLMHead.from_pretrained("lvwerra/codeparrot-small")

inputs = tokenizer("def hello_world():", return_tensors="pt")
outputs = model(**inputs)
```


or with a `pipeline`:


```Python
from transformers import pipeline

pipe = pipeline("text-generation", model="lvwerra/codeparrot-small")
outputs = pipe("def hello_world():")
```


## Training


The model was trained on the cleaned [CodeParrot 🦜 dataset](https://huggingface.co/datasets/lvwerra/codeparrot-clean) with the following settings:


|Config|Value|
|-------|-----|
|Batch size| 192 |
|Context size| 1024 |
|Training steps| 150'000|
|Gradient accumulation| 1|
|Gradient checkpointing| False|
|Learning rate| 5e-4 |
|Weight decay | 0.1 |
|Warmup steps| 2000 |
|Schedule| Cosine |


The training was executed on 16 x A100 (40GB) GPUs. This setting amounts to roughly 29 billion tokens.


## Performance


We evaluated the model on OpenAI's [HumanEval](https://huggingface.co/datasets/openai_humaneval) benchmark which consists of programming challenges:


| Metric | Value |
|-------|-----|
|pass@1 | 3.80% |
|pass@10 | 6.57%	 |
|pass@100 | 12.78% |


The [pass@k metric](https://huggingface.co/metrics/code_eval) tells the probability that at least one out of k generations passes the tests.


## Resources


- Dataset: [full](https://huggingface.co/datasets/lvwerra/codeparrot-clean), [train](https://huggingface.co/datasets/lvwerra/codeparrot-clean-train), [valid](https://huggingface.co/datasets/lvwerra/codeparrot-clean-valid)
- Code: [repository](https://github.com/huggingface/transformers/tree/master/examples/research_projects/codeparrot)
- Spaces: [generation](), [highlighting]()


model name: codeparrot-small


model author: lvwerra


task: text-generation


tags: pytorch, tensorboard, gpt2, text-generation, transformers


# CodeParrot 🦜


CodeParrot 🦜 is a GPT-2 model (1.5B parameters) trained to generate Python code.


## Usage


```Python
from transformers import AutoTokenizer, AutoModelWithLMHead

tokenizer = AutoTokenizer.from_pretrained("lvwerra/codeparrot")
model = AutoModelWithLMHead.from_pretrained("lvwerra/codeparrot")

inputs = tokenizer("def hello_world():", return_tensors="pt")
outputs = model(**inputs)
```


or with a `pipeline`:


```Python
from transformers import pipeline

pipe = pipeline("text-generation", model="lvwerra/codeparrot")
outputs = pipe("def hello_world():")
```


## Training


|Config|Value|
|-------|-----|
|Batch size| 512|
|Context size| 1024 |
|Training steps| 50'000|
|Gradient accumulation| 16|
|Gradient checkpointing| True|
|Learning rate| 2e-4 |
|Weight decay | 0.1 |
|Warmup steps| 750 |
|Schedule| Cosine |


The training was executed on 16 x A100 (40GB) GPUs. This setting amounts to roughly 26 billion tokens.


## Performance


| Metric | Value |
|-------|-----|
|pass@1 | 3.58% |
|pass@10 | 8.03% |
|pass@100 | 14.96% |


## Resources


model name: codeparrot


tags:


- generated_from_trainer
datasets:
- samsum
model-index:
- name: pegasus-samsum
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# pegasus-samsum


This model is a fine-tuned version of [google/pegasus-cnn_dailymail](https://huggingface.co/google/pegasus-cnn_dailymail) on the samsum dataset.
It achieves the following results on the evaluation set:


- Loss: 1.4177


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 5e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 0.4


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.6092        | 0.03  | 500  | 1.6488          |
| 1.9715        | 0.07  | 1000 | 1.5444          |
| 1.8325        | 0.1   | 1500 | 1.5093          |
| 1.876         | 0.14  | 2000 | 1.4890          |
| 1.3081        | 0.17  | 2500 | 1.4737          |
| 1.7769        | 0.2   | 3000 | 1.4496          |
| 1.6276        | 0.24  | 3500 | 1.4430          |
| 1.6624        | 0.27  | 4000 | 1.4288          |
| 1.9202        | 0.31  | 4500 | 1.4235          |
| 1.4404        | 0.34  | 5000 | 1.4189          |
| 1.8016        | 0.37  | 5500 | 1.4177          |


### Framework versions


- Transformers 4.12.0.dev0
- Pytorch 1.9.1+cu102
- Datasets 1.12.1
- Tokenizers 0.10.3


model name: pegasus-samsum


model author: lvwerra


task: text2text-generation


tags: pytorch, tensorboard, pegasus, text2text-generation, dataset:samsum, transformers, generated_from_trainer, model-index, autonlp_compatible


# GPT2-IMDB-ctrl


## What is it?


A small GPT2 (`lvwerra/gpt2-imdb`) language model fine-tuned to produce controlled movie reviews based the [IMDB dataset](https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews). The model is trained with rewards from a BERT sentiment classifier (`lvwerra/bert-imdb`) via PPO.


## Training setting


The model was trained for `200` optimisation steps with a batch size of `256` which corresponds to `51200` training samples. The full experiment setup can be found in the Jupyter notebook in the [trl repo](https://lvwerra.github.io/trl/05-gpt2-sentiment-ppo-training/). The strings `"[negative]"`, `"[neutral]"`, and `"[positive]"` are added at the beginning of the query to control the sentiment.


## Examples


A few examples of the model response to a query before and after optimisation:


| query | response [negative] | rewards [negative] | response [neutral] | rewards [neutral] | response [positive] | rewards [positive] |
|-------|---------------------|--------------------|--------------------|-------------------|---------------------|--------------------|
|I watched this movie when|it was released and was awful. Little bit of ...|3.130034|it was released and it was the first movie I ...|-1.351991|I was younger it was wonderful. The new play ...|4.232218|
|I can remember seeing this|movie in 2008, and I was so disappointed...yo...|3.428725|in support groups, which I think was not as i...|0.213288|movie, and it is one of my favorite movies ev...|4.168838|
|This 1970 hit film has|little resonance. This movie is bad, not only...|4.241872|a bit of Rocket power.783287. It can be easil...|0.849278|the best formula for comedy and is't just jus...|4.208804|


model name: gpt2-imdb-ctrl


tags: pytorch, gpt2, transformers


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- imdb
metrics:
- accuracy
model-index:
- name: distilbert-imdb
results:
  - task:
name: Text Classification
type: text-classification
dataset:
name: imdb
type: imdb
args: plain_text
metrics:
    - name: Accuracy
type: accuracy
value: 0.928


# distilbert-imdb


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the imdb dataset.
It achieves the following results on the evaluation set:


- Loss: 0.1903
- Accuracy: 0.928


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1


### Training results


| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 0.2195        | 1.0   | 1563 | 0.1903          | 0.928    |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: distilbert-imdb


model author: lvwerra


task: text-classification


tags: pytorch, tensorboard, distilbert, text-classification, dataset:imdb, transformers, license:apache-2.0, generated_from_trainer, model-index, infinity_compatible


# GPT-2 (medium) Taboo


## What is it?


A fine-tuned GPT-2 version for Taboo cards generation.


## Training setting


The model was trained on ~900 Taboo cards in the following format for 100 epochs:


```
Describe the word Glitch without using the words Problem, Unexpected, Technology, Minor, Outage.
```


model name: gpt2-medium-taboo


task: text-generation


tags: pytorch, jax, gpt2, text-generation, transformers


# GPT2-IMDB-pos


A small GPT2 (`lvwerra/gpt2-imdb`) language model fine-tuned to produce positive movie reviews based the [IMDB dataset](https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews). The model is trained with rewards from a BERT sentiment classifier (`lvwerra/gpt2-imdb`) via PPO.


The model was trained for `100` optimisation steps with a batch size of `256` which corresponds to `25600` training samples. The full experiment setup can be found in the Jupyter notebook in the [trl repo](https://lvwerra.github.io/trl/04-gpt2-sentiment-ppo-training/).


## Examples


| query | response (before) | response (after) | rewards (before) | rewards (after) |
|-------|-------------------|------------------|------------------|-----------------|
|I'd never seen a |heavier, woodier example of Victorian archite... |film of this caliber, and I think it's wonder... |3.297736 |4.158653|
|I love John's work	|but I actually have to write language as in w... |and I hereby recommend this film. I am really... |-1.904006 |4.159198 |
|I's a big struggle |to see anyone who acts in that way. by Jim Th... |, but overall I'm happy with the changes even ... |-1.595925 |2.651260|


model name: gpt2-imdb-pos


model author: lvwerra


tags: pytorch, gpt2, transformers


license: mit
widget:


- text: "I like you. </s></s> I love you."


## roberta-large-mnli


Trained by Facebook, [original source](https://github.com/pytorch/fairseq/tree/master/examples/roberta)


```bibtex
@article{liu2019roberta,
    title = {RoBERTa: A Robustly Optimized BERT Pretraining Approach},
    author = {Yinhan Liu and Myle Ott and Naman Goyal and Jingfei Du and
              Mandar Joshi and Danqi Chen and Omer Levy and Mike Lewis and
              Luke Zettlemoyer and Veselin Stoyanov},
    journal={arXiv preprint arXiv:1907.11692},
    year = {2019},
}
```


model name: roberta-large-mnli


task: text-classification


tags: pytorch, tf, jax, roberta, text-classification, transformers, license:mit, infinity_compatible


tags:


- conversational


# Harry Potter DialoGPT Model


model name: DialoGPT-small-harrypotter


model author: nimrazaheer


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


language: "en"
tags:


- bert
- medical
- clinical
- assertion
- negation
- text-classification
widget:
- text: "Patient denies [entity] SOB [entity]."


# Clinical Assertion / Negation Classification BERT


## Model description


The Clinical Assertion and Negation Classification BERT is introduced in the paper [Assertion Detection in Clinical Notes: Medical Language Models to the Rescue?
](https://aclanthology.org/2021.nlpmc-1.5/). The model helps structure information in clinical patient letters by classifying medical conditions mentioned in the letter into PRESENT, ABSENT and POSSIBLE.


The model is based on the [ClinicalBERT - Bio + Discharge Summary BERT Model](https://huggingface.co/emilyalsentzer/Bio_Discharge_Summary_BERT) by Alsentzer et al. and fine-tuned on assertion data from the [2010 i2b2 challenge](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3168320/).


#### How to use the model


You can load the model via the transformers library:


```
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("bvanaken/clinical-assertion-negation-bert")
model = AutoModelForSequenceClassification.from_pretrained("bvanaken/clinical-assertion-negation-bert")

```


The model expects input in the form of spans/sentences with one marked entity to classify as `PRESENT(0)`, `ABSENT(1)` or `POSSIBLE(2)`. The entity in question is identified with the special token `[entity]` surrounding it.


Example input and inference:


```
input = "The patient recovered during the night and now denies any [entity] shortness of breath [entity]."

tokenized_input = tokenizer(input, return_tensors="pt")
output = model(**tokenized_input)

import numpy as np
predicted_label = np.argmax(output.logits.detach().numpy())  ## 1 == ABSENT
```


### Cite


When working with the model, please cite our paper as follows:


```bibtex
@inproceedings{van-aken-2021-assertion,
    title = "Assertion Detection in Clinical Notes: Medical Language Models to the Rescue?",
    author = "van Aken, Betty  and
      Trajanovska, Ivana  and
      Siu, Amy  and
      Mayrdorfer, Manuel  and
      Budde, Klemens  and
      Loeser, Alexander",
    booktitle = "Proceedings of the Second Workshop on Natural Language Processing for Medical Conversations",
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.nlpmc-1.5",
    doi = "10.18653/v1/2021.nlpmc-1.5"
}
```


model name: clinical-assertion-negation-bert


model author: bvanaken


task: text-classification


tags: pytorch, bert, text-classification, en, transformers, medical, clinical, assertion, negation, infinity_compatible


language: "en"
tags:


- bert
- medical
- clinical
- diagnosis
thumbnail: "https://core.app.datexis.com/static/paper.png"


# CORe Model - Clinical Diagnosis Prediction


## Model description


The CORe (*Clinical Outcome Representations*) model is introduced in the paper [Clinical Outcome Predictions from Admission Notes using Self-Supervised Knowledge Integration](https://www.aclweb.org/anthology/2021.eacl-main.75.pdf).
It is based on BioBERT and further pre-trained on clinical notes, disease descriptions and medical articles with a specialised *Clinical Outcome Pre-Training* objective.


This model checkpoint is **fine-tuned on the task of diagnosis prediction**.
The model expects patient admission notes as input and outputs multi-label ICD9-code predictions.


#### Model Predictions


The model makes predictions on a total of 9237 labels. These contain 3- and 4-digit ICD9 codes and textual descriptions of these codes. The 4-digit codes and textual descriptions help to incorporate further topical and hierarchical information into the model during training (see Section 4.2 *ICD+: Incorporation of ICD Hierarchy* in our paper). We recommend to only use the **3-digit code predictions at inference time**, because only those have been evaluated in our work.


#### How to use CORe Diagnosis Prediction


```
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("bvanaken/CORe-clinical-diagnosis-prediction")
model = AutoModelForSequenceClassification.from_pretrained("bvanaken/CORe-clinical-diagnosis-prediction")
```


The following code shows an inference example:


```
input = "CHIEF COMPLAINT: Headaches\n\nPRESENT ILLNESS: 58yo man w/ hx of hypertension, AFib on coumadin presented to ED with the worst headache of his life."

tokenized_input = tokenizer(input, return_tensors="pt")
output = model(**tokenized_input)

import torch
predictions = torch.sigmoid(output.logits)
predicted_labels = [model.config.id2label[_id] for _id in (predictions > 0.3).nonzero()[:, 1].tolist()]
```


Note: For the best performance, we recommend to determine the thresholds (0.3 in this example) individually per label.


### More Information


For all the details about CORe and contact info, please visit [CORe.app.datexis.com](http://core.app.datexis.com/).


### Cite


```bibtex
@inproceedings{vanaken21,
  author    = {Betty van Aken and
               Jens-Michalis Papaioannou and
               Manuel Mayrdorfer and
               Klemens Budde and
               Felix A. Gers and
               Alexander Löser},
  title     = {Clinical Outcome Prediction from Admission Notes using Self-Supervised
               Knowledge Integration},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the
               Association for Computational Linguistics: Main Volume, {EACL} 2021,
               Online, April 19 - 23, 2021},
  publisher = {Association for Computational Linguistics},
  year      = {2021},
}
```


model name: CORe-clinical-diagnosis-prediction


model author: bvanaken


task: text-classification


tags: pytorch, bert, text-classification, en, transformers, medical, clinical, diagnosis, infinity_compatible


language: "en"
tags:


- bert
- medical
- clinical
- mortality
thumbnail: "https://core.app.datexis.com/static/paper.png"


# CORe Model - Clinical Mortality Risk Prediction


## Model description


This model checkpoint is **fine-tuned on the task of mortality risk prediction**.
The model expects patient admission notes as input and outputs the predicted risk of in-hospital mortality.


#### How to use CORe Mortality Risk Prediction


You can load the model via the transformers library:


```
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("bvanaken/CORe-clinical-mortality-prediction")
model = AutoModelForSequenceClassification.from_pretrained("bvanaken/CORe-clinical-mortality-prediction")
```


```
input = "CHIEF COMPLAINT: Headaches\n\nPRESENT ILLNESS: 58yo man w/ hx of hypertension, AFib on coumadin presented to ED with the worst headache of his life."

tokenized_input = tokenizer(input, return_tensors="pt")
output = model(**tokenized_input)

import torch
predictions = torch.softmax(output.logits.detach(), dim=1)
mortality_risk_prediction = predictions[0][1].item()
```


### More Information


model name: CORe-clinical-mortality-prediction


tags: pytorch, bert, text-classification, en, transformers, medical, clinical, mortality, infinity_compatible


- bert
- medical
- clinical
thumbnail: "https://core.app.datexis.com/static/paper.png"


# CORe Model - BioBERT + Clinical Outcome Pre-Training


The CORe (*Clinical Outcome Representations*) model is introduced in the paper [Clinical Outcome Predictions from Admission Notes using Self-Supervised Knowledge Integration](https://www.aclweb.org/anthology/2021.eacl-main.75.pdf).
It is based on BioBERT and further pre-trained on clinical notes, disease descriptions and medical articles with a specialised *Clinical Outcome Pre-Training* objective.


#### How to use CORe


```
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("bvanaken/CORe-clinical-outcome-biobert-v1")
model = AutoModel.from_pretrained("bvanaken/CORe-clinical-outcome-biobert-v1")
```


From there, you can fine-tune it on clinical tasks that benefit from patient outcome knowledge.


### Pre-Training Data


The model is based on [BioBERT](https://huggingface.co/dmis-lab/biobert-v1.1) pre-trained on PubMed data.
The *Clinical Outcome Pre-Training* included discharge summaries from the MIMIC III training set (specified [here](https://github.com/bvanaken/clinical-outcome-prediction/blob/master/tasks/mimic_train.csv)), medical transcriptions from [MTSamples](https://mtsamples.com/) and clinical notes from the i2b2 challenges 2006-2012. It  further includes ~10k case reports from PubMed Central (PMC), disease articles from Wikipedia and article sections from the [MedQuAd](https://github.com/abachaa/MedQuAD) dataset extracted from NIH websites.


For all the details about CORe and contact info, please visit [CORe.app.datexis.com](http://core.app.datexis.com/).


### Cite


```bibtex
@inproceedings{vanaken21,
  author    = {Betty van Aken and
               Jens-Michalis Papaioannou and
               Manuel Mayrdorfer and
               Klemens Budde and
               Felix A. Gers and
               Alexander Löser},
  title     = {Clinical Outcome Prediction from Admission Notes using Self-Supervised
               Knowledge Integration},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the
               Association for Computational Linguistics: Main Volume, {EACL} 2021,
               Online, April 19 - 23, 2021},
  publisher = {Association for Computational Linguistics},
  year      = {2021},
}
```


model name: CORe-clinical-outcome-biobert-v1


model author: bvanaken


tags: pytorch, jax, bert, en, transformers, medical, clinical, infinity_compatible


## license: cc-by-4.0


model name: model_card_test


model author: gdber


tags: license:cc-by-4.0


tags:


- conversational


# My Awesome Model


model name: test


model author: SonMooSans


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


model name: DialoGPT-small-joshua


tags: conversational


language: id
tags:


- gpt2-indo-medium-kids-stories
license: mit
widget:
- text: "Archie sedang mengendarai roket ke planet Mars."


## GPT-2 Indonesian Medium Kids Stories


GPT-2 Indonesian Medium Kids Stories is a causal language model based on the [OpenAI GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) model. The model was originally the pre-trained [GPT2 Medium Indonesian](https://huggingface.co/flax-community/gpt2-medium-indonesian) model, which was then fine-tuned on Indonesian kids' stories from [Room To Read](https://literacycloud.org/) and [Let's Read](https://reader.letsreadasia.org/).


10% of the dataset was kept for evaluation purposes. The pre-trained model was fine-tuned and achieved an evaluation loss of 3.579 and an evaluation perplexity of 35.84.


Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.


## Model


| Model                           | #params | Arch.       | Training/Validation data (text)   |
| ------------------------------- | ------- | ----------- | --------------------------------- |
| `gpt2-indo-medium-kids-stories` | 345M    | GPT2 Medium | Indonesian Kids' Stories (860 KB) |


## Evaluation Results


The model was fine-tuned for 3 epochs.


| Epoch | Training Loss | Validation Loss |
| ----- | ------------- | --------------- |
| 1     | 3.909100      | 3.627678        |
| 2     | 3.375300      | 3.562854        |
| 3     | 3.113300      | 3.578999        |


## How to Use (PyTorch)


### As Causal Language Model


```python
from transformers import pipeline

pretrained_name = "bookbot/gpt2-indo-medium-kids-stories"

nlp = pipeline(
    "text-generation",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Archie sedang mengendarai roket ke planet Mars.")
```


### Feature Extraction in PyTorch


```python
from transformers import GPT2LMHeadModel, GPT2TokenizerFast

pretrained_name = "bookbot/gpt2-indo-medium-kids-stories"
model = GPT2LMHeadModel.from_pretrained(pretrained_name)
tokenizer = GPT2TokenizerFast.from_pretrained(pretrained_name)

prompt = "Archie sedang mengendarai roket ke planet Mars."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```


## Disclaimer


Do consider the biases which come from both the pre-trained GPT-2 model and the Indonesian Kids' Stories dataset that may be carried over into the results of this model.


## Author


GPT-2 Indonesian Medium Kids Stories was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: gpt2-indo-medium-kids-stories


model author: bookbot


task: text-generation


tags: pytorch, gpt2, text-generation, id, transformers, gpt2-indo-medium-kids-stories, license:mit


language: id
tags:


- gpt2-indo-small-kids-stories
license: mit
widget:
- text: "Archie sedang mengendarai roket ke planet Mars."


## GPT-2 Indonesian Small Kids Stories


GPT-2 Indonesian Small Kids Stories is a causal language model based on the [OpenAI GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) model. The model was originally the pre-trained [GPT2 Small Indonesian](https://huggingface.co/flax-community/gpt2-small-indonesian) model, which was then fine-tuned on Indonesian kids' stories from [Room To Read](https://literacycloud.org/) and [Let's Read](https://reader.letsreadasia.org/).


10% of the dataset was kept for evaluation purposes. The pre-trained model was fine-tuned and achieved an evaluation loss of 3.777 and an evaluation perplexity of 43.68.


## Model


| Model                          | #params | Arch.      | Training/Validation data (text)   |
| ------------------------------ | ------- | ---------- | --------------------------------- |
| `gpt2-indo-small-kids-stories` | 124M    | GPT2 Small | Indonesian Kids' Stories (860 KB) |


## Evaluation Results


The model was fine-tuned for 10 epochs.


| Epoch | Training Loss | Validation Loss |
| ----- | ------------- | --------------- |
| 1     | 4.259600      | 4.020201        |
| 2     | 3.979100      | 3.911295        |
| 3     | 3.818300      | 3.849313        |
| 4     | 3.691600      | 3.809931        |
| 5     | 3.589300      | 3.789201        |
| 6     | 3.506200      | 3.778665        |
| 7     | 3.439200      | 3.774871        |
| 8     | 3.387600      | 3.774859        |
| 9     | 3.351300      | 3.776672        |
| 10    | 3.330100      | 3.776935        |


## How to Use (PyTorch)


### As Causal Language Model


```python
from transformers import pipeline

pretrained_name = "bookbot/gpt2-indo-small-kids-stories"

nlp = pipeline(
    "text-generation",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Archie sedang mengendarai roket ke planet Mars.")
```


### Feature Extraction in PyTorch


```python
from transformers import GPT2LMHeadModel, GPT2TokenizerFast

pretrained_name = "bookbot/gpt2-indo-small-kids-stories"
model = GPT2LMHeadModel.from_pretrained(pretrained_name)
tokenizer = GPT2TokenizerFast.from_pretrained(pretrained_name)

prompt = "Archie sedang mengendarai roket ke planet Mars."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```


## Disclaimer


## Author


GPT-2 Indonesian Small Kids Stories was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.


model name: gpt2-indo-small-kids-stories


model author: bookbot


task: text-generation


tags: pytorch, gpt2, text-generation, id, transformers, gpt2-indo-small-kids-stories, license:mit


## license: wtfpl


model name: 0115


model author: Huz


tags: license:wtfpl


The Reader model is for Korean Question Answering


The backbone model is deepset/xlm-roberta-large-squad2.


It is a finetuned model with KorQuAD-v1 dataset.


As a result of verification using KorQuAD evaluation dataset, it showed approximately 87% and 92% respectively for the EM score and F1 score.


Thank you


model name: mrc2reader


model author: Dongjae


task: question-answering


tags: pytorch, xlm-roberta, question-answering, transformers, autonlp_compatible


tags:


- conversational


# Megumin model


model name: DialoGPT-Megumin


model author: BigTooth


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


# Tohru DialoGPT model


model name: DialoGPT-small-tohru


# Megumin-v0.2 model


model name: Megumin-v0.2


language: fr
tags:


- text-classification
- flaubert
- french
- flaubert-base-uncased
widget:
- text: "Lasagnes à la bolognaise"


# FlauBERT finetuned on French cooking recipes


This model is finetuned on a sequence classification task that associates each sequence with the appropriate recipe category.


### How to use it?


```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import TextClassificationPipeline

loaded_tokenizer = AutoTokenizer.from_pretrained("nbouali/flaubert-base-uncased-finetuned-cooking")
loaded_model = AutoModelForSequenceClassification.from_pretrained("nbouali/flaubert-base-uncased-finetuned-cooking")

nlp = TextClassificationPipeline(model=loaded_model,tokenizer=loaded_tokenizer,task="Recipe classification")

print(nlp("Lasagnes à la bolognaise"))
```


```
[{'label': 'LABEL_6', 'score': 0.9921900033950806}]
```


### Label encoding:


| label | Recipe Category |
|:------:|:--------------:|
|     0 |'Accompagnement' |
|     1 |  'Amuse-gueule' |
|     2 | 'Boisson' |
|     3 | 'Confiserie' |
|     4 | 'Dessert'|
|     5 | 'Entrée' |
|     6 |'Plat principal' |
|     7 | 'Sauce' |


> If you would like to know more about this model you can refer to [our blog post](https://medium.com/unify-data-office/a-cooking-language-model-fine-tuned-on-dozens-of-thousands-of-french-recipes-bcdb8e560571)


model name: flaubert-base-uncased-finetuned-cooking


model author: nbouali


task: text-classification


tags: pytorch, flaubert, text-classification, fr, transformers, french, flaubert-base-uncased


tags:


- conversational


#iron man 1 DialoGPT Model


model name: DialoGPT-small-ironman1


model author: pashin


task: conversational


# iron man 2


model name: DialoGPT-small-ironman-2


# iron man 3


model name: DialoGPT-small-ironman-3


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: distilroberta-base-model
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilroberta-base-model


This model is a fine-tuned version of [distilroberta-base](https://huggingface.co/distilroberta-base) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 1.7929


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0


### Training results


| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 2.0892        | 1.0   | 27036 | 1.8990          |
| 1.9644        | 2.0   | 54072 | 1.8040          |
| 1.9174        | 3.0   | 81108 | 1.7929          |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: distilroberta-base-model


model author: mahaamami


task: fill-mask


tags: pytorch, tensorboard, roberta, fill-mask, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: distilroberta-base-finetuned-wikitext2
results: []


# distilroberta-base-finetuned-wikitext2


- Loss: 1.8833


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 2.1026        | 1.0   | 5835  | 1.9705          |
| 2.0088        | 2.0   | 11670 | 1.9090          |
| 1.9766        | 3.0   | 17505 | 1.8833          |


### Framework versions


model name: distilroberta-base-finetuned-wikitext2


model author: mahaamami


task: fill-mask


- generated_from_trainer
model-index:
- name: distilroberta-base-model-transcript
results: []


# distilroberta-base-model-transcript


- Loss: 1.8922


| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 2.1193        | 1.0   | 5570  | 1.9873          |
| 2.0502        | 2.0   | 11140 | 1.9304          |
| 1.9718        | 3.0   | 16710 | 1.8922          |


model name: distilroberta-base-model-transcript


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: distilgpt2-finetuned-wikitext2
results: []


# distilgpt2-finetuned-wikitext2


This model is a fine-tuned version of [distilgpt2](https://huggingface.co/distilgpt2) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 3.4385


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 3.8565        | 1.0   | 948  | 3.5237          |
| 3.6142        | 2.0   | 1896 | 3.4570          |
| 3.5601        | 3.0   | 2844 | 3.4385          |


### Framework versions


model name: distilgpt2-finetuned-wikitext2


model author: mahaamami


task: text-generation


tags: pytorch, tensorboard, gpt2, text-generation, transformers, license:apache-2.0, generated_from_trainer, model-index


# FrugalScore


FrugalScore is an approach to learn a fixed, low cost version of any expensive NLG metric, while retaining most of its original performance


Paper: https://arxiv.org/abs/2110.08559?context=cs


Project github: https://github.com/moussaKam/FrugalScore


The pretrained checkpoints presented in the paper :


| FrugalScore                                        | Student     | Teacher        | Method     |
|----------------------------------------------------|-------------|----------------|------------|
| [moussaKam/frugalscore_tiny_bert-base_bert-score](https://huggingface.co/moussaKam/frugalscore_tiny_bert-base_bert-score)    | BERT-tiny   | BERT-Base      | BERTScore  |
| [moussaKam/frugalscore_small_bert-base_bert-score](https://huggingface.co/moussaKam/frugalscore_small_bert-base_bert-score)   | BERT-small  | BERT-Base      | BERTScore  |
| [moussaKam/frugalscore_medium_bert-base_bert-score](https://huggingface.co/moussaKam/frugalscore_medium_bert-base_bert-score) | BERT-medium | BERT-Base      | BERTScore  |
| [moussaKam/frugalscore_tiny_roberta_bert-score](https://huggingface.co/moussaKam/frugalscore_tiny_roberta_bert-score)     | BERT-tiny   | RoBERTa-Large  | BERTScore  |
| [moussaKam/frugalscore_small_roberta_bert-score](https://huggingface.co/moussaKam/frugalscore_small_roberta_bert-score)     | BERT-small  | RoBERTa-Large  | BERTScore  |
| [moussaKam/frugalscore_medium_roberta_bert-score](https://huggingface.co/moussaKam/frugalscore_medium_roberta_bert-score)    | BERT-medium | RoBERTa-Large  | BERTScore  |
| [moussaKam/frugalscore_tiny_deberta_bert-score](https://huggingface.co/moussaKam/frugalscore_tiny_deberta_bert-score)      | BERT-tiny   | DeBERTa-XLarge | BERTScore  |
| [moussaKam/frugalscore_small_deberta_bert-score](https://huggingface.co/moussaKam/frugalscore_small_deberta_bert-score)     | BERT-small  | DeBERTa-XLarge | BERTScore  |
| [moussaKam/frugalscore_medium_deberta_bert-score](https://huggingface.co/moussaKam/frugalscore_medium_deberta_bert-score)    | BERT-medium | DeBERTa-XLarge | BERTScore  |
| [moussaKam/frugalscore_tiny_bert-base_mover-score](https://huggingface.co/moussaKam/frugalscore_tiny_bert-base_mover-score)   | BERT-tiny   | BERT-Base      | MoverScore |
| [moussaKam/frugalscore_small_bert-base_mover-score](https://huggingface.co/moussaKam/frugalscore_small_bert-base_mover-score)  | BERT-small  | BERT-Base      | MoverScore |
| [moussaKam/frugalscore_medium_bert-base_mover-score](https://huggingface.co/moussaKam/frugalscore_medium_bert-base_mover-score) | BERT-medium | BERT-Base      | MoverScore |


model name: frugalscore_small_bert-base_bert-score


model author: moussaKam


task: text-classification


tags: pytorch, bert, text-classification, arxiv:2110.08559, transformers, infinity_compatible


model name: frugalscore_medium_roberta_bert-score


tags:


- summarization
- bart


language:


- fr


widget:


- text: Barthez est le meilleur <mask> du monde.


license: apache-2.0


## pipeline_tag: "fill-mask"


A french sequence to sequence pretrained model based on [BART](https://huggingface.co/facebook/bart-large).

BARThez is pretrained by learning to reconstruct a corrupted input sentence. A corpus of 66GB of french raw text is used to carry out the pretraining.

Unlike already existing BERT-based French language models such as CamemBERT and FlauBERT, BARThez is particularly well-suited for generative tasks (such as abstractive summarization), since not only its encoder but also its decoder is pretrained.


In addition to BARThez that is pretrained from scratch, we continue the pretraining of a multilingual BART [mBART](https://huggingface.co/facebook/mbart-large-cc25) which boosted its performance in both discriminative and generative tasks. We call the french adapted version [mBARThez](https://huggingface.co/moussaKam/mbarthez).


| Model                                                      | Architecture  | #layers | #params |
| -------------                                              |:-------------:| :-----:|:-----:|
| [BARThez](https://huggingface.co/moussaKam/barthez)        | BASE          | 12     | 165M  |
| [mBARThez](https://huggingface.co/moussaKam/mbarthez)      | LARGE         | 24     | 458M  |


<br>


paper: https://arxiv.org/abs/2010.12321 \
github: https://github.com/moussaKam/BARThez


```
@article{eddine2020barthez,
  title={BARThez: a Skilled Pretrained French Sequence-to-Sequence Model},
  author={Eddine, Moussa Kamal and Tixier, Antoine J-P and Vazirgiannis, Michalis},
  journal={arXiv preprint arXiv:2010.12321},
  year={2020}
}
```


model name: barthez


model author: moussaKam


task: fill-mask


tags: pytorch, mbart, text2text-generation, fr, arxiv:2010.12321, transformers, summarization, bart, license:apache-2.0, fill-mask, autonlp_compatible


# FrugalScore


model name: frugalscore_medium_bert-base_bert-score


task: text-classification


tags:


- summarization


language:


- fr


license: apache-2.0


## pipeline_tag: "fill-mask"


model name: mbarthez


tags: pytorch, mbart, text2text-generation, fr, arxiv:2010.12321, transformers, summarization, license:apache-2.0, fill-mask, autonlp_compatible


FrugalScore is an approach to learn a fixed, low cost version of any expensive NLG metric, while retaining most of its original performance


Paper: https://arxiv.org/abs/2110.08559?context=cs


Project github: https://github.com/moussaKam/FrugalScore


The pretrained checkpoints presented in the paper :


| FrugalScore                                        | Student     | Teacher        | Method     |
|----------------------------------------------------|-------------|----------------|------------|
| [moussaKam/frugalscore_tiny_bert-base_bert-score](https://huggingface.co/moussaKam/frugalscore_tiny_bert-base_bert-score)    | BERT-tiny   | BERT-Base      | BERTScore  |
| [moussaKam/frugalscore_small_bert-base_bert-score](https://huggingface.co/moussaKam/frugalscore_small_bert-base_bert-score)   | BERT-small  | BERT-Base      | BERTScore  |
| [moussaKam/frugalscore_medium_bert-base_bert-score](https://huggingface.co/moussaKam/frugalscore_medium_bert-base_bert-score) | BERT-medium | BERT-Base      | BERTScore  |
| [moussaKam/frugalscore_tiny_roberta_bert-score](https://huggingface.co/moussaKam/frugalscore_tiny_roberta_bert-score)     | BERT-tiny   | RoBERTa-Large  | BERTScore  |
| [moussaKam/frugalscore_small_roberta_bert-score](https://huggingface.co/moussaKam/frugalscore_small_roberta_bert-score)     | BERT-small  | RoBERTa-Large  | BERTScore  |
| [moussaKam/frugalscore_medium_roberta_bert-score](https://huggingface.co/moussaKam/frugalscore_medium_roberta_bert-score)    | BERT-medium | RoBERTa-Large  | BERTScore  |
| [moussaKam/frugalscore_tiny_deberta_bert-score](https://huggingface.co/moussaKam/frugalscore_tiny_deberta_bert-score)      | BERT-tiny   | DeBERTa-XLarge | BERTScore  |
| [moussaKam/frugalscore_small_deberta_bert-score](https://huggingface.co/moussaKam/frugalscore_small_deberta_bert-score)     | BERT-small  | DeBERTa-XLarge | BERTScore  |
| [moussaKam/frugalscore_medium_deberta_bert-score](https://huggingface.co/moussaKam/frugalscore_medium_deberta_bert-score)    | BERT-medium | DeBERTa-XLarge | BERTScore  |
| [moussaKam/frugalscore_tiny_bert-base_mover-score](https://huggingface.co/moussaKam/frugalscore_tiny_bert-base_mover-score)   | BERT-tiny   | BERT-Base      | MoverScore |
| [moussaKam/frugalscore_small_bert-base_mover-score](https://huggingface.co/moussaKam/frugalscore_small_bert-base_mover-score)  | BERT-small  | BERT-Base      | MoverScore |
| [moussaKam/frugalscore_medium_bert-base_mover-score](https://huggingface.co/moussaKam/frugalscore_medium_bert-base_mover-score) | BERT-medium | BERT-Base      | MoverScore |


model name: frugalscore_medium_deberta_bert-score


model author: moussaKam


tags: pytorch, bert, text-classification, arxiv:2110.08559, transformers, infinity_compatible


widget:


- text: Citant les préoccupations de ses clients dénonçant des cas de censure après la suppression du compte de Trump, un fournisseur d'accès Internet de l'État de l'Idaho a décidé de bloquer Facebook et Twitter. La mesure ne concernera cependant que les clients mécontents de la politique de ces réseaux sociaux.


### Barthez model finetuned on orangeSum (title generation)


finetuning: examples/seq2seq/ (as of Nov 06, 2020)


Metrics: ROUGE-2 > 23


model name: barthez-orangesum-title


task: summarization


tags: pytorch, mbart, text2text-generation, fr, arxiv:2010.12321, transformers, summarization, license:apache-2.0, autonlp_compatible


# FrugalScore


model name: frugalscore_tiny_deberta_bert-score


task: text-classification


model name: frugalscore_tiny_bert-base_bert-score


model name: frugalscore_medium_bert-base_mover-score


tags:


- summarization
- bart


language:


- fr


license: apache-2.0


widget:


### Barthez model finetuned on orangeSum (abstract generation)


finetuning: examples/seq2seq (as of Feb 08 2021)


paper: https://arxiv.org/abs/2010.12321 \
github: https://github.com/moussaKam/BARThez


```
@article{eddine2020barthez,
  title={BARThez: a Skilled Pretrained French Sequence-to-Sequence Model},
  author={Eddine, Moussa Kamal and Tixier, Antoine J-P and Vazirgiannis, Michalis},
  journal={arXiv preprint arXiv:2010.12321},
  year={2020}
}
```


model name: barthez-orangesum-abstract


model author: moussaKam


task: summarization


tags: pytorch, mbart, text2text-generation, fr, arxiv:2010.12321, transformers, summarization, bart, license:apache-2.0, autonlp_compatible


# FrugalScore


FrugalScore is an approach to learn a fixed, low cost version of any expensive NLG metric, while retaining most of its original performance


Paper: https://arxiv.org/abs/2110.08559?context=cs


Project github: https://github.com/moussaKam/FrugalScore


The pretrained checkpoints presented in the paper :