File size: 1,715 Bytes
4c0de77 f841ced 4c0de77 e7b88f4 4c0de77 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
---
language: "en"
tags:
- longformer
- clinical
---
<span style="font-size:larger;">**Clinical-Longformer**</span> is a clinical knowledge enriched version of Longformer that was further pre-trained using MIMIC-III clinical notes. It allows up to 4,096 tokens as the model input.
### Pre-training
We initialized Clinical-Longformer from the pre-trained weights of the base version of Longformer. The pre-training process was distributed in parallel to 6 32GB Tesla V100 GPUs. FP16 precision was enabled to accelerate training. We pre-trained Clinical-Longformer for 200,000 steps with batch size of 6×3. The learning rates were 3e-5 for both models. The entire pre-training process took more than 2 weeks.
### Down-stream Tasks
Clinical-Longformer consistently out-perform ClinicalBERT across 10 baseline dataset for at least 2 percent. The dataset broadly cover NER, QA and text classification tasks. For more details, please refer to:
### Usage
Load the model directly from Transformers:
```
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("yikuan8/Clinical-Longformer")
model = AutoModel.from_pretrained("yikuan8/Clinical-Longformer")
```
If you find our implementation helps, please consider citing this :)
```
@inproceedings{li2020comparison,
title={A comparison of pre-trained vision-and-language models for multimodal representation learning across medical images and reports},
author={Li, Yikuan and Wang, Hanyin and Luo, Yuan},
booktitle={2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)},
pages={1999--2004},
year={2020},
organization={IEEE}
}
```
### Questions
Please email yikuanli2018@u.northwestern.edu
|