File size: 2,471 Bytes
bbbb7ca 4d2aa5a 1f14f18 f005257 bbbb7ca 4d2aa5a 7ca0aef 4d2aa5a 7ca0aef |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
---
license: bigscience-openrail-m
tags:
- split and rephrase
widget:
- text: >-
Cystic Fibrosis (CF) is an autosomal recessive disorder that affects
multiple organs, which is common in the Caucasian population,
symptomatically affecting 1 in 2500 newborns in the UK, and more than 80,000
individuals globally.
datasets:
- wiki_split
- web_split
language:
- en
---
# T5 model for splitting complex sentences to simple sentences in English
Split-and-rephrase is the task of splitting a complex input sentence into shorter sentences while preserving meaning. (Narayan et al., 2017)
E.g.:
```
Cystic Fibrosis (CF) is an autosomal recessive disorder that affects multiple organs,
which is common in the Caucasian population, symptomatically affecting 1 in 2500 newborns in the UK,
and more than 80,000 individuals globally.
```
could be split into
```
Cystic Fibrosis is an autosomal recessive disorder that affects multiple organs.
```
```
Cystic Fibrosis is common in the Caucasian population.
```
```
Cystic Fibrosis affects 1 in 2500 newborns in the UK.
```
```
Cystic Fibrosis affects more than 80,000 individuals globally.
```
## How to use it in your code:
```python
from transformers import T5Tokenizer, T5ForConditionalGeneration
checkpoint="unikei/t5-base-split-and-rephrase"
tokenizer = T5Tokenizer.from_pretrained(checkpoint)
model = T5ForConditionalGeneration.from_pretrained(checkpoint)
complex_sentence = "Cystic Fibrosis (CF) is an autosomal recessive disorder that \
affects multiple organs, which is common in the Caucasian \
population, symptomatically affecting 1 in 2500 newborns in \
the UK, and more than 80,000 individuals globally."
complex_tokenized = tokenizer(complex_sentence,
padding="max_length",
truncation=True,
max_length=256,
return_tensors='pt')
simple_tokenized = model.generate(complex_tokenized['input_ids'], attention_mask = complex_tokenized['attention_mask'], max_length=256, num_beams=5)
simple_sentences = tokenizer.batch_decode(simple_tokenized, skip_special_tokens=True)
print(simple_sentences)
"""
Output:
Cystic Fibrosis is an autosomal recessive disorder that affects multiple organs. Cystic Fibrosis is common in the Caucasian population. Cystic Fibrosis affects 1 in 2500 newborns in the UK. Cystic Fibrosis affects more than 80,000 individuals globally.
"""
```
|