Bag Of Words (BOW) is a simple and typical encoding for making statistical models discover patterns in language However BOW is a lossy compression that eliminates a very important feature of text: order
This model is trained to learn the most probable order of an unordered token sequence, using a subset of the c4 dataset, and can thus be seen as a "bag-of-words decoder".
Currently, it does not perform well. I'm planning to re-train on a larger subset of c4 later (after may).
How to run:
from transformers import T5ForConditionalGeneration, T5Tokenizer tokenizer = T5Tokenizer.from_pretrained("marksverdhei/t5-deshuffle") model = T5ForConditionalGeneration.from_pretrained("marksverdhei/t5-deshuffle") prompt = ' brown dog fox jumped lazy over quick the the ' ids = tokenizer(prompt, return_tensors="pt").input_ids generated_tokens, = model.generate(ids) print(tokenizer.decode(generated_tokens, skip_special_tokens=True))
- Downloads last month