language:
- en
pipeline_tag: text2text-generation
inference: false
license: mit
ViPE-M-CTX7
ViPE: Visualize Pretty-much Everything, is the first automated model for translating any arbitrary piece of text into a visualizable prompt. It helps any text-to-image model in figurative or non-lexical language visualizations. It has been shown to be more robust than GPT3.5 Turbo (ChatGPT) in generating depictable and semantically meaningful prompts.
Model Description
- Developed by: Computer Graphics Group, University of Tuebingen
- Model type: Auto-Regressive
- Language: English
- License: MIT
- Based on: GPT2-Medium
- Versions: ViPE-M-CTX7 (355M parameters) and ViPE-S-CTX7 (117M parameters)
Model Sources
- Repository: Github
- Paper: ViPE: Visualise Pretty-much Everything (Outstanding Paper Award at EMNLP 2023)
Down Stream Applications
ViPE provides a robust backbone for many practical applications such as music video generation and creative writing.
Music Video Genrations
- Repository: Github
- Demo: ViPE Videos
Creative Writing
- Demo: Hugging Face Playground
Summagery: Document Summarization through Images
- Demo: Hugging Face Playground
Direct Use
You can directly use the model to generate detailed prompts for any arbitrary text.
from transformers import GPT2LMHeadModel, GPT2Tokenizer
def generate(text, model, tokenizer,device,do_sample,top_k=100, epsilon_cutoff=.00005, temperature=1):
#mark the text with special tokens
text=[tokenizer.eos_token + i + tokenizer.eos_token for i in text]
batch=tokenizer(text, padding=True, return_tensors="pt")
input_ids = batch["input_ids"].to(device)
attention_mask = batch["attention_mask"].to(device)
#how many new tokens to generate at max
max_prompt_length=50
generated_ids = model.generate(input_ids=input_ids,attention_mask=attention_mask, max_new_tokens=max_prompt_length, do_sample=do_sample,top_k=top_k, epsilon_cutoff=epsilon_cutoff, temperature=temperature)
#return only the generated prompts
pred_caps = tokenizer.batch_decode(generated_ids[:, -(generated_ids.shape[1] - input_ids.shape[1]):], skip_special_tokens=True)
return pred_caps
device='cpu'
model = GPT2LMHeadModel.from_pretrained('fittar/ViPE-M-CTX7')
model.to(device)
#ViPE-M's tokenizer is identical to that of GPT2-Medium
tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')
tokenizer.pad_token = tokenizer.eos_token
# A list of abstract/figurative or any arbitrary combinations of keywords
texts=['lalala', 'I wanna start learning', 'free your mind; you will see the other side of life', 'brave; fantasy']
prompts=generate(texts,model,tokenizer,do_sample=True,device=device)
for t,p in zip(texts,prompts):
print('{} --> {}'.format(t,p))
lalala --> A group of people chanting "la la la" around a bonfire on a beach at night
I wanna start learning --> A child sitting in a library surrounded by books, excitedly flipping through pages of a book
free your mind; you will see the other side of life --> An astronaut floating in space with a sense of floating weightlessness, looking down towards the earth
brave; fantasy --> A brave knight with shining armor fighting a fierce dragon in a misty forest
Recommendations
You can use either a comma or a semicolon to combine multiple keywords. for example ['dark, fantasy, brave'] or ['This is gonna be the best day of my life; do you agree?']. However, a semicolon draws a stronger boundary between the keywords and encourages the model to transfer the last keyword in a given context (previous keywords).
Training Data
- LyricCanvas dataset: a synthetically generated dataset based on lyrics and synthetically generated prompts
Training Procedure
ViPE has been trained in the standard auto-regressive procedure: given a line (or lines) of lyrics as a prefix, the objective is to generate a plausible prompt that is both despicable and semantically related to the given lyric(c). The loss function does not include the tokens corresponding to the lyrics. So ViPE never generates any original lyrics and only learns to generate visually related prompts.
Evaluation
In all of the following evaluations, ViPE consistently demonstrates its robustness compared to ChatGPT and achieves performance that is competitive with that of human experts.
- Intrinsic evaluations
- General understanding of figurative language using Fig-QA dataset
- Extrinsic evaluations
- Image-text Retrieval on the HAIVMet dataset
- Emotion visualizations: How well does ViPE transfer emotionally charged tweets into a depictable description of a scene in comparison with ChatGPT. The Emotion dataset is utilized.
- Human evaluations
- We conducted a user study involving 30 native English-speaking participants aged between 20 and 40. Participants were presented with 3 images and a metaphor from the HAIVMet dataset. They were asked to select the images that matches the metaphor the best. The images were generated using prompts from ViPE, ChatGPT, and human experts (HAIVMet).
Citation
If you find ViPE useful, please cite our paper.
@inproceedings{shahmohammadi-etal-2023-vipe,
title = "{V}i{PE}: Visualise Pretty-much Everything",
author = "Shahmohammadi, Hassan and
Ghosh, Adhiraj and
Lensch, Hendrik",
editor = "Bouamor, Houda and
Pino, Juan and
Bali, Kalika",
booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2023",
address = "Singapore",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.emnlp-main.333",
pages = "5477--5494"
}