Update README.md

cfa422c 10 months ago

No virus

4.59 kB

	---
	license: mit
	datasets:
	- jmhessel/newyorker_caption_contest
	language:
	- en
	tags:
	- nyc
	- llama2
	widget:
	- text: "This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller.\ncaption: Can I interest you in opening an offshore account?\nexplanation of the caption:\n"
	example_title: "Training prompt format"
	- text: "In this task, you will see a description of an uncanny situation. Then, you will see a joke that was written about the situation. Explain how the joke relates to the situation and why it is funny.\n###\nThis scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller.\ncaption: Can I interest you in opening an offshore account?\nexplanation of the caption:\n"
	example_title: "Paper prompt format"
	- text: "This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller.\ncaption: Can I interest you in opening an offshore account?\nthe caption is funny because"
	example_title: "Suggested prompt format"
	---

	# nyrkr-joker-llama

	New Yorker cartoon description and caption -> attempt at a joke explanation

	Technical details:
	- Based on LLaMa2-7b-hf (version 2, 7B params)
	- Used [QLoRA](https://github.com/artidoro/qlora/blob/main/qlora.py) to fine-tune on [1.2k rows of New Yorker caption contest](https://huggingface.co/datasets/jmhessel/newyorker_caption_contest)
	- Merged LLaMa2 with the adapter weights (from checkpoint step=160, epoch=2.7)

	## Prompt options

	[The original paper](https://arxiv.org/abs/2209.06293), Figure 10 uses this format for joke explanations:

	`In this task, you will see a description of an uncanny situation. Then, you will see a joke that was written about the situation. Explain how the joke relates to the situation and why it is funny.
	###

	{few-shot examples separated by ###, newline after "explanation of the caption:"}
	This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller.
	caption: Can I interest you in opening an offshore account?
	explanation of the caption:
	`

	In training, I used just the individual example:

	`This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller.
	caption: Can I interest you in opening an offshore account?
	explanation of the caption:\n`

	In inference, I had some better results with a more natural prompt (no newline or space at end)

	`This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller.
	caption: Can I interest you in opening an offshore account?
	the caption is funny because`

	## Training script

	Trained on a V100

	```
	git clone https://github.com/artidoro/qlora
	cd qlora

	pip3 install -r requirements.txt --quiet

	! cd qlora && python qlora.py \
	--model_name_or_path ../llama-2-7b-hf \
	--output_dir ../thatsthejoke \
	--logging_steps 20 \
	--save_strategy steps \
	--data_seed 42 \
	--save_steps 80 \
	--save_total_limit 10 \
	--evaluation_strategy steps \
	--max_new_tokens 64 \
	--dataloader_num_workers 1 \
	--group_by_length \
	--logging_strategy steps \
	--remove_unused_columns False \
	--do_train \
	--lora_r 64 \
	--lora_alpha 16 \
	--lora_modules all \
	--double_quant \
	--quant_type nf4 \
	--bits 4 \
	--warmup_ratio 0.03 \
	--lr_scheduler_type constant \
	--gradient_checkpointing \
	--dataset /content/nycaptions.jsonl \
	--dataset_format 'self-instruct' \
	--source_max_len 16 \
	--target_max_len 512 \
	--per_device_train_batch_size 1 \
	--gradient_accumulation_steps 16 \
	--max_steps 250 \
	--eval_steps 187 \
	--learning_rate 0.0002 \
	--adam_beta2 0.999 \
	--max_grad_norm 0.3 \
	--lora_dropout 0.1 \
	--weight_decay 0.0 \
	--seed 0
	```