File size: 4,588 Bytes
7ff9bee 471604c 7ff9bee 471604c cfa422c 471604c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 |
---
license: mit
datasets:
- jmhessel/newyorker_caption_contest
language:
- en
tags:
- nyc
- llama2
widget:
- text: "This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller.\ncaption: Can I interest you in opening an offshore account?\nexplanation of the caption:\n"
example_title: "Training prompt format"
- text: "In this task, you will see a description of an uncanny situation. Then, you will see a joke that was written about the situation. Explain how the joke relates to the situation and why it is funny.\n###\nThis scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller.\ncaption: Can I interest you in opening an offshore account?\nexplanation of the caption:\n"
example_title: "Paper prompt format"
- text: "This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller.\ncaption: Can I interest you in opening an offshore account?\nthe caption is funny because"
example_title: "Suggested prompt format"
---
# nyrkr-joker-llama
*New Yorker* cartoon description and caption -> attempt at a joke explanation
Technical details:
- Based on LLaMa2-7b-hf (version 2, 7B params)
- Used [QLoRA](https://github.com/artidoro/qlora/blob/main/qlora.py) to fine-tune on [1.2k rows of New Yorker caption contest](https://huggingface.co/datasets/jmhessel/newyorker_caption_contest)
- Merged LLaMa2 with the adapter weights (from checkpoint step=160, epoch=2.7)
## Prompt options
[The original paper](https://arxiv.org/abs/2209.06293), Figure 10 uses this format for joke explanations:
`In this task, you will see a description of an uncanny situation. Then, you will see a joke that was written about the situation. Explain how the joke relates to the situation and why it is funny.
###
{few-shot examples separated by ###, newline after "explanation of the caption:"}
This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller.
caption: Can I interest you in opening an offshore account?
explanation of the caption:
`
In training, I used just the individual example:
`This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller.
caption: Can I interest you in opening an offshore account?
explanation of the caption:\n`
In inference, I had some better results with a more natural prompt (no newline or space at end)
`This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller.
caption: Can I interest you in opening an offshore account?
the caption is funny because`
## Training script
Trained on a V100
```
git clone https://github.com/artidoro/qlora
cd qlora
pip3 install -r requirements.txt --quiet
! cd qlora && python qlora.py \
--model_name_or_path ../llama-2-7b-hf \
--output_dir ../thatsthejoke \
--logging_steps 20 \
--save_strategy steps \
--data_seed 42 \
--save_steps 80 \
--save_total_limit 10 \
--evaluation_strategy steps \
--max_new_tokens 64 \
--dataloader_num_workers 1 \
--group_by_length \
--logging_strategy steps \
--remove_unused_columns False \
--do_train \
--lora_r 64 \
--lora_alpha 16 \
--lora_modules all \
--double_quant \
--quant_type nf4 \
--bits 4 \
--warmup_ratio 0.03 \
--lr_scheduler_type constant \
--gradient_checkpointing \
--dataset /content/nycaptions.jsonl \
--dataset_format 'self-instruct' \
--source_max_len 16 \
--target_max_len 512 \
--per_device_train_batch_size 1 \
--gradient_accumulation_steps 16 \
--max_steps 250 \
--eval_steps 187 \
--learning_rate 0.0002 \
--adam_beta2 0.999 \
--max_grad_norm 0.3 \
--lora_dropout 0.1 \
--weight_decay 0.0 \
--seed 0
```
|