jjae/hashtag

modelling

This model is a fine-tuned version of gogamza/kobart-summarization on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.5862

Model description

This model generates hash tag from input text.

Training and evaluation data

This model was trained by the self-instruction process. All data used for fine-tuning this model were generated by chatGPT 3.5.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5.6e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 300
• num_epochs: 50

Training results

Training Loss	Epoch	Step	Validation Loss
1.8136	1.42	500	0.6526
0.4651	2.85	1000	0.5862
0.2643	4.27	1500	0.6752
0.1642	5.7	2000	0.6840
0.1078	7.12	2500	0.7554

Framework versions

• Transformers 4.37.1
• Pytorch 2.1.2+cu118
• Datasets 2.16.1
• Tokenizers 0.15.0

How to Get Started with the Model

Use the code below to get started with the model. You can adjust hyperparameters to fit on your data.

from transformers import PreTrainedTokenizerFast, BartForConditionalGeneration
tokenizer = PreTrainedTokenizerFast.from_pretrained("jjae/kobart-hashtag")
model = BartForConditionalGeneration.from_pretrained("jjae/kobart-hashtag")

def make_tag(text):
  input_ids = tokenizer.encode(text, return_tensors="pt").to(device)
  output = model.generate(input_ids = input_ids, bos_token_id = model.config.bos_token_id,
                          eos_token_id = model.config.eos_token_id, length_penalty = 3.0, max_length = 50, num_beams = 4)
  decoded_output = tokenizer.decode(output[0], skip_special_tokens=True)
  return decoded_output