thealper2/mt5-small-hashtag-generation

mt5-small fine-tuned for multi-hashtag generation

Sequence-to-sequence model that generates a set of hashtags for a social-media post. Fine-tuned from google/mt5-small on sm4rtdev/x_dataset_240.

The task is framed as multi-label semantic generation via seq2seq, not multi-label classification: the tag vocabulary is open, the number of tags per post is variable, and the decoder emits tags as text.

Task format

Input:  Generate relevant hashtags for the following social media post:

        <post text, with any existing hashtags removed>

Output: #hashtag1 #hashtag2 #hashtag3

Training data

Source dataset: sm4rtdev/x_dataset_240 (single train split, X/Twitter posts, predominantly crypto/finance topics, mostly English with a multilingual tail).

Stage	Examples
Raw rows	306,256
Removed by cleaning rules	39,102
Duplicate (post, hashtags) pairs removed	22,348
Usable examples	244,806
Train / validation / test	195,853 / 24,436 / 24,517

Property	Value
Unique hashtags	77,022
Total hashtag occurrences	1,322,438
Mean hashtags per example	5.40
Median hashtags per example	4
Mean post length (words)	24.1

Preprocessing

In this dataset the target hashtags are literally present inside the post text for ~84% of rows. Training on the raw text would therefore reduce the task to substring copying, so hashtags are stripped from the model input. The remaining pipeline:

• Drop rows whose post has fewer than 4 words after stripping.
• Drop rows with 0 hashtags, or more than 20 (keyword-stuffing spam).
• Normalise hashtags (##tag, ＃tag, trailing punctuation) to canonical #tag.
• De-duplicate tags within an example, case-insensitively, preserving order and casing.
• De-duplicate (post, hashtag-set) pairs across the corpus.
• Split 80%/10%/10% with seed 42, grouped by post text so that no post appears in two splits.

URLs, mentions, emojis and punctuation are preserved.

Training configuration

Hyperparameter	Value
Base model	google/mt5-small (~300M parameters)
Learning rate	0.0001
Epochs	3
Training examples	100,000 (seeded random subset)
Batch size (per device)	16
Gradient accumulation	1
Effective batch size	16
Optimizer	AdamW (fused)
Weight decay	0.01
Warmup ratio	0.05
LR schedule	linear
Max grad norm	1.0
Precision	bf16
Gradient checkpointing	False
Max source length	192 tokens
Max target length	64 tokens
Seed	42
Model selection	best hashtag_f1 on validation

fp16 is not supported. mT5 was pre-trained in bfloat16 and overflows to NaN under fp16 mixed precision. Use bf16 or fp32.

Best validation hashtag_f1: 0.4166

Evaluation

Hashtag prediction is a set task: #ai #python and #python #ai are the same answer. The primary metrics are therefore set-based, computed after parsing both prediction and reference into case-insensitive tag sets. ROUGE is reported as a secondary, order-sensitive signal.

Set-based metrics (primary)

Metric	Score
Precision (macro)	0.5288
Recall (macro)	0.3853
F1 (macro)	0.4135
Precision (micro)	0.5722
Recall (micro)	0.3690
F1 (micro)	0.4486
Exact set match	0.0839
Jaccard similarity	0.3205

Generation metrics (secondary)

Metric	Score
ROUGE-1	0.4130
ROUGE-2	0.1380
ROUGE-L	0.3866

Average hashtags generated: 3.46 (reference: 5.37).

Usage

import re
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "thealper2/mt5-small-hashtag-generation"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

PROMPT = "Generate relevant hashtags for the following social media post:\n\n{}"

def generate_hashtags(text: str, num_beams: int = 4) -> list[str]:
    # Match training: strip any hashtags already in the post.
    post = re.sub(r"#[\w\u0080-\uFFFF]+", " ", text)
    post = re.sub(r"\s+", " ", post).strip()

    inputs = tokenizer(
        PROMPT.format(post),
        max_length=192,
        truncation=True,
        return_tensors="pt",
    )
    output = model.generate(
        **inputs,
        num_beams=num_beams,
        max_new_tokens=64,
        no_repeat_ngram_size=3,
        length_penalty=1.0,
        early_stopping=True,
    )
    decoded = tokenizer.decode(output[0], skip_special_tokens=True)

    seen, tags = set(), []
    for tag in re.findall(r"#[\w\u0080-\uFFFF]+", decoded):
        if tag.casefold() not in seen:
            seen.add(tag.casefold())
            tags.append(tag)
    return tags

print(generate_hashtags("Artificial intelligence is changing software development."))

Generation defaults

Parameter	Value	Rationale
num_beams	4	Best measured F1 on validation; 8 is no better.
max_new_tokens	64	Covers >99% of reference tag lists.
no_repeat_ngram_size	3	Best measured F1; disabling it costs ~1.8 points.
length_penalty	1.0	Precision/recall dial: 0.6 gives fewer tags, 2.0 more.
early_stopping	True	Stops once all beams finish.

These defaults are stored in the model's generation_config, so a plain model.generate(**inputs) already behaves correctly.

Do not use greedy decoding. It degenerates on this task, emitting ~12 tags per post against a reference mean of 5.4 (F1 0.21 vs 0.35 with beams).

The tag count is not forced: the model decides how many tags to emit, since the data has a variable number per post.

Limitations

• Domain skew. The training data is dominated by crypto/finance posts (#bitcoin, #crypto, #btc are the most frequent tags). Predictions on other domains fall back to generic tags.
• Multilingual coverage is uneven. Most posts are English; other scripts (CJK, Arabic, Cyrillic, Thai) appear in a small minority of rows, so mT5's multilingual capacity is only lightly exercised.
• Noisy supervision. Author-chosen hashtags are not an exhaustive or objective label set. A "wrong" prediction is often a reasonable tag the original author simply did not use, so precision/recall understate quality.
• Time-bound vocabulary. Tags reflect topics and tickers current at collection time and will drift.
• Spam filtering. Posts with more than 20 hashtags were excluded, so the model will not reproduce keyword-stuffing behaviour.
• Not a safety-filtered model. It reproduces whatever tag conventions, including promotional ones, exist in the source data.

Reproduction

make install
make prepare
make train
make evaluate