gpt-news-classifier

A four-class news topic classifier (World / Sports / Business / Sci/Tech), fine-tuned from gpt2 (124M parameters) on a 2,400-example balanced sample of AG News.

It reaches 88.3% accuracy on the full 7,600-example AG News test split. It was trained on only 2% of the training data, in about five minutes, so treat it as a small, cheap baseline rather than a tuned classifier.

Important: pad your inputs to 128 tokens. This model was trained with every input left-padded to exactly 128 tokens, and it only works if you do the same at inference. Feeding it unpadded text (the default for a single string) drops accuracy from about 88.3% to about a third, and it will predict World for almost everything. The usage example below does this correctly. See Usage for why.

Labels

id	label
0	World
1	Sports
2	Business
3	Sci/Tech

Usage

The repo ships the tokenizer, already configured the way the model was trained: the standard gpt2 vocabulary with the pad token set to EOS, left padding, and a 128-token maximum. Just load it:

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained("salmanhadli/gpt-news-classifier").eval()
tokenizer = AutoTokenizer.from_pretrained("salmanhadli/gpt-news-classifier")

texts = [
    "Manchester United defeat Liverpool 3-1 in dramatic Premier League showdown",
    "Federal Reserve raises interest rates by 25 basis points amid inflation concerns",
]

# padding="max_length" is required: it matches how the model was trained.
inputs = tokenizer(texts, return_tensors="pt", truncation=True,
                   padding="max_length", max_length=128)

with torch.no_grad():
    probs = torch.softmax(model(**inputs).logits, dim=-1)

for text, p in zip(texts, probs):
    print(model.config.id2label[p.argmax().item()], f"{p.max():.1%}", "|", text)

The pipeline("text-classification", ...) helper does not pad to 128 by default, so with its defaults it hits the failure described above. Pass the padding arguments explicitly:

from transformers import pipeline

clf = pipeline("text-classification", model=model, tokenizer=tokenizer)
clf("Manchester United defeat Liverpool 3-1 in dramatic Premier League showdown",
    padding="max_length", max_length=128, truncation=True)

Why padding matters here

GPT-2 uses absolute position embeddings, and the default position_ids count from 0 regardless of the attention mask. During training every example was left-padded to 128 tokens, so the real text always sat at the end of the sequence (for a 50-token headline, positions 78–127), and the model learned to read it there. An unpadded input puts the same text at positions 0–49, a position pattern the classifier never saw. The model is not wrong about the text; it is being shown a layout it was never trained on.

On 200 examples from the AG News test split, the same weights scored 83% with padding="max_length" and 33% without, with 182 of 200 predicted as World. The full-split figure above uses padding="max_length".

Training data

AG News (fancyzhx/ag_news), a corpus of news article titles and descriptions in four topics. A balanced subsample was drawn with a fixed seed (42):

split	source	examples	class balance
train	official train (120,000)	2,400	600 per class
validation	official test (7,600)	400	approximately balanced
test	official test (7,600)	1,000	245 / 250 / 257 / 248

Validation and test are disjoint slices of the official test split, drawn from a balanced pool of 1,400. Shuffling before the split means their class counts are near, not exactly, equal. The dataset card declares no license. The original AG's News corpus was assembled for research use, so check the terms before any commercial use.

Training procedure

Full fine-tuning of all layers with the Hugging Face Trainer, in bf16 on a GPU (about 5 minutes; 287 s reported).

Base model	gpt2 (124.4M parameters)
Learning rate	2e-5, linear decay
Batch size	16 (train and eval)
Epochs	3 (450 steps)
Weight decay	0.01
Max sequence length	128, left-padded to max length
Optimizer	AdamW (fused)
Seed	42
Checkpoint selection	best validation weighted-F1 (epoch 3)

Classification head: GPT2ForSequenceClassification, a linear layer on the last non-pad token, newly initialised. pad_token is set to eos_token.

Framework versions: Transformers 5.16.1, PyTorch 2.11.0+cu128, Datasets 4.8.5.

Results

Full AG News test split (n = 7,600)

Accuracy 0.8834, F1 (macro) 0.8836. The split is exactly balanced (1,900 per class). The model never saw any of these examples in training; 1,400 of them were used as the notebook's validation and test slices below.

class	precision	recall	F1	support
World	0.9178	0.8637	0.8899	1,900
Sports	0.9555	0.9721	0.9637	1,900
Business	0.8136	0.8500	0.8314	1,900
Sci/Tech	0.8506	0.8479	0.8492	1,900

Evaluated on CPU in fp32 with padding="max_length", max_length=128.

Notebook test slice (n = 1,000)

Accuracy 0.9060, F1 (macro) 0.9064, loss 0.2882. This slice happened to score higher than the validation set did (0.865, n = 400) and than the full test split above; with samples this small the difference is within normal variation, so prefer the full-split figure.

class	precision	recall	F1	support
World	0.93	0.91	0.92	245
Sports	0.96	0.96	0.96	250
Business	0.87	0.86	0.87	257
Sci/Tech	0.87	0.90	0.88	248

Validation set (n = 400), per epoch

Epoch	Train loss	Val loss	Accuracy	F1 (macro)
1	0.6235	0.5466	0.8125	0.8102
2	0.4349	0.3967	0.8625	0.8607
3	0.4249	0.3977	0.8650	0.8633

Validation loss was flat between epochs 2 and 3 while accuracy edged up, so there is no sign of overfitting within 3 epochs. Longer training and more data were not tested.

Limitations

• Requires exact 128-token left padding at inference. This is the main way to get bad results; see above.
• Small training set. 2,400 examples, 2% of AG News. A model trained on the full 120,000 would do better.
• Business and Sci/Tech are the weakest classes by F1 (see the per-class table); Sports is the strongest.
• Trained on titles plus descriptions from mid-2000s news text. Each AG News example is a headline followed by a short description. Headlines alone, recent events, opinion pieces and non-news text are out of distribution and were not systematically tested.
• Four fixed topics only. Anything outside World, Sports, Business and Sci/Tech (health, entertainment, politics as its own category) is forced into one of the four.
• Calibration was not measured. Use the scores as a ranking, not as the probability of being correct.
• English only. Inherits the biases of GPT-2's pretraining data and of the AG News sources.

Citation

The dataset comes from:

@inproceedings{Zhang2015CharacterlevelCN,
  title={Character-level Convolutional Networks for Text Classification},
  author={Xiang Zhang and Junbo Zhao and Yann LeCun},
  booktitle={NIPS},
  year={2015}
}

License

MIT, following the base model. The training dataset's license is not declared; see Training data.