bert-rubriques

CamemBERT-base fine-tuned to annotate the section of French news article. 6 categories are allowed: politics, society, international, sport, culture and economics. An article relating to political, social or economic events outside France should be classified in international - so that politics contains only French politics, etc. Culture has no borders, so a Bob Dylan album should end up in the culture section. Economics includes both business news (e.g., some company making profit) or macroeconomics news (e.g. a recession). Politics is strictly about political actors -- even though everything is political, of course.

This has been developed for the Gallicagram project, to avoid 'structure effects': a word could be more present in a corpus or a time period because the section where it tends to appear is larger, and we might want to control for that. More generally, press corpora are really meta-corpora, and lexicometrics/cultural analytics shouldn't mix sports with politics.

This model has been trained on 960 articles of Le Monde archives (1945-...) with the wonderful ActiveTigger (s/o to Émilien Schultz and his colleagues), using active learning to speed up learning. It is meant to work more broadly on any post-war French news articles - and in our experience, it does work. It is probably not as accurate outside of high-brow printed press - I would not use it on tabloid, social media or TV transcript.

Training details

• Base model: camembert/camembert-base
• Epochs: 5
• Batch size: 4 (gradient accumulation: 4)
• Learning rate: 3e-5
• Weight decay: 0.01
• Max sequence length: 512
• Training examples: 960

Evaluation

On a test-set of 241 examples, stratified for time (between 1945 and 2024), overall accuracy is 81.7%, F1 macro 0.813, F1 weighted 0.818. Sports and international are, understandably, better classified (see below). Note that the categories are quite porous, so most errors are ambiguous cases, and this level of accuracy should be understood as a lower bound.

Confusion matrix:

true \ pred	culture	international	politique	société	sport	économie
culture	34	2	1	4	0	1
international	3	66	2	0	0	3
politique	0	3	23	5	0	2
société	5	1	4	39	0	2
sport	0	0	0	0	18	0
économie	0	0	0	5	1	17