faodl/amis-unified-multilabel-xlmr

AMIS Unified Multilabel XLM-R Classifier

This is the best checkpoint from the AMIS unified multilabel classifier training run. It predicts the thematic or commodity relevance of a news chunk across 10 labels in one forward pass.

Labels

topic_index	topic	source_dataset
0	utilization	faodl/amis-agri-utilization
1	wheat	faodl/amis-agri-wheat
2	stocks	faodl/amis-agri-stocks
3	maize_corn	faodl/amis-agri-maize_corn
4	rice	faodl/amis-agri-rice
5	vegetable_oils	faodl/amis-agri-vegetable_oils
6	trade	faodl/amis-agri-trade
7	fertilizers	faodl/amis-agri-fertilizers
8	soybeans	faodl/amis-agri-soybeans
9	production-v03	faodl/amis-agri-production-v03

Training Dataset

The model was trained on the unified dataset faodl/amis-agri-unified. The dataset is created by loading the source binary datasets above, normalizing their text and binary labels, grouping duplicate texts within each topic, outer-joining all topics by exact text, and using -1 for labels that were not annotated for a given topic. The split used for this run is the exact split produced by seed=42.

Split Label Coverage

split	topic	valid_labels	positive_labels	negative_labels
train	utilization	9757	805	8952
train	wheat	3594	1362	2232
train	stocks	4869	411	4458
train	maize_corn	4801	960	3841
train	rice	4774	959	3815
train	vegetable_oils	5359	1296	4063
train	trade	4784	1620	3164
train	fertilizers	4731	1132	3599
train	soybeans	4789	933	3856
train	production-v03	6071	1158	4913
validation	utilization	2075	161	1914
validation	wheat	772	334	438
validation	stocks	1068	95	973
validation	maize_corn	1007	178	829
validation	rice	1048	195	853
validation	vegetable_oils	1145	299	846
validation	trade	1014	361	653
validation	fertilizers	1042	262	780
validation	soybeans	1028	190	838
validation	production-v03	1227	225	1002
test	utilization	2091	165	1926
test	wheat	777	328	449
test	stocks	1029	75	954
test	maize_corn	1030	216	814
test	rice	1026	203	823
test	vegetable_oils	1146	268	878
test	trade	1027	364	663
test	fertilizers	1049	287	762
test	soybeans	1036	199	837
test	production-v03	1307	275	1032

Split Label Totals

split	valid_labels
test	11518
train	53529
validation	11426

Training Parameters

parameter	value
base_model	FacebookAI/xlm-roberta-base
max_length	256
train_batch_size	8
eval_batch_size	16
gradient_accumulation_steps	2
num_epochs	3
learning_rate	2e-05
weight_decay	0.01
warmup_ratio	0.06
validation_size	0.15
test_size	0.15
seed	42

Validation Thresholds

These thresholds were selected independently per label on the validation split by maximizing positive-label F1.

topic	threshold	precision	recall	f1	support
utilization	0.3020	0.6906	0.7764	0.7310	2075
wheat	0.8233	0.9164	0.8862	0.9011	772
stocks	0.5437	0.6545	0.7579	0.7024	1068
maize_corn	0.7905	0.8976	0.8371	0.8663	1007
rice	0.5802	0.8861	0.9179	0.9018	1048
vegetable_oils	0.2602	0.8007	0.8194	0.8099	1145
trade	0.0774	0.7019	0.9197	0.7962	1014
fertilizers	0.6250	0.9300	0.9122	0.9210	1042
soybeans	0.7527	0.8783	0.8737	0.8760	1028
production-v03	0.5786	0.7360	0.8178	0.7747	1227

Validation Metrics

Macro F1 at threshold 0.5: 0.8129
Macro F1 with per-label best-F1 thresholds: 0.8280

metric_set	topic	threshold	precision	recall	f1	support
validation_default_0_5	utilization	0.5000	0.6941	0.7329	0.7130	2075
validation_default_0_5	wheat	0.5000	0.8616	0.9132	0.8866	772
validation_default_0_5	stocks	0.5000	0.6372	0.7579	0.6923	1068
validation_default_0_5	maize_corn	0.5000	0.8315	0.8596	0.8453	1007
validation_default_0_5	rice	0.5000	0.8696	0.9231	0.8955	1048
validation_default_0_5	vegetable_oils	0.5000	0.8352	0.7291	0.7786	1145
validation_default_0_5	trade	0.5000	0.7855	0.7507	0.7677	1014
validation_default_0_5	fertilizers	0.5000	0.9122	0.9122	0.9122	1042
validation_default_0_5	soybeans	0.5000	0.8535	0.8895	0.8711	1028
validation_default_0_5	production-v03	0.5000	0.7154	0.8267	0.7670	1227
validation_best_f1_threshold	utilization	0.3020	0.6906	0.7764	0.7310	2075
validation_best_f1_threshold	wheat	0.8233	0.9164	0.8862	0.9011	772
validation_best_f1_threshold	stocks	0.5437	0.6545	0.7579	0.7024	1068
validation_best_f1_threshold	maize_corn	0.7905	0.8976	0.8371	0.8663	1007
validation_best_f1_threshold	rice	0.5802	0.8861	0.9179	0.9018	1048
validation_best_f1_threshold	vegetable_oils	0.2602	0.8007	0.8194	0.8099	1145
validation_best_f1_threshold	trade	0.0774	0.7019	0.9197	0.7962	1014
validation_best_f1_threshold	fertilizers	0.6250	0.9300	0.9122	0.9210	1042
validation_best_f1_threshold	soybeans	0.7527	0.8783	0.8737	0.8760	1028
validation_best_f1_threshold	production-v03	0.5786	0.7360	0.8178	0.7747	1227

Test Metrics

The tuned test metrics use thresholds selected on the validation split.

Macro F1 at threshold 0.5: 0.8160
Macro F1 with validation-selected per-label thresholds: 0.8181

metric_set	topic	threshold	precision	recall	f1	support
test_default_0_5	utilization	0.5000	0.6867	0.6909	0.6888	2091
test_default_0_5	wheat	0.5000	0.8997	0.9024	0.9011	777
test_default_0_5	stocks	0.5000	0.5938	0.7600	0.6667	1029
test_default_0_5	maize_corn	0.5000	0.8702	0.8380	0.8538	1030
test_default_0_5	rice	0.5000	0.9050	0.8916	0.8983	1026
test_default_0_5	vegetable_oils	0.5000	0.7727	0.7612	0.7669	1146
test_default_0_5	trade	0.5000	0.7771	0.7473	0.7619	1027
test_default_0_5	fertilizers	0.5000	0.8893	0.9233	0.9060	1049
test_default_0_5	soybeans	0.5000	0.8520	0.9548	0.9005	1036
test_default_0_5	production-v03	0.5000	0.7872	0.8473	0.8161	1307
test_validation_best_f1_threshold	utilization	0.3020	0.6760	0.7333	0.7035	2091
test_validation_best_f1_threshold	wheat	0.8233	0.9433	0.8628	0.9013	777
test_validation_best_f1_threshold	stocks	0.5437	0.6064	0.7600	0.6746	1029
test_validation_best_f1_threshold	maize_corn	0.7905	0.9176	0.7731	0.8392	1030
test_validation_best_f1_threshold	rice	0.5802	0.9045	0.8867	0.8955	1026
test_validation_best_f1_threshold	vegetable_oils	0.2602	0.7148	0.8134	0.7609	1146
test_validation_best_f1_threshold	trade	0.0774	0.6909	0.8599	0.7662	1027
test_validation_best_f1_threshold	fertilizers	0.6250	0.9000	0.9094	0.9047	1049
test_validation_best_f1_threshold	soybeans	0.7527	0.9055	0.9146	0.9100	1036
test_validation_best_f1_threshold	production-v03	0.5786	0.8105	0.8400	0.8250	1307

Inference

Use sigmoid over the model logits. For default predictions, use threshold 0.5 for every label. For tuned predictions, use the per-label thresholds shown above in label order.