faodl/agri-vegetable_oils-classifier

AMIS Commodity Classifier

This model repository contains artifacts from an AMIS commodity relevance classifier training run. It includes the Transformer model, any configured TF-IDF or sentence-embedding baselines, prediction files, and the training report.

• Dataset: faodl/amis-agri-vegetable_oils
• Dataset subset: ``
• Dataset revision: main
• Text column: chunk_text
• Label column: label
• Transformer: FacebookAI/xlm-roberta-base
• Generated at: 2026-06-09T16:39:33.521615+00:00

Dataset Summary

Split	Rows	Label 0	Label 1	Unique groups	Mean text length
train	5382	4040	1342	2395	706.0
validation	1186	900	286	513	711.4
test	1082	847	235	514	699.9

Threshold Comparison on Validation Split

Validation metrics document threshold selection and tuning behavior; test metrics remain the primary estimate of out-of-sample performance.

Model	Threshold	Accuracy	Precision	Recall	F1	ROC AUC	Average precision
logistic_tfidf	0.500	0.879	0.738	0.769	0.753	0.929	0.804
logistic_tfidf	0.486	0.880	0.732	0.794	0.762	0.929	0.804
xgboost_tfidf	0.500	0.865	0.797	0.591	0.679	0.926	0.816
xgboost_tfidf	0.357	0.882	0.763	0.741	0.752	0.926	0.816
embedding-logistic_sentence_embeddings	0.500	0.847	0.632	0.881	0.736	0.925	0.802
embedding-logistic_sentence_embeddings	0.620	0.871	0.711	0.783	0.745	0.925	0.802
embedding-svm_sentence_embeddings	0.500	0.853	0.767	0.563	0.649	0.918	0.787
embedding-svm_sentence_embeddings	0.357	0.865	0.685	0.815	0.744	0.918	0.787
embedding-lightgbm_sentence_embeddings	0.500	0.881	0.760	0.741	0.750	0.930	0.829
embedding-lightgbm_sentence_embeddings	0.318	0.879	0.723	0.804	0.762	0.930	0.829
transformer	0.500	0.896	0.762	0.829	0.794	0.940	0.872
transformer	0.625	0.910	0.816	0.808	0.812	0.940	0.872

Threshold Comparison on Test Split

Model	Threshold	Accuracy	Precision	Recall	F1	ROC AUC	Average precision
logistic_tfidf	0.500	0.896	0.766	0.753	0.760	0.937	0.817
logistic_tfidf	0.486	0.896	0.756	0.766	0.761	0.937	0.817
xgboost_tfidf	0.500	0.878	0.772	0.621	0.689	0.923	0.785
xgboost_tfidf	0.357	0.882	0.738	0.706	0.722	0.923	0.785
embedding-logistic_sentence_embeddings	0.500	0.835	0.583	0.834	0.687	0.919	0.763
embedding-logistic_sentence_embeddings	0.620	0.857	0.648	0.745	0.693	0.919	0.763
embedding-svm_sentence_embeddings	0.500	0.871	0.768	0.579	0.660	0.908	0.758
embedding-svm_sentence_embeddings	0.357	0.841	0.611	0.740	0.669	0.908	0.758
embedding-lightgbm_sentence_embeddings	0.500	0.879	0.745	0.672	0.707	0.921	0.793
embedding-lightgbm_sentence_embeddings	0.318	0.872	0.689	0.753	0.720	0.921	0.793
transformer	0.500	0.891	0.741	0.766	0.753	0.930	0.820
transformer	0.625	0.894	0.759	0.749	0.754	0.930	0.820

Confusion Matrices on Test Split

Rows are true labels and columns are predicted labels.

logistic_tfidf at threshold 0.500

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	793	54
RELEVANT	58	177

logistic_tfidf at threshold 0.486

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	789	58
RELEVANT	55	180

xgboost_tfidf at threshold 0.500

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	804	43
RELEVANT	89	146

xgboost_tfidf at threshold 0.357

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	788	59
RELEVANT	69	166

embedding-logistic_sentence_embeddings at threshold 0.500

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	707	140
RELEVANT	39	196

embedding-logistic_sentence_embeddings at threshold 0.620

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	752	95
RELEVANT	60	175

embedding-svm_sentence_embeddings at threshold 0.500

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	806	41
RELEVANT	99	136

embedding-svm_sentence_embeddings at threshold 0.357

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	736	111
RELEVANT	61	174

embedding-lightgbm_sentence_embeddings at threshold 0.500

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	793	54
RELEVANT	77	158

embedding-lightgbm_sentence_embeddings at threshold 0.318

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	767	80
RELEVANT	58	177

transformer at threshold 0.500

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	784	63
RELEVANT	55	180

transformer at threshold 0.625

True / Predicted	NOT_RELEVANT	RELEVANT
NOT_RELEVANT	791	56
RELEVANT	59	176

Validation-Tuned Thresholds

• logistic_tfidf: threshold 0.486 (validation F1 0.762); test F1 change vs 0.5: +0.001.
• xgboost_tfidf: threshold 0.357 (validation F1 0.752); test F1 change vs 0.5: +0.033.
• embedding-logistic_sentence_embeddings: threshold 0.620 (validation F1 0.745); test F1 change vs 0.5: +0.007.
• embedding-svm_sentence_embeddings: threshold 0.357 (validation F1 0.744); test F1 change vs 0.5: +0.009.
• embedding-lightgbm_sentence_embeddings: threshold 0.318 (validation F1 0.762); test F1 change vs 0.5: +0.013.
• transformer: threshold 0.625 (validation F1 0.812); test F1 change vs 0.5: +0.001.

Artifacts

• logistic_tfidf: /content/agri-vegetable_oils-classifier/baselines/logistic
• xgboost_tfidf: /content/agri-vegetable_oils-classifier/baselines/xgboost
• embedding-logistic_sentence_embeddings: /content/agri-vegetable_oils-classifier/baselines/embedding-logistic
• embedding-svm_sentence_embeddings: /content/agri-vegetable_oils-classifier/baselines/embedding-svm
• embedding-lightgbm_sentence_embeddings: /content/agri-vegetable_oils-classifier/baselines/embedding-lightgbm
• transformer: /content/agri-vegetable_oils-classifier/transformer

Inference

Install the runtime dependencies:

pip install transformers torch huggingface_hub pandas joblib scikit-learn xgboost lightgbm

Transformer

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

MODEL_ID = "faodl/agri-vegetable_oils-classifier"

texts = [
    "Rice export prices increased after new procurement rules were announced.",
    "The finance ministry released its monthly fuel tax bulletin.",
]

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, subfolder="transformer")
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID, subfolder="transformer")
threshold = float(getattr(model.config, "threshold", 0.5))

encoded = tokenizer(
    texts,
    truncation=True,
    padding=True,
    max_length=256,
    return_tensors="pt",
)

with torch.no_grad():
    logits = model(**encoded).logits
    probabilities = torch.softmax(logits, dim=-1)[:, 1].tolist()

for text, probability in zip(texts, probabilities):
    label = model.config.id2label[int(probability >= threshold)]
    print({"text": text, "probability_positive": probability, "label": label})

TF-IDF Baselines

Available baseline names in this run: "logistic", "xgboost".

import json
import joblib
from huggingface_hub import hf_hub_download

MODEL_ID = "faodl/agri-vegetable_oils-classifier"
BASELINE = "logistic"

texts = [
    "Maize production forecasts were revised after delayed rains.",
    "The central bank published new exchange rate statistics.",
]

model_path = hf_hub_download(
    repo_id=MODEL_ID,
    repo_type="model",
    filename=f"baselines/{BASELINE}/{BASELINE}_tfidf.joblib",
)
report_path = hf_hub_download(
    repo_id=MODEL_ID,
    repo_type="model",
    filename="report.json",
)

pipeline = joblib.load(model_path)
with open(report_path, encoding="utf-8") as handle:
    report = json.load(handle)

threshold = next(
    result["validation_best_threshold"]["threshold"]
    for result in report["results"]
    if result["model_type"] == f"{BASELINE}_tfidf"
)

probabilities = pipeline.predict_proba(texts)[:, 1]
for text, probability in zip(texts, probabilities):
    label = "RELEVANT" if probability >= threshold else "NOT_RELEVANT"
    print({"text": text, "probability_positive": float(probability), "label": label})

Sentence-Embedding Baselines

Available embedding baseline names in this run: "embedding-logistic", "embedding-svm", "embedding-lightgbm".

import joblib
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel, AutoTokenizer

MODEL_ID = "faodl/agri-vegetable_oils-classifier"
BASELINE = "embedding-logistic"

texts = [
    "Wheat export inspections rose as demand from importers increased.",
    "The sports ministry announced a new stadium renovation plan.",
]

model_path = hf_hub_download(
    repo_id=MODEL_ID,
    repo_type="model",
    filename=f"baselines/{BASELINE}/{BASELINE}.joblib",
)
artifact = joblib.load(model_path)
tokenizer = AutoTokenizer.from_pretrained(artifact["embedding_model_name"])
encoder = AutoModel.from_pretrained(artifact["embedding_model_name"])
encoder.eval()

encoded_batches = []
batch_size = artifact.get("embedding_batch_size", 64)
for start in range(0, len(texts), batch_size):
    batch_texts = texts[start : start + batch_size]
    inputs = tokenizer(
        batch_texts,
        padding=True,
        truncation=True,
        max_length=artifact.get("embedding_max_length", 256),
        return_tensors="pt",
    )
    with torch.no_grad():
        outputs = encoder(**inputs)
    token_embeddings = outputs.last_hidden_state
    attention_mask = inputs["attention_mask"].unsqueeze(-1).to(token_embeddings.dtype)
    embeddings = (token_embeddings * attention_mask).sum(dim=1)
    embeddings = embeddings / attention_mask.sum(dim=1).clamp(min=1e-9)
    if artifact.get("normalize_embeddings", True):
        embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
    encoded_batches.append(embeddings)
embeddings = torch.cat(encoded_batches).numpy()
probabilities = artifact["classifier"].predict_proba(embeddings)[:, 1]
threshold = artifact["validation_best_threshold"]["threshold"]

for text, probability in zip(texts, probabilities):
    label = "RELEVANT" if probability >= threshold else "NOT_RELEVANT"
    print({"text": text, "probability_positive": float(probability), "label": label})

Files

• REPORT.md: Markdown report for this training run.
• report.json: Machine-readable report containing metrics and thresholds.
• transformer/: Fine-tuned Transformer artifacts, when Transformer training is enabled.
• baselines/: TF-IDF and sentence-embedding baseline artifacts, when baseline training is enabled.
• */validation_predictions.csv and */test_predictions.csv: Split-level predictions.