foodseg-models
Trained checkpoints for pixel-level food segmentation, covering four architecture families and two benchmark datasets.
Purpose and motivation
Recognising what is on a plate is only the first step for applications such as dietary tracking and nutrition estimation; knowing where each food item is (and how much of the plate it covers) requires pixel-level segmentation. Food is a hard segmentation target: items overlap, mix, vary widely in appearance and have fuzzy boundaries, and class frequencies are heavily imbalanced.
This project trains and compares several modern segmentation architectures under a common training and evaluation setup, so their strengths can be compared directly on food images. The checkpoints are published so that the results can be reproduced, inspected and reused as starting points for further work.
Scope
Datasets (each model is trained separately on each dataset):
| Dataset | Classes (incl. background) |
|---|---|
| FoodSeg103 | 104 |
| UEC-FoodPix | 103 |
Architectures included:
| Model | Family | Checkpoint |
|---|---|---|
| SegFormer (MiT-B2) | Transformer encoder, lightweight all-MLP decoder | epoch 40 |
| Mask2Former (Swin-T) | Query-based mask classification (Hugging Face format) | last checkpoint (step 7800 / 8400) |
| Swin-UNet | Pure-transformer U-shaped encoder-decoder | epoch 90 |
| Swin Mask R-CNN | Two-stage instance segmentation with a Swin-T backbone | epoch 45 |
Not included: other architectures that were also trained during the project (U-Net, UNet++, Attention U-Net, ResNet-based Mask R-CNN), intermediate epochs, and optimizer states. Only one final checkpoint per model and dataset is published.
Repository layout
foodseg103/ | uecfoodpix/
segformer_b2/ segformer_<dataset>_epoch_40.pth
mask2former/ model.safetensors, config.json
swin_unet/ swin_unet_<dataset>_epoch_90.pth
swin_maskrcnn/ swin_maskrcnn_<dataset>_epoch_45.pth
.pth files are PyTorch checkpoints (either a raw state dict or a dict containing model_state / model_state_dict). Mask2Former is stored in Hugging Face format and can be loaded with Mask2FormerForUniversalSegmentation.from_pretrained(<local folder>). Model definitions and loading code are in utils/model.py and utils/model_loader.py of the accompanying project code.
Evaluation
mIoU over the classes present in the evaluation set:
| Model | FoodSeg103 | UEC-FoodPix |
|---|---|---|
| SegFormer-B2 | 0.661 | 0.793 |
| Mask2Former | 0.652 | 0.743 |
| Swin-UNet | 0.599 | 0.777 |
| Swin Mask R-CNN | 0.543 | 0.673 |
These numbers come from the evaluation CSVs produced during training and were not re-computed for this upload. Scores are not directly comparable across datasets, and rare classes are poorly represented, so global (all-class) mIoU is considerably lower than the values above.
Limitations
- Trained and evaluated only on the two datasets above; behaviour on other cuisines, camera setups or lighting is untested.
- No license has been specified for these weights. Dataset licenses (FoodSeg103, UEC-FoodPix) and the licenses of the pretrained backbones used for initialisation also apply.