shalev396/calorie-lens

Calorie Lens

A photo of a meal goes in; out come the ingredients on the plate, how many grams of each, and the nutrition of every ingredient and of the whole plate (kcal, protein, fat, carbs, fiber, sugars, saturated fat, sodium). It is two models fused into one network, trained on the Nutrition5k dishes, whose every ingredient was weighed on a scale, and on USDA FoodData Central.

Model

photo ─► backbone (ConvNeXt-B or DINOv2-B) ─► 253 ingredient queries (cross-attention)
                                              ├─ is ingredient k on the plate?   (presence)
                                              └─ how many grams, if it is?       (grams)
         backbone ─► totals head ─► plate mass
grams_k = P(k) x grams_k, rescaled to the plate mass  ──►  253 neurons  =  stage 1's output
253 neurons ─► NutritionLayer (frozen, 253 x 8 per-gram table) ─► nutrition per ingredient + totals
food name ─► NutriText (own BPE tokenizer + 4-layer transformer, from scratch) ─► nutrients / 100 g

Stage 1 (IngredientNet)	backbone + a query decoder: one learned query per Nutrition5k ingredient reads the image tokens (2 cross-attention layers, no query self-attention), each query has its own presence and grams outputs. A totals head estimates the plate's mass, kcal and macros directly; the ingredient grams are rescaled to that mass (in log space, blended with their own sum; the blend weight is fitted on validation)
Stage 2a (NutritionLayer)	a bias-free linear layer whose weights are per-gram nutrients and are never trained: kcal / fat / carb / protein are Nutrition5k's own values (its labels are exactly grams x this table), fiber / sugars / saturated fat / sodium come from a reviewed USDA match per ingredient. The 253 gram neurons are its input layer, so the whole thing trains end to end
Stage 2b (NutriText)	open vocabulary: any food name -> nutrients per 100 g. Composition comes from a softmax x 100 g (can never exceed 100 g), sugars and saturated fat are bounded by carbs and fat, kcal stays close to the Atwater energy of the composition. Trained from scratch on ~470k USDA names
Input	any RGB photo, fitted (aspect kept) into a 378 x 504 canvas
Output	{"ingredients": [{name, grams, confidence, kcal, fat_g, carb_g, protein_g, fiber_g, sugars_g, sat_fat_g, sodium_mg}], "other": {...}, "totals": {...}, "direct_estimate": {...}}; the rows add up to the totals exactly
Files	model.safetensors (all three stages), config.json (ingredients, per-gram table, backbone config, fitted settings), nutritext_tokenizer.json, model.py, handler.py + requirements.txt, metrics.json, assets/

config.json records pretrained: false; model.load() rebuilds every network from config.json and loads model.safetensors, so inference never downloads a base model.

Usage

Python

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("shalev396/calorie-lens")
sys.path.insert(0, path)
import model
predictor = model.load(path, device="cpu")            # or "cuda"
result = predictor.predict("meal.jpg")                # a PIL image, bytes or base64 work too
result = predictor.predict("meal.jpg", total_grams=420)   # plate weighed? the scale becomes exact
result, heatmaps = predictor.analyze("meal.jpg")      # + where each ingredient's query looked
predictor.calculate([("grilled salmon", 150), ("white rice", 200)])   # stage 2b food calculator

Space API (free): the Space exposes /predict (photo + optional total grams in, [result, seconds, device] out).

Inference Endpoint: deploy this repo from its page (Deploy -> Inference Endpoints). handler.py takes an image (Content-Type: image/jpeg, or JSON {"inputs": "<base64>", "parameters": {"total_grams": 350}}) and returns the result above; {"inputs": [["oatmeal", 250], ["banana", 120]]} runs the calculator.

Training

• Data: shalev396/calorie-lens-data, built from Nutrition5k: one overhead RGB-D scan per dish where available and 5 frames from each of the 4 side-angle videos. Label bugs were fixed (Cafe 2 totals logged as 0, x1000 scale errors, empty names, duplicate table rows) and every dish was assigned a plate group (plates were scanned once per added item), so validation never shares a plate with training. Test = the official split.
• Recipe: see training/: AdamW with layer-wise learning-rate decay, warmup + cosine, bf16, EMA of the weights, early stopping on validation; augmentation on the GPU (any rotation for overhead scans, which keeps the camera-height scale cue).
• Selection: both backbones are trained with the same recipe; validation picks the winner (ties go to ConvNeXt), which is retrained with an auxiliary depth head. The test split is evaluated once, at the end.

Experiments

experiment	val score	test kcal PMAE (overhead)	test mass PMAE	side-dish kcal PMAE	best epoch	params
E1 ConvNeXt-B (deployed)	17.22	18.35%	14.29%	17.28%	24	90.26M
E2 DINOv2-B	17.92	17.53%	14.09%	16.82%	30	88.95M
E3 ConvNeXt-B + depth	17.36	19.02%	14.27%	17.10%	30	90.26M

Validation picked ConvNeXt-B (17.22 vs 17.92). On the test split DINOv2-B came out lower (17.53 % vs 18.35 %), but the test split makes no choices, so ConvNeXt ships. The depth head (E3) did not help: it was slightly worse on validation and on test. The whole run (NutriText + 3 experiments) took 3.1 h on a Colab GPU.

Evaluation

Numbers: overhead test scans (the official depth_test split, labels corrected), as MAE and percent MAE (MAE / mean of the truth, the Nutrition5k convention). metrics.json also has the side-angle views, test_clean (plates never seen in training), the original labels and the 95 % CI.

metric (test)	value
kcal_mae	46.8789
kcal_pmae	18.3503
mass_pmae	14.2945
fat_pmae	24.2640
carb_pmae	22.3740
protein_pmae	27.8922
side_dish_kcal_pmae	17.2801
test_clean_kcal_pmae	18.3879
presence_map	0.6531
ingredient_grams_mae	11.1806
nutritext_kcal_mae_per_100g	31.6995

Headline: 18.35 % calorie error (46.9 kcal) on the 507 overhead test scans, 95 % CI 16.3-20.2 % (bootstrap over plate groups). The original labels give the same 18.35 % on these scans, so the number compares directly with published results. Plate mass: 14.3 %. Averaging all side-angle frames of a dish: 17.3 %.

method (official overhead test, kcal PMAE)	input	error
Nutrition5k paper (Thames et al., 2021)	RGB	26.1 %
Nutrition5k paper	RGB + depth	18.8 %
Calorie Lens	RGB	18.35 %
best published RGB-only	RGB	14.1 %

Small and medium plates are close: under 100 kcal the error is 11 kcal. Large plates are underestimated: over 600 kcal (38 dishes) the error is 167 kcal, because the model pulls toward a typical plate.

What each part is worth. The totals head read on its own scores 16.4 %, better than the 253 gram neurons (18.4 %): the itemised route, whose rows add up to the totals, costs about 2 points of calorie accuracy. Its estimate is returned as direct_estimate in every result. The plain sum of the soft grams, without rescaling to the plate mass, is the worst (23.6 %).

Ingredients: presence mAP 0.65 (F1 0.73 at the display threshold 0.65); 11.2 g error per present ingredient. Each ingredient's query looks at its own part of the plate:

NutriText (stage 2b), on 633 USDA foods it never saw: kcal R² 0.89 vs 0.78 for a TF-IDF nearest-neighbour lookup, 31.7 vs 47.4 kcal per 100 g, and better on all eight nutrients. On the 253 Nutrition5k names (zero-shot): 56 vs 87 kcal per 100 g (median 18 vs 49).

Limitations

• Every training photo comes from two Google cafeterias: a fixed overhead camera height and 253 ingredients. On other photos the mass scale is uncertain; entering the plate's weight (total_grams) removes that error completely.
• Hidden oil, butter, dressings and sauces can only be inferred from how the food looks.
• Only kcal, fat, carbs and protein are Nutrition5k's own values; the other four nutrients come from a USDA food matched to each ingredient. NutriText values for names it never saw are estimates, and some are off (it gives grilled salmon 7.7 g of carbs per 100 g).
• Plates over ~600 kcal are underestimated (see the chart above).
• Results on corrected labels are not directly comparable with published Nutrition5k numbers; the original-label results are in metrics.json.

Code licence: MIT. Training data: Nutrition5k (CC BY 4.0, Thames et al., CVPR 2021) and USDA FoodData Central (public domain).