MediaAttention: attention model

The default attention model of MediaAttention, an open-source toolkit that predicts where people look in image and video ads. It then scores each ad for focus, cognitive demand, memorability and attention on the brand, product and call to action.

These weights are UNISAL (Droste, Jiao & Noble, ECCV 2020), trained on SALICON and redistributed unchanged under Apache-2.0 (see NOTICE). MediaAttention picked it after benchmarking six open attention models against human eye tracking. It was the most accurate model with a permissive licence, and the fastest.

Use

pip install "mediaattention[unisal,aoi,video,ui] @ git+https://github.com/pranikz/MediaAttention"
mediaattention ad.png --aoi auto        # downloads this model on first use
mediaattention ui                       # local web UI

from PIL import Image
from mediaattention import score
heat, report = score(Image.open("ad.png"))   # heat: (H, W) fixation density

Files:

unisal_salicon.pth: state dict.
unisal_config.json: constructor arguments.
The model code is vendored in mediaattention/unisal/.

Input is RGB resized to one of UNISAL's fixed-aspect sizes and ImageNet-normalised. Output is a log fixation density at the input size.

Benchmark

Human eye tracking on FiWI (Shen & Zhao, ECCV 2014): 149 webpages, 11 viewers, 5 s free viewing. No model here was trained on it. Metrics are from pysaliency, averaged per image; timing is on an Apple M3 Pro. Method and script: benchmarks/.

Model	CC ↑	NSS ↑	sAUC ↑	AUC ↑	SIM ↑	KLD ↓	s / image (M3 Pro)	Licence
DeepGaze MSDB	0.676	1.54	0.748	0.840	0.642	0.462	2.90	no licence file
DeepGaze IIE	0.649	1.50	0.729	0.824	0.629	0.504	0.67	no licence file
UNISAL (default)	0.616	1.31	0.661	0.796	0.613	0.575	0.04	Apache-2.0
TranSalNet-Dense	0.608	1.25	0.646	0.788	0.605	0.603	0.10	MIT
Centre-bias baseline	0.345	0.62	0.500	0.676	0.476	0.899	0.01	–
Spectral residual (fallback)	0.209	0.40	0.562	0.633	0.449	1.052	0.00	–

DeepGaze is more accurate (up to +0.06 CC), but its repository has no licence file. UNISAL is the best openly licensed model and runs ~17× faster than DeepGaze IIE: 0.04 s per image on an M3 Pro.

Limitations

Trained on natural images (SALICON), not ads. Text-heavy layouts are the hardest case (CC 0.53 on FiWI text pages).
Predicts free-viewing attention in the first seconds. It does not model task-driven search.
Single images only here. MediaAttention handles video by scoring frames and smoothing over time.

Citation

@inproceedings{droste2020unisal,
  title={Unified Image and Video Saliency Modeling},
  author={Droste, Richard and Jiao, Jianbo and Noble, J. Alison},
  booktitle={ECCV}, year={2020}
}
@inproceedings{jiang2015salicon,
  title={SALICON: Saliency in Context},
  author={Jiang, Ming and Huang, Shengsheng and Duan, Juanyong and Zhao, Qi},
  booktitle={CVPR}, year={2015}
}

Packaged by Pratyush Mahapatra. Full references: pranikz.github.io/MediaAttention.

Downloads last month: -; Downloads are not tracked for this model. How to track