  Cross-modal recipe retrieval has recently gained substantial attention due to
the importance of food in people's lives, as well as the availability of vast
amounts of digital cooking recipes and food images to train machine learning
models. In this work, we revisit existing approaches for cross-modal recipe
retrieval and propose a simplified end-to-end model based on well established
and high performing encoders for text and images. We introduce a hierarchical
recipe Transformer which attentively encodes individual recipe components
(titles, ingredients and instructions). Further, we propose a self-supervised
loss function computed on top of pairs of individual recipe components, which
is able to leverage semantic relationships within recipes, and enables training
using both image-recipe and recipe-only samples. We conduct a thorough analysis
and ablation studies to validate our design choices. As a result, our proposed
method achieves state-of-the-art performance in the cross-modal recipe
retrieval task on the Recipe1M dataset. We make code and models publicly
available.
