  Fashion plays a pivotal role in society. Combining garments appropriately is
essential for people to communicate their personality and style. Also different
events require outfits to be thoroughly chosen to comply with underlying social
clothing rules. Therefore, combining garments appropriately might not be
trivial. The fashion industry has turned this into a massive source of income,
relying on complex recommendation systems to retrieve and suggest appropriate
clothing items for customers. To perform better recommendations, personalized
suggestions can be performed, taking into account user preferences or purchase
histories. In this paper, we propose a garment recommendation system to pair
different clothing items, namely tops and bottoms, exploiting a Memory
Augmented Neural Network (MANN). By training a memory writing controller, we
are able to store a non-redundant subset of samples, which is then used to
retrieve a ranked list of suitable bottoms to complement a given top. In
particular, we aim at retrieving a variety of modalities in which a certain
garment can be combined. To refine our recommendations, we then include user
preferences via Matrix Factorization. We experiment on IQON3000, a dataset
collected from an online fashion community, reporting state of the art results.
