We present a co-clustering framework that can be used to discover multiple semantic and visual senses of a given Noun Phrase (NP). Unlike traditional clustering approaches which assume a one-to-one mapping between the clusters in the text-based feature space and the visual space, we adopt a one-to-many mapping between the two spaces. This is primarily because each semantic sense (concept) can correspond to different visual senses due to viewpoint and appearance variations. Our structure-EM style optimization not only extracts the multiple senses in both semantic and visual feature space, but also discovers the mapping between the senses. We introduce a challenging dataset (CMU Polysemy-30) for this problem consisting of 30 NPs (∼5600 labeled instances out of ∼22K total instances). We have also conducted a large-scale experiment that performs sense disambiguation for ∼2000 NPs.
REF further investigates an image/text co-clustering framework to disambiguate the multiple semantic senses of some Polysemy words.
Sense discovery via co-clustering on images and text
2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Computer Science