Action classification in still images is an important task in computer vision. It is challenging as the appearances of actions may vary depending on their context (e.g. associated objects). Manually labeling of context information would be time consuming and difficult to scale up. To address this challenge, we propose a method to automatically discover and cluster action concepts, and learn their classifiers from weakly supervised image-sentence corpora. The proposed method obtains candidate action concepts by extracting verb-object pairs from sentences and verifies their visualness with the associated images via two fold crossvalidation. Candidate action concepts are then clustered by using a multi-modal representation with image embeddings from deep convolutional networks and text embeddings from word2vec. More than one hundred human action concept classifiers are learned from the Flickr 30k dataset with no additional human effort and these action concepts are clustered into 81 action clusters. We train a SVM classifier for each of them and we show promising classification results on Flickr30k. We further apply the AdaBoost algorithm to automatically select and combine relevant action concepts given an action query. Promising results have been shown on the PASCAL VOC 2012 action classification benchmark, which has zero overlap with Flickr30k.
Gao et al. REF proposed to learn verb-object pairs as action concepts from image-sentence pairs.
ACD: Action Concept Discovery from Image-Sentence Corpora
ICMR '16
Computer Science