Robust semantic labeling of image regions is a basic problem in representing and retrieving imagehideo content. We propose an SVM-MRF framework to model features and their spatial distributions, leading towards a "semantic" representation. Eigenfeatures of Gabor wavelet features and Gaussian mixture model are used for feature clustering. Since similar feature vectors in one cluster can come from several different semantic classes, SVM is applied to represent conditioned feature vector distributions within each cluster, and a Markov random field is used to model the spatial distributions of the semantic labels. A semantic layout representation is proposed to describe the semantics of the images. Experiments show that this method can improve semantic labeling and is useful in similarity search. Describing imageivideo content with semantic labels is a challenging problem. Semantic representation plays an important role in the analysis and retrieval of imagelvideo content. It is difficult for people to retrieve raw data or annotate imageshideos manually. Automated semantic analysis is necessary for efficient imagehideo retrieval and mining [IO]. The challenge in semantic labeling is that there might be a wide variation in appearance within the same class. Support vector machine (SVM) [4] has been used widely for modeling complex feature distributions in highdimensional spaces. Likewise, the Markov random field (MRF) model has proven its usefulness in image analysis [3, 6, 8,12]. However, despite their effectiveness, a simple SVM or MRF model is inadequate to represent the wide variation of visual features within a semantic class. In this paper, a novel combination of MRF and SVM is proposed for the semantic analysis and representation of aerial imagelvideo content. The major contribution of the paper is the SVM-MRF framework for clustering and classification. A casual greedy algorithm is proposed for optimization. The sites for the MRF are blocks of pixels, each of which is described by a visual feature descriptor. The spatial distribution of semantics labels of each image can be considered as a Markov random field (MRF). Therefore, the analysis phase involves the automatic identification of the semantic classes in a given image, and is a three-step procedure. The first step is to cluster the features of the image blocks using the Gaussian mixture model (GMM) [2] . The GMM is used to model the principal components of the original feature vectors. Each Gaussian represents a cluster in this model. This can improve the clustering pcrformance if the number of clusters is not large. Since similar feature vectors in one cluster can come from several different semantic classes, the second step is the application of the "one-againstothers" SVMs to classify the image blocks into candidate semantic classes within each of the clusters. The third step uses a combination of the SVMs and the MRF model to refine the classification. Semantic layouts based on these labels are then generated for similarity search. The paper is organized as follows. Section 2 describes the use of GMM and SVM for modeling feature distributions. Section 3 focuses on the SVM-MRF model for the identification of semantic classes in a given image. Section 4 discusses the application of this approach to similarity retrieval. Experimental results are presented in Section 5 and we conclude with a discussion in Section 6. In order to recognize the semantics of image blocks, it is necessary to describe the blocks using a visual feature and leam the statistical properties of these features considering their semantic origins. The feature used in our experiments is derived from the homogeneous texture descriptor of MPEG-7. The 2-D frequency plane is partitioned into 24 channels with Gabor filters (6 directions on 4 scales) [9] . The mean of the texture energy in each channel is computed, resulting in a 24-dimensional feature vector. In order to simplify the computation, we propose to classify the features into several clusters before the semantic classification. Due to the non-linearity, each cluster can 0-7803-7750-8/03/$I7.00 02003 IEEE
REF proposed an SVM-MRF framework to model features and their spatial distributions (SVM is applied to represent conditioned feature vector distributions within each cluster, and MRF is used to model the spatial distributions of the semantic labels).
A semantic representation for image retrieval
Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429)
Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429)
Computer Science