We address the problem of automatic image annotation in large vocabulary datasets. In such datasets, there exist three practical issues: (a) Incomplete-labeling: The training samples are not exhaustively tagged with all relevant labels from vocabulary. This is because while building a dataset, human annotators find some labels as "obvious" and miss them in the ground-truth. E.g., an image tagged with "car" might not be tagged with "vehicle". (b) Label-ambiguity: There are some labels that convey same semantic meaning and thus can be used interchangeably, due to which usually only one of them is assigned by annotator. E.g., an image tagged with "flowers" may not be tagged with "blooms", as both convey the same meaning. (c) Structural-overlap: There are some labels that, in spite of being different, share structural properties. E.g., though "tiger" and "lion" are two different labels, structurally they are very similar. Most of the earlier works in this domain have focused on nearestneighbour based models. A recent work [4] also tries to integrate label information in the nearest-neighbour set-up. In this work, first we demonstrate that even the conventional SVM outperforms several benchmark models. Then we propose a new loss function based on the hinge-loss in order to make SVM tolerant against the three issues discussed above. For this, we introduce a tolerance-parameter "t" that adjusts both the margin as well the gradient update-rule for each sample separately. We call this model as Suppor Vector Machine with Variable Tolerance (or SVM-VT). Specifically, we formalize the SVM-VT model as that of solving the following optimization problem: where the additional parameter t j ∈ [0, 1] controls the tolerance against the errors made in the classification of sample x j . The hyperplane w is learnt such that it is more strict towards correctly classifying samples with high value of t j and any such error leads to a large shift in the hyperplane. We propose a heuristic approach for determining the t-value for each sample given a label. For a label l, let S + andS + be the sets of its positive and negative examples respectively. We consider three factors to determine the semantic relatedness of each sample x j ∈S + with that label: (a) Reverse nearest-neighbours based score: For a fixed value of K (= 5), let p k be the number of samples in S + that have x j as their k th nearest neighbour. Then we define (b) Visual similarity based score: We compute the visual similarity score sim(·) (scaled into range [0, 1]) of x j with its nearest neighbour x * i ∈ S + using JEC [3] method and define (c) Label cooccurrence based score: For a label l, let y ∈ {0, 1} m be such that its i th entry is 1 if the i th training image is tagged with l, and 0 otherwise. We compute cooccurrence score co_occur(l i , l j ) between two labels l i and l j using cosine similarity between their corresponding vectors y i and y j . Let x j be tagged with labels L j , then we define From these, we define tolerance parameter for sample x j given label l as We compute t j only for samples inS + , and take t j = 1 for all positive samples assuming that they are correctly annotated. Figure 1: For example labels (in blue) from the three datasets, the top "negative" samples that have least t-scores with corresponding groundtruth labels (smaller t-score implies higher semantic relevance). Dataset → Corel-5k ESP Game IAPRTC-12 Method ↓ P/R/F1/N+ P/R/F1/N+ P/R/F1/N+ MBRM[1] 0
Verma proposed a new loss function to make the SVM more robust, solving three problems in annotation: incomplete labels, fuzzy labels, structural overlap REF .
Exploring SVM for Image Annotation in Presence of Confusing Labels
BMVC
Mathematics Computer Science