We introduce a new algorithm for image segmentation based on crowdsourcing through a game : Ask'nSeek. The game provides information on the objects of an image, under the form of clicks that are either on the object, or on the background. These logs are then used in order to determine the best segmentation for an object among a set of candidates generated by the state-of-the-art CPMC algorithm. We also introduce a simulator that allows the generation of game logs and therefore gives insight about the number of games needed on an image to perform acceptable segmentation.  Categories and Subject Descriptors Crowdsourcing, figure-ground segmentation, labeling game, human computing Semantic annotation of visual content is a process that requires linking pixels within an image with the semantic concepts associated to each group of pixels. This is often done in a user-assisted way. There exist several levels of interaction between users and visual content, ranging from an intentional and accurate annotation targeted at generating high-quality labels (e.g., LabelMe [11]), to a completely unintentional process which is more bound to be noisy (e.g., Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. textual contents surrounding images on web pages and multimedia documents). This paper proposes a method for semantic object segmentation, at a pixel level, based on high-quality annotation data collected from an online game: Ask'nSeek [1] . Ask'nSeek is a two-player game where one player (the master ) places a small square on the image to be segmented, a square which is invisible to the other player (the seeker ). The goal of the seeker is to click inside the hidden square, i.e., to guess the square's location. The seeker is required to solve the problem by requesting clues to the master that consist of the relative position of the square with respect to a semantic object within the image. The name of the object is typed in by the seeker, thereby providing a semantic label for the clicks. Our approach utilizes the information collected from game logs to seed a semi-supervised image segmentation algorithm, which will eventually extract the objects in the image from the surrounding background. Contrary to most existing human-assisted image segmentation solutions, the proposed process is unintentional from the user side, who is engaged in an online game that is scalable to crowds and whose objective is not image segmentation. While the previous work on Ask'nSeek [1] aimed at solving the object detection problem with a bounding box, the work reported in this paper focuses on increasing the spatial accuracy of the results, i.e., performing pixel-based object segmentation. More specifically, this paper explores how the user interaction captured from a crowd-and gaming-based approach influences the quality of the object segmentation process. In particular, we investigate the relationship between the amount of crowdsourcing work and the quality of the corresponding segmentation results by providing an estimation of the minimum amount of clicks which are necessary to reach a certain segmentation accuracy. This paper is structured as follows. Section 2 introduces previous works that are related to this paper. In section 3 we explain how we use game logs from Ask'nSeek in conjunction with a state-of-the-art segmentation algorithm, and also devise a method for simulating game logs. Experiments and results are presented in section 4. Section 5 presents concluding remarks and directions for future work. 15
Ask'nSeek REF also involved the participation of two players to collect the textual labels of the objects contained in an image, as well as selecting the best object candidate based on on clicks labeled as above, below, on the left, on the right or on the objects.
Crowdsourced object segmentation with a game
CrowdMM '13
Computer Science