Original: S ELF - SUPERVISED T RAINING OF P ROPOSAL - BASED S EGMENTATION VIA B ACKGROUND P REDICTION
Revised: S ELF - SUPERVISED T RAINING OF P ROPOSAL - BASED S EGMENTATION VIA B ACKGROUND P REDICTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While supervised object detection and segmentation methods achieve impressive accuracy, they generalize poorly to images whose appearance signiﬁcantly differs from the data they have been trained on.
Revised: While supervised object detection and segmentation methods achieve impressive accuracy, they generalize poorly to images whose appearance signiﬁcantly differs from the data they have been trained on.

Original: To address this in scenarios where annotating data is prohibitively expensive, we introduce a self-supervised approach to detection and segmentation, able to work with monocular images captured with a moving camera.
Revised: To address this in scenarios where annotating data is prohibitively expensive, we introduce a self-supervised approach to detection and segmentation, able to work with monocular images captured with a moving camera.

Original: At the heart of our approach lies the observations that object segmentation and background reconstruction are linked tasks, and that, for structured scenes, background regions can be re-synthesized from their surroundings, whereas regions depicting the object cannot.
Revised: At the heart of our approach lies the observations that object segmentation and background reconstruction are linked tasks, and that, for structured scenes, background regions can be re-synthesized from their surroundings, whereas regions depicting the object cannot.

Original: We encode this intuition as a self-supervised loss function that we exploit to train a proposal-based segmentation network.
Revised: We encode this intuition as a self-supervised loss function that we exploit to train a proposal-based segmentation network.

Original: To account for the discrete nature of the proposals, we develop a Monte Carlo-based training strategy that allows the algorithm to explore the large space of object proposals.
Revised: To account for the discrete nature of the proposals, we develop a Monte Carlo-based training strategy that allows the algorithm to explore the large space of object proposals.

Original: We apply our method to human detection and segmentation in images that visually depart from those of standard benchmarks, achieving competitive results compared to the few existing self-supervised methods and approaching the accuracy of supervised ones that exploit large annotated datasets.
Revised: We apply our method to human detection and segmentation in images that visually depart from those of standard benchmarks, achieving competitive results compared to the few existing self-supervised methods and approaching the accuracy of supervised ones that exploit large annotated datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent object detection and segmentation methods have reached impressive precision and recall rates when trained and tested on large annotated datasets (Lin et al., 2014).
Revised: Recent object detection and segmentation methods have reached impressive precision and recall rates when trained and tested on large annotated datasets (Lin et al., 2014).

Original: However, large and varied datasets do not warrant the best possible performance in a particular application domain, as each comes with its own challenges and opportunities.
Revised: However, large and varied datasets do not warrant the best possible performance in a particular application domain, as each comes with its own challenges and opportunities.

Original: While domain-speciﬁc models are thus needed, annotating separate and sufﬁciently large datasets in all scenarios is impractical.
Revised: While domain-speciﬁc models are thus needed, annotating separate and sufﬁciently large datasets in all scenarios is impractical.

Original: Therefore, weakly- and self-supervised detection and segmentation of salient foreground objects in complex scenes have recently gained attention Croitoru et al.
Revised: Therefore, weakly- and self-supervised detection and segmentation of salient foreground objects in complex scenes have recently gained attention Croitoru et al.

Original: (2018); Eslami et al.
Revised: (2018); Eslami et al.

Original: ; Crawford & Pineau (2019); Rhodin et al.
Revised: ; Crawford & Pineau (2019); Rhodin et al.

Original: ; Bielski & Favaro (2019).
Revised: ; Bielski & Favaro (2019).

Original: These methods promise effortless processing of community videos with little human intervention.
Revised: These methods promise effortless processing of community videos with little human intervention.

Original: However, a closer look at existing techniques reveals that they make strong assumptions, such as the target objects being on top of a static background, or rely on pre-trained supervised object localization, object-boundary detection, and optical ﬂow networks.
Revised: However, a closer look at existing techniques reveals that they make strong assumptions, such as the target objects being on top of a static background, or rely on pre-trained supervised object localization, object-boundary detection, and optical ﬂow networks.

Original: This severely limits their applicability in practice.
Revised: This severely limits their applicability in practice.

Original: To develop a more generic technique, we start from the observation that in most images the background forms a consistent, natural scene.
Revised: To develop a more generic technique, we start from the observation that in most images the background forms a consistent, natural scene.

Original: Therefore the appearance of any background patch can be predicted from its surroundings.
Revised: Therefore the appearance of any background patch can be predicted from its surroundings.

Original: By contrast, a salient object’s appearance is unpredictable from the neighboring scene content and can be expected to be very different from what an inpainting algorithm would produce.
Revised: By contrast, a salient object’s appearance is unpredictable from the neighboring scene content and can be expected to be very different from what an inpainting algorithm would produce.

Original: We incorporate this insight into a proposal-generating deep network whose architecture is inspired by those of YOLO Redmon et al.
Revised: We incorporate this insight into a proposal-generating deep network whose architecture is inspired by those of YOLO Redmon et al.

Original: and
Revised: and

Original: MaskRCNN He et al.
Revised: MaskRCNN He et al.

Original: but does not require explicit supervision.
Revised: but does not require explicit supervision.

Original: For each proposal, we synthesize a background image by masking out the corresponding region and inpainting it from the remaining image.
Revised: For each proposal, we synthesize a background image by masking out the corresponding region and inpainting it from the remaining image.

Original: The loss function we minimize favors the largest possible distance between this reconstructed background and the input image.
Revised: The loss function we minimize favors the largest possible distance between this reconstructed background and the input image.

Original: This encourages the network to select regions that cannot be explained from their surrounding and are therefore salient.
Revised: This encourages the network to select regions that cannot be explained from their surrounding and are therefore salient.

Original: To handle the discrete nature of the proposals, we introduce a Monte Carlo-based strategy to train our network.
Revised: To handle the discrete nature of the proposals, we introduce a Monte Carlo-based strategy to train our network.

Original: It operates on a discrete distribution, is unbiased, exhibits low variance, and is end-to-end trainable.
Revised: It operates on a discrete distribution, is unbiased, exhibits low variance, and is end-to-end trainable.

Original: [Figure]
Revised: [Figure]

Original: We tailor our training algorithm to segmenting human motion from videos in which a single subject moves.
Revised: We tailor our training algorithm to segmenting human motion from videos in which a single subject moves.

Original: The constraints imposed by this well-deﬁned, yet challenging and widely studied task, allow us to generalize to arbitrary, dynamic camera motion.
Revised: The constraints imposed by this well-deﬁned, yet challenging and widely studied task, allow us to generalize to arbitrary, dynamic camera motion.

Original: As a consequence, it lets us perform end-to-end training without the bootstrapping Croitoru et al.
Revised: As a consequence, it lets us perform end-to-end training without the bootstrapping Croitoru et al.

Original: and intermediate optical ﬂow supervision Papazoglou & Ferrari (2013); Jain et al.
Revised: and intermediate optical ﬂow supervision Papazoglou & Ferrari (2013); Jain et al.

Original: ; Tokmakov et al.
Revised: ; Tokmakov et al.

Original: We demonstrate the effectiveness of our unsupervised method on several datasets captured with increasingly mobile cameras, ranging from static to pan-tilt-zoom and hand-held.
Revised: We demonstrate the effectiveness of our unsupervised method on several datasets captured with increasingly mobile cameras, ranging from static to pan-tilt-zoom and hand-held.

Original: We will show that our approach applies to images acquired in conditions signiﬁcantly more general than those of standard benchmarks, without requiring any manual annotation.
Revised: We will show that our approach applies to images acquired in conditions signiﬁcantly more general than those of standard benchmarks, without requiring any manual annotation.

Original: Thus, as shown in Fig. 1, it approaches the quality and sometimes outperforms state-of-the-art detectors that have been trained on large annotated datasets.
Revised: Thus, as shown in Fig. 1, it approaches the quality and sometimes outperforms state-of-the-art detectors that have been trained on large annotated datasets.

Original: Retraining or ﬁne-tuning these methods on this data could be done but would require supervision that is hard to obtain, which makes a self-supervised approach attractive.
Revised: Retraining or ﬁne-tuning these methods on this data could be done but would require supervision that is hard to obtain, which makes a self-supervised approach attractive.

Original: We will make our code and ground-truth segmentations for the existing Ski-PTZ-Dataset and the new Handheld190k dataset publicly available upon acceptance of the paper.
Revised: We will make our code and ground-truth segmentations for the existing Ski-PTZ-Dataset and the new Handheld190k dataset publicly available upon acceptance of the paper.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Most salient object detection and segmentation algorithms are fully-supervised (Redmon et al., 2016;
Revised: Most salient object detection and segmentation algorithms are fully-supervised (Redmon et al., 2016;

Original: He et al., 2017; Song et al., 2018; Cheng et al., 2017) and require large annotated datasets with paired images and labels.
Revised: He et al., 2017; Song et al., 2018; Cheng et al., 2017) and require large annotated datasets with paired images and labels.

Original: Our goal is a purely self-supervised method that succeeds without segmentation and object bounding box annotations.
Revised: Our goal is a purely self-supervised method that succeeds without segmentation and object bounding box annotations.

Original: Note that this differs from the so-called unsupervised object detection methods (Hu et al., 2018;
Revised: Note that this differs from the so-called unsupervised object detection methods (Hu et al., 2018;

Original: Li et al., 2018; Jain et al., 2017), which still require domain-speciﬁc annotations at training but not at test time.
Revised: Li et al., 2018; Jain et al., 2017), which still require domain-speciﬁc annotations at training but not at test time.

Original: We focus our discussion on self- and weakly-supervised methods with regard to the type of training data used and refer to (Koh & Kim, 2017) for the discussion of methods using hand-crafted optimization.
Revised: We focus our discussion on self- and weakly-supervised methods with regard to the type of training data used and refer to (Koh & Kim, 2017) for the discussion of methods using hand-crafted optimization.

Original: Weakly-supervised methods.
Revised: Weakly-supervised methods.

Original: A classical weakly-supervised example is the Hough Matching algorithm (Cho et al., 2015).
Revised: A classical weakly-supervised example is the Hough Matching algorithm (Cho et al., 2015).

Original: It uses an object classiﬁcation dataset and identiﬁes foreground as the image regions that have re-occurring Hough features within images of the same class.
Revised: It uses an object classiﬁcation dataset and identiﬁes foreground as the image regions that have re-occurring Hough features within images of the same class.

Original: Similar principles have been followed using deep networks trained for object detection (Wei et al., 2017;
Revised: Similar principles have been followed using deep networks trained for object detection (Wei et al., 2017;

Original: Jain et al., 2017), optical ﬂow (Tokmakov et al., 2017), and object saliency (Li et al., 2018).
Revised: Jain et al., 2017), optical ﬂow (Tokmakov et al., 2017), and object saliency (Li et al., 2018).

Original: These methods make the implicit assumption that the background varies across examples and can therefore be excluded as noise.
Revised: These methods make the implicit assumption that the background varies across examples and can therefore be excluded as noise.

Original: This assumption is violated in the targeted case of training on domain-speciﬁc images, where foreground and background are similar across examples.
Revised: This assumption is violated in the targeted case of training on domain-speciﬁc images, where foreground and background are similar across examples.

Original: Motion-based methods Given video sequences
Revised: Motion-based methods Given video sequences

Original: , the temporal information can be exploited by assuming that the background changes slowly (Barnich & Van Droogenbroeck, 2011) and linearly (Stretcu & Leordeanu, 2015).
Revised: , the temporal information can be exploited by assuming that the background changes slowly (Barnich & Van Droogenbroeck, 2011) and linearly (Stretcu & Leordeanu, 2015).

Original: However, even a static scene induces non-homogeneous deformations under camera translation, and it can be difﬁcult to handle all types of camera motion (pan, tilt, zoom) within a single video, and distinguish articulated human motion from background motion (Russell et al., 2014).
Revised: However, even a static scene induces non-homogeneous deformations under camera translation, and it can be difﬁcult to handle all types of camera motion (pan, tilt, zoom) within a single video, and distinguish articulated human motion from background motion (Russell et al., 2014).

Original: Some of the resulting errors can be corrected by iteratively reﬁning the crude background subtraction results of Stretcu & Leordeanu (2015) with an ensemble of student and teacher networks (Croitoru et al., 2018; 2019).
Revised: Some of the resulting errors can be corrected by iteratively reﬁning the crude background subtraction results of Stretcu & Leordeanu (2015) with an ensemble of student and teacher networks (Croitoru et al., 2018; 2019).

Original: This, however, induces a strong dependence on the teacher used for bootstrapping.
Revised: This, however, induces a strong dependence on the teacher used for bootstrapping.

Original: [Figure]
Revised: [Figure]

Original: Our approach is conceptually closely related to VideoPCA (Stretcu & Leordeanu, 2015), which models the background as the part of the scene that can be explained by a low-dimensional linear basis.
Revised: Our approach is conceptually closely related to VideoPCA (Stretcu & Leordeanu, 2015), which models the background as the part of the scene that can be explained by a low-dimensional linear basis.

Original: This implicitly assumes that the foreground is harder to model than the background and can therefore be separated as the non-linear residual.
Revised: This implicitly assumes that the foreground is harder to model than the background and can therefore be separated as the non-linear residual.

Original: Here, instead of using motion cues, we propose to rely on the predictability of image patches from their spatial neighborhood.
Revised: Here, instead of using motion cues, we propose to rely on the predictability of image patches from their spatial neighborhood.

Original: Self-supervised Methods.
Revised: Self-supervised Methods.

Original: Most similar to our approach are the self-supervised ones to objectdetection (Eslami et al., 2016; Crawford & Pineau, 2019;
Revised: Most similar to our approach are the self-supervised ones to objectdetection (Eslami et al., 2016; Crawford & Pineau, 2019;

Original: Rhodin et al., 2019; Bielski & Favaro, 2019) that complement auto-encoder networks by an attention mechanism.
Revised: Rhodin et al., 2019; Bielski & Favaro, 2019) that complement auto-encoder networks by an attention mechanism.

Original: These methods ﬁrst detect one or several bounding boxes whose content is extracted using a spatial transformer (Jaderberg et al., 2015).
Revised: These methods ﬁrst detect one or several bounding boxes whose content is extracted using a spatial transformer (Jaderberg et al., 2015).

Original: This content is then passed through an auto-encoder and re-composited with a background.
Revised: This content is then passed through an auto-encoder and re-composited with a background.

Original: In (Rhodin et al., 2019), the background is assumed to be static and in (Eslami et al., 2016; Crawford & Pineau, 2019) even single colored, a severe restriction in practice.
Revised: In (Rhodin et al., 2019), the background is assumed to be static and in (Eslami et al., 2016; Crawford & Pineau, 2019) even single colored, a severe restriction in practice.

Original: Crawford & Pineau (2019) use a proposal-based network similar to ours, but resort to approximating the proposal distribution with a continuous one to make the model differentiable.
Revised: Crawford & Pineau (2019) use a proposal-based network similar to ours, but resort to approximating the proposal distribution with a continuous one to make the model differentiable.

Original: Here, we demonstrate that much simpler importance sampling is sufﬁcient.
Revised: Here, we demonstrate that much simpler importance sampling is sufﬁcient.

Original: In (Bielski & Favaro, 2019), a generative model relies on the assumption that the image region strictly covering the salient object can be subject to random shifts without affecting the realism of the scene.
Revised: In (Bielski & Favaro, 2019), a generative model relies on the assumption that the image region strictly covering the salient object can be subject to random shifts without affecting the realism of the scene.

Original: However, this method can easily be deceived by other background objects whose random displacement can still yield realistic images.
Revised: However, this method can easily be deceived by other background objects whose random displacement can still yield realistic images.

Original: By contrast to all of these methods, our approach works with images acquired using a moving camera and given an arbitrarily colored background.
Revised: By contrast to all of these methods, our approach works with images acquired using a moving camera and given an arbitrarily colored background.

Original: In addition to object detection, the algorithm of Rhodin et al.
Revised: In addition to object detection, the algorithm of Rhodin et al.

Original: (2019) also returns instance segmentation masks by reasoning about the extent and depth ordering of multiple people in a multi-camera scene.
Revised: (2019) also returns instance segmentation masks by reasoning about the extent and depth ordering of multiple people in a multi-camera scene.

Original: However, this requires multiple static cameras and a static background at training time, as does the approach of Baqué et al.
Revised: However, this requires multiple static cameras and a static background at training time, as does the approach of Baqué et al.

Original: (2017) that performs instance segmentation in crowded scenes.
Revised: (2017) that performs instance segmentation in crowded scenes.

Original:  M ETHOD
Revised:  M ETHOD

Original: Our goal is to learn a salient person detector and segmentor from unlabeled videos acquired in as generic a setup as possible, including using hand-held cameras.
Revised: Our goal is to learn a salient person detector and segmentor from unlabeled videos acquired in as generic a setup as possible, including using hand-held cameras.

Original: At inference time, our algorithm takes a single image I ∈ R W × H as input and outputs a bounding box b m ∈ R 4 , expressed as a center location, width, and height, and a segmentation mask S ∈ R 128 × 128 within that window.
Revised: At inference time, our algorithm takes a single image I ∈ R W × H as input and outputs a bounding box b m ∈ R 4 , expressed as a center location, width, and height, and a segmentation mask S ∈ R 128 × 128 within that window.

Original: This is achieved by the multi-stage process visualized in Fig.
Revised: This is achieved by the multi-stage process visualized in Fig.

Original: A ﬁrst network predicts a set of
Revised: A ﬁrst network predicts a set of

Original: C candidate object locations ( b c )
Revised: C candidate object locations ( b c )

Original: Cc =1 and corresponding probabilities ( p c )
Revised: Cc =1 and corresponding probabilities ( p c )

Original: Cc =1 .
Revised: Cc =1 .

Original: To this end, we use a fully-convolutional architecture that divides the image into a grid, with each cell c yielding one probability, p c , and an offset from the cell center to compute b c , similarly to YOLO Redmon et al.
Revised: To this end, we use a fully-convolutional architecture that divides the image into a grid, with each cell c yielding one probability, p c , and an offset from the cell center to compute b c , similarly to YOLO Redmon et al.

Original: Second, the b m with highest probability p m ≥ p c , ∀ c , is chosen and its content is decoded into foreground ˆ I ∈ R 128 × 128 , segmentation mask S , and background B ∈ R W × H with separate encoder-decoder branches.
Revised: Second, the b m with highest probability p m ≥ p c , ∀ c , is chosen and its content is decoded into foreground ˆ I ∈ R 128 × 128 , segmentation mask S , and background B ∈ R W × H with separate encoder-decoder branches.

Original: 3. S ELF -S UPERVISED T RAINING WITH K NOWN B ACKGROUND
Revised: 3. S ELF -S UPERVISED T RAINING WITH K NOWN B ACKGROUND

Original: Given a set of unlabeled images ( I i ) Ni =1 , our goal is to train a neural network D ( I ) (cid:55)→ (( b c ) , ( p c ))
Revised: Given a set of unlabeled images ( I i ) Ni =1 , our goal is to train a neural network D ( I ) (cid:55)→ (( b c ) , ( p c ))

Original: Cc = to propose suitable bounding box candidates and the corresponding probabilities to select the b m that contains an object.
Revised: Cc = to propose suitable bounding box candidates and the corresponding probabilities to select the b m that contains an object.

Original: Because object locations are unknown, we do this with an autoencoder objective,
Revised: Because object locations are unknown, we do this with an autoencoder objective,

Original: L ( F ( I , b c , B ) , I ) , that measures how well the autoencoder F reproduces the input image I on top of a background B , with the attention on b c .
Revised: L ( F ( I , b c , B ) , I ) , that measures how well the autoencoder F reproduces the input image I on top of a background B , with the attention on b c .

Original: As in (Crawford & Pineau, 2019; Rhodin et al., 2019), this is implemented with a spatial transformer T that crops the area of interest, followed by a bottle-neck autoencoder S that produces the foreground and segmentation mask, and a second spatial transformer
Revised: As in (Crawford & Pineau, 2019; Rhodin et al., 2019), this is implemented with a spatial transformer T that crops the area of interest, followed by a bottle-neck autoencoder S that produces the foreground and segmentation mask, and a second spatial transformer

Original: T − 1 that undoes the cropping and blends the synthesized foreground and background.
Revised: T − 1 that undoes the cropping and blends the synthesized foreground and background.

Original: We write
Revised: We write

Original: [Equation]
Revised: [Equation]

Original: S )) , with S
Revised: S )) , with S

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: S )
Revised: S )

Original: [Equation]
Revised: [Equation]

Original: (1) where ◦ is the elementwise multiplication.
Revised: (1) where ◦ is the elementwise multiplication.

Original: The loss L can be a least-square distance between all pixel values, alternatives are discussed at the end of the section.
Revised: The loss L can be a least-square distance between all pixel values, alternatives are discussed at the end of the section.

Original: Because the attention window b selects part of the image for decoding, this loss encourages D to focus on the object so as to model the foreground on top of B .
Revised: Because the attention window b selects part of the image for decoding, this loss encourages D to focus on the object so as to model the foreground on top of B .

Original: Moreover, the autoencoder only approximates the actual image, forcing the segmentation mask to contain only the parts not captured by the background.
Revised: Moreover, the autoencoder only approximates the actual image, forcing the segmentation mask to contain only the parts not captured by the background.

Original: For now, we assume B to be known, but we will remove this assumption later.
Revised: For now, we assume B to be known, but we will remove this assumption later.

Original: In this setting, we derive a probabilistic formulation in which not a single, but multiple candidates can give rise to plausible reconstructions.
Revised: In this setting, we derive a probabilistic formulation in which not a single, but multiple candidates can give rise to plausible reconstructions.

Original: We then reason about the expected loss across all likely candidates,
Revised: We then reason about the expected loss across all likely candidates,

Original: [Equation]
Revised: [Equation]

Original: ) , I )] , with c ∼ p, where p
Revised: ) , I )] , with c ∼ p, where p

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: = p
Revised: = p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where E c denotes the expectation over c drawn from the categorical proposal distribution output by the network D ( I ) .
Revised: where E c denotes the expectation over c drawn from the categorical proposal distribution output by the network D ( I ) .

Original: We optimize Eq.
Revised: We optimize Eq.

Original: (2) with stochastic gradient descent and mini-batches of ( I i ) Ni =1 .
Revised: (2) with stochastic gradient descent and mini-batches of ( I i ) Ni =1 .

Original: Note that minimizing this objective will jointly optimize the detection network D that generates and stochastically selects proposals, and the synthesis network S that models the object appearance and the segmentation mask.
Revised: Note that minimizing this objective will jointly optimize the detection network D that generates and stochastically selects proposals, and the synthesis network S that models the object appearance and the segmentation mask.

Original: Because we have a ﬁnite set of candidates, Eq.
Revised: Because we have a ﬁnite set of candidates, Eq.

Original: (2) can be expressed deterministically as a weighted sum over all candidates as
Revised: (2) can be expressed deterministically as a weighted sum over all candidates as

Original: [Equation]
Revised: [Equation]

Original: This objective is an explicit function of p and could be optimized by gradient descent.
Revised: This objective is an explicit function of p and could be optimized by gradient descent.

Original: However, this sum over C forward passes is inefﬁcient to evaluate in practice when C is large.
Revised: However, this sum over C forward passes is inefﬁcient to evaluate in practice when C is large.

Original: For example, in our experiments C = 64 , which does not ﬁt in memory.
Revised: For example, in our experiments C = 64 , which does not ﬁt in memory.

Original: This was also observed by Crawford & Pineau (2019), who resorted to using a continuous approximation of the discrete distribution p to facilitate end-to-end training.
Revised: This was also observed by Crawford & Pineau (2019), who resorted to using a continuous approximation of the discrete distribution p to facilitate end-to-end training.

Original: Here, we propose a simpler alternative, exploiting Monte Carlo and importance sampling, which provides an unbiased estimator with low variance.
Revised: Here, we propose a simpler alternative, exploiting Monte Carlo and importance sampling, which provides an unbiased estimator with low variance.

Original: Monte Carlo (MC) sampling.
Revised: Monte Carlo (MC) sampling.

Original: In principle, the expectation in Eq.
Revised: In principle, the expectation in Eq.

Original: (2) could be estimated by sampling a small set of J candidate cells from p .
Revised: (2) could be estimated by sampling a small set of J candidate cells from p .

Original: Unfortunately, sampling from such a discrete distribution is not differentiable with respect to its parameters, which precludes end-to-end gradientbased optimization.
Revised: Unfortunately, sampling from such a discrete distribution is not differentiable with respect to its parameters, which precludes end-to-end gradientbased optimization.

Original: Instead of sampling according to p , we can rewrite Eq.
Revised: Instead of sampling according to p , we can rewrite Eq.

Original: (2) using an arbitrary distribution q , by reweighting with the quotient of both distributions.
Revised: (2) using an arbitrary distribution q , by reweighting with the quotient of both distributions.

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: This change of distribution and relative weighting holds for any two probability distributions, as explained in the appendix.
Revised: This change of distribution and relative weighting holds for any two probability distributions, as explained in the appendix.

Original: In practice, we approximate the expectation on a mini-batch, with a single sample drawn from q per image.
Revised: In practice, we approximate the expectation on a mini-batch, with a single sample drawn from q per image.

Original: Importance sampling.
Revised: Importance sampling.

Original: While moving the distribution into the expectation sum provides differentiability, it comes with the drawback of a potentially large variance, i.e., high approximation error for few samples.
Revised: While moving the distribution into the expectation sum provides differentiability, it comes with the drawback of a potentially large variance, i.e., high approximation error for few samples.

Original: For instance, by choosing q to be the uniform sampling distribution U , most of the uniformly drawn samples will have a low probability in p and therefore negligible inﬂuence.
Revised: For instance, by choosing q to be the uniform sampling distribution U , most of the uniformly drawn samples will have a low probability in p and therefore negligible inﬂuence.

Original: To reduce this variance, we leverage importance sampling and set the sampling distribution q to be similar or equivalent to p .
Revised: To reduce this variance, we leverage importance sampling and set the sampling distribution q to be similar or equivalent to p .

Original: To prevent division by very small values that could lead to numerical instability, we deﬁne the new distribution q as
Revised: To prevent division by very small values that could lead to numerical instability, we deﬁne the new distribution q as

Original: [Equation]
Revised: [Equation]

Original: As a side effect, (cid:15) controls the probability that an unlikely case is chosen, which induces a form of exploration that is helpful in the early training stages of the network.
Revised: As a side effect, (cid:15) controls the probability that an unlikely case is chosen, which induces a form of exploration that is helpful in the early training stages of the network.

Original: Note that the fraction p ( c ) q ( c ) cancels numerically for q ( c )
Revised: Note that the fraction p ( c ) q ( c ) cancels numerically for q ( c )

Original: =
Revised: =

Original: p ( c ) .
Revised: p ( c ) .

Original: However, while values cancel, their derivatives do not; the differentiability of p is maintained as the sampling distribution q must be treated as a constant.
Revised: However, while values cancel, their derivatives do not; the differentiability of p is maintained as the sampling distribution q must be treated as a constant.

Original: In this case, the gradient of Eq.
Revised: In this case, the gradient of Eq.

Original: 4 equals that of the likelihood ratio method (Glynn, 1990) used in the
Revised: 4 equals that of the likelihood ratio method (Glynn, 1990) used in the

Original: REINFORCE algorithm
Revised: REINFORCE algorithm

Original: (Williams, 1992).
Revised: (Williams, 1992).

Original: We favor the MC interpretation that explains the effect of the gradient quotient as importance sampling and generalizes to arbitrary q .
Revised: We favor the MC interpretation that explains the effect of the gradient quotient as importance sampling and generalizes to arbitrary q .

Original: While the relation is known for a long time (Glynn, 1990), is overlooked in the recent literature.
Revised: While the relation is known for a long time (Glynn, 1990), is overlooked in the recent literature.

Original: 3. S ELF -S UPERVISED T RAINING WITH M OVING B ACKGROUND
Revised: 3. S ELF -S UPERVISED T RAINING WITH M OVING B ACKGROUND

Original: Having derived an efﬁcient training scheme for proposal-based segmentation when B is given, we would like to address the moving camera scenario, where it is not given, by reducing this more difﬁcult case to the former.
Revised: Having derived an efﬁcient training scheme for proposal-based segmentation when B is given, we would like to address the moving camera scenario, where it is not given, by reducing this more difﬁcult case to the former.

Original: This can be achieved by predicting the background image, which, in the absence of prior shape and appearance information of the foreground, is even harder than segmentation.
Revised: This can be achieved by predicting the background image, which, in the absence of prior shape and appearance information of the foreground, is even harder than segmentation.

Original: To overcome this difﬁculty, we redeﬁne the training strategy and replace background prediction with the simpler task of inpainting local regions, which can easily be trained by removing a region and predicting it from its immediate surrounding (Pathak et al., 2016; Yu et al., 2018).
Revised: To overcome this difﬁculty, we redeﬁne the training strategy and replace background prediction with the simpler task of inpainting local regions, which can easily be trained by removing a region and predicting it from its immediate surrounding (Pathak et al., 2016; Yu et al., 2018).

Original: Under the assumption that the foreground object moves, a network, I , trained on this self-supervised inpainting task would not manage to reconstruct the foreground objects if fully removed in the input because the surrounding background gives no cues of their presence.
Revised: Under the assumption that the foreground object moves, a network, I , trained on this self-supervised inpainting task would not manage to reconstruct the foreground objects if fully removed in the input because the surrounding background gives no cues of their presence.

Original: We therefore cast foreground segmentation as the task of ﬁnding the area b c that, when inpainted, yields the largest image reconstruction error.
Revised: We therefore cast foreground segmentation as the task of ﬁnding the area b c that, when inpainted, yields the largest image reconstruction error.

Original: Instead of doing an expensive search at inference time, we train D to predict the foreground location.
Revised: Instead of doing an expensive search at inference time, we train D to predict the foreground location.

Original: We search probabilistically by sampling likely cases according to the current distribution p and write
Revised: We search probabilistically by sampling likely cases according to the current distribution p and write

Original: [Equation]
Revised: [Equation]

Original: where I takes the image
Revised: where I takes the image

Original: I and the region b c to inpaint as input, L is a pixel loss, a ( b c ) normalizes the loss by the window area, and p and b c are computed with D as before.
Revised: I and the region b c to inpaint as input, L is a pixel loss, a ( b c ) normalizes the loss by the window area, and p and b c are computed with D as before.

Original: We use the same importance sampling strategy to optimize the discrete distribution, but opposed to the foreground objective, we use the negative expectation.
Revised: We use the same importance sampling strategy to optimize the discrete distribution, but opposed to the foreground objective, we use the negative expectation.

Original: This negative reconstruction error encourages the selection of those regions were the true image is dissimilar to the reconstructed background when minimizing Eq.
Revised: This negative reconstruction error encourages the selection of those regions were the true image is dissimilar to the reconstructed background when minimizing Eq.

Original: Training strategy.
Revised: Training strategy.

Original: Unfortunately, the objective G has trivial solutions.
Revised: Unfortunately, the objective G has trivial solutions.

Original: It favors locations with high error density, irrespectively of their size, as illustrated in Fig. 3(b).
Revised: It favors locations with high error density, irrespectively of their size, as illustrated in Fig. 3(b).

Original: Alternatively, removing the normalization by the area favors erasing extensively large regions, containing an object or not, because larger number of reconstructed pixels lead to higher inpainting errors, as shown in Fig. 3(d).
Revised: Alternatively, removing the normalization by the area favors erasing extensively large regions, containing an object or not, because larger number of reconstructed pixels lead to higher inpainting errors, as shown in Fig. 3(d).

Original: To overcome these degenerate cases, we combine the new background objective G with the foreground objective O of Eq.
Revised: To overcome these degenerate cases, we combine the new background objective G with the foreground objective O of Eq.

Original: (4), substituting the known B with the learned inpainting I ( I , b c ) .
Revised: (4), substituting the known B with the learned inpainting I ( I , b c ) .

Original: The reason behind this is that these two terms are complementary: While G prefers locations that cover the object neither precisely nor entirely, O favors a tight ﬁt over partial coverage but has a trivial solution when b c is on a background region, i.e., not covering the object and having nothing to encode.
Revised: The reason behind this is that these two terms are complementary: While G prefers locations that cover the object neither precisely nor entirely, O favors a tight ﬁt over partial coverage but has a trivial solution when b c is on a background region, i.e., not covering the object and having nothing to encode.

Original: Finding a balance between these two adversarial objectives by relative weighting turned out to be difﬁcult, if not impossible.
Revised: Finding a balance between these two adversarial objectives by relative weighting turned out to be difﬁcult, if not impossible.

Original: Instead, we separate them such that their inﬂuence on the individual network components is mutually exclusive.
Revised: Instead, we separate them such that their inﬂuence on the individual network components is mutually exclusive.

Original: Probabilities p c are only optimized according to G .
Revised: Probabilities p c are only optimized according to G .

Original: Therefore, O can not impose bias to background regions where it has a trivial solution.
Revised: Therefore, O can not impose bias to background regions where it has a trivial solution.

Original: In turn, b c is optimized only by O to ﬁnd a tight ﬁt without the bias of G to excessively large or small proposals b c .
Revised: In turn, b c is optimized only by O to ﬁnd a tight ﬁt without the bias of G to excessively large or small proposals b c .

Original: Also S is solely optimized by O to output the best possible reconstruction, instead of the largest distance to the background as induced by G .
Revised: Also S is solely optimized by O to output the best possible reconstruction, instead of the largest distance to the background as induced by G .

Original: The separation can conveniently be computed in a single forward-backwards pass by treating the excluded variables as constants in the respective objectives, that means cutting of gradient ﬂow through them.
Revised: The separation can conveniently be computed in a single forward-backwards pass by treating the excluded variables as constants in the respective objectives, that means cutting of gradient ﬂow through them.

Original: Note that the stable training through effective separation into coarse and ﬁne localization of adversarial termsis only possible with the chosen proposal-based detection framework; not with direct regression.
Revised: Note that the stable training through effective separation into coarse and ﬁne localization of adversarial termsis only possible with the chosen proposal-based detection framework; not with direct regression.

Original: [Figure]
Revised: [Figure]

Original: Implementation details.
Revised: Implementation details.

Original: Since no labels for intermediate supervision are available, naive end-toend training is difﬁcult.
Revised: Since no labels for intermediate supervision are available, naive end-toend training is difﬁcult.

Original: To counteract this, we use ImageNet-trained weights for initialization; rely on a perceptual loss on top of the per pixel L 2 loss for L ; exploit the Focal Spatial Transformers (FST) of Rhodin et al.
Revised: To counteract this, we use ImageNet-trained weights for initialization; rely on a perceptual loss on top of the per pixel L 2 loss for L ; exploit the Focal Spatial Transformers (FST) of Rhodin et al.

Original: (2019) to speed up convergence; and scale the erased region in I to be 1.
Revised: (2019) to speed up convergence; and scale the erased region in I to be 1.

Original: the size of that predicted by D to increase the chances of covering the object.
Revised: the size of that predicted by D to increase the chances of covering the object.

Original: Moreover, we limit the location offsets to 1.5 the cell width and discard those outside the image.
Revised: Moreover, we limit the location offsets to 1.5 the cell width and discard those outside the image.

Original: In addition, we rely on
Revised: In addition, we rely on

Original: L 2 priors on the output of D , and a v-shaped prior L v on S that stabilizes early training iterations by pushing the solution at the beginning of the training to be larger than a value λ , yet sparse when exceeding this threshold.
Revised: L 2 priors on the output of D , and a v-shaped prior L v on S that stabilizes early training iterations by pushing the solution at the beginning of the training to be larger than a value λ , yet sparse when exceeding this threshold.

Original: We performed a grid search on the relative weights of the terms, the offset limits, and λ .
Revised: We performed a grid search on the relative weights of the terms, the offset limits, and λ .

Original: The pixel reconstruction and perceptual losses are weighted 1:2, and the priors have a weight of 0.1, 1, and 0.1, respectively, to compensate for their different magnitudes.
Revised: The pixel reconstruction and perceptual losses are weighted 1:2, and the priors have a weight of 0.1, 1, and 0.1, respectively, to compensate for their different magnitudes.

Original: Additional details are given in the appendix.
Revised: Additional details are given in the appendix.

Revised: Regarding the inpainting network, using an off-the-shelf inpainting method leads to hallucinated objects.

Revised: Therefore, we train the inpainting network in a self-supervised manner on the available input videos.

Revised: Details are given in the appendix, Section A.1.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate our approach to self-supervised salient object detection and segmentation.
Revised: In this section, we evaluate our approach to self-supervised salient object detection and segmentation.

Original: Note that our algorithm works on single images at inference time and requires the background inpainting model only at training time.
Revised: Note that our algorithm works on single images at inference time and requires the background inpainting model only at training time.

Original: Our focus is on people-detection.
Revised: Our focus is on people-detection.

Original: Nevertheless the method could generalize to arbitrary objects, as long as videos or picture collections of a single object in front of the same scene are available.
Revised: Nevertheless the method could generalize to arbitrary objects, as long as videos or picture collections of a single object in front of the same scene are available.

Original: We ﬁrst demonstrate the use of our method in a controlled environment with static background, to compare it to a state-of-the-art self-supervised approach.
Revised: We ﬁrst demonstrate the use of our method in a controlled environment with static background, to compare it to a state-of-the-art self-supervised approach.

Original: Then, we introduce skiing footage acquired using PTZ-cameras and footage of people performingeveryday activities recorded using hand-held cameras to demonstrate that existing supervised methods that do well in the controlled environment struggle to adapt to such challenging conditions, whereas our approach delivers promising results.
Revised: Then, we introduce skiing footage acquired using PTZ-cameras and footage of people performingeveryday activities recorded using hand-held cameras to demonstrate that existing supervised methods that do well in the controlled environment struggle to adapt to such challenging conditions, whereas our approach delivers promising results.

Original: We provide additional qualitative results, experiments on the loss function, and details on the parameter search in the appendix material.
Revised: We provide additional qualitative results, experiments on the loss function, and details on the parameter search in the appendix material.

Original: 4. P EOPLE IN A C ONTROLLED E NVIRONMENT
Revised: 4. P EOPLE IN A C ONTROLLED E NVIRONMENT

Original: We compare our method against state-of-the-art ones on the Human3.6m dataset (Ionescu et al., 2014) that comprises 3.6 million frames and 15 motion classes.
Revised: We compare our method against state-of-the-art ones on the Human3.6m dataset (Ionescu et al., 2014) that comprises 3.6 million frames and 15 motion classes.

Original: It features 5 subjects for training andfor validation, seen from different viewpoints against a static background and with good illumination.
Revised: It features 5 subjects for training andfor validation, seen from different viewpoints against a static background and with good illumination.

Original: Comparative Results.
Revised: Comparative Results.

Original: On the left side of Table 1, we compare our detection accuracy to that of a very recent self-supervised deep learning method (Rhodin et al., 2019), using the mean detection precision (mAP), the mean precision of having an intersection-over-union (IoU) of more than 50%.
Revised: On the left side of Table 1, we compare our detection accuracy to that of a very recent self-supervised deep learning method (Rhodin et al., 2019), using the mean detection precision (mAP), the mean precision of having an intersection-over-union (IoU) of more than 50%.

Original: Our slightly lower accuracy stems from not explicitly assuming a static background, which Rhodin et al.
Revised: Our slightly lower accuracy stems from not explicitly assuming a static background, which Rhodin et al.

Original: (2019) do.
Revised: (2019) do.

Original: While valid in a lab, this assumption results in total failure in outdoor scenes with moving backgrounds, such as those discussed the next sections.
Revised: While valid in a lab, this assumption results in total failure in outdoor scenes with moving backgrounds, such as those discussed the next sections.

Original: Ablation Study.
Revised: Ablation Study.

Original: Here we show that our model choices for training and probabilistic inference are important.
Revised: Here we show that our model choices for training and probabilistic inference are important.

Original: Using uniform sampling instead of importance sampling, does not converge, as shown in Fig. 3(a).
Revised: Using uniform sampling instead of importance sampling, does not converge, as shown in Fig. 3(a).

Original: Fig.
Revised: Fig.

Original: 3(b) illustrates that joint training of D with O and G , instead of our separate one, produces bounding boxes that are too large.
Revised: 3(b) illustrates that joint training of D with O and G , instead of our separate one, produces bounding boxes that are too large.

Original: Fig. 3(c) shows that using only the background objective leads to small detections that miss the subject and (d) that direct regression without multiple  candidates diverges.
Revised: Fig. 3(c) shows that using only the background objective leads to small detections that miss the subject and (d) that direct regression without multiple [Figure] [Figure] candidates diverges.

Original: These failure cases are representative of the behavior on the whole dataset.
Revised: These failure cases are representative of the behavior on the whole dataset.

Original: Fig.
Revised: Fig.

Original: 3(e) demonstrates that our full model using the separate training strategy and importance sampling can accurately detect the person and estimate tighter bounding boxes.[Figure] [Figure]
Revised: 3(e) demonstrates that our full model using the separate training strategy and importance sampling can accurately detect the person and estimate tighter bounding boxes.

Original: Multiple people.
Revised: Multiple people.

Original: Although our focus is on handling single objects or persons, our probabilistic framework can handle several at test time by sampling more than once.
Revised: Although our focus is on handling single objects or persons, our probabilistic framework can handle several at test time by sampling more than once.

Original: Fig.
Revised: Fig.

Original: 4 shows the predicted cell probability as blue dots whose size is proportional to the probability.
Revised: 4 shows the predicted cell probability as blue dots whose size is proportional to the probability.

Original: The fully-convolutional architecture operates locally and thereby predicts a high person probability close to both subjects.
Revised: The fully-convolutional architecture operates locally and thereby predicts a high person probability close to both subjects.

Original: As a result, both the detection and segmentation results remain accurate as long as the individuals are sufﬁciently separated.
Revised: As a result, both the detection and segmentation results remain accurate as long as the individuals are sufﬁciently separated.

Original: 4. S KIERS F ILMED U SING A PTZ-C AMERAS
Revised: 4. S KIERS F ILMED U SING A PTZ-C AMERAS

Original: We now turn to the out-of-the-ordinary motions of six skiers on a slalom course featured in the
Revised: We now turn to the out-of-the-ordinary motions of six skiers on a slalom course featured in the

Original: Ski-PTZ-Dataset (Rhodin et al., 2018b).
Revised: Ski-PTZ-Dataset (Rhodin et al., 2018b).

Original: The six skiers are split four/one/one to form training, validation, and test sets, with, respectively, 7800 , 1818 and 1908 frames.
Revised: The six skiers are split four/one/one to form training, validation, and test sets, with, respectively, 7800 , 1818 and 1908 frames.

Original: The intrinsic and extrinsic parameters of the pan-tilt-zoom cameras are constantly adjusted to follow the skier.
Revised: The intrinsic and extrinsic parameters of the pan-tilt-zoom cameras are constantly adjusted to follow the skier.

Original: As a result, nothing is static in the images, the background changes quickly, and there are additional people standing as part of the background.
Revised: As a result, nothing is static in the images, the background changes quickly, and there are additional people standing as part of the background.

Original: We use the full image as input, evaluate detection accuracy in relation to the available 2D pose annotation, and segmentation accuracy by manually segmenting 9 frames from each of the six cameras which add up to 108 frames for two test sequences.
Revised: We use the full image as input, evaluate detection accuracy in relation to the available 2D pose annotation, and segmentation accuracy by manually segmenting 9 frames from each of the six cameras which add up to 108 frames for two test sequences.

Original: For the hyperparameter selection we use the manually segmented 3 frames from each of six cameras which is in total 36 frames for two validation sequences.
Revised: For the hyperparameter selection we use the manually segmented 3 frames from each of six cameras which is in total 36 frames for two validation sequences.

Original: In Table 1, we evaluate detection accuracy.
Revised: In Table 1, we evaluate detection accuracy.

Original: Note that our method delivers an mAP 0 .
Revised: Note that our method delivers an mAP 0 .

Original: 5 score that is signiﬁcantly better than that of the general YOLO Redmon et al.
Revised: 5 score that is signiﬁcantly better than that of the general YOLO Redmon et al.

Original: (2016) detector trained on MSCOCO.
Revised: (2016) detector trained on MSCOCO.

Original: In Table 2, we compare our approach to several state-of-the-art segmentation baselines in terms of precision, recall, F-, and J-measure as deﬁned in (Pont-Tuset et al., 2017).
Revised: In Table 2, we compare our approach to several state-of-the-art segmentation baselines in terms of precision, recall, F-, and J-measure as deﬁned in (Pont-Tuset et al., 2017).

Original: To be fair, we compensate for different segmentation masks quantiﬁcation levels by a grid search (at 0.05 intervals) to select the best threshold in terms of J-measure for each method.
Revised: To be fair, we compensate for different segmentation masks quantiﬁcation levels by a grid search (at 0.05 intervals) to select the best threshold in terms of J-measure for each method.

Original: Interestingly, MaskRCNN trained on a large generic dataset is outperformed by ARP on this dataset.
Revised: Interestingly, MaskRCNN trained on a large generic dataset is outperformed by ARP on this dataset.

Original: Without using any object localization data, our method is on par with MaskRCNN and close to weakly supervised methods that train on large datasets with motion boundary and segmentation mask annotations.
Revised: Without using any object localization data, our method is on par with MaskRCNN and close to weakly supervised methods that train on large datasets with motion boundary and segmentation mask annotations.

Original: More importantly, our approach  outperforms the self-supervised one of Stretcu & Leordeanu (2015).
Revised: More importantly, our approach [Figure] [Figure] outperforms the self-supervised one of Stretcu & Leordeanu (2015).

Original: While our F- and J-measures are slightly lower than that of (Croitoru et al., 2019), part of this difference can be attributed to Croitoru et al.
Revised: While our F- and J-measures are slightly lower than that of (Croitoru et al., 2019), part of this difference can be attributed to Croitoru et al.

Original: (2019) using a segmentation mask discriminator that is trained on the combination of the
Revised: (2019) using a segmentation mask discriminator that is trained on the combination of the

Original: [Figure]

Original: [Figure]

Original: ImageNet VID and YouTube Objects datasets.
Revised: ImageNet VID and YouTube Objects datasets.

Original: Albeit also trained in a self-supervised fashion, it thereby leverages additional information.
Revised: Albeit also trained in a self-supervised fashion, it thereby leverages additional information.

Original: In Fig.
Revised: In Fig.

Original: 6, we show the scenes reconstructed by Bielski & Favaro (2019) along with the resulting segmentation masks.
Revised: 6, we show the scenes reconstructed by Bielski & Favaro (2019) along with the resulting segmentation masks.

Original: Note that their generative model fails to segment the foreground object alone and instead segments background objects and sometimes even the ground.
Revised: Note that their generative model fails to segment the foreground object alone and instead segments background objects and sometimes even the ground.

Original: This method relies on the property that foreground regions can undergo random perturbations without altering the realism of the scene.
Revised: This method relies on the property that foreground regions can undergo random perturbations without altering the realism of the scene.

Original: However, in the Ski-PTZ-Dataset , some background objects, such as poles, also satisfy this property, and the generator can choose to keep these regions.
Revised: However, in the Ski-PTZ-Dataset , some background objects, such as poles, also satisfy this property, and the generator can choose to keep these regions.

Original: Since the results on the training samples are very poor, we do not provide any quantitative results of this work on the test data.
Revised: Since the results on the training samples are very poor, we do not provide any quantitative results of this work on the test data.

Original: Further qualitative results are shown in Fig.
Revised: Further qualitative results are shown in Fig.

Original: The probability distribution, visualized as blue dots that increase in magnitude with the predicted likelihood, show clear peaks on the persons.
Revised: The probability distribution, visualized as blue dots that increase in magnitude with the predicted likelihood, show clear peaks on the persons.

Original: The limitations include slightly blurred and bleeding masks and occasional false positives, reducing precision.
Revised: The limitations include slightly blurred and bleeding masks and occasional false positives, reducing precision.

Original: We investigate the effectiveness of different mask priors and ImageNet pre-training on the validation part of the Ski-PTZ-Dataset in the appendix.
Revised: We investigate the effectiveness of different mask priors and ImageNet pre-training on the validation part of the Ski-PTZ-Dataset in the appendix.

Original: Our (cid:96) v prior achieves the highest scores in all measures with consistently reliable results.
Revised: Our (cid:96) v prior achieves the highest scores in all measures with consistently reliable results.

Original: They demonstrate that imposing regularization on the segmentation masks allows us to obtain sharper masks, removing the noise around the foreground object [Table]
Revised: They demonstrate that imposing regularization on the segmentation masks allows us to obtain sharper masks, removing the noise around the foreground object.

Original: 4. D AILY A CTIVITIES C APTURED U SING H AND -H ELD C AMERAS
Revised: 4. D AILY A CTIVITIES C APTURED U SING H AND -H ELD C AMERAS

Original: We introduce a new Handheld190k dataset that features three training, one validation and one test sequences, comprising 120 855 , 23 076 and 46 326 images, respectively, with a single actor performing actions mimicking those in H36M .
Revised: We introduce a new Handheld190k dataset that features three training, one validation and one test sequences, comprising 120 855 , 23 076 and 46 326 images, respectively, with a single actor performing actions mimicking those in H36M .

Original: We manually annotated 112 frames in the validation and 240 frames in the test sequence to provide ground-truth segmentation masks which we believe will be useful for evaluating other self- and weakly-supervised methods. 
Revised: We manually annotated 112 frames in the validation and 240 frames in the test sequence to provide ground-truth segmentation masks which we believe will be useful for evaluating other self- and weakly-supervised methods. The camera operators moved [Table]

Revised: and Handheld190k dataset.

Revised: Ours exceeds or is on par with the self-supervised methods (marked with s ), and approaches the accuracy of weakly-supervised (marked with w ) and fully-supervised methods (marked with d ).

Original: The camera operators moved laterally, to test robustness to camera translation and hand-held rotation.
Revised: laterally, to test robustness to camera translation and hand-held rotation.

Original: We provide examples of our detection and segmentation results in Fig.
Revised: We provide examples of our detection and segmentation results in Fig.

Original: 5, more are given in the appendix.
Revised: 5, more are given in the appendix.

Original: Our method is robust to[Table] the undirected camera motion and to dynamic background motion, such as branches swinging in the wind and clouds moving, and to salient textures in the background, such as that of the house facade.
Revised: Our method is robust to the undirected camera motion and to dynamic background motion, such as branches swinging in the wind and clouds moving, and to salient textures in the background, such as that of the house facade.

Original: To perform a quantitative comparison, we use 240 manually segmented test images taken from different motion classes with the subject in many different poses.
Revised: To perform a quantitative comparison, we use 240 manually segmented test images taken from different motion classes with the subject in many different poses.

Original: We evaluate against the same methods using the same quantities as for skiing in Table 2.
Revised: We evaluate against the same methods using the same quantities as for skiing in Table 2.

Original: In this scenario, MaskRCNN yields the highest scores, which is not surprising as the tested sequences are similar to its training set MSCOCO.
Revised: In this scenario, MaskRCNN yields the highest scores, which is not surprising as the tested sequences are similar to its training set MSCOCO.

Original: It is followed by our work in F- and J-measure.
Revised: It is followed by our work in F- and J-measure.

Original: Despite using a discriminator that is trained in an unsupervised fashion on another, larger dataset, (Croitoru et al., 2019) performs comparably to ours.
Revised: Despite using a discriminator that is trained in an unsupervised fashion on another, larger dataset, (Croitoru et al., 2019) performs comparably to ours.

Original: Contrary to Ski-PTZ-Dataset results, we outperform ARP Koh & Kim (2017) in F- and J-measure and have a larger margin on VideoPCA Stretcu & Leordeanu (2015).
Revised: Contrary to Ski-PTZ-Dataset results, we outperform ARP Koh & Kim (2017) in F- and J-measure and have a larger margin on VideoPCA Stretcu & Leordeanu (2015).

Original: Both of these methods often fail in separating the non-homogeneously moving background due to the hand-held camera motion.
Revised: Both of these methods often fail in separating the non-homogeneously moving background due to the hand-held camera motion.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have proposed a self-supervised method for object detection and segmentation that lends itself for application in domains where general purpose detectors fail.
Revised: We have proposed a self-supervised method for object detection and segmentation that lends itself for application in domains where general purpose detectors fail.

Original: Our core contributions are the Monte
Revised: Our core contributions are the Monte

Original: Carlo-based optimization of proposal-based detection, new foreground and background objectives, and their joint training on unlabeled videos captured by static, rotating and handheld cameras.
Revised: Carlo-based optimization of proposal-based detection, new foreground and background objectives, and their joint training on unlabeled videos captured by static, rotating and handheld cameras.

Original: Our experiments demonstrate that, even if trained only on single persons, our approach generalizes to multi-person detection, as long as the persons are sufﬁciently separated.
Revised: Our experiments demonstrate that, even if trained only on single persons, our approach generalizes to multi-person detection, as long as the persons are sufﬁciently separated.

Original: In contrast to many existing solutions Barnich & Van Droogenbroeck (2011); Russell et al.
Revised: In contrast to many existing solutions Barnich & Van Droogenbroeck (2011); Russell et al.

Original: ; Croitoru et al.
Revised: ; Croitoru et al.

Original: (2018), our approach does not exploit temporal cues.
Revised: (2018), our approach does not exploit temporal cues.

Original: In the future, we will integrate temporal dependencies explicitly, which will facilitate addressing the scenario where multiple people interact closely, by incorporating physics-inspired constraints enforcing plausible motion.
Revised: In the future, we will integrate temporal dependencies explicitly, which will facilitate addressing the scenario where multiple people interact closely, by incorporating physics-inspired constraints enforcing plausible motion

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In this document, we supply additional implementation details, qualitative results, and a mathematical justiﬁcation for the importance sampling transformations.
Revised: In this document we supply additional implementation details, qualitative results and a mathematical justiﬁcation for the importance sampling transformations

Original: A. I MPLEMENTATION D ETAILS
Revised: A. I MPLEMENTATION D ETAILS

Original: For efﬁciency, W and p , and I and S share parameters, except for their output layers.
Revised: For efﬁciency, W and p , and I and S share parameters, except for their output layers.

Original: In the following, we explain the architecture of each sub-network.
Revised: In the following, we explain the architecture of each sub-network.

Original: Detection network.
Revised: Detection network.

Original: We predict one candidate relative to each cell in a regular grid using a fullyconvolutional architecture similar to YOLO Redmon et al.
Revised: We predict one candidate relative to each cell in a regular grid using a fullyconvolutional architecture similar to YOLO Redmon et al.

Original: We use a ResNet-18 backbone He et al.
Revised: We use a ResNet-18 backbone He et al.

Original: (2016), that reduces the input dimensionality by a factor 8, forming a low resolution grid of features, e.g., to spatial resolution 8 × 8 from 128 × 128 .
Revised: (2016), that reduces the input dimensionality by a factor 8, forming a low resolution grid of features, e g., to spatial resolution 8 × 8 from 128 × 128 .

Original: The feature size is set to 5; two for bounding box location, two for scale, and one for the probability.
Revised: The feature size is set to 5; two for bounding box location, two for scale, and one for the probability.

Original: Each feature output represents one cell, the offset is predicted relative to the cell center as sketched in Fig.
Revised: Each feature output represents one cell, the offset is predicted relative to the cell center as sketched in Fig.

Original: 2 (main document).
Revised: 2 (main document).

Original: The output of p is forced to be positive and form a proper distribution, with (cid:80) Cc =1 p c = 1 , by a soft-max activation unit.
Revised: The output of p is forced to be positive and form a proper distribution, with (cid:80) Cc =1 p c = 1 , by a soft-max activation unit.

Original: To prevent this network from constantly predicting bounding boxes at the borders of the image, we zero the outer cell probabilities.
Revised: To prevent this network from constantly predicting bounding boxes at the borders of the image, we zero the outer cell probabilities.

Original: The synthesis network.
Revised: The synthesis network.

Original: S has the form of a bottle-neck auto-encoder, based on the publicly available implementation of Rhodin et al.
Revised: S has the form of a bottle-neck auto-encoder, based on the publicly available implementation of Rhodin et al.

Original: (2018a).
Revised: (2018a).

Original: The encoding part is a 50-layer residual network and weights are initialized with ones trained on ImageNet classiﬁcation.
Revised: The encoding part is a 50-layer residual network and weights are initialized with ones trained on ImageNet classiﬁcation.

Original: The hidden layer is 728 dimensional, split into andimensional space that goes through two fully connected layers and a 128 dimensional space that is replicated spatially to an 128 × 8 × 8 feature map to encode spatially invariant features.
Revised: The hidden layer is 728 dimensional, split into andimensional space that goes through two fully connected layers and a 128 dimensional space that is replicated spatially to an 128 × 8 × 8 feature map to encode spatially invariant features.

Original: The decoding is done with the second half of a U-Net architecture Ronneberger et al.
Revised: The decoding is done with the second half of a U-Net architecture Ronneberger et al.

Original: (2015) with 32, 64, 128, 256 feature channels in each stage.
Revised: (2015) with 32, 64, 128, 256 feature channels in each stage.

Original: The ﬁnal network layer outputs four feature maps, three to predict the color image
Revised: The ﬁnal network layer outputs four feature maps, three to predict the color image

Original: ˆ I and one for the segmentation mask.
Revised: ˆ I and one for the segmentation mask.

Original: Discussion We use a YOLO detection architecture combined with spatial transformers and syntesis network S .
Revised: Discussion We use a YOLO detection architecture combined with spatial transformers and syntesis network S .

Original: To improve inference speeds, one could merge D and S to share the same base network through RoI pooling as in MaskRCNN.
Revised: To improve inference speeds, one could merge D and S to share the same base network through RoI pooling as in MaskRCNN.

Original: Nevertheless, the MC training approach is necessary, as the inpainting network I can not share the same features for multiple bounding box locations since the blacked-out input region varies for each b c .
Revised: Nevertheless, the MC training approach is necessary, as the inpainting network I can not share the same features for multiple bounding box locations since the blacked-out input region varies for each b c .

Original: The inpainting network.
Revised: The inpainting network.

Original: I is a 6 layer U-Net model with 8, 16, 32, 64, 128, 256 feature channels in each stage.
Revised: I is a 6 layer U-Net model with 8, 16, 32, 64, 128, 256 feature channels in each stage.

Original: The input to this network is an image where the selected bounding box region is hidden and the output is the entire image with the initially hidden patch being reconstructed.
Revised: The input to this network is an image where the selected bounding box region is hidden and the output is the entire image with the initially hidden patch being reconstructed.

Original: It is trained independently from the rest of the pipeline by feeding images with randomly occluded regions of varying sizes.
Revised: It is trained independently from the rest of the pipeline by feeding images with randomly occluded regions of varying sizes.

Original: In our full pipeline, the weights of the inpainting network are frozen and to remove the image evidence corresponding to the foreground person, the hidden patch in the input image to the inpainting network is selected to be bounding box region expanded by 20% in both dimensions.
Revised: In our full pipeline, the weights of the inpainting network are frozen and to remove the image evidence corresponding to the foreground person, the hidden patch in the input image to the inpainting network is selected to be bounding box region expanded by [Figure]

Original: In principle, any off-the-shelf inpainting network trained on large and generic background datasets could be used.
Revised: In principle, any off-the-shelf inpainting network trained on large and generic background datasets could be used.

Original: For instance, the methods of Pathak et al.
Revised: For instance, the methods of Pathak et al.

Original: ; Yu et al.
Revised: ; Yu et al.

Original: (2018) can produce very plausible results.
Revised: (2018) can produce very plausible results.

Original: However, in domain-speciﬁc images, they tend to hallucinate objects as shown in Fig.
Revised: However, in domain-speciﬁc images, they tend to hallucinate objects as shown in Fig.

Original: and are therefore ill-suited to our goal.
Revised: and are therefore ill-suited to our goal.

Original: Instead, we train I from scratch, by reconstructing randomly removed rectangular image regions, as shown 
Revised: Instead, we train I from scratch, by reconstructing randomly removed rectangular image regions, as shown in Fig.

Original: [Figure]
Revised: Note that it is ﬁne for this network not to generalize well to new scenes as it is not needed at test time.

Original: Overall training All training stages are performed on a single NVIDIA TITAN X (Pascal) GPU with Adam and a learning rate of 1e-3.
Revised: Overall training All training stages are performed on a single NVIDIA TITAN X (Pascal) GPU with Adam and a learning rate of 1e-3.

Original: First, the inpainting network is optimized for 200k iterations and subsequently the complete network for additional 100k iterations.
Revised: First, the inpainting network is optimized for 200k iterations and subsequently the complete network for additional 100k iterations.

Original: The decoding pars of the synthesis network S uses a reduced learning rate of 1e-4, to prevent occasional diverging behavior.
Revised: The decoding pars of the synthesis network S uses a reduced learning rate of 1e-4, to prevent occasional diverging behavior.

Original: We use a batch size of 16 and an input image resolution of 640px × 360px for the Ski-PTZ-camera dataset, 640px × 360px for the in-house handheld posing dataset, and 500px × 500px for H36M.
Revised: We use a batch size of 16 and an input image resolution of 640px × 360px for the Ski-PTZ-camera dataset, 640px × 360px for the in-house handheld posing dataset, and 500px × 500px for H36M.

Original:  the importance sampling function q we use (cid:15) = 0 .
Revised: Importance sampling For the importance sampling function q we use (cid:15) = 0 .

Original: 001 , which makes the method numerically stable while the probability of choosing a random bounding box stays low, i.e., 6 . 4% for 64 cells.
Revised: 001 , which makes the method numerically stable while the probability of choosing a random bounding box stays low, i.e., 6 . 4% for 64 cells.

Original: A. Ski-PTZ-Dataset
Revised: A. Ski-PTZ-Dataset

Original: In Table 3 we compare different mask priors and analyze the effect of using pre-trained weights.
Revised: In Table 3 we compare different mask priors and analyze the effect of using pre-trained weights.

Original: Although (cid:96) 1 yields better segmentation masks than (cid:96) 2 , it might suppress the mask values too strictly which causes convergence problems.
Revised: Although (cid:96) 1 yields better segmentation masks than (cid:96) 2 , it might suppress the mask values too strictly which causes convergence problems.

Original: This is mitigated by our (cid:96) v prior, which achieves the highest scores in all measures with consistently reliable results.
Revised: This is mitigated by our (cid:96) v prior, which achieves the highest scores in all measures with consistently reliable results.

Original: Our unsupervised training method sufﬁces for obtaining correct results from scratch.
Revised: Our unsupervised training method sufﬁces for obtaining correct results from scratch.

Original: Nevertheless, given our relatively small dataset, using a pre-trained network signiﬁcantly improves the stability of the training as well as the results.
Revised: Nevertheless, given our relatively small dataset, using a pre-trained network signiﬁcantly improves the stability of the training as well as the results.

Original: To this end we use an ImageNet pre-trained model since it is both practicaland the weights are readily available for diverse architectures, although from a different domain and task.
Revised: To this end we use an ImageNet pre-trained model since it is both practical

Original: To quantify the reliability of our method with the (cid:96) v regularizer, we repeated the Ski-PTZ-Dataset experiment three times with the best-performing conﬁguration and computed the mean and std

Original: [Table]
Revised: [Table]

Original: the Precision, Recall, F- and

Original: J-Measure are consistent, respectively

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] In Table 4 we compare the performance of our method for different values of hyper-parameters, where the subscript of b corresponds to the minimum and maximum size of the bounding box and λ is deﬁned as in Eq.
Revised: 5] , (cid:96) v,λ =0 .0.

Original: [Table] [Table] 5] , (cid:96) v,λ =0 .0. b [0 . 15 , 0 . 5] , (cid:96) v,λ =0 .0.
Revised: b [0 . 15 , 0 . 5] , (cid:96) v,λ =0 .0.

Original: b [0 . 15 , 0 . 5] , (cid:96) v,λ =0 .0.
Revised: b [0 . 15 , 0 . 5] , (cid:96) v,λ =0 .0.

Original: b [0 . 20 , 0 . 5] , (cid:96) v,λ =0 .0. b [0 . 20 , 0 . 5] , (cid:96) v,λ =0 .0.
Revised: b [0 . 20 , 0 . 5] , (cid:96) v,λ =0 .0. b [0 . 20 , 0 . 5] , (cid:96) v,λ =0 .0.

Original: b [0 . 20 , 0 . 5] , (cid:96) v,λ =0 .0. b [0 . 20 , 0 . 5] , (cid:96) v,λ =0 .0.
Revised: b [0 . 20 , 0 . 5] , (cid:96) v,λ =0 .0. b [0 . 20 , 0 . 5] , (cid:96) v,λ =0 .0.

Original: b [0 . 25 , 0 . 5] , (cid:96) v,λ =0 .0. b [0 . 25 , 0 . 5] , (cid:96) v,λ =0 .0.
Revised: b [0 . 25 , 0 . 5] , (cid:96) v,λ =0 .0. b [0 . 25 , 0 . 5] , (cid:96) v,λ =0 .0.

Original: b [0 . 25 , 0 . 5] , (cid:96) v,λ =0 .0. b [0 . 25 , 0 . 5] , (cid:96) v,λ =0 .0.
Revised: b [0 . 25 , 0 . 5] , (cid:96) v,λ =0 .0. b [0 . 25 , 0 . 5] , (cid:96) v,λ =0 .0.

Original: b [0 . 30 , 0 . 5] , (cid:96) v,λ =0 .0. b [0 . 30 , 0 . 5] , (cid:96) v,λ =0 .0.
Revised: b [0 . 30 , 0 . 5] , (cid:96) v,λ =0 .0. b [0 . 30 , 0 . 5] , (cid:96) v,λ =0 .0.

Original: b [0 . 30 , 0 . 5] , (cid:96) v,λ =0 .0. b [0 . 30 , 0 . 5] , (cid:96) v,λ =
Revised: b [0 . 30 , 0 . 5] , (cid:96) v,λ =0 .0. b [0 . 30 , 0 . 5] , (cid:96) v,λ =0 .0.

Original: [Table]

Original: b [0 . 35 , 0 . 5] , (cid:96) v,λ =0 .0. b [0 . 35 , 0 . 5] , (cid:96) v,λ =
Revised: b [0 . 35 , 0 . 5] , (cid:96) v,λ =0 .0. b [0 . 35 , 0 . 5] , (cid:96) v,λ =0 .0.

Original: [Table]

Original: b [0 . 35 , 0 . 5] , (cid:96) v, [Table]
Revised: b [0 . 35 , 0 . 5] , (cid:96) v,λ =0 .0.

Original: = [Table] b [0 . 35 , [Table]
Revised: b [0 . 35 , 0 . 5] , (cid:96) v,λ =0 .0.

Original: [Table] b [ [Table]
Revised: b [0 . 20 , 0 . 55] , (cid:96) v,λ =0 .0.

Original: [Table] [Table] b [0 . 20 , [Table]
Revised: b [0 . 20 , 0 . 60] , (cid:96) v,λ =0 .0.

Original: [Table] b [0 . 20 , 0 . 65] , (cid:96) [Table]
Revised: b [0 . 20 , 0 . 65] , (cid:96) v,λ =0 .0.

Original: [Table]

Original: b [0 . 20 , 0 .
Revised: b [0 . 20 , 0 .

Original: 70] , (cid:96) v, [Table] =
Revised: 70] , (cid:96) v,λ =0 .0.

Original: [Table] b [0 . 20 , 0 . 75] , (cid:96) v,λ =
Revised: b [0 . 20 , 0 . 75] , (cid:96) v,λ =0 .0.

Original: [Table]

Original: b [0 . 20 , 0 . 80] , (cid:96) v,
Revised: b [0 . 20 , 0 . 80] , (cid:96) v,λ =0 .0.

Original: [Table]
Revised: In this table we analyze the effectiveness of our hyper-parameter choice for the minimum and maximum bounding box sizes (given in square brackets) as well as the threshold λ for the (cid:96) v loss. and the weights are readily available for diverse architectures, although from a different domain and task. To quantify the reliability of our method with the (cid:96) v regularizer, we repeated the Ski-PTZ-Dataset experiment three times with the best-performing conﬁguration and computed the mean and std; the Precision, Recall, F- and J-Measure are consistent, respectively, 0 . 54 ± 0 .

Original:  In this table we analyze the 
Revised: 60 ± 0 . 01 , 0 . 74 ± 0 . 51 ± 0 . In Table 4 we compare the performance of our method for different values of hyper-parameters, where the subscript of b corresponds to the minimum and maximum size of the bounding box and λ is deﬁned as in Eq.

Original: [Equation]
Revised: [Equation]

Original: = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:32)
Revised: = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:32)

Original: [Equation]
Revised: [Equation]

Original: W (cid:88) x
Revised: W (cid:88) x

Original: H (cid:88)
Revised: H (cid:88)

Original: y
Revised: y

Original: T − 1 ( S ) xy (cid:33) − λ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) + λ , (7)
Revised: T − 1 ( S ) xy (cid:33) − λ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) + λ , (7)

Original: A. Handheld190k D ATASET
Revised: A. Handheld190k D ATASET

Original: The capture setup of our Handheld190k dataset is depicted in Fig. 8.
Revised: The capture setup of our Handheld190k dataset is depicted in Fig. 8.

Original: The dataset is composed of ﬁve actors performing the same actions as those available in the H36M dataset Ionescu et al.
Revised: The dataset is composed of ﬁve actors performing the same actions as those available in the H36M dataset Ionescu et al.

Original: (2014), namely directions, discussion, eating, greeting, phone talk, posing, buying, sitting, sitting down, smoking, taking photo, waiting, walking, walking dog and walking in pair .
Revised: (2014), namely directions, discussion, eating, greeting, phone talk, posing, buying, sitting, sitting down, smoking, taking photo, waiting, walking, walking dog and walking in pair .

Original: We excluded lying on the ﬂoor actions, to not making our actors lie in the dirt.
Revised: We excluded lying on the ﬂoor actions, to not making our actors lie in the dirt.

Original: The data from three actors compose our training set and the other two form the test set.
Revised: The data from three actors compose our training set and the other two form the test set.

Original: The data was obtained using 3 GoPro6 cameras recording FullHD videos at 30 FPS in linear lens mode.
Revised: The data was obtained using 3 GoPro6 cameras recording FullHD videos at 30 FPS in linear lens mode.

Original: As seen in the Fig.it is an outside recording.
Revised: As seen in the Fig.it is an outside recording.

Original: For the entire duration, the cameras were subject to lateral movement and varying hand-held rotation.
Revised: For the entire duration, the cameras were subject to lateral movement and varying hand-held rotation.

Original: The motion stablization of the GoPros was deactivated at the time of the recording.
Revised: The motion stablization of the GoPros was deactivated at the time of the recording.

Original: A. Q UALITATIVE R ESULTS
Revised: A. Q UALITATIVE R ESULTS

Original: We provide additional examples of our detection and segmentation results on the Handheld190k test set in Fig.
Revised: We provide additional examples of our detection and segmentation results on the Handheld190k test set in Fig.

Original: Fig.
Revised: Fig.

Original: 9 also shows results from the ski test set.
Revised: 9 also shows results from the ski test set.

Original: Additional segmentations on the established H36M dataset are given in Fig.
Revised: Additional segmentations on the established H36M dataset are given in Fig.

Original: Notably, our method is robust to the undirected camera motion and to dynamic background motion, and works equally well for the very different domains of skiing and every-day activities.
Revised: Notably, our method is robust to the undirected camera motion and to dynamic background motion, and works equally well for the very different domains of skiing and every-day activities.

Original: A. I MPORTANCE S AMPLING T HEORY
Revised: A. I MPORTANCE S AMPLING T HEORY

Original: In the following, we give an explanation for the change of distribution when computing an expectation.
Revised: In the following, we give an explanation for the change of distribution when computing an expectation.

Original: Let p, q be two discrete probability distributions and f ( c ) an arbitrary function of c .
Revised: Let p, q be two discrete probability distributions and f ( c ) an arbitrary function of c .

Original: Reasoning about the limit towards inﬁnitely many samples c j drawn from q ( · ) , we derive
Revised: Reasoning about the limit towards inﬁnitely many samples c j drawn from q ( · ) , we derive

Original: [Equation]
Revised: [Equation]

Original: where J is the number of samples drawn and 1 c j = c is one if c j equals c .
Revised: where J is the number of samples drawn and 1 c j = c is one if c j equals c .

Original: In the second line, we exploit that we have a ﬁnite number of classes C .
Revised: In the second line, we exploit that we have a ﬁnite number of classes C .

Original: Each sample must fall into one of them and the probability of coming from cell c is q ( c ) .
Revised: Each sample must fall into one of them and the probability of coming from cell c is q ( c ) .

Original: This relation provides us with a tool to change the sample distribution for expectations.
Revised: This relation provides us with a tool to change the sample distribution for expectations.

Original: Next, we analyze te variance of such estimator depending on the chosen sampling distribution.
Revised: Next, we analyze te variance of such estimator depending on the chosen sampling distribution.

Original: A. I MPORTANCE S AMPLING V ARIANCE
Revised: A. I MPORTANCE S AMPLING V ARIANCE

Original: The variance of an estimator gives us a measure for the expected accuracy after a limited number of samples.
Revised: The variance of an estimator gives us a measure for the expected accuracy after a limited number of samples.

Original: We use it to compare the three candidates discussed in the main document.
Revised: We use it to compare the three candidates discussed in the main document.

Original: The variance of estimating the objective
Revised: The variance of estimating the objective

Original: O with a monte carlo sum over c 1 , . . .
Revised: O with a monte carlo sum over c 1 , . . .

Original: c J samples drawn independently from p is
Revised: c J samples drawn independently from p is

Original: Var [ O ( I )]
Revised: Var [ O ( I )]

Original: ≈ Var (cid:34)
Revised: ≈ Var (cid:34)

Original: J
Revised: J

Original: J (cid:88) j
Revised: J (cid:88) j

Original: =
Revised: =

Original: L ( F ( I , c j ) , I ) (cid:35) =
Revised: L ( F ( I , c j ) , I ) (cid:35) =

Original: 1 J
Revised: 1 J

Original: J (cid:88) j
Revised: J (cid:88) j

Original: =
Revised: =

Original: Var [ L ( F ( I , c j ) , I )]
Revised: Var [ L ( F ( I , c j ) , I )]

Original: = 1 J Var [ L ( F ( I , c j ) , I )] .
Revised: = 1 J Var [ L ( F ( I , c j ) , I )] .

Original: [Figure]
Revised: [Figure]

Revised: Rhodin

Original: [Figure]
Revised: [Figure]

Original: Here Var [ L ( F ( I , c j ) , I )] is the variance over the random variable c j , and we utilized the identity Var[ ax ] = a 2 Var[ x ] and the independence of samples.
Revised: Here Var [ L ( F ( I , c j ) , I )] is the variance over the random variable c j , and we utilized the identity Var[ ax ] = a 2 Var[ x ] and the independence of samples.

Original: The variance reduces linearly with the number of samples and is proportional to that of L ( · ) .
Revised: The variance reduces linearly with the number of samples and is proportional to that of L ( · ) .

Original: In contrast, using uniform sampling (Eq. 4, main document, with q = U c ), yields a quadratic variance growth with the number of cells,
Revised: In contrast, using uniform sampling (Eq. 4, main document, with q = U c ), yields a quadratic variance growth with the number of cells,

Original: [Equation]
Revised: [Equation]

Original: From which we conclude that uniform sampling leads to a higher variance, since the remaining Var[ p ( · ) L ( · )] term is not expected to improve on Var[ L ( · )] .
Revised: From which we conclude that uniform sampling leads to a higher variance, since the remaining Var[ p ( · ) L ( · )] term is not expected to improve on Var[ L ( · )] .

Original: With importance sampling according to q (Eq. 4, main document) the variance is,
Revised: With importance sampling according to q (Eq. 4, main document) the variance is,

Original: [Equation]
Revised: [Equation]

Original: If q ≈ p it is equivalent to the one of Eq.
Revised: If q ≈ p it is equivalent to the one of Eq.

Original: In general, q should be constructed to minimize Eq.
Revised: In general, q should be constructed to minimize Eq.

Original: In our case, F depends on each individual image and cell c , it is thereby difﬁcult to impose assumptions to reduce the variance of L without evaluating it for each c .
Revised: In our case, F depends on each individual image and cell c , it is thereby difﬁcult to impose assumptions to reduce the variance of L without evaluating it for each c .

Original: Therefore, setting q ≈ p is a good choice from the perspective of variance reduction.
Revised: Therefore, setting q ≈ p is a good choice from the perspective of variance reduction.

Original: A. A DDITIONAL C OMPARISONS TO R ELATED W ORK
Revised: A. A DDITIONAL C OMPARISONS TO R ELATED W ORK

Original: Motion-based approaches are prone to fail when there is no or too complex motion information to separate foreground and background and in textureless areas.
Revised: Motion-based approaches are prone to fail when there is no or too complex motion information to separate foreground and background and in textureless areas.

Original: These failure cases occur in all our scenarios, as shown in the optical ﬂow images in Fig.
Revised: These failure cases occur in all our scenarios, as shown in the optical ﬂow images in Fig.

Original: 11 generated from our datasets.
Revised: 11 generated from our datasets.

Original: To this extend, we excluded motion cues in this study.
Revised: To this extend, we excluded motion cues in this study.

Original: We are planning on integrating their complementary merits in the future.
Revised: We are planning on integrating their complementary merits in the future.

Original: We also investigate the possibility of using a pre-trained localization network from Zhou et al.
Revised: We also investigate the possibility of using a pre-trained localization network from Zhou et al.

Original: and obtain the heatmaps and the corresponding bounding box predictions for H36M S9 and S11 test images using their
Revised: and obtain the heatmaps and the corresponding bounding box predictions for H36M S9 and S11 test images using their

Original: GoogLeNet-CAM pre-trained model on ImageNet.
Revised: GoogLeNet-CAM pre-trained model on ImageNet.

Original: As the output heatmaps in Fig.
Revised: As the output heatmaps in Fig.

Original: 12 show, the pre-training on ImageNet alone is not reliable to have accurate detections.
Revised: 12 show the pre-training on ImageNet alone is not reliable to have accurate detections.

Original: This is mainly due to the multiple activated regions that yield very big bounding boxes covering not just the human subject but also the other objects in the scene matching with the ImageNet classes.
Revised: This is mainly due to the multiple activated regions that yield very big bounding boxes that frame. We report has Our can accurately detect reader

Original: Among all the bounding box predictions that Zhou et al.
Revised: Table

Original: (2016) generates, we select the one yielding the highest IOU score against the groundtruth bounding box corresponding to that frame. We report that Zhou et al.
Revised: ImageNet Another part of our approach where we can provide additional comparison is the sampling method. To explore an alternative strategy to Monte-Carlo based sampling, we tried replacing the importance sampling part in our method with the categorical reparameterization used in Crawford & Pineau (2019).

Original: (2016) has an mAP 0 .
Revised: Since both strategies approximate the same objective, they should lead to very similar outcomes with a possible difference in the convergence speed. To this end we used Gumbel-Softmax distribution and tried this estimator with several different temperature values. Our experiments show that Gumbel-Softmax based categorical reparameterization didn’t lead to faster convergence as shown in Fig.

Original: 5 score of 0.1 which is signiﬁcantly lower than our mAP 0 .
Revised: This might be partly due to the high value of Gumbel noise added to the log probabilities.

Original: 5 score of 0.58.
Revised: However, to have a more solid claim, a principled grid search for hyperparameters such as the temperature is necessary.

Original: This gap indicates that doing the detection using only their pre-trained model is not reliable and selection of the best bounding box among the candidates isunfeasible in a self-supervised setting. Our model can accurately detect the human subject without needing an ImageNet pre-training in thedetection component.
Revised: We will provide the exact comparison in our ﬁnal version after conducting an extensive hyperparameter search. Overall our importance sampling approach is simpler compared to their approach and has the advantage of being an unbiased estimator. In addition to that it does not need custom layers that behave differently in the forward and backwards passes during optimization, is the for the

Revised: Gumbel- categorical reparameterization.