Paper:
There are myriad kinds of segmentation, and ultimately the "right" segmentation of a given scene is in the eye of the annotator. Standard approaches require large amounts of labeled data to learn just one particular kind of segmentation. As a first step towards relieving this annotation burden, we propose the problem of guided segmentation: given varying amounts of pixel-wise labels, segment unannotated pixels by propagating supervision locally (within an image) and non-locally (across images). We propose guided networks, which extract a latent task representationguidance-from variable amounts and classes (categories, instances, etc.) of pixel supervision and optimize our architecture end-to-end for fast, accurate, and dataefficient segmentation by meta-learning. To span the few-shot and many-shot learning regimes, we examine guidance from as little as one pixel per concept to as much as 1000+ images, and compare to full gradient optimization at both extremes.
To explore generalization, we analyze guidance as a bridge between different levels of supervision to segment classes as the union of instances. Our segmentor concentrates different amounts of supervision of different types of classes into an efficient latent representation, non-locally propagates this supervision across images, and can be updated quickly and cumulatively when given more supervision.

Review:
Summary
This paper proposes to formulate diverse segmentation problems as a guided segmentation, whose task is defined by the guiding annotations.
The main idea of this paper is using meta-learning to train a single neural network performing guidance segmentation.
Specifically, they encode S annotated support image into a task representation and use it to perform binary segmentation.
By performing episodic optimisation, the model's guidance to segmentation output is defined by the task distribution.
Strength
Learning a single segmentation algorithm to solve various segmentation problem is an interesting problem that worth exploring.
This paper tackles this problem and showed results on various segmentation problems.
Weakness
The proposed method, including the architecture and training strategy, is relatively simple and very closely related to existing approach. Especially, the only differences with the referenced paper (Shaban et al., 2017) is how the support is fused and how multiple guidance could be handled, which can be done by averaging. These differences are relatively minor, so I question the novelty of this paper.
This paper performs experiments on diverse tasks but the method is compared with relatively weak baselines absolute performance looks bad compared to existing algorithms exploiting prior knowledge for each of the tasks.
For example, the oracle performance in semantic segmentation (fully supervised method) is 0.45 IOU in PASCAL VOC dataset, while many existing algorithms could achieve more than 0.8 mean IOU in this dataset. 
In addition, I question whether foreground / background baseline is reasonable baseline for all these tasks, because a little domain knowledge might already give very strong result on various segmentation tasks.
For example, in terms of video segmentation, one trivial baseline might include propagating ground truth labels in the first frame with color and spatial location similarity, which might be already stronger than the foreground / background baseline.
There are some strong arguments that require further justification. 
- In 4.3, authors argue that the model is trained with S=1, but could operate with different (S, P).
However, it's suspicious whether this would be really true, because it requires generalisation to out-of-distribution examples, which is very difficult machine learning problem. The performance in Figure 5 (right) might support the difficulty of this generalisation, because increasing S does not necessarily increase the performance.
- In 5.3, this paper investigated whether the model trained with instances could be used for semantic segmentation. I think performing semantic segmentation with model trained for instance segmentation in the same dataset might show reasonable performance, but this might be just because there are many images with single instance in each image and because instance annotations in this dataset are based on semantic classes. So the argument that training with instance segmentation lead to semantic segmentation should be more carefully made.
Overall comment
I believe the method proposed in this paper is rather incremental and analysis is not supporting the main arguments of this paper and strength of the proposed method. 
Especially, simple performance comparison with weak baselines give no clues about the property of the method and advantage of using this method compared to other existing approaches.