  Aerial Image Segmentation is a particular semantic segmentation problem and
has several challenging characteristics that general semantic segmentation does
not have. There are two critical issues: The one is an extremely
foreground-background imbalanced distribution, and the other is multiple small
objects along with the complex background. Such problems make the recent dense
affinity context modeling perform poorly even compared with baselines due to
over-introduced background context. To handle these problems, we propose a
point-wise affinity propagation module based on the Feature Pyramid Network
(FPN) framework, named PointFlow. Rather than dense affinity learning, a sparse
affinity map is generated upon selected points between the adjacent features,
which reduces the noise introduced by the background while keeping efficiency.
In particular, we design a dual point matcher to select points from the salient
area and object boundaries, respectively. Experimental results on three
different aerial segmentation datasets suggest that the proposed method is more
effective and efficient than state-of-the-art general semantic segmentation
methods. Especially, our methods achieve the best speed and accuracy trade-off
on three aerial benchmarks. Further experiments on three general semantic
segmentation datasets prove the generality of our method. Code will be provided
in (https: //github.com/lxtGH/PFSegNets).
