  Mapping new and old buildings are of great significance for understanding
socio-economic development in rural areas. In recent years, deep neural
networks have achieved remarkable building segmentation results in
high-resolution remote sensing images. However, the scarce training data and
the varying geographical environments have posed challenges for scalable
building segmentation. This study proposes a novel framework based on Mask
R-CNN, named HTMask R-CNN, to extract new and old rural buildings even when the
label is scarce. The framework adopts the result of single-object instance
segmentation from the orthodox Mask R-CNN. Further, it classifies the rural
buildings into new and old ones based on a dynamic grayscale threshold inferred
from the result of a two-object instance segmentation task where training data
is scarce. We found that the framework can extract more buildings and achieve a
much higher mean Average Precision (mAP) than the orthodox Mask R-CNN model. We
tested the novel framework's performance with increasing training data and
found that it converged even when the training samples were limited. This
framework's main contribution is to allow scalable segmentation by using
significantly fewer training samples than traditional machine learning
practices. That makes mapping China's new and old rural buildings viable.
