  Estimating the heightmaps of buildings and vegetation in single remotely
sensed images is a challenging problem. Effective solutions to this problem can
comprise the stepping stone for solving complex and demanding problems that
require 3D information of aerial imagery in the remote sensing discipline,
which might be expensive or not feasible to require. We propose a task-focused
Deep Learning (DL) model that takes advantage of the shadow map of a remotely
sensed image to calculate its heightmap. The shadow is computed efficiently and
does not add significant computation complexity. The model is trained with
aerial images and their Lidar measurements, achieving superior performance on
the task. We validate the model with a dataset covering a large area of
Manchester, UK, as well as the 2018 IEEE GRSS Data Fusion Contest Lidar
dataset. Our work suggests that the proposed DL architecture and the technique
of injecting shadows information into the model are valuable for improving the
heightmap estimation task for single remotely sensed imagery.
