  In this work we address the task of observing the performance of a semantic
segmentation deep neural network (DNN) during online operation, i.e., during
inference, which is of high importance in safety-critical applications such as
autonomous driving. Here, many high-level decisions rely on such DNNs, which
are usually evaluated offline, while their performance in online operation
remains unknown. To solve this problem, we propose an improved online
performance prediction scheme, building on a recently proposed concept of
predicting the primary semantic segmentation task's performance. This can be
achieved by evaluating the auxiliary task of monocular depth estimation with a
measurement supplied by a LiDAR sensor and a subsequent regression to the
semantic segmentation performance. In particular, we propose (i) sequential
training methods for both tasks in a multi-task training setup, (ii) to share
the encoder as well as parts of the decoder between both task's networks for
improved efficiency, and (iii) a temporal statistics aggregation method, which
significantly reduces the performance prediction error at the cost of a small
algorithmic latency. Evaluation on the KITTI dataset shows that all three
aspects improve the performance prediction compared to previous approaches.
