  Instance level detection and segmentation of thoracic diseases or
abnormalities are crucial for automatic diagnosis in chest X-ray images.
Leveraging on constant structure and disease relations extracted from domain
knowledge, we propose a structure-aware relation network (SAR-Net) extending
Mask R-CNN. The SAR-Net consists of three relation modules: 1. the anatomical
structure relation module encoding spatial relations between diseases and
anatomical parts. 2. the contextual relation module aggregating clues based on
query-key pair of disease RoI and lung fields. 3. the disease relation module
propagating co-occurrence and causal relations into disease proposals. Towards
making a practical system, we also provide ChestX-Det, a chest X-Ray dataset
with instance-level annotations (boxes and masks). ChestX-Det is a subset of
the public dataset NIH ChestX-ray14. It contains ~3500 images of 13 common
disease categories labeled by three board-certified radiologists. We evaluate
our SAR-Net on it and another dataset DR-Private. Experimental results show
that it can enhance the strong baseline of Mask R-CNN with significant
improvements. The ChestX-Det is released at
https://github.com/Deepwise-AILab/ChestX-Det-Dataset.
