  In this paper, we present a holistically end-to-end algorithm for line
segment detection with transformers that is post-processing and
heuristics-guided intermediate processing (edge/junction/region detection)
free. Our method, named LinE segment TRansformers (LETR), tackles the three
main problems in this domain, namely edge element detection, perceptual
grouping, and holistic inference by three highlights in detection transformers
(DETR) including tokenized queries with integrated encoding and decoding,
self-attention, and joint queries respectively. The transformers learn to
progressively refine line segments through layers of self-attention mechanism
skipping the heuristic design in the previous line segmentation algorithms. We
equip multi-scale encoder/decoder in the transformers to perform fine-grained
line segment detection under a direct end-point distance loss that is
particularly suitable for entities such as line segments that are not
conveniently represented by bounding boxes. In the experiments, we show
state-of-the-art results on Wireframe and YorkUrban benchmarks. LETR points to
a promising direction for joint end-to-end detection of general entities beyond
the standard object bounding box representation.
