ylh22/DR2Seg

DR²Seg is a reasoning segmentation framework for multimodal large language models that improves both segmentation accuracy and reasoning efficiency through a decomposed two-stage rollout strategy.

DR$^2$Seg decomposes reasoning segmentation into two rollout passes of the same MLLM during training:

1. First pass: (I, Q) -> (R^1, D, A^1)

   • R^1: first-pass reasoning chain
   • D: explicit description of the target
   • A^1: spatial answer, including boxes and points

2. Second pass: (I, D) -> (R^2, A^2)

   • the original complex query is replaced by the generated description
   • the model reasons again based on a more explicit target description

In the first stage, the MLLM generates a self-contained description that explicitly specifies the target object. In the second stage, this description replaces the original complex query so the model can verify whether the inferred target is correct and sufficiently localized. Based on this design, DR²Seg introduces two self-rewards during reinforcement learning:

1. Description self-reward: encourages the first-pass description to be faithful and self-contained when the second-pass answer is correct.
2. Length-based self-reward: encourages the second-pass reasoning to be shorter than the first pass, reducing redundant thinking tokens and mitigating overthinking.