Paper: Symbol as Points: Panoptic Symbol Spotting via Point-based Representation (ICLR-2024)

Reviewer comment (Novelty):
Perhaps I missunderstood something, but I do not see the motivation of symbol segmentation in CAD drawings. As far as I understand CAD drawings should already contain information about the symbols included in the floorplan and where they are located.

Author response:
While tools like AutoCAD enable the rapid creation of  CAD drawing, it typically lack detailed categorization information for each graphical primitive or symbol. The task of  panoptic symbol spotting aims to effectively extract and interpret this information in graphic primitive granularity.  Accurately spotting symbols is crucial for  Building Information Modeling(BIM), as the picture shows here: https://anonymous.4open.science/r/x-BB39/bim.png .

Reviewer comment (Theory):
In equation (2) I understand that l_k is the distance between v_1 and v_2. Then, what about circles and ellipses? How is the lengh computed? And for arcs, this definition does not account for the curvature. Two arcs with very different curvature can have the same representation.

Author response:
* **Regarding Equation (2) and Primitive Representation**:  $l_k$ represents the distance between $v_1$ and $v_2$ for linear primitives. For circular primitives like circles and ellipses, $l_k$ is defined as the circumference.  For arcs, $l_k$ means the length of curve, not the distance between $v_1$ and $v_2$. Since all arcs in CAD are quadratic Bezier curves, we can easily compute the length of curves using the python package `svgpathtools`

Reviewer comment (Writing):
In equation (3), it is not clear how the neighbourhood M(p_i) is defined. Do adjacent points mean connected primitives? Or primitives inside a certain distance? Which is exactly the difference with A(p_i) defined later in section 3.3 (given that the threshold used in section 3.3 is just one pixel). In this sense, the role of the ACM module is not very clear.

Author response:
* **On Defining Neighborhood $M(p_i)$ in Equation (3):**   In $M(p_i)$, adjacent points means top K nearest primitives which are measured by the distance between the primitive positions (Eq 1) and $p_i$, not locally connected primitives.  $A(p_i)$ means a combination of $M(p_i)$ and $C(p_i)$ which means locally connected primitive points (Eq. 8). The reason that we use a small threshold in Eq. 7 to define the connectivity of two primitives is that for most CAD drawings, the interconnected endpoints of two connected lines are not the same point, but two points that are very close to each other.  We therefore regard lines whose endpoints are close enough as connected.

Reviewer comment (Reproducibility):
Related to the previous point, It is not clear how it is performed downsamplind and upsampling in the Point Transformer. The same as in the original Point Transformer?

Author response:
* **Downsampling and Upsampling in Point Transformer:** The downsampling and upsampling processes of point features is shown in https://anonymous.4open.science/r/x-BB39/pool&unpool.png, which is from [R2]. Adjacent point features are downsampled by pooling to get a point feature, which could also be upsampled through unpooling.

Reviewer comment (Writing):
In equation (12) it is not clear what is e_i and L(e_i).

Author response:
* **Explanation of Equation (12):**  $e_i$ is a graphical entities, $L(e_i)$ is the length of the graphical entitity $e_i$

Reviewer comment (Evaluation):
In the experiments, which is the difference between Semantic and Panoptic Symbol Segmentation? Why in table 1 (semantic segmentation) the evaluation measure is F1? How are F1 and wF1 defined in this context?

Author response:
* **On Experimental Evaluation:**  (1) Semantic Symbol Segmentation does not distinguish between objects of the same category, while Panoptic Symbol Segmentation does. (2) We follow previous works such as PanCADNet [R3], and GAT-CADNet [R4] to use F1 as metric for fair comparison;  (3) The F1 is the harmonic mean of Precision and Recall, The wF1 is length-weighted F1. Both metrics are well defined in [R3]. We will provide the detail definition in the supplementary material in future.

Reviewer comment (Novelty):
A primary concern from the reviewer centers on the paper's predominant reliance on existing methodologies to address the issue. Specifically, in Sec3.1 (From Symbol to Point), many parameterizations echo those found in FloorplanCAD, albeit this paper seeks to enhance the diversity of encoded features. The point-based representation in Sec 3.2 directly employs the Point Transformer, reminiscent of CADTransformer. Both Contrastive Connection Learning (Sec3.4) and KNN Interpolation (Sec3.5) have been thoroughly examined in other scholarly works. While the "Attention with Connection Module" presents as novel to the reviewer, it would be beneficial to undertake a comprehensive review to discern if analogous concepts have been in previous literature.

Author response:
* While our approach to primitive features bears a resemblance to the vertex features defined in GAT-CADNet, our method focuses on leveraging the most straightforward and fundamental features in CAD drawings, such as type, length, and clockwise angle. These features, while simple, are highly effective in capturing the essence of individual primitives. Although more complex features could potentially be utilized, we found that these basic attributes yield impressive results, striking a balance between simplicity and effectiveness.
* Compared to CADTransformer, our method is simple yet effective, marked by three key distinctions:
  * Our process eliminates the need for rasterization of CAD drawings to images, which is a time-intensive step required for many CAD segmentation methods including CADTransformer.
  * We rely on simple and direct primitive features, as opposed to CADTransformer's use of HRNet for feature extraction. This approach not only simplifies the process but also enhances the scalability of our method to complex, high-resolution CAD drawings.
  * Our method employs the Point Transformer for interactions with each graphic primitive, whereas CADTransformer utilizes the Vision Transformer (ViT), which more easily occurs OOM when dealing with high resolution CAD drawings;
  * Our methodology is designed for end-to-end training without the necessity for any post-processing operations. In contrast, CADTransformer requires the use of a clustering algorithm to derive the final instance results, adding additional steps to the process;

Reviewer comment (Experiments):
In Table 4, where the benchmark approach registers a PQ of 73.1, could you detail the design of how this baseline method is formulated?

Author response:
Our baseline is a simple combination of Point Transformer and Mask2Former's transformer decoder with input of point-based representation.  This serves as a foundation for comparing the enhanced performance of SymPoint.

Reviewer comment (Evaluation):
In Table 4, it appears the newly introduced "ACM" module inadvertently undermines performance. Could the authors shed light on the causative factors behind this decline?

Author response:
Our original intention in introducing ACM was to utilize these connections between each graphic primitive.  we conduct experiments in SESYD-floorplans dataset that is smaller than floorplanCAD,  ACM can significantly promote performance and accelerate the model convergence.  An convergence curve between without/with ACM is shown here: https://anonymous.4open.science/r/x-BB39/sesyd_val_loss.png, https://anonymous.4open.science/r/x-BB39/sesyd_val_PQ.png, https://anonymous.4open.science/r/x-BB39/sesyd_val_RQ.png, https://anonymous.4open.science/r/x-BB39/sesyd_val_SQ.png,   The quantitative results are shown below,

|              | PQ    | RQ    | SQ    |
| ------------ | ----- | ----- | ----- |
| baseline     | 88.23 | 91.61 | 96.01 |
| baseline+ACM | 91.98 | 95.99 | 95.85 |

But, floorplanCAD is more complex compared with SESYD-floorplans, and more noisy connections could be introduced. As this figure shows : https://anonymous.4open.science/r/x-BB39/noisy-connection.png . Therefore, we have introduced an additional Contrastive Connection Learning to mitigate the impact of noise connections and more effectively utilize connection information with category consistency.  Ablation study in Table 4 (a) verifies the effectiveness of these modules.

Reviewer comment (Presentation):
Again, referencing Table 4, the KInter technique emerges as a salient contributor to performance enhancement. Could the authors offer a more clear explanation and visualization? It might also be worthwhile to highlight this module within the methods section.

Author response:
While bilinear interpolation, as utilized in Mask2Former, is tailored for regular data, such as image,  but it is  unsuitable for irregular sparse primitive points. Here, We provided some visualizations of point masks for KNN interpolation and bilinear interpolation as shown in https://anonymous.4open.science/r/x-BB39/vis2-knn_interp-vs-bilinear_interp.png.  Note that these point masks are soft masks (ranging from 0 to 1) predicted by intermediate layers. After downsampling the point mask to 4x and 16x, we can clearly find that KNN interpolation well perverse the original mask information interpolation, while  bilinear interpolation causes a significant information loss, which could harm the final performance.

Reviewer comment (Novelty):
The originality and technical contribution of this work is quite limited. Point Transformer, Mask2Former and InfoNCE are all well-established methods or models.

Author response:
**1. Regarding Originality and Technical Contribution:**

While SymPoint incorporates well-established methods like Point Transformer, Mask2Former, and InfoNCE, our innovation lies in adapting and optimizing these methods for the specific challenge of CAD vector graphics recognition.  **It is worth noting that the combination of Point Transformer and Mask2Former is just our baseline.**

Specifically, the novelties of our paper lie in several aspects: 

* We carefully analyzing the data characteristics of CAD drawings and design novel and effective way of transferring CAD primitive entities into point feature in Section 3.1; 
* We propose Attention with Connection Module (ACM) , which is novel in efficiently capturing the local connection information of primitives and enhancing their discriminability when in conjunction with Contrastive Connection Learning (CCL) module in the context of CAD graphics in Section 3.3 and 3.4 ;
* We propose KNN Interpolation to downsample  attention mask for masked attention calculation, which shows superior performance compared with the naive bilinear interpolation used in Mask2Former in Section 3.5.  

Our ablation study in Table 4 (a) has demonstrated that these modules effectively promoted the performance of the model. These strategies represent a unique approach not previously explored in this field.  **We achieves 77.3/87.1/ 88.7 (PQ/RQ/SQ) with these novel designs, significantly outperforming the baseline 73.1/83.3/87.7 (PQ/RQ/SQ) . Note that our baseline method has already surpassed many state-of-the-art methods, such as PanCADNet and CADTransformer.**

Reviewer comment (Novelty):
The potential application range of the proposed method can be narrow (CAD vector graphics), because it is unclear whether the idea and techniques presented in this work can be extended to other tasks.

Author response:
**2. On the Potential Application Range:**

In response to the concern regarding the application range of SymPoint, we acknowledge that our current focus is primarily on CAD vector graphics. However, we believe there is potential for applying our method to a broader range of fields, like **3D modeling[R1]** and **circuit design[R2]**, **sketch segmentation[R3]**  as shown here:  https://anonymous.4open.science/r/x-BB39/application.png. **We have conduct some preliminary experiments on circuit design in the appendix A.2. and achieve state-of-the-art performance on two datasets as shown in Table 7.** Although this beyond the scope of this paper, we see it as a valuable direction for future research.

[R1] Lv, Xiaolei, et al. "Residential floor plan recognition and reconstruction" ICCV. 2021.

[R2] Jiang, Xinyang, et al. "Recognizing vector graphics without rasterization" NeurIPS. 2021.

[R3]Yang, Lumin , et al. "Sketchgnn: Semantic sketch segmentation with graph neural networks" TOG. 2023.

Reviewer comment (Novelty):
The authors should explain and verify the originality and technical contribution of the proposed method.

Author response:
**1. Regarding Originality and Technical Contribution:**

While SymPoint incorporates well-established methods like Point Transformer, Mask2Former, and InfoNCE, our innovation lies in adapting and optimizing these methods for the specific challenge of CAD vector graphics recognition.  **It is worth noting that the combination of Point Transformer and Mask2Former is just our baseline.**

Specifically, the novelties of our paper lie in several aspects: 

* We carefully analyzing the data characteristics of CAD drawings and design novel and effective way of transferring CAD primitive entities into point feature in Section 3.1; 
* We propose Attention with Connection Module (ACM) , which is novel in efficiently capturing the local connection information of primitives and enhancing their discriminability when in conjunction with Contrastive Connection Learning (CCL) module in the context of CAD graphics in Section 3.3 and 3.4 ;
* We propose KNN Interpolation to downsample  attention mask for masked attention calculation, which shows superior performance compared with the naive bilinear interpolation used in Mask2Former in Section 3.5.  

Our ablation study in Table 4 (a) has demonstrated that these modules effectively promoted the performance of the model. These strategies represent a unique approach not previously explored in this field.  **We achieves 77.3/87.1/ 88.7 (PQ/RQ/SQ) with these novel designs, significantly outperforming the baseline 73.1/83.3/87.7 (PQ/RQ/SQ) . Note that our baseline method has already surpassed many state-of-the-art methods, such as PanCADNet and CADTransformer.**