mr CAD : Multimodal Communication to Refine Computer-aided Designs (2025)

Abstract: In collaborative creation tasks, people steer artifacts towards specific goals by _refining_ them with _multimodal_ communication over multiple rounds of interaction. In contrast, generative AI excels at creating artifacts in a single turn but can struggle to make precise refinements that match our design intent. To close this gap, we present mrCAD, a dataset of multi-turn interactions in which pairs of humans iteratively created and refined computer-aided designs (CADs). In each game, a _Designer sent instructions to a _Maker_, explaining how to create and subsequently refine a CAD to match a target design that only the _Designer_ could see. mrCAD consists of 6,082 communication games, 15,163 instruction-execution rounds, played between 1,092 pairs of human players. Crucially, _Designers_ had access to two communication modalities – text and drawing. Analysis finds that players relied more on text in refinement than in initial generation instructions, and used different linguistic elements for refinement than for generation. We also find that state-of-the-art VLMs are better at following generation instructions than refinement instructions. These results lay the foundation for modeling multi-turn, multimodal communication not captured in prior datasets.

## introduction recent advances in generative ai allow people to ask for the creation of artifacts in natural, conversational language. however, while many models generate high-quality output in response, they often struggle to process complex, multi-turn requests @xcite . this challenge is especially pronounced in tasks such as image editing @xcite and code generation @xcite , resulting in frustration with ai-generated content that is "nearly there" but difficult to refine to precisely match user intent. on the other hand, the ability of humans to iteratively refine communicated concepts is a core mechanism used to ensure robustness of communication @xcite , as well as a crucial part of the design process @xcite . training ai systems to understand these forms of repair is particularly challenging given that they are typically multimodal, relying on the interaction between different communication modalities to make their meaning precise. we present mrcad, a resource for studying and benchmarking multimodal refinement communication in a computer-aided design (cad) setting. our task (figure 1 ) elicits the following phenomena: (1) contextual dependence, where the refinement instructions must be interpreted in the context of the current cad ( § 4); (2) interactions between modalities, where drawings and texts must be interpreted together ( § 4); and (3) monotonicity, where participants communicate to iteratively shrink the distance between the current cad and the target cad ( § 4 and § 5). these phenomena capture how humans can communicate flexibly to accomplish precise tasks such as producing cad designs. we highlight two main uses of mrcad: (1) as a benchmark for evaluating multimodal, multiturn instruction following systems such as those powered by vision-language models, and (2) as a resource for understanding the nature of human refinement communication. benchmarking cad reconstruction is particularly attractive as an instruction following benchmark for the following reasons: (1) the cad reconstruction accuracy can be programmatically defined, rather than relying on learned metrics such as clip @xcite . (2) cad programs can be easily manipulated using clicks and drags (similar to a slide design software), making it easy to evaluate both tool-using agents and human participants. (3) the target cad designs in mrcad are sourced from sketchgraph @xcite , consisting of naturalistic 2d designs created by human designers, rather than synthetically generated from a domain-specific language that the authors have crafted by hand. understanding as a resource for understanding the nature of multimodal refinement in humanhuman collaboration, mrcad is large scale, containing 6,082 human-to-human plays of the game, with a total of 15,163 rounds of instruction. a large number of cads in our dataset are recreated multiple times by different dyads, ranging from 2 to 30 games per design, capturing the natural variation in refinement strategies across people. a notable feature of cad is that while it only contains simple curves, they combine form rich semantic objects and sub-parts (like the "flower" and "petals" in figure 1 ), which must be parsed contextually @xcite . this is in contrast to works such as zitnick and parikh (2013), where each object in the scene has a canonical, pre-defined name. our work makes the following contributions: the mrcad task (fig. 1 ) is a two player, multiturn communication game. following mccarthy et al. (2024), a designer issues multimodal instructions to guide a maker to take actions in a mutually observable cad environment to construct a target cad, known only to the designer. 2.1 mrcad environment state the state d is a cad data structure. d -> { curve , ...} curve -> line | circle | arc line -> l(p , p) // end points circle -> c (p ,p ) // points on diameter arc -> a(p ,p ,p ) // start , mid , end each curve is defined by its control points that the maker selects. a render function allows players to view the state by rendering the design as an image. action the maker can alter the cad by making, removing, and moving curves via translation. they can also change the shape of a curve by moving its control points. if multiple curves share the same control point, moving the point would modify all the curves, and deleting the point would delete all the adjacent curves. action -> make_curve curve | remove_curve curve | move_curve curve vxy | move_point p p | delete_point p transition actions ⃗ @xmath0, a 2 , . . . , a n ] can be applied to a design, which results in a new design. the asymmetric chamfer distance from cad d to cad e is calculated by sampling 10 points on every curve of d, and calculating (symbolically) the minimum distance from each point to e. each distance is then normalized by multiplying 1 4 of the maximum size of the canvas, making it invariant to the canvas size. these distances are summed, making the asymmetric chamfer distance. the symmetric chamfer distance we use is the average of both directions. distance metric cads are programmatic entities that exist in a visual space, where very different elements could be used to create two cads of similar appearance. for example, a series of straight lines can be used to represent a curve. therefore, we devise a distance metric ∆ based on chamfer distance (butt and maragos, 1998) that respects the vector-like nature of cads while also accounting for geometric similarity (fig. 2 ). ## effects of multimodality and refinement on communication we first confirm that multimodality and refinement do indeed play important roles in communication of designs. we conduct an ablation study by manipulating the kind of communication schemes available to the participants (details in appendix b.1): results are summarized in figure 3 . in conclusion: (1) drawing is important, and (2) refinement is important. note that while drawing-only performed as well as multimodal in this study (figure 3a ), in the multimodal condition participants still used more text and less drawing over time (figure 3b , c ). this points to a preference to use more drawing in communicating generation and more text in communicating refinements. statistical analysis of results are given in appendix b.2. ## the mrcad dataset we detail the collection of the dataset, present the main statistics of our dataset, and a set of basic analyses that reveals the striking effects of multimodality and refinement on communication. we also include a gallery of multimodal refinement rounds (figure 5 ), highlighting the intriguing yet unexplored aspects of the dataset for future research. ## evaluation of frontier models we evaluate both api access models and openweights models (which we finetune) on the mrcad dataset. we are interested in the following. rq1 what is the difference in performance between humans and models in following the instructions in mrcad? rq2 what is the effect of generation vs refinement in model performance ? rq3 what is the effect of multimodal vs singlemodal in model performance? ## discussion we present mrcad, a large-scale dataset of humans playing multi-turn, multimodal communication games in a cad environment. analysis revealed significant differences between instructions to generate and instructions to refine, notably a trend towards the complementary use of drawing and text when refining designs. this distinction is mirrored by a severe performance gap in frontier vlms ability to generate vs to refine cads. we posit that this is due to the lack of multimodal refinement data on the internet, from which these models are trained: (i) lack of drawing-as-instruction data: while the internet is a rich source of textual and image data, data of drawings used as instructions is much sparser -particularly the ephemeral drawings that accompany language. (ii) lack of refinement trajectories: furthermore, people typically upload only finished artifacts to the internet, often with accompanying contextual text data, making these datasets well-suited for generation tasks. on the other hand, the process of editing artifacts based on further instructions is rarely uploaded, which is why these models struggle with interactive refinements. as a multimodal refinement dataset, mrcad serves as a valuable resource for investigating these phenomena.