From Words to Wires: Generating Functioning Electronic Devices from Natural Language Descriptions (2023)

Abstract: In this work, we show that contemporary language models have a previously unknown skill – the capacity for electronic circuit design from high-level textual descriptions, akin to code generation. We introduce two benchmarks: PINS100, assessing model knowledge of electrical components, and MICRO25, evaluating a model’s capability to design common microcontroller circuits and code in the Arduino ecosystem that involve input, output, sensors, motors, protocols, and logic – with models such as GPT-4 and Claude-V1 achieving between 60% to 96% Pass@1 on generating full devices. We include six case studies of using language models as a design assistant for moderately complex devices, such as a radiation-powered random number generator, an emoji keyboard, a visible spectrometer, and several assistive devices, while offering a qualitative analysis performance, outlining evaluation challenges, and suggesting areas of development to improve complex circuit design and practical utility. With this work, we aim to spur research at the juncture of natural language processing and electronic design.

## introduction the realm of science fiction often presents us with captivating visions of technology's future. a case in point is the replicator from the tv series star trek, a machine capable of creating various physical objects -from food and medicine to functioning devices -based solely on a user's high-level description of those objects. contemporary language models hint at the precursors to some of this capacity for design, including the ability to design novel 2d and 3d object models @xcite , predict molecular structures for drug discovery @xcite 1 python library and data: https://github.com/ cognitiveailab/words2wires 2 companion video: https://youtu.be/pz1rr0ddapi ## experiment 1: component knowledge to design electronic circuits, the designer needs a knowledge of the individual electrical components that can be used to build a circuit. one of the most fundamental aspects of this knowledge is the component pinouts, or the specific function of each electrical terminal (or pin) on a component. for example, a light-emitting diode (led) typically has two pins, one an anode where positive voltage is applied, and one a cathode where the negative terminal or ground is applied. here we measure large language models' knowledge of component pinouts by asking them to generate pinouts for a large number of common electrical components. pinout scoring method g p t 3 . 5 g p t -4 c l a u d e strict scoring (exact) 55% 74% 56% permissive scoring 74% 86% 72% benchmark: we assembled a benchmark of electronic component pinouts, pins100, containing 100 common parts frequently used in circuits found on high-traffic electronic tutorial websites such as the arduino project hub and au-todesk tinkercad circuits. components range from 2 to 40 pins, and span a large assortment of part categories including passives (e.g. resistors/capacitors), input (e.g. switches), output (e.g. leds, motors, relays), sensors, integrated circuits, power regulators, logic (e.g. 7400-series and and or gates), and microcontrollers (e.g ar-duino, raspberry pi). models: we evaluate on instruction-tuned models including openai's chatgpt (gpt-3.5-turbo) and gpt-4 (openai, 2023), and anthropic claude-v1 4 . model prompts are identi-cal across models, and include a static 1-shot exemplar (a 14-pin 7400-series logic integrated circuit) that provides an example of the pinout task, as well as the requested json output format. additional hyperparameters, evaluation details, and the full prompt are provided in the appendix. evaluation: we evaluate using two binary measures of accuracy analogous to the pass@1 codegeneration metric @xcite . the first scoring method, strict, requires a given model to output all of a component's pins correctly to be considered correct, otherwise it will be considered incorrect. the second method, permissive, requires only the function-critical pins of a component to be present to be considered correct, while failing to include non-critical pins still counts as success. foot_2 results: model performance in the pinout generation task is shown in ## experiment 2: circuit generation how well can contemporary language models leverage their component knowledge to design simple but functioning electronic devices? in this experiment, we investigate end-to-end generation of working devices, which includes generating four core elements: (1) a bill of materials (bom), or list of components in the device, (2) the pinouts for each component, (3) a complete electrical circuit diagram called a schematic that details how the components are to be connected, and (4) the code to be programmed onto a microcontroller -a lightweight processor that controls embedded circuits. an example of a complete model-generated design for a trivial device that turns on an led in response to a button being pressed is shown in figure 3 . benchmark: to assess a model's ability to create microcontroller-driven electronic devices, we developed a benchmark, micro25, that includes 25 tasks intended for the common arduino microcontroller ecosystem.. these tasks, shown in representations: models were given format prompts to export all generated elements (bill of materials, pinouts, schematics, and code) in an annotated json format, shown in figure 3 . the annotated format allows the model to add comments for each generated element (e.g. specifying the uses of each component, or the purpose of each connection in the schematic), analogous to chain-of-thought reasoning @xcite applied to circuit generation, as well as code generation from requirements specifications @xcite , where here the requirements are the schematics and other device specifications generated immediately preceding the code. additional details of this representation format are provided in appendix b.4 and c. ## experiment 3: open device generation while we've observed that language models have the capacity to design comparatively simple devices in experiment 2, this result is tempered by these benchmark tasks being representative of fairly common skills and capacities that are frequently taught in microcontroller-oriented curricula found in books, internet tutorials, and blog posts -and as such, the micro25 tasks likely exist in some form in the voluminous (but closed) training data of these models. here, we examine how well the best-performing model, gpt-4, can create comparatively more complex devices in a more realistic and qualitative setting, where it is used as a design assistant like github copilot @xcite to create initial plans that are then vetted and corrected by a domain expert before being physically constructed. to further increase task difficulty, each of the device specifications in this experiment were explicitly crafted to be highly unusual -either using uncommon components, or combining common components in unusual ways -such that the likelihood of similar devices appearing in the closed model training data is low. the six devices are shown in figure 4 . methods: initial specifications (in the form of a natural language textual description of a device) were created for all devices, and iteratively refined several times to provide clarifications in response to undesired or errorful model-generated output. after several attempts at refining device descriptions, any remaining errors were manually corrected by a domain expert, then the devices were physically manufactured. all devices were constructed by a human, and physical aspects of the design (e.g. printed circuit boards, 3d printed or laser cut components) were designed by a human. full device description strings, model prompts, and detailed qualitative descriptions of manual corrections re-quired by a domain expert to reach functionality are described in the appendix. qualitative challenges: generation challenges can be organized into two categories: promptspecific challenges, and hardware-specific challenges. with respect to prompts, device generation is highly sensitive to the specific prompt, and small (and seemingly helpful) changes in the prompt to address an error can cause new errors to occur in other aspects of the device that were previously generated correctly. similarly, having many composite requirements in the project (such as adding the requirement for each key in the emoji keyboard to generate its own musical tune) generally decreases performance, and suggests that iteratively generating devices from the core requirements (such as generating a functioning emoji keyboard) through to more fine-grained details (like adding in musical tunes) may reduce the inference load at each generation step, improving generation performance. with respect to hardware, a number of pragmatic issues occur. electronic parts regularly reach endof-life cycles and are no longer manufactured or easily available, yet the model frequently generates these, likely due to the abundance of examples that use these components in internet tutorials. similarly, the model frequently uses deprecated versions of libraries, or combines the features from different versions of libraries. finally, the model generally performs poorly at generating low-level device drivers for specific hardware (such as sensors), and favors using existing device driver libraries. when an existing library isn't available, the model will either hallucinate one, or generate a reasonable first-pass at a device library that requires extensive modification to low-level details (like clock timings) to function. ## challenges and discussion we identify the following challenges and opportunities in developing this capacity for automated device design further: prompt sensitivity: currently, small changes in the prompt can cause large changes in the output, affecting overall performance. while this is evident from small changes in task description strings producing novel errors in experiment 3, this phenomenon is also visible for simpler cases. for example, the pinouts for some components -such as the mlx90614 temperature sensor -are incorrect when tested independently in experiment 1, but emoji usb keyboard that has keys for 9 common emojis. pressing an emoji types its ascii string, just as if entered on a normal keyboard. a short musical tune with similiar affect to the emoji (e.g. a love song for the heart emoji) is also played. random number generator that uses an ambient radiation sensor to continuously update the random seed. random numbers are generated every few seconds, and displayed on a vintage nixie tube using a high-voltage driver. visible light spectrometer that measures spectra using a hamamatsu micro-spectrometer and displays the spectrum on a 128x128 pixel oled screen. here, the device is shown measuring the characteristic spectrum of a white led. non-contact temperature sensor that displays the temperature on a strip of 8 leds. higher temperatures show as red colors and illuminate more leds, while lower temperatures show as blue colors and illuminate fewer leds. pill alarm assistive device that has alarms for three pills. when it's time to take a pill, the alarm continuously waives a physical flag saying "take pill x" back and forth using a servo motor to get the users attention, until the button is pressed. ultrasonic glasses assistive device that uses an ultrasonic distance sensor to measure the distance of the nearest object in front of a person with a visual impairment. the distance is converted into an audible tone of varying frequency. correct when generated as part of a full circuit in experiment 2. this suggests that task performance is not currently robust, and may benefit from creating and fine-tuning on a task-specific dataset. manual evaluation: like copilot, we have observed that the electronic designs generated by lan-guage models are rarely perfect and frequently have errors. currently these have to be discovered and corrected by a human. contemporary work in code generation aims to use reflection (e.g. @xcite to iteratively run generated code in an external interpreter (like python), report any errors to the language model, then continue this process until the generated code runs error-free. the lack of electronic simulators with large libraries of simulated devices presents a significant barrier to this form of automatic evaluation in the near-term. ultimately this may be addressed by constraining circuit generation to only parts available within a given simulator, or pushing a focused effort to developing more capable simulators with a larger repertoire of components. generating devices with common-sense knowledge: language models contain a variety of common-sense reasoning abilities @xcite , and leveraging these abilities may enable new applications. for example, in the context of assistive devices, gpt-4 is able to infer that an ultrasonic sensor can be used to create assistive glasses to aid the visually impaired with navigation. similarly, the model can use its common-sense knowledge to design devices that contain the most common emojis, or keys for all the prime numbers up to 20 just as easily. ultimately, electronic devices may be distributed as templates, that can be semi-automatically customized to a variety of applications based on user preferences. quantifying time savings of automatic versus human device design: precisely quantifying the benefits of automated coding assistants (such as github copilot) is challenging, and currently measured at least in-part with qualitative measures @xcite . these assistants may provide large time savings when they function correctly, but likely increase debugging time when they generate problematic code, complicating measuring their precise benefit. a similar situation likely exists here for device generation, and we provide only the following anecdotal account: the best-performing gpt-4 model described in this work produces in minutes what undergraduates in our course might initially take hours to days to perform, as they learn to adapt their existing computer science skills to the electronics and microcontroller domain. as such, in the near-term, systems such as words2wires might be viewed as productivity assistants that allow (for example) scientists with existing coding skills but minimal electronics knowledge to quickly design instrumentation (such as data loggers) or other customized devices that are relatively modest in scope and complexity with a minimal time investment. ## conclusion this study empirically characterizes the previously unknown potential of contemporary language models to move from words to wires -that is, to generate working electronic device designs from highlevel text descriptions. our analysis demonstrates these models have moderate-to-high proficiency in generating component-level knowledge on the pins100 benchmark, while gpt-4 significantly outperforms claude-v1 at generating 25 fullyfunctional devices from the micro25 benchmark, reaching near-perfect performance. when used as a design assistant for generating six more complex devices, language models can generate devices that nearly meet specifications, but still require moderate correction by domain experts to function. while this novel application of language models inspires the democratization of electronic device creation, further development is currently tempered by the lack of simulators to automatically evaluate designs, and the highly manual nature of this process.