Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities (2024)

Abstract: When presented with questions involving visual thinking, humans naturally switch reasoning modalities, often forming mental images or drawing visual aids. Large language models have shown promising results in arithmetic and symbolic reasoning by expressing intermediate reasoning in text as a chain of thought, yet struggle to extend this capability to answer text queries that are easily solved by visual reasoning, even with extensive multimodal pretraining. We introduce a simple method, whiteboard-of-thought prompting, to unlock the visual reasoning capabilities of multimodal large language models across modalities. Whiteboard-of-thought prompting provides multimodal large language models with a metaphorical ‘whiteboard’ to draw out reasoning steps as images, then returns these images back to the model for further processing. We find this can be accomplished with no demonstrations or specialized modules, instead leveraging models’ existing ability to write code with libraries such as Matplotlib and Turtle. This simple approach shows state-of-the-art results on four difficult natural language tasks that involve visual and spatial reasoning. We identify multiple settings where GPT-4o using chain-of-thought fails dramatically, including more than one where it achieves 0% accuracy, while whiteboard-of-thought enables up to 92% accuracy in these same settings. We present a detailed exploration of where the technique succeeds as well as its sources of error.

## introduction which lowercase letter is a circle with a vertical line touching it to the right going down? this question may sound rather trivial. likely, you solved it by forming a mental image as you proceeded through the sentence, first placing the circle then adding the line to recognize the letter 'q.' if there had been more pieces to keep track of, you might instead resort to pen and paper, but follow a similar process. humans excel at this type of visual thinking. we readily interweave reasoning in words and reasoning in images to solve problems and communicate ideas @xcite ; we form images not only for directly visual reasoning, but also to draw maps during spatial navigation @xcite and even process abstract concepts @xcite . large language models (llms) have revolutionized nlp @xcite , showing the power of scaling parameter count and training data in all sorts of reasoning tasks. chainof-thought (cot) prompting @xcite , which allows a language model to solve complex problems by writing out intermediate steps as text before producing a final answer, and similar techniques have shown huge promise in tasks such as arithmetic and symbolic reasoning. it is thus natural to ask: can llms solve tasks that we, as humans, solve with visual thinking? we find that for some tasks involving visual and spatial reasoning, even the best llms fail dramatically. in fig. 1 , we see @xcite ) claim the answer to our earlier question is in fact 'b'; it does not perform the correct implicit visual reasoning about left, right, up, and down to reach the correct answer. 'b' was observed sampling deterministically; with temperature, it picks randomly among @xcite ., the letters with the correct features but in different arrangements. it references the 'bowl' and the 'stem' to narrow down potential options then fails to perform the spatial reasoning to determine what the described orientation means. a formal background in typography should not be needed to answer the question; anyone who can create the correct visual with a basic level of visual recognition will solve it easily. this highlights the significant differences between the two reasoning processes. query: which lowercase letter is a circle with a vertical line touching it to the right going down? when we combine these two elements, we get a shape that looks like this: "b". the lowercase letter "b" consists of a circular shape (the bowl) with a vertical line (the stem) attached to the right side of the circle, extending downward. therefore, the lowercase letter you are describing is "b". ## preliminaries we provide a brief overview of the main concepts underlying this work here. llms and mllms large language models have seen substantial success across a wide range of natural language tasks by scaling data and parameter counts @xcite . recent work has extended advances in language modeling to the multimodal input setting, leading to multimodal large language models (mllms). these models generate text conditioned on not other text as context, but also inputs from other modalities, enabling tasks such as captioning or visual question answering. of particular note for this work, the large amount of image-andtext data available on the internet has enabled this effort to prove especially successful for image inputs, achieving state-of-the-art results across a wide range of computer vision tasks @xcite @xcite . ## whiteboard-of-thought the goal of this work is to equip mllms with the ability to create images and visually process them to better answer queries. our approach operates this whiteboard by synthesizing drawing code, exe-cuting this code to create the drawing, and parsing the resulting image before producing a final answer. fig. 1 shows an example of the full procedure. creating visuals with mllms. current mllms typically do not inherently possess the ability to produce outputs in the visual domain. instead, we will show how we can create visuals using a model that only produces text. the images we create for visual reasoning tend to be minimal, abstract, and symbolic @xcite . we use code as a natural way to create such visuals. leveraging what models already know about common python libraries like matplotlib or turtle enables this capability to emerge zeroshot, without needing any specialized, hand-crafted modules (though these could be made to adapt the technique to specific domains). we discuss other approaches such as text-to-image models in the appendix. in order to generate the code, we provide the mllm with the query and prompt it to write code to visualize it. for each query, we prompt the mllm with the input you write code to create visualizations using the {matplotlib/turtle} library in python, which the user will run and provide as images. do not produce a final answer to the query until considering the visualization. along with the query. the model then decides what visualization code to write based on the query. full prompting and inference details for code generation can be found in the appendix. the resulting code is then passed to a runtime environment to render it in image form. in this case, we use the python interpreter with the previouslymentioned visualization libraries. processing the generated visuals. to process the resulting image, we make use of the mllm's intrinsic multimodal input capacity. this obviates the need for external tools, like handcrafted visual modules @xcite , leading to a tightly self-contained method. in summary, the model is given a prompt including the query, the knowledge that it can use the aforementioned visualization libraries, and that it will be provided the resulting images along with the query to perform further steps towards producing a final answer. it produces code, which is then executed to create an image. the resulting visual is returned back to the model to perform the next step or produce an answer. @@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@@#*@@@@ @@@@@@@@@@@@%#@@@@@@%-:+@@@@ @@@@@@@@@@#-.=@@@@@+.:#@@@@@ @@@@@@@@#-.-#@@@@%=.=%@@@@@@ @@@@@@*=...:---=+-.=%@@@@@@@ @@@@@+.:=*#%@@*..-%@@@@@@@@@ @@@@@@@@@@@@@#-.-%@@@@@@@@@@ @@@@@@@@@@@#=.=#@@@@@@@@@@@@ @@@@@@@@@#:.=%@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@@@@@@@@ ## experiments we conduct experiments in two broad classes of natural language tasks that involve visual reasoning. first, we consider three datasets from big-bench @xcite that involve understanding information represented as ascii text graphics. next, we consider tasks involving natural language navigation under different spatial conditions @xcite . we perform all experiments in the zero-shot regime, and compare to two baselines without visualization: directly prompting the model for an answer ('direct') and zero-shot chain-of-thought @xcite ('cot'). we use a temperature of 0 and greedy decoding for generation. for all experiments, we use gpt-4o (gpt-4o-2024-05-13) as the underlying mllm as it has each of the necessary capabilities enabling our model and the baselines -zero-shot chain-of-thought as well as the ability to produce code outputs and to accept image inputs. we explore preliminary experiments with alternative models in appendix f, with results reflecting similar trends. full prompts and other generation details can be found in the appendix. ## in-the-wild examples the poem is arranged in a way that creates a visual image. the lines are centered and vary in length, creating a shape that resembles a tongue or a speech bubble. the visual layout of the poem is not just a random arrangement of words but a deliberate design to evoke the image of the eiffel tower, symbolizing paris. this adds a layer of meaning to the poem, connecting the text to the iconic structure and the cultural significance of paris. ## conclusions we propose whiteboard-of-thought, a simple, zeroshot method to unlock visual reasoning across modalities in multimodal large language models. we accomplish this by generating code that can create a visual, then returning the visual back to the model for further reasoning. this work demonstrates whiteboard-of-thought's capabilities across multiple tasks requiring visual and spatial reasoning that have thus far proved challenging for current state-of-the-art models with text reasoning. as the abilities of these models to generate code, understand visual inputs, and perform general reasoning continue to improve, we expect the results of whiteboard-of-thought to similarly grow.