Testing the Depth of C hat GPT ’s Comprehension via Cross-Modal Tasks Based on ASCII -Art: GPT 3.5’s Abilities in Regard to Recognizing and Generating ASCII -Art Are Not Totally Lacking (2024)

Abstract: In the months since its release, ChatGPT and its underlying model, GPT3.5, have garnered massive attention, due to their potent mix of capability and accessibility. While a niche industry of papers have emerged examining the scope of capabilities these models possess, language — whether natural or stylized like code — has been the vehicle to exchange information with the network. Drawing inspiration from the multi-modal knowledge we’d expect an agent with true understanding to possess, we examine GPT3.5’s aptitude for visual tasks, where the inputs feature ASCII-art without overt distillation into a lingual summary. In particular, we scrutinize its performance on carefully designed image recognition and generation tasks. An extended version of this write-up is available at: https://arxiv.org/abs/2307.16806 .

## introduction chatgpt has rapidly been adopted since its release in november 2022. this large language model (llm) builds off of version 3.5 of the generative pre-trained transformer model family developed by openai, a child whose lineage has been marked by one massive step after another in regard to the size of llm networks and their training data. active utilization in industry @xcite and education @xcite are already a reality, though with growing concerns on the impacts on the workforce and academic integrity. fueled by the model's unprecedented popularity, accessibility, and power, a niche industry of papers attempting to rigorously investigate the abilities of chatgpt -and the gpt3/gpt3.5 family underlying it more broadly -have materialized in short order. however, efforts thus far have almost exclusively focused on language-centric tasks @xcite . filling this gap, we explore gpt3.5's abilities to "see" and "draw" -critically, doing so without first summarizing the inputs into a verbal description for the model. our vehicle in order to conduct this analysis is ascii-art (aart) @xcite . ultimately, gpt3.5 demonstrates noticeable visual acumen. we uncover that gpt3.5 has subtly more vision-related acumen than has been appreciated. ## related work most work on chatgpt has considered canonical nlp problems @xcite @xcite . as pointed out in @xcite , chatgpt's diverse capabilities and accessibility have fueled a deluge of papers exploring its potential and limitations. the model has proven performant in areas ranging from poetry (cushman, 2022) to programming @xcite to verbally-enabled room navigation @xcite . within this space, most relevant to us are efforts treating chatgpt's spatial reasoning, as well as those exploring its integration into multicomponent pipelines geared toward text-based image recognition, manipulation, or generation. both @xcite and @xcite -examining, respectively, the network's performance in an introductory engineering course and from surveying across the literatureobserved limitations in gpt3.5's abilities to handle "diagrams or figures" and to "perform spatial, temporal, or physical inferences". muddying their conclusions, however, are a subset of reported instances where the network produced aart-but with major qualifiers of being rare and generic enough to likely be rote memorization. there have been attempts to integrate recent gpt-family models into vqa @xcite @xcite @xcite @xcite @xcite , image generation @xcite @xcite @xcite , graph analysis (ex., layout descriptions, scene graphs, etc.) @xcite @xcite @xcite , and in other problem settings where visual-content could play either input or output roles @xcite . the diversity of implementation-specifics notwithstanding, the takeaways are largely the same: these works either (1) prior to querying the llm, summarize context verbally or in a humanreadable data structure via different foundation model specifically engineered for image-related tasks or (2) modify the language models in question to explicitly include visual knowledge, often coupling this with additional training of parts that are woven intimately into the llms. for our purposes, adopting either strategy disqualifies a work from bearing fully on our main question. that is, many existing works simply "let gpt3.5 see" by either modifying it to the point of being a fundamentally different model, or giving it a seeing-eye dog (i.e., another foundation model that addresses all the seeing and manipulation). each of @xcite , @xcite , and @xcite examine gpt3.5's spatial reasoning, navigation, and interaction tasks, but yet again all exchanges were mediated through verbal descriptions of the world state and action space, though sparing the use of a separate foundation model to produce the words. we comment further on the nature of this distinction in appendix a. the aforementioned aside, some substantive works exist that somewhat resonate with our work. under the impetus of differentiating content generated by chatgpt versus humans, @xcite curated questions that emphasized the areas where llms' aptitude most differed -for better or worse -from that of a human. among the eight tests considered, identification of aart was one, exposing a patent gap between human and chatgpt performance -94% and 8% accuracy respectively on 50 cataloged drawings. in addition to the limited show-verbatim-and-describe nature of these trials, we highlight that all samples came from a public website existent for years before chatgpt's release, the ascii art archive, foot_1 risking membership in gpt3.5's training data; moreover, the images' online popularity may predate their inclusion in the catalog. while 8% accuracy is not astounding, it is not nothing; questions remain as to how much is from memorization, actual recognition ability, and random chance. under similar inspiration, the massive, collaborative effort of the "big benchmark" @xcite showcases 204 diverse tasks examining language model's capabilities. three such tasks nominally featured aart, but all concerned the recognition of text that was "rendered" in that fashion. the only other germane task we saw was the "text navigation game", foot_2 which featured a small input grid containing an aart "maze", requiring the models to verbally specify moves from the start to the goal; no instances of "success" were observed by any model for board sizes above 5-by-5, and moreover the authors made reference to success rates on smaller boards being on par with random movement. overall, we find a lack of sufficient subtlety in the benchmark's pertinent tasks, them failing to be sensitive enough -at least as explored -to detect all but the most obvious performance. furthermore, probing specific to evaluating vision systems -such as robustness to rotation, noise or translation -were not carried out, leaving insights only at the high-level outcomes of the raw tests. both of these aspects help distinguish our work from theirs, not to mention the fact that we examine generation of visual content in addition to its recognition. like us, @xcite ) study capabilities of openai's gpt model family, version 3.5 and 4 in their case, 4 using a series of prompts without additional training or system modifications. their endeavor partially examined rudimentary aart produced to explore the models' recursive generation abilities -however, whether this is a "visual" task or essentially an algebraic computation is debatable. the authors note that certain examples displayed are likely memorized from training data, but also point out (rightfully) that the more exotic figures produced are less subject to this concern. either way, the concern underscores the fact that their prompts for aart did not (obviously) impose novelty-constraints on the output, thus failing to rule out preprepared responses as "correct" outputs. in contrast, out experiments require responses to correspond with unique, freshly generated structures provided in our prompts, reducing the feasibility of context-independent, pre-canned responses passing scrutiny. finally, we remark on a certain degree of "folk knowledge" about chatgpt's drawing abilitiesfor instance @xcite @xcite . however, exchanges in this category directly featuring aart (i.e., not code for diagrams, etc.) were mostly sporadic acts, not systematic or deep explorations. a theme throughout is the appearance of aart of reasonable quality, but occurring at inappropriate times in respect to the prompts -hallmarks of shallow memorization, repeating training examples without deeper, semanticallymeaningful interpretation or modification. as a result, the casual consensus judges chatgpt's abilities in this regard as poor. we endeavor to perform more rigorous analysis than the loose folkperceptions. ## the ascii-art used in experiments we use aart of box diagrams (aads) to nontrivially probe gpt3.5's vision-related capabilities. we briefly share the inspiration for this particular choice, since we believe the observations valuable: first, we realized that aads are used as illustrations in many settings -e.g., electrical-circuit diagrams, placement charts, and flowcharts online. indeed, mini-languages like pic @xcite exist to aid their creation, though manual drawing is rarely difficult. gpt3.5 may therefore have a substantial amount of varied training data available for these drawings, e.g., as part of common-crawl. 5 additionally, owing to their common role as a visual aids accompanying verbal descriptions, these depictions likely have appreciable amounts of granular visio-lingual coupled data. second, we encountered quite promising results during early investigations into chatgpt's germane abilities. in a trial, we requested drawings of several town layouts, each with certain buildings and accompanying labels. illustrations were generated that matched our specification, a feat not easily dismissed as mere memorization. reasonable success continued during additional requests (e.g., for roads) that followed. following these leads, we have run experiments featuring randomly generated aads to gauge chat-gpt's aptitude in typical vision-related tasks: content recognition despite changes due to rotation, scale, "pixel" noise, and translation. if gpt3.5 can handle these tasks, then it suffices to say it is not entirely incapable of "doing well at aart", despite impressions held in folk knowledge. ## generation experiments we examine gpt3.5's ability to generate aart, tasking it to transform input images as specified. ## conclusion drawing inspiration from the comprehension we'd expect an intelligent agent to possess across multiple signal modalities, in this work we examined gpt3.5's aptitude for visual tasks, where the inputs featured diagrams rendered as ascii-art. in sharp contrast to the large majority of prior works, we made no attempt to overtly distill the image content into a lingual summary. we conducted experiments analyzing the model's performance on image matching tasks after various transforms typical in visual settings, as well as tasks requesting such transforms be generated. in each of these categories of experiment, we found that while gpt3.5 had room for notable improvement, results suggested it was not totally lacking in regard to visual and pictorial aptitude. given that gpt3.5 is a model nominally trained on text-only input, we were pleasantly intrigued by these outcomes.