--- title: Critical AI Prompt Battle author: Sarah Ciston editors: - Emily Martinez - Minne Atairu category: critical-ai --- # p5.js Critical AI Prompt Battle By Sarah Ciston With Emily Martinez and Minne Atairu ## What are we making? In this tutorial, you can build a tool to run several AI chat prompts at once and compare their results. You can use it to explore what models 'know' about various concepts, communities, and cultures. This tutorial is part 2 in a series of 5 tutorials that focus on using AI creatively and thoughtfully. Part 1: [Making a ToolBox for Making Critical AI] Part 3: [Training Dataset Explorer] Part 4: [Machine Learning Model Inspector & Poetry Machine] Part 5: [Putting Critical Tools into Practice] The code and content in this tutorial build on information from the prior tutorial to start creating your first tool for your p5.js Critical AI Kit. It also builds on fantastic work on critical prompt programming by Yasmin Morgan (2022), Katy Gero et al.(2024), and Minne Atairu (2024). ## Why compare prompts? When you're using a chatbot to generate code or an email, it's easy to imagine its outputs are neutral and harmless. It seems like any system would output basically the same result. Does this matter for basic uses like making a plain image or having a simple conversation? Absolutely. Training datasets are shaping even the most innocuous outputs. This training shows up in subtle insidious ways. Unfortunately, the sleek chatbot interface hides all the decision-making that leads to a prompt output. To glimpse the differences, we can test many variations by making our own tool. With our tool, we can hope to understand more about the underlying assumptions contained in the training dataset. That gives us more information to decide how we select and use these models — and for which contexts. ## Steps ### 1. Make a copy of your toolkit prototype. Use [Tutorial One]([XXX]) as a template. Make a copy and rename the new Space "Critical AI Prompt Battle" to follow along. To jump ahead, you can make a copy of the [finished example in the editor]([XXX]). But we really encourage you to type along with us! ### X. Import the Hugging Face library for working with Transformer models. Put this code at the top of `sketch.js`: ```javascript import { pipeline, env } from 'https://cdn.jsdelivr.net/npm/@xenova/transformers@2.10.1'; env.allowLocalModels = false; // skip local model check ``` The import phrase says we are bringing in a library (or module) and the curly braces let us specify which specific functions from the library we want to use, in case we don't want to import the entire thing. It also means we have brought these particular functions into this "namespace" so that later we can refer to them without using their library name in front of the function name — but also we should not name any other variables or functions the same thing. More information on importing [Modules]([XXX]). ### X. Create global variables to use later. Declare these variables at the top of your script so that they can be referenced in multiple functions throughout the project: ```javascript var PROMPT_INPUT = `The woman has a job as a [MASK].` // a field for writing or changing a text value var OUTPUT_LIST = [] // a blank array to store the results from the model ``` We will be making a form that lets us write a prompt and send it to a model. The `PROMPT_INPUT` variable will carry the prompt we create. The `OUTPUT_LIST` will store results we get back from the model. Think about what `PROMPT_INPUT` you'd like to use first to test your model. You can change it later; we're making a tool for that! A basic prompt may include WHAT/WHO is described, WHERE they are, WHAT they're doing, or perhaps describing HOW something is done. For fill-mask tasks, it will replace one `[MASK]` with one word (called a "token"). It's a bit like MadLibs, but the model makes a prediction based on context. When writing a fill-mask prompt, consider what you can learn about the rest of the sentence based on how the model responds (Morgan 2022, Gero 2023). Its replacement words will be the most probable examples based on its training. Often fill-mask tasks are used for facts, like "The capital of France is [MASK]. For our critical AI `PROMPT_INPUT` example, we will something quite simple that also has subjective social aspects: `The woman has a job as a [MASK].` ### X. Select the task and type of model. Let's write a function to keep all our machine learning model activity together. The first task we will do is called a "fill mask," which uses an "encoder-only" transformer model [XXX-explain] to fill in missing words. Call the function `fillInTask()` and put `async` in front of the function call. About `async` and `await`: Because [inference][XXX-explain] processing takes time, we want our code to wait for the model to work. We will put an `await` flag in front of several functions to tell our program not to move on until the model has completely finished. This prevents us from having empty strings as our results. Any time we use `await` inside a function, we will also have to put an `async` flag in front of the function declaration. For more about working with asynchronous functions, see [Dan Shiffman's video on Promises]([XXX]). Here's our basic model: ```js async function fillInTask(){ const pipe = await pipeline('fill-mask', 'Xenova/bert-base-uncased'); let out = await pipe(PROMPT_INPUT); console.log(out) // Did it work? :) // yields { score, sequence, token, token_str } for each result return await out } await fillInTask() ``` Inside this function, create a variable and name it `pipe`. Assign it to the predetermined machine learning pipeline using the `pipeline()` method we imported. The 'pipeline' represents a string of pre-programmed tasks that have been combined, so that we don't have to program every setting manually. We name these a bit generically so we can reuse the code for other tasks later. Pass into your method the `('fill-mask', 'Xenova/bert-base-uncased')` to tell the pipeline to carry out a fill mask task, using the specific model named. If we do not pick a specific model, it will select the default for that task. We will go into more details about switching up models and tasks in the [next tutorial]([XXX]). Finally, in the `README.md` file, add `Xenova/bert-base-uncased` (no quote marks) to the list of models used by your program: ``` title: P5tutorial2 emoji: 🌐 colorFrom: blue colorTo: yellow sdk: static pinned: false models: - Xenova/bert-base-uncased license: cc-by-nc-4.0 ``` ### X. Add model results processing Let's look more closely at what the model outputs for us. In the example, we get a list of five outputs, and each output has four properties: `score`, `sequence`, `token`, and `token_str`. Here's an example: [REPLACE][XXX] ```js { score: 0.2668934166431427, sequence: "the vice president retired after returning from war.", token: 3394, token_str: "retired" } ``` The `sequence` is a complete sentence including the prompt and the replaced word. Initially, this is the variable we want to display. You might also want to look deeper at the other components. `token_str` is the fill-in word separate from the prompt. `token` is the number assigned to that word, which can be used to look up the word again. It's also helpful to understand how frequently that word is found in the model. `score` is a float (decimal) representing how the model ranked these words when making the selection. We can isolate any of these properties to use them in our toolkit: ```js // a generic function to pass in different model task functions async function getOutputs(task){ let output = await task await output.forEach(o => { OUTPUT_LIST.push(o.sequence) // put only the full sequence in a list }) console.log(OUTPUT_LIST) } //replace fillInTask with: await getOutputs(fillInTask()) ``` By putting the [XXX] ### X. Add elements to your web interface. ### X. [PSEUDOCODE] Connect form, test with console.log() ```js // let PREPROMPT = `Return an array of sentences. In each sentence, fill in the [BLANK] in the following sentence with each word I provide in the array ${blankArray}. Replace any [FILL] with an appropriate word of your choice.` ``` ### X. [PSEUDOCODE] Test with simple example. ### X. [PSEUDOCODE] Parse model results. ### X. [PSEUDOCODE] Send model results to interface ### X. [PSEUDOCODE] Test with more complex example (add a model, add a field) ### X. [PSEUDOCODE] Add a model to the tool. You can change which model your tool works with by README.md and to sketch.js Search the list of models available. ### X. [PSEUDOCODE] Make a list of topics that interest you to try with your tool. - Experiment with adding variety and specificity to your prompt and the blanks you propose. Try different sentence structures and topics. - What's the most unusual or obscure, most 'usual' or 'normal', or most nonsensical blank you might propose? - Try different types of nouns — people, places, things, ideas; different descriptors — adjectives and adverbs — to see how these shape the results. For example, do certain places or actions often get associated with certain moods, tones, or phrases? Where are these based on outdated or stereotypical assumptions? - How does the output change if you change the language, dialect, or vernacular (e.g. slang versus business phrasing)? (Atairu 2024). - >"How do the outputs vary as demographic characteristics like skin color, gender or region change? Do these variances reflect any known harmful societal stereotypes?" (Atairu 2024) - >"Are stereotypical assumptions about your subject [represented]? Consider factors such as race, gender, socioeconomic status, ability. What historical, social, and cultural parallels do these biases/assumptions reflect? Discuss how these elements might mirror real-world issues or contexts. (Atairu 2024) ### Reflections Here we have created a tool to test different kinds of prompts quickly and to modify them easily, allowing us to compare prompts at scale. By comparing how outputs change with subtle shifts in prompts, we can explore how implicit bias emerges from [repeated and amplified through] large-scale machine learning models. It helps us understand that unwanted outputs are not just glitches in an otherwise working system, and that every output (no matter how boring) contains the influence of its dataset. ### Compare different prompts: See how subtle changes in your inputs can lead to large changes in the output. Sometimes these also reveal large gaps in the model's available knowledge. What does the model 'know' about communities who are less represented in its data? How has this data been limited? ### Reconsider neutral: This tool helps [reveal/us recognize] that [no version of a text, and no language model, is neutral./there is no 'neutral' output]. Each result is informed by context. Each result reflects differences in representation and cultural understanding, which have been amplified by the statistical power of the model. ### Consider your choice of words and tools: How does this help you think "against the grain"? Rather than taking the output of a system for granted as valid, how might you question or reflect on it? How will you use this tool in your practice? ## Next steps ### Expand your tool: This tool lets you scale up your prompt adjustments. We have built a tool comparing word choices in the same basic prompt. You've also built a simple interface for accessing pre-trained models that does not require using [a login/another company's interface]. It lets you easily control your input and output, with the interface you built. Keep playing with the p5.js DOM functions to build your interface & the HuggingFace API. What features might you add? You might also adapt this tool to compare wholly different prompts, or even to compare different models running the same prompt. Next we will add additional aspects to the interface that let you adjust more features and explore even further. ## Further considerations Consider making it a habit to add text like "AI generated" to the title of any content you produce using a generative AI tool, and include details of your process in its description (Atairu 2024). ## References Atairu, Minne. 2024. "AI for Art Educators." AI for Art Educators. https://aitoolkit.art/ Katy Ilonka Gero, Chelse Swoopes, Ziwei Gu, Jonathan K. Kummerfeld, and Elena L. Glassman. 2024. Supporting Sensemaking of Large Language Model Outputs at Scale. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '24). Association for Computing Machinery, New York, NY, USA, Article 838, 1–21. https://doi.org/10.1145/3613904.3642139 Morgan, Yasmin. 2022. "AIxDesign Icebreakers, Mini-Games & Interactive Exercises." https://aixdesign.co/posts/ai-icebreakers-mini-games-interactive-exercises ============================================ Tutorial 1: ### X. Get to know the terms and tools. API: Model: Dataset: ### X. Create a Hugging Face Space. A Hugging Face Space is just like a GitHub repo with GitHub Pages, except it's hosted by Hugging Face and already attached to its datasets, models, and API. Visit `https://huggingface.co/new-space?`. Name your Space. Maybe something like `p5jsCriticalAIKit` or `criticalAITutorial1` Select `static` and then select a `Blank` template. Make sure you keep the default settings of `FREE` CPU basic, and you can choose whether you want your space to be public or private. Your new Space should load on the `App` page, which is its web page. It should say `Running` at the top in green if it worked. Click on the drop down menu next to Running. Select Files to see your file tree and repository (repo) in the web interface. ![screenshot of new space app page]() ![screenshot of new space app page with file menu]() Click "Add File" to make a new `sketch.js` file. Go ahead and click "Commit" to save the new file before you get started editing it. You can hit "Edit" to make changes to the file in your browser. Alternately, you can clone the whole repository to work from your desktop. Refer to [p5.js Tutorial on Setup]() or [Dan Shiffman’s Hosting a p5.js Sketch with GitHub Pages](https://youtu.be/ZneWjyn18e8) which also works for HF Spaces)] for more detailed information about setting up your workspace. We recommend this, especially if you’re already familiar or willing to dive in! ### X. Add p5.js to the Space Edit `index.html` to include the p5.js library by including this line inside the ` tags: `` If you like, you can change the title of your page to your preference. You can also remove any elements inside the `` tags, since we will replace them. Next, update `index.html` to reference the `sketch.js` file we created. Add this line inside the `` tags: `` // make sure it has the type attribute "module" The script element importing the sketch file may be familiar to you. Importantly, it also needs the `type="module"` attribute, so that we can use both p5.js and other libraries in the file. Let's set that up next... ### X. Create a class instance of p5 in `sketch.js`. Our p5.js Instance is basically a wrapper that allows us to hold all of our p5.js functions together in one place and label them, so that the program can recognize them as belonging to p5.js. First we declare a `new p5()` class instance: ```javascript new p5(function (p5) { // }) ``` Then, all our usual p5.js coding will happen within these curly braces. ```js new p5(function (p5) { p5.setup = function(){ // } p5.draw = function(){ // } }) ``` Important: When using any functions specific to p5.js, you will start them out with a label of whatever you called your p5.js instance. In this case we called it `p5` so our functions will be called `p5.setup()` and `p5.draw()` instead of the `setup()` and `draw()` you may recognize. This will apply to any other function that is special to p5.js, like `p5.noCanvas`, but *not* to other functions which are standard to Javascript. Anything code written outside of the `new p5(){}` instance will not understand any p5.js syntax. Let's add the instance mode version of `p5.noCanvas()` because we will be working directly with the DOM and don't need a canvas. ```js new p5(function (p5) { p5.setup = function(){ p5.noCanvas() console.log('p5 instance loaded') } p5.draw = function(){ // } }) ``` We can also check that the p5 instance is working correctly by adding `console.log('p5 instance loaded')` to `p5.setup()`, since you won't yet see a canvas or any DOM elements Check that the page loaded, since we don't have a canvas. Add this outside of the p5.js instance: ```js window.onload = function(){ console.log('DOM loaded, sketch.js loaded') } ``` ### X. Create a web interface and add template features. We'll build an easy p5.js web interface so that we can interact with our Critical AI Kit. Create three new functions and run them in the In the `p5.setup()` function. Add a fourth function named `displayResults()` but don't run it in Setup. Instead it will run with a button press we make later. ```js new p5(function (p5){ p5.setup = function(){ p5.noCanvas() console.log('p5 instance loaded') makeTextDisplay() makeFields() makeButtons() } function makeTextDisplay(){ // } function makeFields(){ // } function makeButtons(){ // } function displayResults(){ // } }) ``` For a deep dive into how to use the p5.DOM features, see [DOM TUTORIAL]((XXX)). Here we'll quickly put some placeholder text, input fields, and buttons on the page that you can expand on later. First, add a title, a description, and some alt text for accessibility. Don't forget to add `p5.` in front of every function that is specific to p5.js. ```js function makeTextDisplay(){ let title = p5.createElement('h1','p5.js Critical AI Kit') let intro = p5.createP(`Description`) let altText = p5.describe(p5.describe(`Pink and black text on a white background with form inputs and buttons. The text describes a p5.js tool that lets you explore machine learning interactively. When the model is run it adds text at the bottom showing the output results.`)) } ``` For now, we'll just add a single input field, for writing prompts. It won't work yet because we'll need to connect it to the rest of the form. We describe its size, give it a label, and give it the class `prompt`. ```js function makeFields(){ let pField = p5.createInput(``) pField.size(700) pField.attribute('label', `Write a prompt here:`) p5.createP(pField.attribute('label')) pField.addClass("prompt") } ``` We'll add one button that will let us send prompts to the model. We create a variable called `submitButton`, use it to create a button with the `p5.createButton` function, and display the text `"SUBMIT"` on the button. We also size the button and give it a class. For now it won't do anything because we haven't used its `.mousePressed()` method to call any functions, but we'll add that later. ```js function makeButtons(){ let submitButton = p5.createButton("SUBMIT") submitButton.size(170) submitButton.class('submit') // submitButton.mousePressed() } ``` And how about somewhere to display the results we get from our model? We won't see them yet, because we haven't run the model, but let's add a header and a paragraph for our outputs to come. When we are up and running we'll put this together with our model outputs to display the results on our web page. ```js function displayResults(){ let outHeader = p5.createElement('h3',"Results") let outText = p5.createP('') ``` ### X. Add CSS flair. Create a `style.css` file and paste in this code: [Link to raw file]([XXX) You can also write your own or revamp this code! See [CSS Tutorial [XXX]]([XXX]) for more details on playing with styles. ### X. Add authorization to your space. [MAY NOT BE NEEDED][XXX] We'll use some template configuration code to make sure our program talks to the Hugging Face API. Paste this code into your `sketch.js` file: ```js [XXX-TO-DO][MAY NOT NEED] ``` Also add this to your `README.md`: ```markdown hf_oauth: true hf_oauth_scopes: - read-repos - write-repos - inference-api ``` When you next load your app, click `Authorize` ![screenshot of Hugging Face app authorization screen]() To check if your authorization has worked, visit the Settings for your Hugging Face profile. Click `Connected Apps` and you should see the name of your `Space`. ![screenshot of authorized space in Hugging Face Settings interface]() To adjust the configuration of your HF space: https://huggingface.co/docs/hub/spaces-config-reference **ALT if not using HF Spaces:** ### X. Get a HuggingFace API key. https://huggingface.co/docs/hub/spaces-overview#managing-secrets ### X. Connect your API key to your p5.js instance. Reflections & Next Steps We’ve now put together all the basic foundations of a web page ready to host some Critical AI tools. As we move on to [XXX]